introduction to federation server - · pdf fileintroduction to federation server ... lotus...
TRANSCRIPT
© 2006 IBM Corporation
Introduction to Federation Server
Alex LeeIBM Information Integration SolutionsManager of Technical Presales Asia Pacific
2
WebSphere Federation Server
Federation overview
Tooling support
Case studies
Summary and references
3
What if you could…
Using standard SQL and any tool that supports JDBC/ODBC…
while looking to the user like a single database
BI toolsBusinessAnalysis
MgmtReports
Access data anywhere in your enterprise
No matter where it resides
Regardless of what format it is in
Regardless of vendor
Without creating new databases and without disruptive changes to existing ones…
4
Then you could …Produce information needed by the organization faster
– Melbourne Health – built the worlds first solution to access public medical history –discoveries were made days after implementation versus months with the prior process.
Improve the productivity of your people
– Taikang Life - saved 90% in people costs to compile real time reports (1 instead of 10 people)
Reduce business process costs– Neckermann – reduced labor costs
equivalent to 5 full-time employees per year– Pioneer Display – production efficiency
increased over 25%
BI toolsBusinessAnalysis
MgmtReports
5
And without…
BI toolsBusinessAnalysis
MgmtReports
Building new databases for data you already have stored in multiple places
– Acquiring hardware and software infrastructure to support them
– Keeping them up to date– Keeping them secure– Assuring their reliability
and availability for the next 5-7 years
6
What is Federation?
Federation is an integration pattern that allows a collection of resources to be viewed and manipulated as if they were a single resource while retaining their autonomy and integrity. It is the technology on which EII is based.
7
IBM WebSphere Information ServerDelivering information you can trust
Parallel ProcessingRich Connectivity to Applications, Data, and Content
Information Server
Understand Cleanse Transform Federate
QualityStageInformation Analyzer Federation ServerDataStage
Information Services Director
Metadata Server
8
Data FederationTransparent
Appears to be one sourceIndependent of how and where data is storedApplications continue to work despite of any change in how data is stored
HeterogeneousAccesses data from diverse sourcesRelational, Structured, XML, messages, Web, …
Extensible Bring together almost any data source.Wrapper Development Toolkit
High FunctionFull query support against all dataCapabilities of sources as well
AutonomousNon-disruptive to data sources, existing applications, systems.
High PerformanceOptimization of distributed queries
9
Relationaldatabases
Web,Non-Relational
Sources
CollaborationSystems
XMLWeb services
Packagedapplications
SQL, SQL/XML
Content Repositories
and Imaging Systems
Workflow systems
Content
Mainframefiles
Mainframedatabases
SQL
Virtualized Information AccessAccess diverse and distributed information as if it were in one system
Single sign on – Unified views – Common language – Web services or Java API Query and update – Optimized access
Classic Federation Server for z/OS
Federation Server
II Content Edition
10
VSAMSequential
IMS AdabasCA-DatacomCA-IDMS
Federated Sources
DB2InformixOracleSybase Teradata Microsoft SQL Server ODBC
OLE DBExcelFlat filesLife sciencesCustom-built
DB2 CMFamilyDomino.doc DocumentumFileNetOpen TextStellentInterwovenHummingbird
WebSphereFileNet
Lotus NotesMicrosoft Index ServerIBM Lotus Extended Search
SametimeQuickPlaceMicrosoft Exchange
WebSphere BI Adaptors
SAPPeopleSoftSiebel
Plus partner tools and custom-built connectors extend access to more sources
Content& Imaging
Workflow systems
Relationaldatabases
WebOther
XMLWeb services
Packagedapplications
Mainframefiles
Mainframedatabases
SQL
ContentSQLFederation Server
Classic Federation Server for z/OS
II Content Edition
CollaborationSystems
11
Incorporate data sources using wrappers
Access to a particular class of data sources or protocols
Contains information about data source characteristics
High-function relational wrappers from IBM
Read/Write access
Clean, simple interface for nonrelational wrappers
Written by IBM, third parties, customers
Read Only
Data Federation Approach
Server2Server1
Nicknam
e
Nicknam
e
Nicknam
e
(OptionalLocal Data)
(remote data source 1)
(remote data source 2)
WrapperB
Server3
Nicknam
e
Wrapper A
Table
Federated views
(remote data source 3)
12
Powerful query processing engine in federated server
Decomposes, rewrites and distributes queries
Cost-based optimizer chooses query plan with pushdown as appropriate
Query execution engine drives wrappers, combines results
Compensates for missing function in data source
Invokes functions at remote sources as needed
Data Federation ApproachN
ickname
Nicknam
e
Table
Rel. WrapperClient library
NR. WrapperClient library
Local + Remote Execution Plans
Federated Server
Non-SQL
DB2 cost-based optimizer
13
Agenda
Federation overview
Tooling support
Case Studies
Summary and References
14
Tools for modeling
Visualize and define mappings between remote schema and federated schema
Generate federated schema based on transformations and joins– Nicknames
– Views
Simplify creation of virtual schemas
15
Control Center– Tools to configure and
administer standard wrappers
– Plug-in architecture allows custom wrappers to be administered
Administration Tools
16
Tools help manage the complexity
Configuration wizard– Guides you through federation
configuration process
Discovery– Server discovery: Automatically
discovers and configures external servers
– Nickname discovery
Deploy– Capture configuration to a script,
save and deploy
– Facilitates cloning system configuration for horizontal scaling...
17
WFS tools help manage the complexity
Health Monitoring– Monitor health of servers, nicknames that affect
configuration
Statistics refresh– Refresh nickname statistics on demand or by
scheduled task
Snapshot Monitoring– Snapshots are useful for determining the status of
a database system.
Event Monitoring – Collect information about the database and any
connected applications when specified events occur.
18
Agenda
Federation overview
Tooling support
Case Studies
Summary and References
19
When to use Federation
Too big - Data from multiple sources is just too big to integrate on a permanent basis
Too ad hoc - Data is too varied and unpredictable to make an ETL process worthwhile
Too proprietary - Data is owned by disparate entities/organizations that do not want to support ongoing ETL processes
Too recent - Data from multiple sources is required that must be current or must not be updated while being read
Application or tool does not support native access to the sources being accessed
20
Providing on-demand relational access to multiple types of dataRequirements
Warehouse users need access to up-to-the-minute data from an external sourceExternal data may not be integrated into the warehouse until later (or never) because it is not practical or possible/allowed
SolutionAccess remote data via Federation Server nicknames and combine with DW dataAlso enables prototyping of ETL development
Federation Server
External Web services
Client
Data Warehouse
ODSFlat files or spreadsheets
21
Enabling transparent drill-through to detail data from summary data
Requirements
Users query a summary warehouse that is fed from a detail warehouse by an ETL processMost queries satisfied from summary alone. Some need to retrieve detail data after initial filtering by the summaryBe able to retrieve detail on-demand within the context of the summary query without making a new connection
Solution
Extract, transform and load relevant source data into summary and detail databases using bulk data movementUsers query summary data using existing applicationDetail data visible on-demand from summary warehouse using federation
Federation Server
DataStageClient
DataStage
Summary Database
Bulk data
Bulk data
requestsDetail Database
requestsdatadata
22
Unified view of regionally distributed data with same data model
RequirementsSeveral regional databases with similar logical data models, but unique dataApplication needs to see the data as one large database with a single schemaImpractical to physically consolidate data
SolutionAccess relevant remote tables via Federation Server nicknamesConnect matching nicknames from different sources via a UNION ALL viewCan optionally cache common data at the federated server or create local aggregates
Client
Federation Server
Seattle
Phoenix
San Jose
23
Placement, Consolidation, and Access Choices
ETL or replication preferred: – Centralized data needed for access performance or availability
• Complex, multi-dimensional queries
– Point-in-time consistency needed e.g. close of business
– Complex transformation needed for semantically consistent data
Federation preferred:– Access performance and load on sources traded for overall lower cost
• Queries returning small result sets among federated systems• Large volume data that is infrequently accessed• Data that changes rapidly
– Business requirements demand current data
– Data security, licensing or regulations restrict data movement
– Unique functions must be accessed at data source
– Data semantics consistent and managed across domains
– Read/write access is required
24
Agenda
Federation overview
Tooling support
Case Studies
Summary and References
25
Value of Federation
Speed time to market for new applications– Simplify and enrich portal development
– Reduce hand-coding by half
– Reduce skills requirements
– Use familiar SQL programming model and existing tools
– Build on a standards-based, strategic integration platform
Enhance value and insight from existing assets and applications– Work within your existing infrastructure
– Extend existing warehouses
– Combine existing data and content assets in new ways
– Facilitate cross-divisional reporting
Increase control over IT costs– Reduce need to rip and replace
– Reduce need to manage redundant data
26
ReferencesFor product information on WebSphere Federation Server
http://www.ibm.com/software/data/integration/federation_serverFor the system and data source requirements:
http://www.ibm.com/software/data/integration/federation_server/requirements.html
WebSphere Federation Server V9.1 infocenter:http://publib.boulder.ibm.com/infocenter/db2luw/v9/index.jsp
Some whitepapers on federation technology:“IBM Federated Database Technology”:
http://www.ibm.com/developerworks/db2/library/techarticle/0203haas/0203haas.htmlTwo-part series on using data federation technology:
http://www.ibm.com/developerworks/db2/library/techarticle/dm-0506lin/http://www.ibm.com/developerworks/db2/library/techarticle/dm-0507lin/
“Maximizing the performance of WebSphere Information Integrator with MQTs”:http://www.ibm.com/developerworks/db2/library/techarticle/dm-0605lin/
“Use federated procedures in WebSphere Federation Server”:http://www.ibm.com/developerworks/db2/library/techarticle/dm-0605bhatia
27