bhawani prasad data integration-ppt
DESCRIPTION
Bhawani prasad data integration-pptTRANSCRIPT
1
Data Integration Strategy
April 10, 2013 – BHAWANI NANDAN PRASAD – BI Practice Head SMP
– IIM Calcutta, MBA – Stratford University USA, B.E. (IT)
2 2
IIF Moment
Integration Scope and Frame
Integration Strategy Agenda
Business Requirements
Appendix – Lessons Learned Best Practices and Industry Research
Recommendation - Integration Strategy and Architecture
Integration Strategy Decisions
Integration Technology Comparison
3 3
IT Integration
Scope and Frame
4 4
IN THE FRAME
• Define IT enablement needs for the key Business Process Areas that are not met by Advanced Technical Applications. e.g., L&D Support Systems, Portals to facilitate workflow, Management Reporting System(s).
• Define standards for Data Definition, Information flow. Design architecture data and repositories to enable Reporting and data sharing across applications, including the “Given” applications.
• Define standards for easy integration of new applications (design for growth).
• Support Advanced Technical Apps team to meet ESAS and other standards and to ensure data integration.
• Validate the infrastructure needs of applications against the PCIS environment. Initiate change if necessary.
• Phase 4 planning – Proof of Concepts e.g. implement database structures, BI tools (internal pilots).
• Application, and Information Governance design.
4
R00#1 Frame and Data Integration Charter
“Robust”
Messaging,
Data transformation,
Federation
architecture
Data warehousing,
archive
Enterprise data layer
Business
integration/analytics
Bi-directional
transfer of data
Shows the R00#1 Match-play selection
5 5 5
IT Integration Scope
Key Focus Area Related Area
6
Conceptual Architecture – Integration Focus area
Presentation
Refining Optimization
Center
Business
Analytics
(Dimensional
Models)
Integration
Services
Physical Integration
Project DBExtract, Transform &
Load
Virtual Integration
Information IntegrationInformation Integration
Services
Process &
Workflow
Process &
WorkflowBusiness
Services Scheduling
Simulations
Planning
Shipping Maintenance
Blending Historian
Production
Operations
Information
Sources
Data Integration
Application
DataDocument
Repositories
Data
Sources
Data
Go
vern
an
ce
Master Data
Top 10
• Material
• Production
• Warf
• TankDCS &
instruments
OE/Reliability
Test Results
Field Reliability Center
Key Focus Area Related Area
7 7
Solution Integration Architecture and Information Architecture
• Solution Integration Architecture focus areas:
– Process Integration: Workflow, Orchestration, ESB
– Business Performance Mgmt: Business Activity Monitoring
– Collaboration: Communications
Information Architecture focus areas:
– Data Integration: EAI, ETL, EII, CDC, Replication
– Data Management: Repository for Core Data standardization; Data Governance; Master Data Management; Taxonomy
– Data Standardization: Chevron/Industry standard model Definition, Semantic model,
– Business Intelligence (BI): Reporting; Dashboard
– Unstructured Data Management: Semantic Web; Ontology
8 8
IT Integration
Business Requirements
9 9
Console Operator’s Data View
Console
Operator
HMI
360 view
ROC
Oil Movement
Systems
Optimization
(V-mesa, GDOT,
DMCplus)
Early Event
Detection (EED)
Mass Balance
Alarm System
(CAMMS)
Procedure Mgmt
System (ExaPilot)
Root Cause Analysis Plant Resource
Mgmt System (PRM)
Business Event Log
Global Notification
Communication
Alert Mgmt System
Document Mgmt
Knowledge Mgmt
Learning Mgmt
Console Scheduling
Instruction
Enterprise Asset
Management
(Maximo)
Erroneous Data Log
IT Asset Mgmt (ITSM)
KPI Tracking &
Dashboard
Lab System
(STARLIMS)
Optimization Log
Scheduling Systems
Lynx
SIMTO/MB
O/RVT
Simulator
Shift Turnover
Structure Round
Safety Instrumented
System (SIS)
Legend
IT scope
AT scope
Existing
LP System Flying
Petro
Loss Prevention &
HES Systems
10 10
System Data Flow (Partial example)
Legend
IT scope
AT scope
Existing
Based on BPR Business Requirements and IA Assessment details, every systems and data input/output are depicted in diagrams as below and vetted with SMEs to assure Business Requirements are captured correctly in relation to data. This then serves as basis for our RICEF (Report, Interface, Conversion, Extension & Forms ) detail list.
11 11
RICEF List Summary
Reports
Web Reports Dashboard
Conversions
Extensions (Forms)
6 >100 10 3
Shift Turnover Report
General KPIs
Document Management System (DMS)
Maximo
Console Scheduling Instructions
Blending KPIs Knowledge Management System (KMS)
Lab
Training Level Assessment EED & Alarms KPIs
Turnover documents into Shift Turnover System
Knowledge Management System
Equipment Maximo List Reliability KPIs
Plant Procedures into ExaPilot
Lab Stream Reports SIS KPIs Console Scheduling Instructions (CSI)
Energy KPIs Learning Management System (LMS)
Optimization KPIs
Plant economics information to Area of Optimization Petro
Process Control KPIs (?) P&IDS, drawings
Others ITSM conversion
Facilities Phone list to Refinery Phone directory
RICEF is an acronym for Reports, Interfaces, Conversions, Extensions, and Forms, all of which are basis for Data Integration.
RICEF List Summary
12
Example Total of 36 “system to system” data interfaces in IT Scope
25 interfaces with bulk data on scheduled basis
7 interfaces on demand with low volume of data
4 interfaces with real-time streaming data
6+ of the 36 have process & workflow requirements
RICEF List Interfaces in IT Scope
Bulk Data (large volumes) on a Schedule Basis (25)
Scheduling (SIMTO/MBO) to Console Scheduling Instructions (CSI) *
Scheduling (SIMTO/MBO) to RVT
Erroneous Data Log (EDL) to RVT (Lynx)*
Structure Rounds to Shift Turnover
Event Log to Shift Turnover
Maximo to Shift Turnover
Shift Turnover to Knowledge Management System (KMS) **
Optimization Log to KMS **
Document Management System (DMS) to KMS **
PI – Health Environmental and Safety (HES)
(continued )
PI - Accounting systems
PI-Lab
Lab to DMS
Learning Management System (LMS) to DMS
LMS to People Data
IT Asset Management System (ITSM) to Simulator Scheduler
CSI/SIMTO - RVT
SIMTO to Area of Optimization
Oil Movement System to Wharf/Pipeline Scheduling (2)
Maximo / Lab / Reliability / PI to KPI (4+)
13
* Indicates possible workflow * Indicates Unstructured data
RICEF List Interfaces in IT Scope
14
On Demand & Lower Volume Data (7)
Maximo to Maintenance Schedule
Maximo to Maintenance Request (approval/denial) *
Learning Management System (LMS) to Simulation Systems
LMS to Simulator Scheduling System
Document Management System (DMS) to Simulator Scheduling System
DMS to IT Asset Management System (ITSM)
LMS to ITSM
* Indicates possible workflow
Real-time and streaming data (4)
Oil Movement Systems (OMS) to Alert System
Lab to Alert System
Task System to Alert System
Alert System to Global Notification System
Process Interfaces – Workflow (6)
Shift Turnover- Knowledge Management System (KMS)
(approval) *
Maximo to Maintenance Request (approval/denial) *
Document Management System (DMS) to KMS (Approval)
*
Optimization Log to KMS * (approval)
Scheduling (SIMTO/MBO) to Console Scheduling
Instructions (CSI) *
Erroneous Data Log (EDL) to RVT (Lynx) *
Maximo to Maintenance Request (approval/denial) *
RICEF List Interfaces in PCIS Scope
15
PCIS Scope (11)
PI to Mass Balance
PRM to Alarms System
DCS to Alarm System
PRM to Maximo
Lab to-Mass Balance
DCS to APC
DCS to Exapilot
DCS-OMS
DCS-PI
PI-APC
Scheduling (SIMTO/MBO) - OMS
16 16
Integration Landscape
Oil Movement System
Early Event
Detection
Mass Balance
Alarm System
Procedure Mgmt
System (Exa-Pilot)
Plant Resource
Mgmt System
Business Event Log
Alert Mgmt
System Knowledge Mgmt
System
Learning Mgmt
System
Console Scheduling
Instruction
Enterprise Asset
Management
Erroneous Data Log
IT Asset Mgmt (ITSM)
KPI Tracking &
Dashboard Lab Systems
(Starlims)
Optimization Log
Shift Turnover
Structure Round
PI (ODS with History)
DCS
Simulator
Scheduling
Outlook People
Data
Safety Instrumented
System
Real Time
Scheduled
On Request
Scheduling RVT
Legend
IT scope
AT scope
Existing
Level 3.5
P
C
I
S
S
c
o
p
e
I
T
S
c
o
p
e
Scheduling Systems
SIMTO/MBO
Simulation
Communication
Global
Notification
Document Mgmt
System
Flying Petro
Optimization
17 17
Integration
Technology Comparison
18
The Integration Stack & Products
Low
High
High Integration Stack Vendor Capability
Bu
sin
es
s C
ap
ab
ilit
y o
f a
do
pti
ng
In
teg
rati
on
File Transfer
ETL
Data Transfer File Transfer
ETL
Ad-Hoc Interfaces
Point-to-point Interfaces ETL
Adapters
EAI
WS*-Communication
ESB
Orchestration Engine
BAM
Service Registry
BPM & SOA
WS*-Communication
ESB
Orchestration Engine
BAM
Service Registry
Web 2.0
Composite Solutions
WS*-Communication
ESB
Orchestration Engine
BAM
Service Registry
Web 2.0
EDA
Integration 2.0
• Innovative integration techniques
• Complex & flexible Technology
• Mature integration techniques
• Proven & Robust Technology
Integration technology today
moving data...
…to synchronize and
rationalize data for systems...
…leveraging functionality
in applications…
…to create new processes and services
to support business needs
…predicting future business
while sharing business
capabilities with partners...
…anywhere, anytime, and
through any standard means
More Less
Centralized Integration
18
19 19
Key Integration Components
Custom code remains a popular option
Key Data Integration Methods
EII
( Enterprise Information Integration )
CDC
( Change Data
Capture )
EAI
( Enterprise Application Integration )
Data Replication
ETL
( Extract , Transform ,
Load )
SOA (Service Oriented Architecture) Framework
Key Process Integration Methods
Workflow EAI
Orchestration +ESB
ESB (Enterprise
Service Bus)
Web Services
EII (Enterprise Information Integration)
EAI (Enterprise Application
Integration)
ETL (Extract,
Transform, Load)
CDC (Change
Data Capture)
Data Replication
20 20
Overview of Integration Components
21 21
Enterprise Information Integration (EII)
Data Virtualization
As a project-oriented DI middleware, data virtualization is often referred to as virtual data federation, high-performance query or EII. As enterprise architecture, it is frequently described as a virtualized data layer, an information grid, an information fabric or as data services in SOA environments.
EII is a middle tier query server:
contains a metadata layer with
consolidated business
definitions.
Communicates through web
services, database connections,
or XML;
Listener waits for a request –
sends whatever queries are
needed across whatever data
sources are required to return
data to the requestor;
Metadata robustness is the
differentiator.
Federated data stores produce
accessibility to enterprise data
without forcing central control.
22 22
Technology Overview: EII (Enterprise Information Integration)
EII tools
Create virtual views of distributed enterprise data through queries executed in real time
without physically moving or copying data
* Also known as data federation; virtual data warehousing; data virtualization
Benefits
Latency: Through federated queries, information can be accessed within milliseconds
Storage: Data is not moved or copied from source systems, so additional storage is not required
Drawbacks
Volume: Should only be used for small targeted data sets
Quality: Minimal transformation capabilities — efforts to include will negatively impact
latency
While never making a material impact as a pure-play market, data
federation is an important part of the data integration platform, but need
to watch out for high level integration and maintenance effort
23 23
Enterprise Application Integration (EAI)
•EAI is focused on moving data between Enterprise Applications with business logic applied. It picks up an application transaction and initiates a transaction in another system, for example CRM system picks up a new order and enters it into your Financial Application.
Driven by business events
Connectivity between applications
Information consistency a key requirement
Bus/hub with application adaptors
Wire-level messaging protocols
24 24
Technology Overview: EAI (Enterprise Application Integration)
EAI Tools
These products, which started out as rudimentary software that supported basic messaging, routing, and data transformation needs, have grown into more sophisticated tools that now also provide full support for SOA as well as electronic data interchange (EDI).
Benefits
Latency: Through message based orchestration, information can be transferred within
seconds to service real-time data integration
Event based: Data transfer can be triggered by event
Drawbacks
Proprietary: Traditional EAI vendors used proprietary protocols
Quality: Data validation can be performed, however doing it with match &
merge multiple source systems is not the strength of this toolset
EAI toolset can be brought in as a middleware framework to support SOA,
however with insufficient in-house experience, a POC is recommended.
25 25
Extract Transform Load (ETL)
As a data integration hub, ETL products connect to a broader array of databases, systems, and applications as well as other integration hubs. ETL batch architecture is generally split into 4 major components: Extract, Clean, Transform and Load.
Provide expanded functionality, especially in
the areas of data quality, transformation, and
administration
Coordinate and exchange meta data among
heterogeneous systems to deliver a highly
integrated environment that is easy to use and
adapts well to change
Capture and process data in batch or near-
real time using a standardized information
delivery architecture
Provide greater performance, throughput, and
scalability to process larger volumes of data at
higher speeds
Load data more quickly and reliably, by adding
change data capture techniques, continuous
processing, and improved runtime operations.
26 26
Technology Overview: ETL (Extract, Transform and Load)
•ETL tools
Batch or incremental extraction of high volumes of data from one or more sources
Able to run complex transformations on the data which can include cleansing, reformatting, standardization, aggregation, or the application of any number of business rules
Loads the resulting data set into specified target systems
Benefits
Volume: Manages extremely high volumes of data movement
Quality: Allows for complex data transformations, enabling much higher quality, hence more usable information
Re-Use: Routines to extract/transform/load can be re-used by many applications
Drawbacks
Latency: Optimized when scheduled as batch data movement as opposed to real-time or on-demand. By reducing the volume throughput (with CDC) ETL can be used to meet operational near real-time requirements.
Performance: Extracts can cause performance impacts to source production systems, so low-impact batch extraction “windows” need to be identified, or use CDC.
ETL has spread beyond data warehousing and can supports near real-time
data integration for both operational and BI applications
27
Change Data Capture (CDC)
•CDC Integration Suite provides secure, high-volume, real-time, and bi-directional data integration and transformation between applications. This product supports a wide range of databases, including those that run on legacy, back-office, and other operational systems on different platforms.
27
Journal Log
Redo/Archive
Logs
Publisher
Engine
And Metadata
Subscriber
Engine
And Metadata
TCP/IP
GUI
Unified Admin
Point
With Monitoring
Databas
e
Audit
Database
Message
Queue
Web
Services
Business
Process
Publisher Subscriber
• Provide a pseudo to actual real-time update capabilities
• Heterogeneous system and platform support
• Real-time selective data capture and delivery
• Limited data transformation
• High performance even with very high volumes;
• Guaranteed integrity of data transactions- 2 phase commit
28 28
Technology Overview: CDC Change Data Capture
CDC tools
The optimal approach is to capture “deltas” or changes in the source data created or updated in operational systems as they are written to the DBMS log files and make them immediately available in real-time to downstream applications
Benefits
Volume: Captures only changes or “deltas” since last pull from source databases, reducing amount of data that needs to be moved
Performance: With the option to access database log files versus production database — no performance impact to source operational systems
Latency: Can enable continuous updates throughout the day
Drawbacks
Latency: No latency issue. Due to source log reading, source system down time could cause extra administrative task to synchronize and monitor data
Performance: Data transformation ability is more limited unlike ETL tools
CDC could be an option to combine with ETL for the enablement of near real-
time and more throughput, with less impact to source systems.
29 Copyright © 2006 Accenture All Rights Reserved.
Enterprise Service Bus (ESB)
• ESB is a product category in the integration market
• ESBs (as products) are a set of technologies developed to support program-to-program communication/integration (such as Web services, Object request brokers, Remote procedure calls, MOM-Message Oriented Middleware, etc.) and SOA
• ESBs are seen as a potential evolution of middleware technologies
All in one package (Administration and Management services, Service Definition tools and Repository services)
Combine features from previous integration technologies
Provide value added services:
Intelligent Routing
Message validation
Transformation
Security
Load balancing, etc.
Support highly distributed
architectures
ENTERPRISE SERVICE BUS
.NET
Application
SOAP/HTTP
J2EE
Application
JMS/JCA
Legacy
Application
JCA / MQ
Gateqay
Partner Web Service
SOAP/HTTP SOAP/HTTP Adapters
Enterprise
AppsEnterprise
Apps
Distributed
Query
Engiine
Database
Database
30 30
Technology Overview: ESB (Enterprise Service Bus)
•ESB tools
These technologies typically incorporate adapter technology to connect to a variety of application and database types, ability to route transactions according to business rules and transport transactions from source to target with low latency.
Benefits
Open: More open than EAI tools. Universal support for distributed processing
External Entities: ESB is easier to configure and implement; hence often chosen to
support B2B applications
Drawbacks
Vendor: ESB only vendors are smaller than EAI vendors
Experience: Lack of Chevron internal working and support knowledge
ESB is best used for establishing business processes (BPM) and
orchestration infrastructure that will leverage a business services layer to
support SOA across the entire enterprise. ESB federation can also be
implemented to mitigate drawbacks.
31 31
• Focus on the Differences…
Differences Between EAI and ETL
EAI ETL
Focus Application Integration
Process, B2B
Data Integration
Analytic, KPI
Timing Real-Time Batch, Near-real time
Data Transactional Historical
Transformation Minimal Complex
Interfaces Predictable Evolutionary
Volume Single Message or
Transaction
Bulk (Hour, Day, Week,
etc.)
32 32
IT Integration
Strategy Decision
33 33
Approach for Integration Strategy Decision
1. Translate BPR business process and functional requirements into system data flow diagrams, vetted and confirmed with business SMEs and AT teams. Categorize data integration requirements based on system data flow diagrams,
and RICEF list was created
2. Study standards and seek to understand environment
3. Leverage other Information Management initiatives and EA direction. Capture lessons learned from others projects.
4. Conduct technology scanning and gather industry information from Gartner, Forrester, Open O&M and vendors
5. Develop Integration strategy focused on Data Integration, that supports Requirements, Process Integration and Data Management
6. Vet with Architects - IT AA, IA and SIA teams to get feedbacks
7. Present Integration strategy recommendation to technical review team
8. Include stakeholder and technical review board feedbacks and update recommendation
34 34
Objective and Criteria for Data Integration Strategy Decision
Decision Objective
Must support the business’ need in delivering timely & well integrated data with consistent naming,
content and meaning; providing Console Operators a complete and concise view of trusted data.
Requirements: How well does the DI decision satisfy the requirements?
Standards: How well does the DI decision align with Chevron standards?
Reliability: Is the DI decision proven with robust technologies?
Interoperability: How well does the DI components interoperate?
Supportability: Does the DI decision match organization capability?
Total Cost of Ownership: Does this decision offer the optimum TCO?
Sustainability: Can the DI decision easily adapt to business changes?
Data Management: Does the DI decision support information management disciplines?
Decision Criteria
35 35
Key Integration Alternatives
Custom code remains a popular option
Key Data Integration Methods
EII
( Enterprise Information Integration )
CDC
( Change Data
Capture )
EAI
( Enterprise Application Integration )
Data Replication
ETL
( Extract , Transform ,
Load )
SOA (Service Oriented Architecture) Framework
Key Process Integration Methods
Workflow EAI
Orchestration +ESB
ESB (Enterprise
Service Bus)
Web Services
EII (Enterprise Information Integration)
EAI (Enterprise Application
Integration)
ETL (Extract,
Transform, Load)
CDC (Change
Data Capture)
Data Replication
36
Step 1 Business Requirement
36 “system to system” data interfaces in R00#1 IT Scope
25 bulk data on scheduled basis
7 on demand with low volume of data
4 real-time streaming data
37 37
Component Technology Standard
Data Integration
ETL tools
Managed Choice:
1. IBM InfoSphere DataStage
2. MS SSIS
Process Integration
EAI tools
Managed Choice:
1. SAP XI (version 3)
2. BizTalk
Integration
Middleware
Managed Choice:
1. Integration Brokers (EAI tools)
2. Web Services
3. Batch file transfer
4. Direct access
5. Intermediate databases
6. Custom built
Step 2 Standard and Usage
38
Step 3 Lessons learned from other projects
• Selected SOA architecture to facilitate multiple data integration points with real time BI integration
• Realized the value of Master Data Management with their SOA implementation
• Selected a hub and spoke architecture to facilitate multiple data integration points with complex data translations. Selected ETL platform for data movement for all planning and scheduling data.
• Realized the value of web services to facilitate work flow for data validation processes within the Refineries.
• Selected a hybrid architecture to facilitate multiple data integration points with complex data translations. Most data was required in real time to capture trade deals.
• Selected an ETL platform for application integration with robust transformation.
• Selected orchestration to facilitate work flow and data integration with external parties and systems
38
39
Step 4 Integration Capability Comparisons
Data Integration Technologies
ETL
EAI
(Orchestration)
Bulk Data Transfer
Real Time Messaging Routing
On Demand Data Integration
Metadata Data Management
Data Transformation
Process Orchestration
Distributed Processing
Data Standardization
Human Interfaces
SOA and Web Services Integration
Workflow
EII ESB
full support partial support no support
Step 5 Integration Components Chosen
40
Custom code remains a popular option
Key Data Integration Methods
EII
( Enterprise Information Integration )
CDC
( Change Data
Capture )
EAI
( Enterprise Application Integration )
Data Replication
ETL
( Extract , Transform ,
Load )
SOA (Service Oriented Architecture) Framework
Key Process Integration Methods
Workflow EAI
Orchestration +ESB
ESB (Enterprise
Service Bus)
Web Services
EII (Enterprise Information Integration)
EAI (Enterprise Application
Integration)
ETL (Extract,
Transform, Load)
CDC (Change
Data Capture)
Data Replication
Selection
41
Integration Strategy Decision
• Include both Process and Data Integration as a hybrid architecture
• Process Integration includes EAI Orchestration and Workflow
• Application Integration includes EAI for real-time data integration
• Data Integration includes ETL for non real-time bulk data integration
• ETL platform can be used to add on Data Management toolset
• Exclude technologies that do not meet requirements or criteria
• Custom coding does not meet supportability & TCO criteria
• EII does not meet Info Mgmt - data standardization criteria
• CDC for near real-time data can be handled by EAI
• Replication/CDC is already used by PI (ODS), but is not extensible
• ESB is not yet in CVX standard, EAI has some ESB features
41
42 42
Integration Conceptual Architecture Hybrid Technology
Data Mart
Human
Workflow
Process Integration
Orchestration (EAI)
Filter
Route
Other
Requesting
applications
Receiving
applications
XML
messages
Transform
Filter
Route
Service Calls Service wrapper
Guarantee
Data Integration
Hub (ETL)
Extract
Transform
Load
Profile
Quality
Metadata
Mgmt
MDM
Staging
Area
Data
Warehouse
ODS
(PI)
Source
Systems
OLAP
cube
HMI
Operational BI
Analytical BI
Target
Systems
43 43
Data Integration Strategy Decision Rationale
Requirements
• The combination of ETL, EAI & Workflow components satisfy the business requirements of bulk data transfer, real-time data integration and workflow.
• ETL platform is necessary to facilitate analytical BI environment. Mature ETL platform incorporates information management - data standardization toolset.
• EAI is required to facilitate the real-time application integration and automated work processes defined by the BPR teams.
Standards
• Ttechnology standards and best practices include Data Integration (ETL) and Application Integration (EAI) Toolsets.
• Select a Data Management standard for Master Data Management, Metadata Management and Data Quality.
Reliability
• Using ETL toolset to provide bulk and scheduled data interfaces as baseline. This technology has been proven and used by many projects.
• EAI technology is mature, EAI toolset (BizTalk).
44 44
Data Integration Strategy Decision Rationale - continued
Interoperability
• Partner orchestration of BizTalk and Share point portal.
• Standard ETL and BI toolsets with web service to prove interoperability across the platforms.
Supportability
• Continue to use PI as data transfer hub between Plant Information Network (PIN) and Process Control Network (PCN) - Leverage what exists increases supportability. Apply PCIS standards to utilize OPC for PI integration.
• GDST and Refinery have experiences with implementing and supporting ETL. ITC provides services for ETL support, database support and BI support.
Total Cost of Ownership
• Using EAI and ETL standard toolset to facilitate refinery centrally managed process and data flow would bring cost benefit in leveraging enterprise support and license costs.
• Infrastructure and resources for ETL toolset may be shared with Lynx within refineries which can greatly reduce license and support cost.
45 45
Data Integration Architecture Decision Rational - continued
Sustainability
• EAI and workflow provides the foundation for SOA framework and adaption of newer technology (composite software and Web 2.0 ) is feasible
• Once we gain more experiences with EAI and workflow toolset, it can be expanded to handle more integrations to accelerate SOA.
• We will maintain ETL as a foundation to add on real-time and on-demand components.
• SOA still maturing. Work with ITC to ensure we remain consistent with the company direction of SOA
Data Management
• Info Mgmt disciplines provide Data Governance, Master Data Management, Metadata Management and Data Quality improvement.
• Using data integration hub to provide standardized data layer provides a good foundation for information management.
46 46
IT Recommendation
Integration Strategy and Architecture
47 47
Integration Strategy Recommendation
Business Requirements ETL EAI Workflow
Near real time and scheduled Bulk Data
Conversion and Interfaces X
Integrate data from Operational BI to
Analytical BI (load ODS and staging data
into DW/Data Mart/OLAP) X
Real-time Integration (application to
application or Integration Hub to HMI) X
On Demand low volume of data (event
triggered data delivery) X
Human-centric Workflow with
orchestration X X
Provide services to HMI in connecting all
portals, application data, workflow data,
integration hub data and collaboration
data.
X X
Data Transformation, Meta Data
Management, Data Cleansing X
48
Integration Architecture
49
Next Steps
– Review Proof of Technology findings of EAI Tools
– Gather and review feedback to update the DI Strategy recommendations:
• Internal – IT EA and AA teams
AT team, if feasible
• External - Information Architects
– Recommend Data Integration Toolsets
50 50
IT Integration
Appendix – Lessons Learned
51
Lessons learned 1
– SOA architecture to facilitate multiple data integration points with real time BI integration instead of using an integration middleware
• However, this alternative carries a very large architectural footprint, higher. costs, and demands for technology expertise.
– Master Data Management with their SOA implementation
• A reference data model was added to the SOA implementation when data quality issues were surfaced due to disparate data sources.
51
52
Upstream Foundation Services vs Data Integration
SO
– Small messages on demand
– Transformations tend to be simple
BI
– Infrequent exchanges of (large) amounts of data
– Transformations complex
– Increasing drive for Real Time DW
SOBI
– Leverages the strengths at the extremes
– Exploits the middle ground
Messages vs. Data
SO BI
Fine Grain
Services / Real-
time events
Medium Grain
Services
Coarse Grain
Import / Export /
ETL
53
SOBI Summary
Service Orientation (SO) Business Intelligence (BI)
• Provides application-to-application integration
• Well suited to events and real-time data – high frequency
• Allows agile change in business processes
• Supports reuse of enterprise components
• Encapsulates and abstracts functionality
• Tightly defined data formats and structures
• Well suited for data-to-data integration
• Can handle large data volumes
• Provides foundation for business decisions
• Provides a combined model of the enterprise data
• Good tools and mechanisms for transforming data
• Ability to question the data and to answer key business questions
54
Solution Architecture Services Integration pattern
Business Message Standards (schemas & semantics)
Presentation Presentation Services
(Analysis & Reporting)
Business Analytics
& Analysis Services
(Dimensional Models)
Integration
Services
Physical Integration
Project
DB
Extract, Transform &
Load
Virtual Integration
Data Integration Message Standards (schemas & semantics)
Services
Atomic & Composite
Entity Services
Proc
ess &
Workf
low
Business
Services Production
Drilling HES
Maintenance Financial
Well Reservoir
Surveillance
Analytics
Notification
Data • Enterprise
• OPCO
• SBU
• Asset
Message Standards (schemas & semantics)
Applicati
on Data
Facade
Document
Repositories
Facades
Data
Sources
Facades Systems of Record
(SoR)
Hie
rarc
hy &
Cro
ss R
efe
ren
ce S
erv
ices
Master,
Reference
& Hierarchy
SUPER 7
• Well
• Reservoir
• Equipment
• Field
• Property
• Location
• Facility
55
Lessons learned 2
– Select a hub and spoke architecture to facilitate multiple data integration points with complex data translations. Most data required for 1-7 day plans.
– Use ETL platform for data movement for all planning and scheduling data.
• Several ODS tables and data warehouse structures were built in the central hub (San Ramon) with supporting individual hubs within each refinery
• A robust cross reference model was used for the numerous codes and data sources to provide a consistent name and definition of master data across the supply chain.
– Use the value of web services to facilitate work flow for data validation processes.
• A web services front end was added to the Validation Tool that provides updates and corrections for data to be used in the scheduling tool (SIMTO)
55
56
Conceptual Architecture Hub and Spoke Pattern
External
Source
Systems
SRA
(Crude)
ICTS
SAP
PS DF RSPF RBS&OPTI
(SRA)
ETL
“Full visibility” with
limited event
notification
capabilities
“Integration”
P to P
Interfaces
(Driven by SubTeams) –
(Stored Procs, ETL
or Connect Direct)
We
b a
cc
es
s D
as
hb
oa
rd
/KP
I
Ly
nx
Re
po
rtin
g/A
na
lytic
s A
rc
hite
ctu
re
AD
HO
C R
ep
ortin
g/Q
ue
rie
sD
rill D
ow
n, O
LA
P
Metadata
Re
po
rtin
g/A
na
lytic
s T
oo
l
Lynx Data
Warehouses
(regional & global)
Operational Data
Stores/StagingETL
Common Data Model
Master Data
Management
Common Business Transformations
SQL Server database
ETL“Availability”
57
Lessons learned 3
– Hybrid architecture
– To facilitate multiple data integration points with complex data translations. Most data was required in real time to capture trade deals.
– ETL platform for application integration with robust transformation.
– Orchestration tool (Bitzttalk) facilitates work flow and data integration with external parties and systems.
57
58
Logical Architecture - Hybrid integration pattern
ServicesServices
Service Providers
Transport Providers
Pipeline
3rd Party
Leases
Extex(Royalty Payments)
Market Data
Providers
Deal Confirm
Exchanges
Banks
Counterparty
Inspection
Terminal
Ship
Ports
Rail
Trucks
4GEN
Tax
“SOG”
“Corporate Credit”
Cashflow
Netback
“Master Data”
- EA Master Data
- EA Facilities
- SAP
SAPSAP
Rolfe & Nolan
NAVARIK
Trading 1 Trading 2 Trading 3
Price
Credit
MDM +Xref
O
R
C
H
E
S
T
R
A
TI
O
N
O
R
C
H
E
S
T
R
A
TI
O
N
RTR
Intraday PositionSnapshot DB
ETL
BI
Document
ManagementMRA
SAPXI
MPA Price
Noms
Confirms
Ship Status
Lifting
Schedules
Ratings
Deals
Statements
Royalty
Vols
Deals
Corporate
Credit
Services
SOG
Services
SAPXI
TAX
Clients
Port Activity
Credit Limit
Master Data
Credit Services
Risk
Algo
License
Mgm
R&N
Services
Risk Services
Price Services Master &
Xref Services
Unstructured
Market/CP dataMaster Contracts
Lease Vols
Rates
Payments
Actual
Volumes
Inspection
Reports
Consolidated
Position
Viewer
Brokers
Rating Agency
RailTrac
CVMS/Shipnet
Clients
Confirms
Tickets
Schedules
Refineries
Exchange
Allocation
CP
Services
Banking
Services
Exchange
Cuts
News/Data
AR/AP/GL
Invoices
Credit
Exposure
Plans
Rail Car
Ship info
Ship
Schedules
Movement
Tools
Enterprise
Facilities
Credit
Engine
Valuation
Libraries
59 59
IT Integration
Best Practices
60 60
Best Practices for Data Integration
1. Don’t loose sight of DI Architecture vision, however include tactical data integration solution for specific business requirements. (phased approach)
2. Categorize data in business value and usage. (prioritize)
3. Prioritize the sequence of implementing data integration. (sequence)
4. Document data migration and infrastructure deployment roadmap.
5. Establish new standards for naming, data types and metadata. (governance)
6. Publish metadata definitions and glossaries of business terms.
7. Establish a coexistence strategy with legacy systems. Always have a migration plan.
8. Establish physical reference architecture and tools.
9. Implement environments for the foundation components ahead of time.
10.Begin data migration into the integrated environment.
61 61
Planning for Data Migration
•Data Migration (Conversion) from legacy system to the newly integrated environment needs to be considered carefully by weighing highest value vs. highest usage.
– Foundation Data Migration - Implementing the main lookup data, or master data, for enterprise
– Core transactional data migration - Detailed transactions for the basic enterprise events
– Application data migration – Supports specific company functions
•This strategy leverages the building of foundational master data that will be most often queried by end users, then adding core transactional data that adds value and incrementally allows more business value as data becomes richer in content.
62 62
Data
Governance
Data
Structure
Data
Quality
Data Management Capabilities
Data
Creation
Data
Storage
Data
Movement
Data
Usage
Data
Retirement
• Data Ownership
• Data Stewardship
• Data Policies
• Data Standards
• Data Workflow
• Data Modeling
• Data Taxonomy
• Business Process
Flows
• Data Profiling
• Data Cleansing
• Data Transformation
• Data Monitoring
• Data Compliance
• Data Traceability
Master Data &
Metadata
• Master Data
Management
• Reference Data
Management
• Metadata
Management
Data Management Foundation
63 63
Integrating Data Content and Meaning
•Another aspect of data integration is standardizing the usage of data content and meaning. This type of data content integration yields business efficiencies and quality of data.
– Integration of content standardizes data values, e.g. lookup codes, across different data bases. (For example, if PI Tag or P&ID needs to be uniquely identified at the global level across all refineries, a newly defined unique ID can be created and tied with existing ID.) Depending on local operation or global data analysis, two sets of ID can be translated and delivered to satisfy user request.
– Besides the physical data movement and storage of integrated data bases, the common integration of data meaning needs to be standardized. Metadata provides definitions of subject areas, tables, and columns in a data repository.
– When all users refer to the data repository, the meaning of each data element is standardized to a common definition.
– Additional metadata can be provided that displays calculations for derived data elements, glossaries of business terms, and lineage of the source of data.
64 64
Data Integration Architecture considerations
Commonality, consistency and interoperability of DI components:
Minimal number of products or product suites supporting all data
deliveries
Single metadata repository and/or the ability to share metadata
across all components
Common design environment to support all deliverables
Interoperability with other integration tools and applications
Efficient support for all data deliveries regardless of runtime
architecture (centralized vs. distributed )
65 65
Decision Making methodology Top-down
•Integrate Use Case with Pattern Matching
•Using integration-pattern matching, look for matches by comparing their specific use cases with “typically deployed” Data Integration patterns. Examples:
• To improve Global Manufacturing-wide reliability reporting, the appropriate integration pattern would be an enterprise data warehouse that physically consolidates and summarizes OE data from across all refineries.
• To provide operational DCS information to business level applications and for operational BI, an replicated operation data store that stores up-to-the-second transactional data would be the best fit.
• To support upstream or downstream product movement analysis and establish a performance, a data mart or an OLAP cube sourced from the ODS or Data Warehouse would be the best pattern.
66 66
Decision Making methodology Bottom-up
Assessing integration factors
This is often valuable where the DI decision is complex and/or where a clear integration pattern match is not obvious. For example, to determine whether virtual, physical or a hybrid combination:
If data extracted from many source systems could be used by many other systems, then physical data store is good for data reuse and future expansion.
If significant data cleansing and complex transformation are required, then physical data consolidation is typically the most practical choice.
If harmonized data need to be aggregated, summarized to provide for analytical dashboard, then physical data store is needed to load into Data Warehouse/Data Mart and/or OLAP cubes.
If source systems are mostly available as system of record, data can be passed between systems without significant data matching, merging or harmonizing, then virtual makes sense.
Hybrid combination may be a good choice if a project has both real-time business process integration and large amount of data interfaces.
67 67
Data Integration Patterns
68 68
IT Integration
Industry Research
69 69
Integration Product Comparison
70 70
Magic Quadrant Data Integration Tools
71 71
Traditional EAI vs. ESB
Lightweight, distributed, standards-based and inexpensive Complex, proprietary, centralized, and costly integration
Flexible and adaptive business logic Lack of support for new business logic
Abstraction Known Implementation
Message Oriented Object and Message Oriented
Loosely Coupled with coarse-grained Business Services Tightly Coupled with use of proprietary adapters
Services Orchestration Application Block
Designed to change Designed to last
Process Oriented Functionality Oriented
Service Oriented Architectures hub-and-spoke architecture
ESB Traditional EAI
72 72
Use Cases of EAI, ETL, EAI + ETL
•EAI Software An example - During the Internet boom, companies flocked to EAI to connect e-commerce with back-end inventory and shipping systems to reflect product availability and delivery times.
•ETL toolset in an ‘always awake’ mode – near real time
To deliver near-real-time capabilities. The ETL tools typically use application-level interfaces to detect new transactions or events as soon as they are generated by the source application. They then deliver these events to any application that needs them either immediately (near real time), at predefined intervals (scheduled), or when the target application asks for them (publish and subscribe).
•EAI plus ETL
EAI tools captures data and application events in real time and passes them to the ETL tools, which transform the data and loads it into the BI environment.
73 73
73
What vendors say about ESB?
– Some stress the role of the ESB in eBusiness, its inter-organizational. Rather than intra-organizational role
– Almost all believe, that the ESB is more than the bus it runs on. Essentially, they are describing a service-oriented architecture from another viewpoint
– Some see orchestration as part of the ESB architecture, others do not
– Some package MOM and EAI in their ESB products
– Some identify event monitoring as the major differentiator from MOM
– Some consider services management as part of the ESB solution
– Some see an ESB as strictly related to Web services and describe it as a Web Services
Network.
All Vendors are “flexible” in defining ESB. Their definition always manages to show that their current solutions are using it
74 74
ESB, When to Consider
– When deploying SOA across the enterprise
– When establishing business processes (BPM) and orchestration infrastructure that will leverage a business services layer
– When moving from a complex point-to-point or ‘spaghetti’ architecture to a more manageable and flexible IT infrastructure
– When integrating to multiple and heterogeneous data sources and applications
– When there is heavy business logic and security through the service bus to multiple end points
– When further separation from composite applications is required (away from underlying implementations)
– When flexible coupling is required
75 75
Information (Data) Services in SOA
•For data to be a first-class citizen in the SOA world, a clear separation must exist between data consumers and data providers. This separation mirrors the principle that service consumers and providers must be distinct and separate in an SOA. Furthermore, this separation must be delineated by an interface, or contract, that both providers and consumers share
76 76
Gartner on SOA and Data Services
Gartner suggested that success in loosely-coupled service-oriented business applications (SOBAs) becomes more difficult since each design point has to verify it own semantics, context and data structures.
Key Findings Under a loosely-coupled architecture, data stewardship and governance best practices can be supported by data services within an SOA instead of embedding such practices within application logic. Where people and processes were formerly embedded in application design, they now fall under the domains of business process platforms and EIM - Enterprise Information Management.
Predictions Based on lessons learned through data warehouse, data mart and operational data store implementation practices, 60% of failed information-as-a-service initiatives through 2009 will list a lack of an effective data governance strategy as one root cause of failure.
Recommendations
Organizations should begin their selection of data profiling, quality, mining and master data management tools with the end goal of deploying all the logic and processing within these tools as services that can interoperate and execute actions on behalf of and against data used by SOBAs, and as a callable service by business context services.
77 77
Composite solutions
• Some of the approaches promoted by the Web 2.0 movement (mash-ups, RIA - Rich Internet Applications) are moving the Integration challenges up to the presentation layer
SAP Work Management
& Purchasing Personal
Management
Drilling Information
Collaboration
"As Is"
Business Process: 3.0 Set-up New Well
Sub process: 3.3 Set-up Well Ownership
Company: APC
Verison 1.0, Version Date 2/28/01
3.3.2
CREATE TEMP WELL
FILE AND CHECKLIST
OF STEPS TO
COMPLETE D.O.
PROCESS
(LAND CLERK)
R.O.W.L.DRILLING
TITLE
OPINION
TITLE CURATIVES
CONTRACTS AND
LEASES FOR UNIT
PLAT (IF NEEDED)
SPACING/ POOLING
INFORMATION
3.3.5
DELIVER WELL
FILE TO D.O.
MANAGER
(LAND CLERK)
3.3.6
ASSIGN WELL FILE
TO LAND ADMIN
DIVISION ORDER
ANALYST
(D.O. MANAGER/
SUPERVISOR)
3.3.7
REVIEW WELL
FILE FOR
COMPLETENESS
(LAND ADMIN)
3.3.9
ANALYZE AREA
TO DETERMINE IF
IN A PRIORITY
MARKETING AREA
(LAND ADMIN)
PAPER PAPER
3.1.19
TRACK PARTNER
AFE RESPONSES
(LAND ADMIN)
3.5.1
PLACE DRILLING
REPORT WITH "FINAL
REPORT" STATUS ON
NETWORK DRIVE
(PROD CLERK)
3.3.1
SET-UP 100% APC
BILLING SCHEDULE
IN EXCALIBUR
(JIB)
A
B
3.3.3
SEND R.O.W.L. TO
JIB
(LANDMAN)
PAPER
3.3.8
COORDINATE WITH LANDMAN
FOR MISSING FILE INFO. (LAND
ADMIN DIVISION ORDER
ANALYST)
3.3.4
UPDATE BILLING
SCHEDULE WITH
TRUE JIB
INTEREST
(JIB)
PRE-DRILL ACTIVITIES
"As Is"
Business Process: 3.0 Set-up New Well
Sub processes: 3.1 Set-up Drilling AFE
Company: UPR
Version 1.1, Version Date 3/5/01
3.1.1
RUN WELL
ECONOMICS IN
OGRE
(RESVR ENGR)
3.1.2
TEAM MTNG TO
COMMUNICATE
NEED FOR AFE,
LEASE AND
WELL STATUS
3.1.3
SET-UP WELL
NUMBER IN
WINS
(ENGR TECH)
3.1.4
CREATE $0.00
PENDING AFE
IN WINS
(LAND SPEC)
3.1.5
COMPLETE AND
PRINT AFE (LAND
SPEC)
3.1.6
ENTER $0.00 AFE
IN EXCALIBUR
(FIN SPEC)
3.1.9
APPROVE AFE
BY COMMITTEE
MEETING
(CROSS-DEPT)
E-MAIL,
PHONE or
FAX
3.1.7
NOTIFY
LANDMAN AFE IS
COMPLETE
(LAND SPEC)
PRINTED
INTERNAL AFE3.1.10
SEND SIGNED
AFE TO
FINANCIAL SPEC
(LAND SPEC)
3.2.2
SET-UP WELL
NUMBER IN
PERC/ DIMS
(AUTO)
AUTO
3.1.8
NOTIFY
ENGINEERING
TECH AFE IS
COMPLETE
(LAND SPEC)
SIGNED
AFEA
3.2.1
SET-UP WELL
NUMBER IN
EXCALIBUR
(AUTO)
MARKETING
PRICE
INFORMATION
G + G
FORECAST
ECONOMIC
FORECAST
WELL-UNIT
OWNERSHIP
(LANDMAN)
"To Be" for 2001
Business Process: 3.0 Set-up New Well
Sub-process: 3.3 Set-up Well Ownership
Version 1.5, Version Date 7/18/01
3.3.1PREPARE STAKE/
PERMIT PACKAGE
IN WORD
(LANDMAN)
3.3.4
BEGIN RELEASEOF WELL
LOCATION MEMO
(ROWL) IN WORD
(LANDMAN/LAND
EXPLORATION
SPEC)
3.3.2
ORDER
TITLEOPINION(S)
(LANDMAN)
3.3.3
BUILD
WELL/ UNITFILES
(LANDMAN)
3.3.7
EVALUATE PIPELINECONNECTIONS TO
WELL, PRIORITY OF
MARKETING AREA
(FIELD SERV)
A
3.3.5REVIEW JOA
CONTRACT
OWNERSHIP IN
CONTRACTS
(LANDMAN)
3.3.6
REVIEW OR
CREATE CROSS
REFERENCE OFJOA TO WELL(S) IN
WINS
(LANDMAN)
3.3.8
CAPTURE
PRELIMINARY
WELL OWNERSHIP
IN ROWL(LANDMAN)
PRE-DRILL ACTIVITIES
3.1.10
RECEIVE REQUEST
FOR NEW WELL
DRILL AFE
OWNERSHIP (LAND
ADMIN SPEC)
3.3.10
ENTER LEASES AND
CONTRACTS INTO WINS;
SET UP APO INTERESTS;SET INTEREST FINAL-LAND
FLAG
(LAND EXPLORATION
SPEC)
3.3.13
REVIEW MKTG
ARRANGEMENT
SET-UP FOR ANYOWNER CHANGES
(FIELD SERV)
3.3.14
REVIEW JIB
DECK FOR ANY
OWNER
CHANGES
(JIB ACCT)
3.3.12COMPLETE AND
APPROVE ROWL
(LANDMAN/LAND
EXPLORATION
SPEC)
3.3.9
SEND WELL WORKING
INTEREST PARTNERS
AND PERCENTAGES TO
BUSINESS SERVICES(LAND EXPLORATION
SPEC)
EMAIL AND POST TO NETWORK DRIVE
3.3.15
ANALYZE ROWL FOR
DRILLING/
COMPLETION INFO IN
DIMS, WINS, PDB(OPERATIONS TECH)
3.1.33
UPDATE FINAL
INTERESTS BASED ON
PARTNERS' RESPONSES
IN ROWL (LAND ADMIN
SPEC)
EMAIL 3.3.11ADD/COMPLETE
NACU DATA TO
ROWL
(LANDMAN/LAND
ADMIN ANALYST)
TITLE CURATIVE,
TITLE OPINIONS,
ETC.
3.3.16
3.4.1
PRELIMINARY D.O.HEADER AUTO
ESTABLISHED IN
DOMAIN
Documents
Knowledge
Management
Planning
Process Guides
• Presentation Integration Servers
enables the creation of Composite
Applications by introducing a level
of orchestration between the
presentation layer of “legacy”
and composed applications
• Business processes are packaged
and reused by BPM tools
introducing business process
layer composition
• Solutions are built by combining
capabilities at every level of
the software stack: data,
process and presentation
Web 2.0 (1)
78
One’s view of Web 2.0 is highly dependent on one’s background and interest, and can best be described by these three anchor points:
Technology and architecture – consists of the infrastructure of the Web and the concept of Web platforms. Examples of specific technologies include Ajax, Representational State Transfer (REST) and Really Simple Syndication (RSS.) Technologists tend to gravitate toward this view. Community and Social – looks at the dynamics around social networks, communities and other personal content publish/share models, WIKIs, and other collaborative-content models. Most people tend to gravitate toward this view, hence, there is a lot of Web 2.0 focus on “the architecture of participation.” Business and process – Web services-enabled business models and mashup/remix applications. (A mashup is a Web site of Web application that combines content from more than one source.) Examples include long-tail economics and advertising and subscription models such as a service (SaaS.) Of course, business people tend to zero in on this angle.
79
Web 2.0 (2)
• What's Old Is New Again
• Most of what people call Web 2.0 is not entirely new. Many of the concepts and
technologies have existed for some time:
• For example, RSS is essentially the same as resource definition framework, a format
popularized by Netscape during Web 1.0 and the hype around push technology.
• Ajax is essentially JavaScript, dynamic HTML and asynchronous XML, all of which
have existed for more than five years and have become well-known with the advent
of high-profile implementations such as Google Maps.
• Certainly, collaboration and advertising are not new.
• Mashups bear a striking similarity to the SOA-derived term "composite
applications." What is new is how some of these are used, and in what
combinations.