real-time data integration best practices and architecture
Post on 26-Jan-2015
135 Views
Preview:
DESCRIPTION
TRANSCRIPT
Real-Time Data IntegrationBest Practices and Architectures
Presented by Nelson Petracek, Informatica
Real-timeData Quality
Missed Opportunities
Lower Customer
Service Levels
The Need for Timely Information
2
Customers
Indirect Sales Channel
Sales & Marketing
Enterprise
Product Designers
EnterpriseInformation System
Retail Stores
Call Center
Increased Costs
Market Share Erosion
Increased Exposure to
RiskRT Data
Warehouse
Data Synchronization
OperationalData Hub
DataWarehouse
OperationalData Hub
3
Applications created in 2012 using traditional architecture models will be an IT-constraining
legacy by 2016
The leading business applications of 2016 are designed today using Nexus-enabled
application architecture principles
Changing Perspectives on Data
• It is no longer sufficient to view information “after the fact”.
• Business demands information sooner, with more accuracy, in order to meet competitive and regulatory demands.
• Business needs to respond to “threats” and “opportunities sooner.
• Reduce decision latency.
• Proactive Alerts and Notifications
4
What does “Real-Time” mean to you?
5
DaysPerformance Management
Sales by Business Unit
HourInventory Management
Compliance
MinutesOrder ConfirmationOperational Metrics Second
Straight-through ProcessingCall-center Analytics
Point-of-entry Applications
InstantaneousSecurities Exchange
Operational Data Integration
Analytical Data Integration
Application 2
Traditional Integration
6
Application 1
Application Logic
Data
Application Logic
Data
Application 3
Application Logic
Data
Custom Extracts and Interfaces
Query/pull
Interfaces Interfaces
Query/pull
“Siloed” Applications, Embedded Interfaces and Transformation
Interfaces
Critical Considerations
• Complexity of source and targets.
• Data formats.
• Level of quality.
• Availability of source/target interfaces.
• Volume, velocity; and variety.
• Delivery requirements.
• Real-time, near real-time, batch/scheduled.
• Loose coupling/reusability.
• Performance/Availability.
8
Real-time Data Integration Patterns
• Transactional Data Processing
• Data Replication
• Data Integration Hub (DIH)
• Event Driven Architecture (EDA)
9
Applications Processes BI Tools Portals Web Services
Batch Trickle Feed Real Time
Applications Databases Messages Flat Files XML UnstructuredData
Mainframe
Data Consumers
Data SourcesOperational Data (Field Devices, Applications, etc.)
LocationContext
(GIS)
BI, DW, Other Targets
Event BasedApplications
Communication “Cloud”
Data Receipt / Transformation & Mapping
Real Time Event Transport / Delivery
Event Processing
Real Time Web Content Delivery
IDRPWC/PX
SourceApplications/Technologies
Interaction management(Publications & subscriptions)
Data Governance
Persistence
Hub Management
ArchivingDelivery & Schedule
CatalogService
Data & Event Monitoring
Validation & Quality Rules
On-boardingMonitoring &
Auditing
Data Transformation & Enrichment
Cloud App (HR)
POS App
CRM
Planning
Master Data
DataWarehouse
Big Data(Analytics)
Finance
Transactional Data Processing
10
1
Operational BI & Real-Time DWProvide freshest information to business w/o impact, off-load reporting, & do high volume updates (ODS/OLTP/DW/Appliance)
2
Operational (Sync)Ensure all users are seeing the same,trusted, up-to-date data that is synchronizedacross the organization.
Transactional/ProductionApplications
Merge/Apply,Reports & Queries
Source System Target Systems
ODS/OLTP/DW/Appliance
TransactionalApplications
Merge/Apply,Reports & Queries
Source System Target Systems
ODS/OLTP/DW
Transactional Data Processing
11
EXTRACT
SERVER
MANAGER
SERVER
MANAGERhttp://
APPLY
Console
Source System Target System
SQL Apply Merge ApplyAudit Apply
Intermediate Files
CommittedCheckpoint
CheckpointHigh SpeedExtraction
High SpeedParallel Apply
JMS*
In-Memory Cache Synchronization
12
Cache Node
Cache Node
Cache Node
Data Sources
Real Time Data
Replication
In Memory Grid
Consuming Applications
Data Integration Hub
13
Data Integration
Hub(Publish/
Subscribe)
Define Topic
Define Publishers
(Connection properties,
Mode, Frequency)
Define Transformation and validation
rules
Define Security &
Access Policies
Define Subscribers (Connection properties,
Mode, Frequency)
Deploy (Instantiate persistence
and data integration
flows)
Monitor, Govern, Alert
& Audit
Cloud App (HR)
POSApp
CRM
Planning
MasterData
DataWarehouse
Big Data(Analytics)
Finance
Data Integration Hub
14
Pu
b.
Ide
nti
fic
ati
on
Da
ta Q
ua
lity
Fir
ew
all
State logging, Error handling
XM
L N
orm
ali
zati
on
Su
b.
Tra
ns
form
ati
on
Sy
nc
-up
Co
ntr
ol
Su
b.
Ide
nti
fic
ati
on
Su
b.
De
liv
ery
Su
b.
Ac
kn
ow
led
ge
me
nt
Command Interface
Monitor and Alert Interface
TransferInterface
Applications Applications
TransferInterface
EnterpriseScheduler
EnterpriseMonitor
File Manager
Connector Connector
Prompt:$ SNMP
Prompt:$
Event Driven Architecture
• An architecture in which the activity is driven by changes in state within an environment.
• Events drive the execution of logic.
15
Ev
en
t S
ou
rce
s (
Pro
du
ce
rs)
De
tec
ted
Situ
atio
ns
(Co
ns
um
ers
)
Inte
gra
tio
n/D
ire
ct In
teg
ratio
n/D
irec
t
Pre-Process
Process
Post-Process
Patterns
Alerts
Event Streams
Actions
Devices
Systems
Applications
People
Sample EDA Reference Architecture
16
Operational Data (Field Devices, Applications, etc.)
LocationContext
(GIS)
BI, DW, Other Targets
Event BasedApplications
Connectivity & Data Replication
Transformation & Mapping
Real Time Delivery
Event Processing
Real Time Web Content Delivery
Event Enablement and Transformation
17
• Event-enable and collect events from underlying systems and applications.• Data Replication
• PowerExchange/PowerCenter
• Transform events to normalized format for downstream processing.• B2B Data Transformation
• Enrich events as needed.
Transformation & Mapping
Connectivity & Data Replication
Event Transport
18
• Deliver transformed events to downstream applications, systems, and event processors.• Ultra Messaging
• Dynamic Routing
• Parallel Persistence
• Guarantees delivery, and decouples event producers from event consumers.
Real Time Delivery
Event Rules and Delivery
19
• Processes events, applies user/business defined rules, and generates responses.
• RulePoint
• Rules may be applied across events and over time/space.
• Alerts/notifications may be sent to BI environments, web applications, etc.
Real Time Web Content Delivery
Event Processing
Architectural Implications
20
Information and Data Architecture
Application Architecture and Development
Infrastructure Architecture
• Shift to modeling events as enterprise assets. Incorporate the processing of immediate, individual events.
• Deliver quality data as it is needed by the business.
• Complement existing SOA strategies with eventdriven applications.
• Move to a single environment that supports different integration delivery models.
• Shift from centralized, database-centric client-server applications to distributed systems.
• Deploy infrastructure that can support data delivery over different latencies.
Summary
• Augment traditional integration solutions with real-time data integration to:
• Reduce decision latency.
• Improve decision making and responsiveness.
• Increase visibility.
• Build upon the principles of SOA and an event-driven architecture (EDA).
• Informatica provides the capabilities for adding real-time data integration to your environment.
21
top related