how cme group used informatica to merge three major ... · pdf fileto merge three major...

1

2

How CME Group Used Informatica to Merge Three Major Exchanges While

Reducing Support and Processing Times

Hadi Abdul @Hadi_Sid

Lead Systems Analyst

Amanda Morris @ProManda

Lead Software Engineer

CME Group @CMEGroup

3

Agenda

• About CME Group

• Business Challenge

• Solution Case Studies

• Benefits and Use of Informatica

• Future Roadmap

• Lesson Learned

• Q & A

4

• The world's leading and most diverse derivatives marketplace

• Result of mergers between three major exchanges

• 2007 merger of Chicago Mercantile Exchange and Chicago Board of Trade

• 2008 acquisition of New York Mercantile Exchange

• Headquarters in Chicago – offices around the world

• Partner Exchanges – DME, Korean, and Green

5

CME Group Trading Floor

Also…Electronic Trading and

Privately Negotiated Trades

6

CME Group Mergers and Acquisitions

2007

2008

2010

2011

Chicago Mercantile Exch merges with Chicago Board of Trade

New York Mercantile Exch and Commodity Exch Acquisition

Dow Jones Acquisition

Partner Exchanges – Dubai and Korea

• Combined organization became known as CME Group

• Integration completed in 2008

• Now four exchanges operating under CME Group

• Integration completed in 2009

• 90% investment in Dow Jones Indexes

• Including the Dow Jones Industrial Average

• Continuing to seek global opportunities

7

CME Group Futures & Options Products

• Crude Oil (WTI) , Natural Gas, Heating Oil Energy

• Gold, Silver, Copper Metals

• Corn, Wheat, Soybeans, Cattle Agriculture

• Eurodollar, Treasury Notes and Bonds Interest Rate

• S&P 500, Dow Jones Industrial Average Equity Indexes

• G10 and Emerging Market Currency Pairs FX

• Interest Rate Swaps, Credit Default Swaps OTC

• Home Price Index Futures Real Estate

• Hurricane, Snowfall, Rainfall Weather

8

Business Challenge

• Data from 3 different exchanges needed to be in

a single place in a single format

• Complicated by complex code base and

divergent systems

• Aging Technologies needs to be replaced

• Mainframe decommission deadline

• Volume increased exponentially, SLAs became

harder and harder to meet

• Number of trades doubled from 1 to 2 Billion over 4 years

9

CME Group Data Processing Facts

• Data in many formats • Oracle, SQL Server, DB2, flat files, XML, FIXML,

Spreadsheets

• Over 300 distinct production Oracle Schemas

• Large data sets • 15 TB data warehouse

• Globex Audit Data tops at about 500+ million records a day

• Millions of trades and quotes a day, copies everywhere

• Diverse technology • Java, .NET, shell scripts, perl, PL/SQL, COBOL, DB2,

other ETL tools

10

Pre-Integration Data Flow

DB2 Chicago

Mercantile

Exchange

Chicago Board

Of Trade

New York

Mercantile

Exchange

Oracle

Mainframe ETL

Flat Files

Java

Oracle ETL

Oracle/Excel

Streetbook/

Bridgeback

11

Post-Integration Conceptual View

Market

Public Data

Pre Clearing

Trade Data

Post Clearing

Trade Data

Reference

Data

ERP

Other Data

Extract Control

Source Data

Cleansing

Data Integrity

Reference Data

Management

Business Rules

Error Handling

Market

Data Liquidity

Volume Order

Quantity

Order

Execution

Trade

Revenue

Market

Maker Regulatory

Other

Internal Reports

Ad Hoc Requests

External Reports

Dashboards

Business Objects

Source Data ETL Architecture Data Warehouse Applications

Client Applications

12

EFFICIENCIES GAINED IN POST TRADE REPORTING

Case Study 1

13

Stats

Data Warehouse

Report Engine

CME Group and Partner Exchanges

Reports

Prices

Other Customers

Product Volume/OI Settlements

Data Cloud

Forward Contracts

Post Trade Reporting Informatica Utilization

14

Users and Reports

• Users

• Over 350+ internal users

• Reports, Dash Boards

• 3000+ hits per day for trade data reports

• Traders, brokers, investors, and universities.

• Reports

• Business Objects, Qlikview, and Crystal Reports

• Over 300+ reports for external customers

• Over 1000+ internal reports

15

CME Group - Published Data

Crude Oil and Gold Futures Contracts

16

Efficiencies Gained

• Mainframe ETL Processing Time Lines

• ETL Processes Completed around 11 PM

• Reports were being generated at 1:30 AM

• Informatica ETL Processing Time Lines

• ETL processing completes at 8:30 PM

• Reports Generation completes at 9:30 PM

• Trade Data Load

• Mainframe ETL: 55 Minutes

• Informatica ETL: 22 Minutes

• Gain of 60%

17

Batch Processing Efficiencies Gained

• Less jobs running due to simplified processes

• SLAs consistently met

• Improved Performance

• Reuse versus customization

• Ease of Maintenance

• Much Shorter Learning Curve

18

Informatica Processing Benefits

• Ease of Troubleshooting

• Java error vs. Informatica error

• In the middle of the night, which would you rather dig through?

• Standardized error handling

• Programmers often “forget” to include error handling or miss possible unexpected exceptions

• Standardized monitoring

• All processes monitored side by side in standard format

19

Informatica Project Development Benefits

• Safely retire legacy applications

• Resources can easily retrain to Informatica

• Keep people with valuable business knowledge

• Even as systems are decommissioned

• Share and reuse of code for faster delivery

• Faster development cycle

• Minimize business outages

20

SIMPLIFYING REGULATORY BATCH PROCESSING

Case Study 2

21

Past: Regulatory Trade Processing

Flat Files

DB2

Oracle

PL/SQL

SQL

Loader

Mainframe

Flat File

Interface

Java

“Splitter”

COBOL

Flat Files

External

Tables

Java

Processing

Informatica

Distributed

22

Complex Code

• Overly complex code used to move data

repeatedly

• Mainframe decommissioning – opportunity for simplifying?

• Java programs that were essentially ETL

• PL/SQL packages, perl scripts, shell scripts, etc.

• Lots of duplicate, obsolete, and unnecessary

coding

• Hard to support

23

Some people see a problem and think “I

know, I’ll use Java!”

Now they have a ProblemFactory.

@Lonnen Chris Lonnen Twitter

24

Informatica to Simplify

• Take a hard look at your code base

• Do you really need a custom app and framework for what is essentially an ETL process?

• Push back from “coders”

• Gut reaction is to code and go with what they know

• Need a strong champion of Informatica

• Proving that it CAN be done in Informatica

• “Why not?” vs “Why?”

25

Simplified: Regulatory Trade Processing

Oracle

Regulatory Data

Warehouse

Compliance Distributed

Systems

Oracle Informatica

Informatica

Informatica

Java

Processing PL/SQL

Staging Load Processing and

Final Table Load

26

Simplified Process

• 80% + of data movement and processing

performed via Informatica

• Every new process starts with, “Can we do this in Informatica?”

• Still using PL/SQL and Java

• Compelling arguments for usage

• Iterative functionality

• Complex business logic

• Cursor-driven logic required

• Multi-threading

27

Future Roadmap

• Upgrade to PowerCenter 9.1

• Upgrade to Exadata

• Add Informatica Source Control

• Enforce development standards

• Create in-house user group and share resources

and knowledge

• Continue to look for opportunities to use

Informatica to simplify processes

28

Exadata Performance Improvement

Records

Baseline

Time

Oracle

Exadata

Time

Savings /

N(X) Faster

60 GB

Flat File

Load 365 Million 2:00:00 0:14:00 1:49:30

Staging

Load 9 Million 0:16:00 0:15:00 1:00

Query 1 103,147 0:13:55 0:00:27 40

Query 2 2,605 0:18:15 0:00:06 183

29

Uses of Informatica at CME Group

• Recurrent reports

• Data loads

• Data updates

• Automated testing (QA)

• Ad hoc extracts

• Historical data loads from legacy systems

• Archiving data off database to files

30

Lesson Learned

• No SQL overrides

• Hides code and table usage

• Lose benefits of Informatica

• Define Standards Upfront

• We defined, but didn’t enforce – enforce!

• Code Reviews

• To enforce those standards!

• Often overlooked because it isn’t “real” code

31

Thank You