Eli Lilly and Company
• Fortune 500 Pharmaceutical and Bio-Tech Company• Headquartered in Indianapolis, IN• Over $14 Billion in Sales (2005)• 24 Manufacturing Sites World-wide excluding 3rd Party Manufacturers• Key products: Prozac, Zyprexa, Cialis, Insulins
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
3
Agenda
• Overview of ISA S95 Standard• Review of Logical Standby• Facts & Figures• Lessons Learned• Conclusions
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
5
Process DeviceContinuous/ Batch Manufacturing
I/O andSensors
Discrete Manufacturing
High AlarmLow AlarmComm. Err.
Business Planning and
Logistics Systems
ManufacturingOperations and
Control Systems
AutomationControl Systems
Process ControlDedicated Systems
Business Process Information Network
Operations Information Network
Automation and Control Network
Discrete Device Process Device
Aseptic AreaLevel 0
BuildingLevel 2
PlantLevel 3
EnterpriseLevel 4
ISA S95 Physical Architecture
Aseptic AreaLevel 1
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
6
S95 Bottom Line
All systems at a given level (and all levels below it) should be able to operate autonomously and independently of the level above it for a given period of time (as determined by the implementer).
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
8
Benefits of Logical Standby
• Seamless replication of enterprise data warehouse (data, structures, accounts) to all sites
• 7 warehouses loaded at once! (near real-time replication of data)
• Low cost solution (no software cost, no ETL development cost, minimal maintenance cost)
• Low risk solution (i.e. high data quality – if it works in the primary, it will work in the standby)
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
9
Benefits of Logical Standby
• Easy to add additional sites (4-6 hours US, 12-18 hours OUS)
• No performance impact on the primary database during replication
• Allows sites to “extend” enterprise warehouse by adding local schemas and data as necessary to satisfy local needs
• Allows sites to easily suspend replication during local site processing
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
10
Implications of Logical Standby
• Requires homogeneous computing environment (i.e. chip set, bit size, oracle version)• Impacts if and how hardware, operating system,
and/or database upgrades may occur• Standby databases must be upgraded prior to the
primary (10.1.0.3 and up supports rolling upgrades)• Requires supplemental logging
• Increases the amount of redo generated by ~ 8% (storage issue, not performance)
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
11
Implications of Logical Standby
• Increases demand for archived redo log file system spacePrimary
• Automated gap resolution is cool, but … logs must remain on-line for gap resolution (possible change to existing archive logmaintenance strategies)
• Query V$ARCHIVED_LOG table to confirm deliveryStandby
• Logs from primary cannot be deleted until applied (automatic in v10.2)
• exec DBMS_LOGSTDBY.PURGE_SESSION• Query DBA_LOGMNR_PURGED_LOG
• SQL Apply transactions roll through the standby’s online redo logs
• 1 primary redo log yields ~ 1.6 archived redo logs on the standby• Improved/fixed in v10.2
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
12
Why Not Data Streams?
Our prototype comparison of Data Guard and Data Streams proved Data Guard to be the better solution for Eli Lilly• Simple replication of an entire database• Log mining may be done on primary db server
(performance impact) or downstream db server (homogeneous computing environment)
• More complex to set up• More complex to support and maintain• Data Guard proto-type worked “right out of the box”
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
14
Current State – Primary DB
P
• 90 Gigabyte Warehouse• Unix platform running Oracle v10.1.0.4• Located in Indianapolis, IN• Delta loads occur every other hour• Average load generates 100M of redo• Protection Mode: Maximum Performance• Redo Shipment: ARCH
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
15
Current State – Logical Standbys
s5s3s1
s4
s6
s2P
s7Standby sites:
• Indianapolis, IN (2)• Tippecanoe, IN • Carolina, Puerto Rico• Baskingstoke, England• Fegersheim, France• Kinsale, Ireland
Data Guard setting: Standby
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
16
Init.ora Parameters – Primary Database
• LOG_ARCHIVE_CONFIG='DG_CONFIG=(P,s1,s2,s3,s4,s5,s6,s7)'• LOG_ARCHIVE_DEST_1=
'LOCATION=/oracle/data/arch01/P/VALID_FOR=(ALL_LOGFILES,ALL_ROLES)DB_UNIQUE_NAME=P'
• LOG_ARCHIVE_DEST_2='SERVICE=s1VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)DB_UNIQUE_NAME=s1‘
• LOG_ARCHIVE_DEST_STATE_1=ENABLE• …• LOG_ARCHIVE_DEST_STATE_8=ENABLE• LOG_ARCHIVE_MAX_PROCESSES=10
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
17
Init.ora Parameters – Logical Standby
• DB_NAME=s1• DB_UNIQUE_NAME=s1• SERVICE_NAMES=s1• LOG_ARCHIVE_CONFIG='DG_CONFIG=(P,s1)'• LOG_ARCHIVE_DEST_1=
'LOCATION=/oracle/data/arch01/s1/VALID_FOR=(ONLINE_LOGFILES,ALL_ROLES)DB_UNIQUE_NAME=s1'
• LOG_ARCHIVE_DEST_2='LOCATION=/oracle/data/arch01/s1_dest_2/VALID_FOR=(STANDBY_LOGFILES,STANDBY_ROLE)DB_UNIQUE_NAME=s1'
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
18
Init.ora Parameters – Logical Standby
• LOG_ARCHIVE_DEST_STATE_1=ENABLE• LOG_ARCHIVE_DEST_STATE_2=ENABLE• PARALLEL_MAX_SERVERS=9• REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE• INSTANCE_NAME=s1
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
19
Future State *
Ps2
s3 s5 s14
s13
s12
s1
s4s8
s7s10
s11s6s9
* proposed
• Up to 14 sites (8 OUS)• Cascaded standbys• Schema-based data filtering• Use of standby redo logs (LGWR ASYNC)
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
20
Cascaded Standby
• Having one logical standby serve as the primary database to another logical standby
• Capability documented in Oracle Data Guard manual beginning with version 9.2
• Provides the following significant benefits:• Overcomes 9 standby limit• Redo log shipped to zone 2 once regardless of the number
of zone 2 logical standbys• Unfortunately, a bug exists preventing the startup of the
SQL apply process at the cascaded (end point) site• Bug has been filed with Oracle Support and eagerly!
awaiting resolution
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
22
Lessons Learned
• Speed at which all standby databases receive redo is limited to the speed of the slowest network path to any one standby• Use standby redo logs• For high speed networks, use ASYNC
• Standard maintenance processes may have to be changed• Processes to compress archived redo• Process to delete archived redo
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
23
Lessons Learned
• Be aware of large loads and help Data Guard out when necessary• DEFER slower sites via LOG_ARCHIVE_DEST_STATE_#
(active sites will receive redo as normal)• Execute large load• Tar, compress, and manually transfer archived redo• Register archived redo in standby• When all log registered at slower site, re-ENABLE the site
• Set redo shipment for inaccessible standby sites to DEFER
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
24
Lessons Learned
• If redo shipment can’t occur, archive processes die (Bad! Very bad!) – bug being worked with Oracle
• Make sure your standbys are regularly applying redo
• Status and notification messages are recorded as “errors” in alert logs
• Password for SYS must be identical for all databases in the standby configuration (primary or standby) for standby to receive redo
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
25
Lessons Learned
• Backup/recovery of logical standby is not intuitive – awaiting official documentation from Oracle
• The Oracle Data Guard product team has been totally committed to our success as an early adopter of Data Guard technology
12/11/2006 Copyright© 2006 Eli Lilly and CompanyWarehouse Replication Using Oracle Data Guard and Logical Standby
26
Conclusion
• Data Guard with Logical Standby database has proven to be extremely successful in allowing Eli Lilly to satisfy site reporting needs much quicker without incurring the costs typically associated with standard ETL delivery methods