zero data loss recovery appliance: insider’s ... - oracle · global financial services company...

Copyright©2018, Oracleand/oritsaffiliates.Allrightsreserved.|

ZeroDataLossRecoveryAppliance:Insider’sGuidetoArchitecture&BestPracticesTimChienDirectorofProductManagementOracleDevelopment

StefanReinersDBAMETRO-nomGmBH

October24,2018

JonySafiSeniorManagerOracleDevelopment


SafeHarborStatement

Thefollowingisintendedtooutlineourgeneralproductdirection.Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,timing, andpricingofanyfeaturesorfunctionalitydescribedforOracle’sproductsmaychangeandremainsatthesolediscretionofOracleCorporation.


Agenda

Introduction&Architecture

Operational&MonitoringBestPractices

METRONOMCaseStudy

Resources&NextSteps

1

2

3

4


TraditionalBackupSolutionsAreNotDesignedforDatabaseTreatDatabasesasJustFilestoPeriodicallyCopy

DailyBackupWindow

Largeperformanceimpactonproduction

DataLossExposure

Losealldatasincelastbackup– novalidation

ManySystemstoManage

Scalebydeployingmorebackupappliances

PoorDatabaseRecoverability

Manyfilesarecopiedbutprotectionstateofdatabaseisunknown


1010001001101001010101010001010101

0010101010100010101010010101010010100010101010010101000101010100

0011010010101010100010101010010

101000100110100101010101000101

Reliable&CompleteRecovery

EliminateLongBackupWindows

End-to-EndDataProtection

Control&Monitoring

Scale-OutHardware&Storage

ContinuallyValidatesRecoveryStatus

EngineeredDataProtectionForCompleteRecoverabilityoftheOracleDatabase

ZeroDataLossRecoveryAppliance

ArchiveBackuptoCloudStorage

Copyright©2018Oracleand/oritsaffiliates.Allrightsreserved.|

CloudArchive

RemoteReplica

DBBackupCloudService

TapeBackupVirtualFull

DayN

Full

Day0

Incremental

Day1

Incremental

DayN

RecoveryApplianceArchitecture

DeltaPush

CompressedDeltaStore

VirtualFull

ProtectedDBs

ChangedblocksandReal-TimeRedo(nofullbackups)

Datavalidationonreceipt,copy,

restore,periodically

Copyright©2018Oracleand/oritsaffiliates.Allrightsreserved.|

Agenda



METRONOMCaseStudy

Resources&NextSteps

1

2

3

4


MAARecommendations

• One1 RecoveryAppliance(RA)systemperdatacenter2

• BackupprimaryandstandbydatabasestotheirrespectivelocalRA

• NoRAreplicationforanydatabasewitharemotestandby

• RestoreoperationcanuseanyRAinanylocation

• DonotmakeanychangestotheRecoveryAppliance

1 ItcanbeaninterconnectedZDLRAconfiguration2 ThedefinitionofaDataCenteralsoincludes“firerooms”


TapeLibrary

OneWay

Bi-Directional

Hub&Spoke

RemoteDataCenterLocalDataCenter

RAReplica:DataLossProtectionfromSiteDisasters

BENEFITS• ReplicationtoRemote

Applianceprotectsdatafromdisastersorsitefailures

• AutomatedrestorefromLocalApplianceordirectlyfromRemoteAppliance


LeadingNorthAmericaHealthcareCompany1000+ProtectedDBs,3DataCenters,Bi-Directional+Hub&SpokeReplication

EastCoastDC#2“Hub”(200DBs)

DR NON-PROD

StandbyDBs->LocalRA->DBClone

Standalone->LocalRA->Replicated

MidwestDC(50DBs)

PROD,NON-PROD

Standalone->LocalRA->Replicated->DBClone

EastCoastDC#1(1000DBs)

PROD NON-PROD

StandaloneDBs->LocalRA->Replicated

PrimaryDBs->LocalRA

Replication Replication

Copyright©2018,Oracleand/oritsaffiliates.Allrightsreserved.|

NEW: ZDLRAHighAvailabilityforBackup&Recovery

• Databasebackupandredooperationsautomaticallyfailovertodownstreamreplicaappliance.

• ZDLandpoint-in-timerecoveryoperationsfromdownstreamappliancearefullysupported.

• Virtualfullbackupsonprimaryappliancearesynchronizedtocurrentstatewhenbackonline

PreserveBackup&RecoveryContinuityduringPlannedorUnplannedZDLRAOutages


RAOsakanormallyreplicatestoRATokyo– allDBsfullyrecoverable

Whenupstreamappliance(RAOsaka)isnotavailable,backupsandredoareredirectedtodownstreamappliance(RATokyo)

• Virtualfullsarecreatedasnormal

Whenupstreamisbackonline,downstreamappliancebackupsaretransferred

• Transferredbackupsareingestedandprocessedintovirtualfulls

• Normalbackupstoupstreamappliancecanberestartedimmediately

Benefits

• PreserveHighAvailabilityduringplannedorunplanneddowntime

• Databasebackup&restore/recoverabilityavailablefromUSorDS

• MAAPresentation andMOSNote2432144.1 NOWAVAILABLE

• www.oracle.com/goto/maa ->ZeroDataLossRecoveryAppliance

X

ZDLRAHighAvailabilityforBackup&Recovery

RAOsaka

Replication

RAOsaka RATokyo

RATokyo

Replication

RAOsaka RATokyo

BackupstoDSAppliance

DSApplianceBackupsTransferredtoUpstream


BackupFailovertoAlternateAppliance• IncrementalsandRedonormallysenttoPrimaryRA

• AlternateRAservesasbackupstagingareawhenprimaryRAisunavailable,thensyncswithprimaryRAafterwards– Novirtualfullscreatedonalternate,hencerecoverabilitynotsupported

– Spacesizedfor‘n’incrementalsandarchivedlogbackupsduringprimarydowntimeperiod

• Benefits:– Preservesbackupandredoshipmentcontinuityduringplannedmaintenance/upgrades

– PreventslocalFastRecoveryAreasfromfillingupwitharchivedlogs

– Incrementalforeverbackupscontinue

RABostonPrimaryAppliance

XRANYCAlternateAppliance

AlternateAppliancetakesoverbackupsandredotransport

Incrementals&

Redo

“BF_FORWARD”PolicyonAlternateRA:STORE_AND_FORWARD=‘YES’


GlobalFinancialServicesCompany8000+ProtectedDBs,GlobalDataCenters,BackupFailovertoAlternateAppliance

DB+RAPOD1(600-700DBs)

DBGroup2->RA2->Tape

Failoverto:->RA1

DBGroup1->RA1->Tape

Failoverto:->RA2

DBGroup1 DBGroup2

DB+RAPOD2(600-700DBs)

DBGroup3 DBGroup4

StandbyDBs->LocalRAFailoverto:->RA3

DBGroup3->RA3->Tape

Failoverto:->RA4

EachRAinapodisconfiguredasfailoverfortheother.

• Spaceisreservedfornormal&failoverbackups

StandardizedDBOn-Boarding:

• DataGuardorStandalone?• FailoverConfigRequired?• DBSize+Change%+

Retention->ReservedSpace• ReservedSpaceperiodically

adjustedperspaceneeds


NEW: ZeroDataLossCross-PlatformDatabaseMigration

• Significantreductionindowntime–lessthan2hourread-onlydowntime,regardlessofDBsize.

• Newautomationtoolsimplifies platformmigrationsteps- especiallyusefulforlargedatabases

• Supportssameandcross-endianplatformmigrationactivities

SimpleDatabaseMigrationtoOn-PremiseorCloud@CustomerusingZDLRA


SourceDB(11g-12c) DestinationDB(12c)

SOETBS Day1:FullBackup

Export Import

Read-only

MetaData

SOETBS

ZeroDataLossCross-PlatformDatabaseMigrationDramaticallyReduceMigrationTime– FromHours/DaystoFewHoursorLess

• CentralizedRecoveryAppliance“migrationengine” +minimaldowntime(shortread-onlyatend)• Dailyincrementalbackups->virtualfullbackupsonRecoveryAppliance

• Atdestination,restorelatestvirtualfullbackup,priortomigrationwindow• RESTOREFROMPLATFORMXXXFOREIGNDATAFILEYYY

• Whenreadytoswitchover:Atsource,takefinalincrementalandmetadatatablespaceexportinread-only• RECOVERFROMPLATFORMXXXFOREIGNDATAFILECOPYYYYatdestination• IMPORTDataPumpexportfileatdestination

• Destinationopeninread-onlytoverifymigrateddata,thenopenread-writeforbusiness

VirtualFullRestore

Restore&RecoverFinalIncremental

Day2->N:IncrementalBackups

Dayn:FinalIncremental

Read-write


LeadingGlobalSemiconductorManufacturer

ResultsAchieved• Consolidateandstandardize– Consolidatedseveraldatabaseservers

– CompatiblewithExadata– Multi-Tenantoption

• Reliable,ScalableandHighPerforming

• ImprovedthetimeandcosttobuildandmaintainAnalyticsplatform

• NearzerodowntimemigrationusingZDLRA–RMANDUPLICATE.

• Deliverexceptionalservicetobusinessusers

• Eliminatedfullbackups• ImprovedRTOby4X• Reducedbackupwindowsby2X

ExadataX4-2 ExadataX2-2

CascadeStandby

Incrementalforeverbackups

ActiveDataGuard

ExadataC@C

NewArchitecture

BusinessNeeds• AccelerateGrowth• DriveOperationalExcellence

• CustomerExperience• OperationalEfficiency

• Groworganizationalcapabilities– optimizeinnovation

• Addresscurrent&plannedbusinessgrowthobjectives

SolutionNeeds• Stability• ZeroPreventableOutages• FocusonBusinessOps• IncreaseITagility,self-

serviceandalignmenttobusinessdrivers

ExadataX4-2 ExadataX2-2

ActiveDataGuard

WeeklyFullanddailyincrementalbackup

NASDevice

LegacyArchitecture


NEW: RASystemActivityReportinBIPublisherTracksRecoveryApplianceActivityandHighlightsAreasofAction

• Displaysinformationforthefollowing:– StateofProtectedDatabases– RASpaceUtilization– StateoftheRecoveryAppliance– Tasks&TaskHistory


RASystemActivityReport:Healthy?• Itemstowatchfor:Badreport

Watchfor:

IftherearetasksofWORKTYPEinRUNNINGstateandcreatedadayearliertheninvestigate.

IftherearetasksofMAINTENANCEorSBTTYPEpresentandtheircreationtimeisolderthanoneweekthenitshouldbeinvestigated.

AlargenumberoftasksinEXECUTABLEstateforthesameTASK_TYPE

ThesameTASK_TYPEisincreasinginthenumberofjobs(CURRENT_COUNT)overtime.Asystemwithorderingwaitsolderthan1day.

AsystemwithtasksinSTALL_WHEN_WAIT.ThisshouldonlybeseenifOracleSupportistroubleshootingthesystem.

ZDLRA1


RASystemActivityReport:Healthy?

TheGood:

Minimumcreationtimeforactivetasksiswithinthelast24hoursforworktasks.

Taskhistorystateshouldindicateworkisbeingcompleted.

Reviewothersections

ZDLRA1


BackupBestPractices• UseTransparentDataEncryption(TDE)insteadofRMANencryption– RMANencryptionwillpreventZDLRAfromcreatingVirtualFullBackups(VB$).

• TDEbackupswillnotgetcompressedontheZDLRA– Encryptedbackupsdon’tgetcompressedbythestorageingeneral– Asizingexercisemustbe(re)-conductedwhenpreparingforTDE– TheincrementalforeverstrategystillappliestoTDEbackups

• UseRAbuilt-incompressioninsteadofRMANcompression– RMANcompressionincursDBCPUutilization&backupsaredecompressed+recompressedonRA

– $rmantarget<targetstring>catalog<catalogstring>backupdevicetypesbtcumulativeincrementallevel1filesperset1sectionsize64gdatabaseplusarchivelognotbackedupfilesperset32;


UseRecommendedRASoftware• SubscribetoMOSalertsandreferperiodicallytothefollowingnotes:– RecoveryApplianceCriticalIssuesMOSnoteforcriticalissuesalerts (DocID1927928.1)– RecoveryApplianceSupportedVersionsMOSnoteforlatestsoftwareupdate (DocID1927416.1)

• UseRecommendedRASoftwaretoavoidknowncriticalissues– Number1methodtoavoidproblems:Upgradetorecommendedsoftwarerelease– ZeroDataLossRecoveryApplianceSupportedVersions(DocID1927416.1)– ZeroDataLossRecoveryApplianceUpgradeandPatching(DocID2028931.1)– Patchesarecumulativeandinclude

- Bugfixes(Avoidabugbeforeithappens)- Enhancements(Newfeatures,optimizations,etc.)

• CoordinatewithPlatinumPatching– Scheduleearly&OpenProactiveSR

VeryImportant


Importantthingstoavoid

1. Don’tignore incidents:Resolveandunderstand2. Don’tneglect theRA:Monitorthesystem3. Don’tmake modificationstotheRAconfiguration

i. ZeroDataLossRecoveryAppliance- InstallingThird-PartySoftwareandModifyingInternalApplianceSoftware(DocID2014361.1)

4. Don’t takeperiodicLevel0:VirtualLevel0requiresonlyonelevel0followedbylevel1s5. Don’t backuptoanothermedia:Switchingtoanothermediacanimpactpastandfuture

backupsmadetotheRA


1. MonitortheRAi. OEMUnifiedManagementDashboard:Reviewtwicedailyandsetupalertsandnotificationsii. RuntheSystemActivityReportdailyandmonitortrends:

ZeroDataLossRecoveryApplianceSystemActivityScript(DocID2275176.1)iii. RunExachkMonthlyandreviewfindings:

HowToupdateexachkoutsideZDLRAInstall,PatchingandUpgrade(DocID2399688.1)iv. ReviewCapacityPlanningReportMonthlyorBi-Monthly

2. UseMultiSection: Setsectionsizeto64GB,thisallowsforefficientprocessinginZDLRA’sflashcachei. Largedatafiles>16TBwillautomaticallyseeanincreaseinsectionsize- sectionsize=sizeof(datafile)/256ii. Smalldatafiles<64GBwillnothavesectionsiii. ForcesFILESPERSETto1

3. Usethelatestlibra (RMANclientsbtlibrarythatsupportsZDLRA):ZDLRA:Wheretodownloadnewsbtlibrary(libra.somodule)(DocID2219812.1)

4. Validatethebackup/restorenetworkfirst:a. UseZeroDataLossRecoveryApplianceNetworkTestThroughputscript(DocID2022086.1) –

requiresqperf,supportedonspecificOSesb. UseHowtomeasurenetworkperformancefromRMANforZDLRAorCloudBackups(DocID2371860.1) –

usesRMAN’s“NETTEST”option,OSagnostic

4 importantthingstodo


Agenda



METRONOMCaseStudy

Resources&NextSteps

1

2

3

4

ZDLRA @ METRONOM

10.24.2018

Agenda Introduction

Mission

Best Practices

Target

Summary

1

2

3

4

5

29

Introduction

Introduction

Introduction

Team of 13 DBAs

• ~ 2100 Oracle databases

• AIX, Linux, Windows - Servers

• Database features including:

• RAC

• Data Guard

• GoldenGate for Minimal Downtime Maintenance (MDM)

32

Mission

Mission

• Started POC in 2016

• Replication between two DCs

• Easy Migration

• Go Live April 2017

• Backup of more than 350 productive databases by end of 2017

• Zero Downtime

34

Best Practices

Easy migration• Calculate how much space is needed• Enroll the database to the local ZDLRA• Grant access to the protected database to

a backup user• Modify sqlnet.ora• Create and verify a Wallet• Copy libra.so to the LIB directory• Register database• Set new backup configuration• Enable Block Change Tracking• Run test Backup

Best Practices

addzdlra.sh -d testdb -p gold

MAX_RETENTION_WINDOW

• Set it, but not too aggressive

Don‘t Submit multiple requests to Delete Databases

• A delete can take a lot of time

• If database delete does not progress for some time, contact Support

Don‘t Neglect the RA

• Monitor and adjust the System

• System Activity Report• Clarify Findings with Support

Best Practices

37

Target

Target

• Use ZDLRA to Backup ~900 databases within the next 12 months

• Add more storage to ZDLRA for growth and replication needs

• Implement Minimal Downtime Maintenance patching solution via new RA High Availability for Backup & Recovery procedure (aka Auto-Failover to Replica RA)

• Outsource Backups to Oracle Cloud

RA-based Migration of IBM AIX to ExaCC/OCCATTEND OUR MIGRATION SESSION [TRN4033] TOMORROW, 9 AM, MOSCONE WEST 3004

40

Summary

• Less time for Backups

• Fast restore and recovery

• Easy Migration due to automated processes

• Scalable for data growth

Summary

Thank you foryour attention!

Stefan Reiners


Agenda



METRONOMCaseStudy

Resources&NextSteps

1

2

3

4


Resources&NextSteps

• RecoveryApplianceProductCentral–www.oracle.com/recoveryappliance

• RecoveryApplianceMAABestPracticesCentral–www.oracle.com/goto/maa ->ZeroDataLossRecoveryAppliance

• OpenWorldMAAPresentations–www.oracle.com/goto/maa ->Presentations

• AskTOMBackup&RecoverySessions– asktom.oracle.com->OfficeHours->Searchfor‘backup’

• Getourbusinesscardforfollowupquestions!

zero data loss recovery appliance: insider’s ... - oracle · global financial services company...

Documents