prs3782 ds8000 data replication best practices 20090928

IBM Advanced Technical Support - Americas

© 2009 IBM Corporation

IBM DS8000 Data Replication Best PracticesIBM DS8000 Data Replication Best Practices

Bob Kern – [email protected] Sedgwick – [email protected] Sautter – [email protected]


© 2009 IBM Corporation2

Agenda

DS8000 Copy Services - Best PracticesDS8000 Data Replication Technology

–Key Concepts–Selecting the Right Solution to Match your Clients Business

NeedsPlanning for Data Replication

–Configuration–Data Collection–BandWidth Studies–Automation

Case Studies – What you do not want to do…..



DS8000 Copy Services - Best Practices Map solution to Customer’s Business Requirements (RTO/RPO)

– PiT Solution vs Continuous Mirror, Sync vs Async– For several small customers doing PTAM -> a PiT Inc FC + GC can be attractive. – High Availability – z/OS Basic HyperSwap or GDPS HyperSwap Mgr, – Dist. System – Software Dual Write across 2 Luns– Metro Mirror on Same Data Center Floor -> outage, re-ipl minimizes outage time.

Configuration Guidelines for Primary & Secondary– Balance Primary & Secondary Performance (subsystems/cache/drives/etc.)– 1:1 or 2:1 Configurations tend to be simpilier to configure & Manage.– Be careful reusing “old” technology boxes as targets. – Ability to Test D/R while Maintaining D/R Protection.

• Standard for most customer.• Emerging Standard -> Test the Way the Recover & Recover the Way they Test.

– Failover/Failback Functionality• Site Toggle can perhaps reduce D/R Test costs. Run 6 months in each site.

Bandwidth Analysis -> Data Collection – Use Tools: Disk Magic, RMF Magic etc.– Analysis is VERY Important to understand BW for MB/Sec Update rates.– MB/SEC Update Rates can also help customer Manager his business. Understand activity 24X7…– Planning for Capacity Growth

• Initial Deployment should have some capacity Growth planned into Solution.• Customer needs to understand how to do this as the years go by, workloads change etc.

Review - D/R Lessons Learned– White Paper: IBM Storage Infrastructure for Business Continuity – Updated This year– http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS2511

Management Software -> TPC-R or GDPS– DSCLI can be made to work, but can be very complicated.– Best Implementations involve TPC-R or GDPS Software.



Map the Right Solution to the Clients Business Requirements



Aspects of Availability

High AvailabilityFault-tolerant, failure-

resistant infrastructure supporting continuous application processing

Continuous OperationsNon-disruptive backups and system maintenance coupled with continuous availability of

applications

Disaster RecoveryProtection against unplanned outages such as disasters through reliable,

predictable recoveryProtection of critical business data

Recovery is predictable and reliable

Operations continue after a disaster

Costs are predictable and manageable



Business Continuity Tiers

Recovery Time Objective 15 Min. 1-4 Hr.. 4 -8 Hr.. 8-12 Hr.. 12-16 Hr.. 24 Hr.. Days

Cos

t

Tier 4 - Point in Time disk copy

Tier 3 - Electronic VaultingTier 2 - Hot Site, Restore from Tape

Tier 5 –Software replication

Tier 1 –Restore from Tape

Recovery from a disk image Recovery from tape copy

Tier 7 – Site Mirroring with automated recovery

Tier 6 - Disk mirroring (with/without automation)

TAPE

DIS

K

MA

NA

GEM

ENT

SWN

ETW

OR

KD

/R

AU

TOM

ATI

ON

REP

LIC

ATI

ON

SW

Best Business Continuity practice is to blend solutions in order to maximize application coverage at optimum cost

SER

VIC

ES



Copy data command issued Copy is immediately available

Read and write to both source and copy possible

Write Read

When copy is complete,relationship betweensource and target ends

Time

Optional background copy

Source Target

PiT Incremental FlashCopy+ Metro Mirror

Available on:

DS6000, DS8000, ESS

FlashCopy Internal Copy

Available on:

DS6000, DS8000, ESS

SAN Volume Controller, XIV,

DS4000, DS5000,

PiTIncrementalCopy

Primary

Prod

PPRC-XD Secondar

y

PiTCopies

Global Copy(Asynchronous

)

PiT Incrementa

l FlashCopy

FlashCopy

WAN

REMOTE

Data Replication Enabling Core Technologies -



IBM Copy Services Technologies

Primary

Site A

Metro

Site B

Out of

Region

Site C

Metro / Global MirrorThree site synchronous and asynchronous mirroringAvailable on:

DS8000N Series

FlashCopyPoint in time copyDS8K FCSEAvailable on:

DS8000, DS6000, SAN Volume ControllerDS4000/DS5000N SeriesXIV

Within Storage System

Out of

Region

Site

B

Primary

Site A

Global MirrorAsynchronous mirroringAvailable on:

DS8000, DS6000SAN Volume Controller DS4000/DS5000N Series

Primary

Site A

Metro distance

<300km

Site B

Metro Mirror Synchronous mirroringDS8K HyperSwapAvailable on:

DS8000, DS6000, ESSSAN Volume ControllerDS4000/DS5000N SeriesXIV



Configuration Guidelines



Configuration Guidelines for Global MirrorWhitepapers:

– Global Mirror http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100642– GM Secondary http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101105

TPC for Replication or GDPS– Required– Managing GM with dscli scripts is not a realistic customer option

Do not undersize the GM Secondary

Solution Planning Considerations– Asymmecrical .vs symmetrical– Volume size and layout

• LH, RH, RJ (A, B, C) are exactly the same size– Failover & failback considerations

• More on layout later– LSS Considerations – balance resources

• Dedicate two LSS(s) per DS8000 to an application, one even, one odd– Tends to group volume numbers in ranges – easily recognizable ranges of volumes– Management is easier with fewer LSSs

– Performance evaluation & Bandwidth Sizing between sites– Use dedicated PPRC link ports, 2 minimum, on separate host adapters

Testing !!



Global Mirror Prerequisite –> Performance at the Primary Site

Global Mirror Primary Performance must be good on volumes to be replicated

– Rank Performance• Backend response times• Frontend Write response times ≤ 2ms• No cache issues

– Host Port Performance– Good I/O balance across Ranks, Host Adapters– Performance Tools:

• TPC for Disk• RMF Magic

Plan and Monitor for Growth– Workloads increase– Additional replication requirements

• New applications

Global Mirror

Local Site

Global Mirror Primary

Remote Site

Global Mirror Secondary



Global Mirror Secondary - Performance Capacity

Equal to or Greater than the performance capacity of the primary

Global Mirror places more write performance stress on the secondary

– More storage capacity– Flash Copy processing

Cache required per volume count

Do not undersize the Secondary

DS8300 model 932DS8300 model 922

DS8300 any modelDS8100 model 931

DS8100 model 931DS8100 model 921

DS8300DS8100

DS8100DS8100

ESS 800 with Arrays Across Loops

ESS 800

ESS 800ESS 800

Global Mirror Secondary Global Mirror Primary

Global Mirror

Local Site

Global Mirror PrimaryRemote Site


3600012000DS8000 / 256GB

180006000DS8000 / 128GB

90003000DS8000 / 64GB

45001500DS8000 / 32GB

45001500DS8000 / 16GB

N/A350DS6800

N/A1000ESS 800

Recommended max number of GMir volumes DS8000 LIC Release 3.0 or later

Recommended max number of GMirvolumes

Ok.



Placement of B and C volumes

On Secondary– All ranks contain equal numbers of B

and C volumes– B and C copies for particular volumes

kept on separate ranks– Activity for busy volumes spread over

two ranks– DDM size equal to or one size greater

than primary

CB

CB

CB

CB

On Secondary– B and C copies on

same rank results in hotspot being concentrated on single rank

A

Same RPM

Same RAID type

Primary Secondary



Placement of B, C and D volumes

All ranks contain equal numbers of B, C and D volumes

B and C copies for particular volumes kept on separate ranks

Placement of D volumes as shown above less critical but easier to manage/track/implement

CB D

CB D

CB D

CB

CB

DD

D volumes placed on separate ranks to keep testing and backup activity separate

B and C copies for particular volumes on different ranks

This does reduce available ranks for B & C volumes



FlashCopy Source & Target Placement

In general:– Spread evenly across disk subsystems– Within each disk subsystem, spread evenly across clusters– Within each cluster, spread evenly across device adapters– Within each device adapter, spread evenly across ranks

Place FlashCopy target in same cluster as source– If using BACKGROUND COPY, target on a different device adapter

FlashCopy Space Efficient– Use FlashCopy Space Efficient when economy is more important than performance and

for short-lived relationships with low update rate on source volumes• Short Term FlashCopy relationships• Good for read only applications

– Tape Backup, 24 hour online backup, etc

Different ranksDoesn’t matterSame clusterFlashCopy Impact to Applications

Different ranksDifferent device adapter

Same clusterBackground Copy Performance

Different ranksDoesn’t matterSame clusterFlashCopy Establish Performance

RankDevice AdapterCluster



FlashCopy SE Relationships

Full volume only

NOCOPY only in first release – Background copy cannot be initiated to a SE volume by any means

Must specify “SE target ok” at establish

Recommended Usage– Use FlashCopy Space Efficient when economy is more important than

performance and for short-lived relationships with low update rate on source volumes

• Short Term FlashCopy relationships• Good for read only applications

– Tape Backup, 24 hour online backup, etc



Data CollectionPerformance and Bandwidth

Studies



Data Collection Essentials

Historical Data– Data selection is critical for obtaining valid results

• Single data points or averages are not very useful• Size and duration of peaks are important

– Need to identify daily peaks and the workload profile over time• End of month, end of quarter, end of year, etc.

– Identify active volumes for workload balancing– Quantify expected growth

Configuration details – Production vs. test data– Temporary data may not be part of Global Mirror– Volume layout by array and storage pool– Network configuration including bandwidth available

Monitoring Performance and Status– TPC Standard Edition– RMF data (enable ESS data collection)– TPC for Replication– Global Mirror Monitor



Data Collection Essentials (2)

Data needed for evaluation– Performance data for 1 week– I/O rates, Data Rates, Response times– Configuration details and event timelines (when are peaks expected)

Data Sources– RMF for zSeries– iSeries PT reports– Total Storage Productivity Center (TPC) reports– iostat reports, windows perfmon reports

Evaluation tools– Disk Magic Capacity planning – RMF Magic Performance evaluation– DS8Qtool DS8000 physical and logical configuration details

Contact IBM ATS for Performance and Bandwidth Studies– add a link to partner world.



Lessons & Automation



Lessons Learned About IT Survival

Repeated Testing before a disaster is crucial to successful recovery after a disaster

– TTWYR – Test The Way You Recover– RTWYT – Recover The Way You Test

After a disaster, everything is different– Staff well-being will be 1st priority– Company will benefit greatly from well-documented, tested, available and

automated (to the extent possible) recovery procedures

May be necessary to implement in-house D/R solution to meet RTO/RPOPlan geographically dispersed IT facilities

– IT equipment, control center, offices, workstations, phones, staff, . . .– Network entry points

Installed server capacity at second data center can be utilized to meet normal day-to-day needsFailover capacity can be obtained by

– Prioritizing workloads– Exploit new technology: Capacity Back Up (CBU)

Data backup planning and execution must be flawless– Disk mirroring required for <12hr RTO (need 2x capacity)– Machine-readable data can be backed up; not so for paper files

Check D/R readiness of critical suppliers, vendors

– Repeated Testing before a disaster is crucial to successful recovery after a disaster

–TTWYR – Test The Way You Recover–RTWYT – Recover The Way You Test

– After a disaster, everything is different

–Company will benefit greatly from well-documented, tested, available andautomated … recovery procedures



Automation: Critical for successful rapid recovery & continuity

The benefits of automation:– Allows business continuity processes to be built on a reliable,

consistent recovery time– Recovery times can remain consistent as the system scales to provide

a flexible solution designed to meet changing business needs– Reduces infrastructure management cost and staffing skills – Reduces or eliminates human error during the recovery process at time

of disaster– Facilitates regular testing to help ensure repeatable, reliable, scalable

business continuity– Helps maintain recovery readiness by managing and monitoring the

server, data replication, workload and the network along with the notification of events that occur within the environment

Automate - Automate - Automate



Tivoli Storage Productivity Center for Replication (TPC-R) Overview

Replication management solution – Simplified replication management

& monitoring– Powerful commands and logic

Multiple Storage subsystems– DS8000, DS6000, ESS800, SVC

Multiple logical volume types– Open systems (FB) LUNs– z/OS (CKD) volumes

Multiple replication types– FlashCopy– Metro Mirror– Global Mirror– Metro/Global Mirror

High performance and scalability

TPC for Disk, TPC for Data and TPC for Fabric are not required but can coexist on the same server

– Shared instance of DB2– Shared SNMP port



TPC Replication Manager

TPC ForReplication

ESSDS6000DS8000

SAN Volume Controller

ESSDS6000DS8000

SAN Volume Controller

Primary/Source Site Second/Target Site

• Setup Copy Sessions• Execute Copy Operations• Monitor Copy Status• Manage/Monitor Consistent

Groups• Alert Operations on Exceptions /

Failures

• Automated copy services configuration

• Central operations for copy services• Operational status on copy services

operations• Assistance with recovery on failures

• DS6000, DS8000 support• Global Mirror Support• Replication Progression

Monitoring• High Availability• Disaster Recovery

Automation (failover, failback)



TPC for Replication GUI

My Work hyperlinks on left

Display area for panels on right

Select session, select action (from dropdown list) and GO

Tables with hyperlinks and sortable columns

Health Overview on every panel

Session view– Triangle Indicates application

access (active host)– Arrows between roles indicate

direction of active replication



TPC-R Video Series (new on Techdocs)

Series of live demonstrations captured on video managing variousenvironmentsLink to Summary of all the videos:

– http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS3687

Series includes the following demonstrations:– CLI vs. TPC-R – (2:13) – adding copysets using CLI and TPC-R– TPC-R 3.3 GM – (17:36) – GM setup with FO/FB using TPC-R 3.3– TPC-R 3.4 GM with Practice – (6:39) – GM with Practice Volumes setup with FO/FB– TPC-R 4.1 overview – (14:24) – Overview of TPC-R with MM setup demonstration – TPC-R 4.1 adding HW – (10:01) – how to add DS8000 and SVC to TPC-R– TPC-R 4.1 MM setup – (19:20) – using TPC-R 4.1 to manage MM with FO/FB – TPC-R 4.1 GM setup – (17:36) – Using TPC-R 4.1 to manage GM with FO/FB– TPC-R 4.1 MM with Practice – (8:15) Using TPC-R 4.1 to create practice volumes



TPC-R Simplification: Starting a Global Mirror Copy

Using the DS8000 Hardware Commands

1. Determine where to place Master GM session given the PPRC paths.

2. Establish PPRC links between Master and Subordinate DS8000’s.

3. Establish PPRC paths between A and B volumes

4. Establish Subordinate sessions on the A volumes of the DS8000’s

5. Establish a GC relationship between A and B

6. Query A to determine first pass complete

7. Establish Flash copy between B and C with incremental

8. Add A to the subordinate Global Mirror session

9. If first A volume on this DS8000, then start the Global Mirror Master with new configuration

Monitor the Global Mirror Master with 051 queries and calculate RPO.

Monitor for failures and fatal conditions

Using TPC-R Commands

1. START



TPC-R Simplification: Recover a Global Mirror Copy

Using the DS8000 Hardware Commands

1. Establish PPRC B to A Failover

2. Query all B to C Flash Copy relationships and determine if they are revertible and have the same sequence number

3. If the sequence numbers are all the same AND at least one relationship is not revertible, issue a “withdraw Flash Copy with commit” to all of the revertible relationships

4. If all of the Flash Copy relationships are Revertible, issue a “withdraw Flashcopy with revert” to all Flashcopy relationships.

5. Issue “establish Flashcopy C to B”with Fast Reverse Restore

Using TPC-R Commands

1. RECOVER



AA BB

CC

Continuous Availability /

Disaster Recovery Metropolitan Region

Continuous Availability Regionally and Disaster Recovery

Extended Distance

Continuous Availability of Data

within a Data Center

Disaster Recovery atExtended Distance

Near-continuous availability to data

Single Data CenterApplications remain active

GDPS/PPRC HyperSwap Manager

Automated D/R acrosssite or storage failure

No data loss

Two Data CentersSystems remain active

GDPS/PPRC HyperSwap Manager

GDPS/PPRC

Automated Disaster Recovery

“seconds” of Data Loss

Two Data Centers

GDPS/GMGDPS/XRC

Data availabilityNo data loss

Extended distances

Three Data Centers

GDPS/MGMGDPS/MzGM

The right level of business continuity protection for your business….GDPS family of offerings

SDM



The right level of protection for your business – Distributed Platforms

Tivoli SA AppMan Platforms:

Symantec VCS Platforms:

IBM System P & pHype - AIX 5.3IBM System x (Intel / AMD x86_64) - Suse SLES 9 & RH 4HP (Itanium / PA RISC) – HP-UX 11.23.SUN (SPARC) – Solaris 9 & 10.VMWare ESX 3.0 (Intel / AMD x86_64) - SuseSLES 9 & RH 4 & Windows AS & Windows 2300.

DR at extended distance

GDPS/XRC

Rapid systems recoverywith only ‘seconds” of data loss

K-sysSDM

VCS VCS and

GDPS DCMAgent

GCO

Site-1 Site-2

CA / DR within a metropolitan region

GDPS/PPRCK-Sys

K-Sys

Two data centers - systems remain active; designed to provide no data

loss

VCS or SA AppMan

VCS or SA AppMan &GDPS DCM

AgentGCO

Site-1 Site-2

IBM System p AIX 5.2, 5.3, 6.1, Linux: SUSE SLES 9,10 RedHat RHEL 4,5

IBM System x Linux: Suse SLES 9,10, RedHatRHEL 4,5; Windows 2003,2008

IBM System I Linux: Suse SLES 9,10, RedHatRHEL 4,5

IBM System z z/OS V1.7+, Linux: Suse SLES 9,10, RedHat 4,5

VMWARE ESX Win Server- Linux: Suse SLES 9,10, RedHat RHEL 4,5; Windows 2003, 2008

Ref IBM Tivoli System Automation 3.1 Installation & Customization Guide in the Release notes for a more detailed reference on GDPS DCM Supported configurations.



Case Study #1

Bandwidth Requirements



Data Replication Case #1 - Summary

Global Mirror consistency group formation fails– Drain time exceeded– Suspended volumes– Link incidents (Frame transmission retries <1%)

Configuration Details– Two 9 Mbit links between sites– 80 volumes mirrored– 3:1 compression on links– PPRC ports on the same HBA

Operational Details– Script written to monitor Out-of-Sync Tracks– TPC for Disk used to provide performance data– TPC-R used to automate Global Mirror– Started with a few mirrored volumes – no issues with Global Mirror– Increased number of volumes and workload

• Noticed host impact due to slow links • Switched to Global Copy• Large number of OOS Tracks



Data Replication Case #1 - Bandwidth Requirements

The following chart shows the mirrored write MB/s profile with indications of the Link Bandwidth required– 3:1 Compression and 80% Link efficiency – Exceeding the bandwidth will result in higher RPO

Workload values above the “9 Mb” links indicate that the capacity of the link is exceeded– 1-3 Distance Links do not handle the workload– 4- Distance Links are sufficient for the current workload

• With some longer RPO times during peaks• Should suspend GM during large peaks

– 8-12- Distance Links would be needed for the 25 MB/s peak

Available bandwidth is 2 links – Not dedicated – Link timeouts occur when over-driven



2 Links = available bandwidth

4 Links = min required bandwidth



Data Replication Case #1 – Out-of-Sync Tracks

The 1st chart shows the PPRC Link activity– Available Link bandwidth was insufficient– Over driving the links resulted in redriving frames (timeout)– Delays resulted in full track transfers instead of sending data from cache– Full track transfers increased the bandwidth requirement

The 2nd chart shows the OOS Tracks and suspended volumes– Peak workload occurred at 2:00-3:00 am on most days– Large number of OOS Tracks could not be copied before the next peak– OOS Tracks must be low to for Consistency Groups to form– OOS Tracks fully copied about 2 times per week



Overdriving links causes delays

CGs possible only when

links are not overdriven

Full Track Transfers increase required

bandwidth



Overdriving links causes OOS Tracks

and suspended volumes

CGs possible

only when OOS Tracks

are low

Cannot “catch-up”before the next

peak



Data Replication Case #1 – Conclusion

Bandwidth equivalent to 4 – “9 Mbit” links to handle the workload– Sufficient for the “normal” workload– May need to suspend Global Mirror during large peaks (measured 25

Mb/sec)– Should consider future growth

Follow Best Practices for Data Replication– Links should be dedicated to guarantee required bandwidth and reduce

link timeouts. – PPRC Ports should be on dedicated cards and

• Do not share cards with host activity• Do not put both links on the same HBA (single point of failure)• 4 links on 2 HBAs would be preferred

– Monitor Status and Performance• Use TPC for Disk to monitor link activity and workload growth• Do not ignore TPC-R messages• Test DR procedures



Case Study #2

Performance Requirements



Data Replication Case #2 - Summary

Global Mirror consistency group formation fails– Drain time exceeded– Host performance impacted while GM is active

Configuration Details– Distance between sites 37 miles / 60 km– Link information GigE = 100 MB/sec– Primary DS8000 300G 15K Drives Raid 5– Secondary DS8000 300G 15K Drives Raid 5

Operational Details– TPC for Disk used to provide performance data– TPC-R used to automate Global Mirror– Increased workload since initial GM Design

• Noticed host impact when GM was implemented• Switched to Global Copy



Data Replication Case #2 – Performance & Bandwidth

Bandwidth analysis – Available bandwidth is not the cause of Global Mirror issues – Usually 20-30 MB/sec short peaks 50, 90 MB/sec – GigE Link can handle 100 MB/sec without compression

Current Global Mirror Configuration Follows Best Practices– Volume placement is not the cause of performance issues – PPRC ports are separate from Host ports (good)

• Should use every other port on HBA card for best performance– Volumes spread over all Arrays (good)

• PPRC volumes share arrays with Flash Copy targets• Do Not have the FlashCopy source and target in the same array• Target should be in the came cluster

Performance analysis – Primary disk utilization is too high without any Mirror activity– Secondary disk utilization is at the maximum when Global Mirror is active– Flash Copy activity adds to HDD utilization– When Global Copy is active HDD utilization decreases– Raid 10 will reduce the HDD Utilization (double number of arms)



Global Mirror

Global Copy



Global Mirror

Caution levelWarning level

Part 1

06-21 14:20 to 06-22 13:40



Global Copy

Part 2

06-22 15:35 to 06-23 14:45



Global Mirror

GM Flash Copy adds to HDD activity

Utilization is too high!

Part 1

06-21 14:20 to 06-22 13:40



Global Copy

Part 2

06-22 15:35 to 06-23 14:45



Data Replication Case #2 – Conclusion

Previous Disk Magic Study Showed Good Results– Original workload

• 4,800 I/O per second HDD Utilization 35%– However the workload has grown

• 7,300 I/O per sec HDD Utilization >90%• Other systems were added

HDD Utilization is too high– This condition will cause Host performance issues even without Global Mirror

active– Overloaded arrays at remote site causes Host performance issues when

Global Mirror is active– Added copy activity at the primary site causes Host performance issues

when Global Copy is active

Current workload has reached the limit of the current configuration– Need to spread the workload over more arrays and use Raid 10– Need to plan for future growth– Continue to use TPC Standard Edition to monitor performance



Further Assistance

DS8000 Architecture & Best Practices - Replay

TIME Given: August 13; 11:00 a.m. New York, 4:00 p.m. London, 5:00 p.m. Paris, 15:00:00 GMT

IBMers: http://w3.ibm.com/sales/support/ShowDoc.wss?docid=Q824765C05775Z31&infotype=SK&infosubtype=N0&node=doctype,N0|doctype,TLC|brands,B5000|brands,B8S00&appname=CC_CFSS

BPs: http://www.ibm.com/partnerworld/wps/servlet/ContentHandler/tsee081309km

Contact ATS:Business Partners:

PARTNERWORLD CONTACT SERVICES (US & Canada) 1-800-426-9990 or fill out a request online:http://ibm.com/partnerworld/techline

IBMer’s: Open Techline Request



Additional References



Copy data command issued Copy is immediately available

Read and write to both source and copy possible

Write Read


Time


Source Target

PiT Incremental FlashCopy+ Metro Mirror

Available on:

DS6000, DS8000, ESS

FlashCopy Internal Copy

Available on:

DS6000, DS8000, ESS

SAN Volume Controller, XIV,

DS4000, DS5000,

PiTIncrementalCopy

Primary

Prod

PPRC-XD Secondar

y

PiTCopies

Global Copy(Asynchronous

)

PiT Incrementa

l FlashCopy

FlashCopy

WAN

REMOTE

Data Replication Enabling Core Technologies -



DS8000 FlashCopy Options

Once Source/Target “Logical” relationship established – Both volumes are available for Read/Write.

Multiple Relationships - Single Source may have up to 12 Targets.

Background Copy Optional–NoCopy to Copy – Background Copy

Persistent & Incremental

Consistent FlashCopy (across volumes in single DS8K or across multiple DS8Ks)

Target device may be in any LSS

Space Efficient FlashCopy (single FC repository for all target volumes)– For Backup typically requires 10-20% of actual space.

Remote Pair FlashCopy



applicationapplication

UCB

MetroMirror

UCB

IBM HyperSwap Technology -> Higher Availability for Parallel SYSPLEX !

Ability to swap enterprise class System z Disk Subsystems in seconds.HyperSwap substitutes Metro Mirror secondary for primary deviceNo operator interaction, Designed to scale to multi-thousands of z/OS volumes Includes volumes with SYSRES, page data sets, catalogsNon-disruptive - applications keep using same device addressesHyperSwap integration with z/OS yielding Higher Availability for z/OS.

P S

Basic HyperSwap (GA 2008)•Single site continuous availability function

•Unplanned failures•Planned fail over (testing)

•Aimed at masking disk failures•IBM Disk Subsystems ONLY (ESS 800, DS6000, DS8000)

GDPS/PPRC HyperSwap Manager (GA 2006)•Single site or multiple sites•Continuous availability and/or Entry level DR solution•Any Vendors disk subsystem that supports the IBM PPRC Architecture (ex. IBM, EMC, Hitachi (HP & SUN)

GDPS/PPRC w/HyperSwap (GA 2002)•Full function HyperSwap across multiple Sites for D/R and High Availability.•Includes Server, Workload & Network Management across sites in addition to Storage•Supports GDPS/MzGM and GDPS/MGM environments.•Any vendors disk subsystem that supports the IBM PPRC Architecture. (ex. IBM, EMC, Hitachi (HP & SUN)



DS8000 Global Mirror:Concept

Asynchronous long distance copy (Global Copy), i.e., little to no impact to application writesVerify End to End Data Integrity

CRC sent & verified w/each changed Block/record.GM detects dropped FCP FramesFor ECKD devices track format also sent & verified in Metadata.

CG “Marked” at primary, but CG is formed at Target Site, yields continuous BW utilization.Momentarily pause application writes (fraction of millisecond to few milliseconds)Create point in time consistency group across all primary subsystems (in OOS bitmap)

New updates saved in Change Recording bitmapRestart application writes and complete write (drain) of point in time consistent data to remote siteStop drain of data from primary (after all consistent data has been copied to secondary)Logically FlashCopy all data (i.e., 2ndary is consistent, now make tertiary look like 2ndary)Restart Global Copy writes from primaryAutomatic repeat of sequence every few seconds to minutes to hours (selectable and can be immediate)

Intended benefitLong distance, no application impact (adjusts to peak workloads automatically), small RPO, remote copy solution for zSeries and Open Systems data, and consistency across multiple subsystems

Host I/O

TertiaryPrimary

Secondary

RemoteSite

LocalSite

(Asynchronous)

Global Copy

FlashCopy (record, nocopy, persistent, inhibit target

write)

Global Copy (PPRC-XD) over long distance

Could require channel extenders

FCP links only



Metro/Global Mirror architecture

Local Site (Site A) Intermediate Site (Site B) Remote Site (SiteC)

BA C

D

Metro Mirror

Global MirrorMetro Mirror networksynchronoussmall distance

Global Mirror networkasynchronouslarge distance

Global Mirror FlashCopyNOCOPY

***Server or Servers

normal application I/Os

12

3

4

Metro Mirror write1. application to VolA2. VolA to VolB3. write complete to A4. write complete to application

a bc

Global Mirror consistency group formation (CG)a. write updates to B volumes paused (< 3ms) to create CGb. CG updates to B volumes drained to C volumesc. after all updates drained, FlashCopy changed data from C to D

volumes



Metro Mirror - Synchronous mirroring available on DS8000, DS6000, and ESS

Designed to provide:No data loss Industry leading replication performanceHigh Availability with GDPS™ HyperSwap™:

System z™ and open systems dataEase of use, lower cost

SECONDARY

VOLUME A

VOLUME B

PRIMARY

IBM Host

Send I/O write (2)

Confirm I/O write (3)

Req (1)

Ack (4)

Read and write to both source and copy possibleWrite Read


Time


Source Target

Delivered by IBM Global Services

RCMF/PPRC

GDPS/PPRC

GDPS/PPRCHyperSwap Manager

GDPS/XRC

RCMF/XRC

GDPS/GMGDPS/MGM

DS6000/DS8000 -Data Replication

Data Replication Options: • FlashCopy® (FC) (within Box)• Metro Mirror (MM/PPRC) (Sync Copy) • Global Copy (GC) Async Copy, No Consistency

• PiT Incremental FC to MM, GC, or GM Primary

• Global Mirror (GM) Async Copy • Global Mirror for zSeries® (zGM)• Metro Global Mirror (MGM)• zGM + MM/PPRC Multi-Target (MzGM)

Managed by:• TotalStorage® Productivity Center –Replication Manager (TPC-R)

• Geographically Dispersed Parallel Sysplex (GDPS)

(See integrated solutions on inside flap.)For more information, please contact your local IBM Representative or IBM Business Partner.

May 2008

Integrated solutions

FlashCopy - Point in time copy available on DS8000™, DS6000™, and ESSFeatures:

Multiple relationships - Single Source may have up to 12 Targets.Background Copy optionalNoCopy to Copy – Background CopyPersistent and IncrementalConsistent FlashCopyTarget device may be in any LSSPiT Inc FlashCopy to MM, GC, or GM Primary

Delivered by IBM Global Services

GDOC Veritas Clusters + IBM MM, GM or EMC SRDF, SRDF/A or VVR

AIX/HACMP 5.1 +

Metro Mirror

Windows GeoDistanceMSCS + Metro-Mirror

Scripts

TECHNICAL SALES SUPPORT AMERICAS



Metro / Global Mirror – Three-site synchronous and asynchronous mirroring Available on DS8000

Global Mirror - Asynchronous mirroring available on DS8000™, DS6000™, and ESS

z/OS Global Mirror -- Asynchronous mirroring for System z available on DS8000 and ESS 800

PrimaryHost

SecondaryHost

3PrimaryDS8000

1

2

4

System DataMover

System DataMover

'A‘Primary

Native performance

Consistent Data

FlashCopy

REMOTEHOSTS

SAN

PRIMARYHOSTS

‘B’Global CopySecondary

SAN

DS8000, ESSGlobal Mirror

–

A B

C

I/O Write

I/O Write

Metro

Global

Designed to provide:Performance, scalability

Metro Mirror, Global MirrorSatisfy all 3-site requirements:

Fast failover / failback to any siteFast re-establishment of three-site recovery, without production outagesResynchronize any site with incremental changes only*

Ease of use, autonomic, self-monitoring

Premium performance and scalabilityData moved by DFSMS™ System Data Mover (SDM) address space(s) running on z/OS. Supports heterogeneous disk subsystems

Unlimited distancesTime consistent data at the recovery site

RPO within secondsSupports System z™ and System z Linux® dataOver 200 installations worldwide3-Site GDPS®/PPRC HyperSwap and GDPS/XRC “Multi-Target” Supported

Designed to provide:

Metro / zGlobal Mirror Multi-Target (MzGM) - Three-site synchronous and asynchronous mirroring Available on DS8000

Designed to provide:• Performance, scalability

•Metro Mirror, zGlobal Mirror• Satisfy all 3-site requirements:

•HA and CA capability for Metro distance•Out of region DR with fast recovery•Resynchronize any site with incremental changes only*

• Ease of use, autonomic, self-monitoring

DS6K/DS8K/ESS key design to points:

Capability to achieve an RPO of 3-5 seconds with sufficient bandwidth and resources

Do not impact production applications when insufficient bandwidth and/or resources are available

Scalable; providing consistency across multiple primary and secondary disk subsystems

Allow for removal of duplicate writes within a consistency group before sending data to remote site

Allow for less than peak bandwidth to be configured by allowing RPO to increase without restriction at peak times

Provide consistency between System z and open systems data and between different platforms on open systems.

© International Business Machines Corporation, 2007.IBM, the IBM logo, GDPS, Global Dispersed Parallel Sysplex, HyperSwap, z/VM, z/OS, z.VSE, System z, are trademarks of International Business Machines Corporation in the United States, other countries or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Other company, product and service names may be trademarks or service marks of others.



DS6K/DS8K Copy Services Matrix

Yes

No

No

No

Yes3

Yes9

Yes

Yes

Yes

Yes

No

Incremental FLC Source

Yes

No

No

No

No

No

Yes

Yes

No6

No5

No

Incremental FLC Target

YesNoYesYes9YesYesYesYesNo7Incremental FLC Source

YesNoNoNoNoNoYes 2No5No7Incremental FLC Target

Yes

No

Yes

No

No

Yes1

No6

Yes

Yes

Global Mirror

Primary

No

No

Yes

No

No

No

Yes1

No5

No


YesYesYesYes1No6YesYesGlobal Mirror Primary

NoYes8YesNoYes1No5NoGlobal Mirror Secondary

YesYesYesNoYesYesYesConcurrent Copy Source

NoYes4Yes 4NoYes 2No5NoFlashCopy Target

YesYes 4Yes 3,4YesYesYesYesFlashCopy Source

NoYes8YesNoYes 1No5NoMetro Mirror or Global

Copy Secondary

YesYesYesYes 1NoYesYesMetro Mirror or Global

Copy Primary

YesNo5YesNo5YesNoYes11GMz10

(XRC) Secondary

YesYesYesNoYesYes11NoGMz10

(XRC) Primary

Concurrent Copy

Source

FlashCopy Target

FlashCopy Source

Metro Mirror or Global

Copy Secondary

Metro Mirror or Global Copy

Primary

GMz10

(XRC) Secondary

GMz10

(XRC) Primary

DeviceIs

MayBecome



DS6K/DS8K Copy Services Matrix Notes1. Only in a Metro/Global Copy (supported on ESS) or a Metro/Global Mirror Environment

(supported on ESS and DS8000).2. FlashCopy V2 at LIC 2.4.0 and higher on ESS800 (DS6000 and DS8000 utilize FlashCopy V2 by

default).– You must specify the proper parameter to perform this– Metro Mirror primary will go from full duplex to copy pending until all of the flashed data

is transmitted to remote– Global Mirror primary cannot be a FlashCopy target

3. FlashCopy V2 Multiple Relationship.4. FlashCopy V2 Data Set FlashCopy (only available for z/OS volumes).5. The Storage Controller will not enforce this restriction, but it is not recommended.6. A volume may be converted between the states Global Mirror primary, Metro Mirror primary

and Global Copy primary via commands, but it two relations cannot exist at the same time (i.e. multi-target).

7. GMz (XRC) Primary, Global Mirror Secondary, Incremental FlashCopy Source and Incremental FlashCopy Target all use the Change Recording Function. For a particular volume only one of these relationships may exist.

8. Updates to the affected extents will result in the implicit removal of the FlashCopy relationship, if the relationship is not persistent.

9. This relationship must be the FlashCopy relationship associated with Global Mirror – i.e. there may not be a separate Incremental FlashCopy relationship.

10. Global Mirror for zOS (GMz) is supported on ESS and DS800011. In order to ensure Data Consistency, the XRC Journal volumes must also be copied.



Reference Resources



References

“Global Mirror Whitepaper”, V1-3– By Nick Clayton, 13/09/2005– http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100642

z/OS DFSMS Advanced Copy Services– SC35-0428, January 2006

DSx000 Command-line Interface User’s Guide– DS6000 GC26-7922, September 2006– DS8000 SC26-7916, November 2006

Device Support Facilities User’s Guide and Reference, Release 17– GC35-0033, March 2005

Redbooks– www.redbooks.ibm.com or w3.itso.ibm.com– Search on “DS6000” & then “DS8000”

• Many choices available– Also search on “copy services”

• Again, many choices available



To COPY or to NOCOPY?…. That is the question!

BACKGROUND NOCOPY is typically the best choice to minimize rank and DA activity within the physical box

But…. You must ask why are you making a copy? And…. What type of application workload do I have?– For example:

• Is the copy only going to be used for creating a tape backup?– BACKGROUND NOCOPY should be used and the relationship withdrawn after the tape

backup is complete• Is the copy going to be used for testing or development?

– NOCOPY again is typically the best choice• Will you need a copy of the copy?

– BACKGROUND COPY must be used so that the target will be withdrawn from its relationship after all of the tracks are copied thereby allowing it to be a source in a new relationship

> Possibly use NOCOPY to COPY option • Is the workload OLTP (NOCOPY typically is the choice) or are there a large number of

random writes and are not cache friendly (COPY may be the better choice)



References

SC26-7916 DS8000 Command-Line Interface User’s Guide

GC26-7922 DS6000 Command-Line Interface User’s Guide

SC35-0428 DFSMS Advanced Copy Services

SG24-6788 IBM System Storage DS8000 Series: Copy Services in Open Environments

SG24-6787 IBM System Storage DS8000 Series: Copy Services with IBM System z

SG24-6783 IBM System Storage DS6000 Series: Copy Services in Open Environments

SG24-6782 DS6000 Series: Copy Services with IBM System z Servers

GC35-0033 Device Support Facilities User’s Guide and Reference, Release 17



References

“Global Mirror Whitepaper”, V1-3 By Nick Clayton, 13/09/2005 WP100642– http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100642

Performance White Paper– http://w3-

1.ibm.com/sales/systems/portal/_s.155/254?navID=f320s260&geoID=All&prodID=System%20Storage&docID=ditlDS8000PerfWPPower5

DS8000/DS6000 Copy Services: Getting Started WP100905– http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100905

DS8000 Disk Mirroring Licensing - Frequently Asked Questions – http://w3-

1.ibm.com/sales/systems/portal/_s.155/?navID=f220&geoID=AM&prodID=System%20Storage&docID=ditlDS8000MirroringLicFAQ

Redbooks– www.redbooks.ibm.com or w3.itso.ibm.com

• Search on “DS6000” & then “DS8000” Also search on “copy services”– IBM System Storage DS8000 Series: Copy Services in Open

Environments, SG24-6788-02Redbook, published 29 November 2006

– IBM System Storage DS8000 Series: Copy Services with IBM System z, SG24-6787-02Redbook, published 14 December 2006



References

Technical information on IBM TotalStorage Business Continuity Solutions:

– IBM ITSO Redbook: TotalStorage Business Continuity Solutions Guide SG24-6547-02 :

• http://www.redbooks.ibm.com/abstracts/sg246547.html?Open– IBM ITSO Redbook: TotalStorage Business Continuity Solutions Overview SG24-6684

:• http://www.redbooks.ibm.com/redbooks/pdfs/sg246684.pdf

Technical ITSO Redpapers on Business Continuity Solutions:–REDP4062 TotalStorage Business Continuity Solution Selection

Methodology• This ITSO Redpaper describes an IT Business Continuity Solution Selection Methodology and how to apply it to your computing environment. There are several scenarios which demonstrate the application of this methodology.

–REDP4063 Planning for Heterogeneous Platform BC and DR• This ITSO Redpaper discusses how to plan for IT Business Continuity in a highly heterogeneous platform server and storage installation. Discussion is included of of IBM storage-based tools that can be useful to provide Business Continuity and Disaster Recovery in this diverse environment.

• It is a 2005 version of this presentation



References

Technical ITSO Redpapers on Business Continuity Solutions:

–REDP4064 Small and Medium Business Considerations for IT BUsiness Continuity

• Small and Medium Business (SMB) enterprises have IT Business Continuity needs and concerns similar to large enterprises; yet in other ways, SMB companies have key differences. This redpaper discusses those differences, and describes an overview of IT Business Continuity Solution selection in the SMB business environment.

–REDP4066 Networking Tutorial for IT Business Continuity Planners• Confused by terms such switches, routers, bridges, OC3, DWDM, dark fibre?

• This Redpaper is intended to enable the IT Business Continuity planner to better understand many commonly used networking concepts, in order to be able to better evaluate and select the appropriate networking components for your IT Business Continuity solution.



References

IBM Implementation Services for Geographically Dispersed Open Clusters (GDOC)

– http://www-03.ibm.com/servers/storage/solutions/business_continuity/continuous_availability/technical_details.html

IBM Geographically Dispersed Parallel Sysplex: – http://www-

03.ibm.com/servers/storage/solutions/business_continuity/continuous_availability/technical_details.html

IBM System Storage Business Continuity Solutions website:

http://www-03.ibm.com/servers/storage/solutions/business_continuity/

IBM Global Services Business Continuity and Recovery Services:

http://ibm.com/services/continuity/

IBM Business Continuity and Recovery Services– Your local IBM Global Services ITS representative



System Storage Enterprise Disk Practices Resources System Storage Business Continuity Solutions website

http://www-03.ibm.com/servers/storage/solutions/business_continuity/index.html

System Storage Technology Centerhttp://www-03.ibm.com/system/storage/ Storage Education http://www-03.ibm.com/systems/education/cust/crossprod/custcp.html

System Storage Interoperation Centerhttp://www-01.ibm.com/systems/support/storage/config/ssic/index.jsp

System Storage Serviceshttp://www-03.ibm.com/systems/storage/services/index.html

Redbooks/Redpapershttp://www.redbooks.ibm.com/redbooks.nsf/portals/Storage The IBM TotalStorage DS8000 Series: Concepts and Architecture (SG24-6452-00)IBM System Storage Business Continuity Solutions Overview (SG24-6684-01)IBM System Storage DS8000 Series: Copy Services with IBM System z (SG24-6787-02)IBM System Storage DS8000 Series: Copy Services in Open Environments (SG24-6788-02)IBM System Storage Solutions Handbook (SG24-5250-06)

White papersIBM Storage Infrastructure for Business Continuity SolutionGlobal Mirror Technical Whitepaper



References

IBM System Storage Business Continuity Solutions website:

http://www-03.ibm.com/servers/storage/solutions/business_continuity/

IBM Global Services Business Continuity and Recovery Services:

http://ibm.com/services/continuity/

IBM Business Continuity and Recovery Services– Your local IBM Global Services ITS representative



Following are the IBM links to presentations on theEnterprise Disk Mirroring Sales Kit on System Sales Web Site

IBM Three Site Enterprise Disk Mirroring Executive SummaryIBM Three Site Mirroring Competitive Marketing Summary for IBM*

IBM Two and Three Site Enterprise Disk Mirroring OverviewIBM Three Site Disk Mirroring for Open Systems Presentation GuideIBM Three Site Disk Mirroring for zSeries Presentation Guide

IBM z/OS Global Mirror and Global Mirror Positioning GuideIBM Two and Three Site Disk Mirroring Technical Presentation GuideDeep Dive on IBM Global Mirror – from US Storage Symposium 2005IBM Disk Mirroring Update – from US Storage Symposium 2005

IBM DS8000 DS6000 ESS Disk Mirroring Link Efficiency TCO Studies*IBM Two and Three Site Mirroring Competitive Marketing Guide*

* IBM Confidential documents



Trademarks

CICS*ClearCaseDB2*e-business logoFICON*GDPS*HyperSwap

The following are trademarks of the International Business Machines Corporation in the United States and/or other countries.

The following are trademarks or registered trademarks of other companies.

* All other products may be trademarks or registered trademarks of their respective companies.

Intel is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States, other countries or both.Microsoft is a trademark of Microsoft Corporation in the United States, other countries, or both.Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.UNIX is a registered trademark of The Open Group in the United States and other countries.Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

* Registered trademarks of IBM Corporation

IBM*IBM eServerIBM logo*IMS*MQSeries*On Demand Business logo

Parallel Sysplex*Rational*System z9Tivoli*WebSphere*

z/OS*z/VM*zSeries*



DisclaimersCopyright © 2009 by International Business Machines Corporation.

No part of this document may be reproduced or transmitted in any form without written permission from IBM Corporation.

Product data has been reviewed for accuracy as of the date of initial publication. Product data is subject to change without notice. This information could include technical inaccuracies or typographical errors. IBM may make improvements and/or changes in the product(s) and/or programs(s) at any time without notice.

Any statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such such products, programs or services available in all countries in which IBM operates or does business. Any reference to an IBM Program Product in this document is not intended to state or imply that only that program product may be used. Any functionally equivalent program, that does not infringe IBM’s intellectually property rights, may be used instead. It is the user’s responsibility to evaluate and verify the operation of any on-IBM product, program or service.

THE INFORMATION PROVIDED IN THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IBM EXPRESSLY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NONINFRINGEMENT. IBM shall have no responsibility to update this information. IBM products are warranted according to the terms and conditions of the agreements (e.g., IBM Customer Agreement, Statement of Limited Warranty, International Program License Agreement, etc.) under which they are provided. IBM is not responsible for the performance or interoperability of any non-IBM products discussed herein.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents or copyrights. Inquiries regarding patent or copyright licenses should be made, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 10504-1785U.S.A.

prs3782 ds8000 data replication best practices 20090928

Documents