ibmsystemstorageds layout for sap ha instllations v3

7/31/2019 IBMSystemStorageDS Layout for SAP HA Instllations V3

1/13

IBM System Storageand

SAP High Availability Installations onAIX

including PowerHA, Power Server and VIOS aspects

Version Date: December 20, 2010Version 2

Katharina ProbstWalter Orb

IBM Boeblingen Lab, Germany

SAP on Power Systems Development @ SAP Copyright IBM Corp. 2009

Copyright IBM Corp. 20101


2/13

Table of contents

Table of contents.................................................................................................................................. 2

Abstract................................................................................................................................................. 3

Introduction........................................................................................................................................... 4

Dual VIOS................................................................................................................................................... 4..........

AIX Operating System................................................................................................................................ 4..........

PowerHA..................................................................................................................................................... 5..........

SAP............................................................................................................................................................. 6..........

Storage Setup........................................................................................................................................ 7

Eliminate Hardware SPoF........................................................................................................................... 8..........

Eliminate SPoF in the Path......................................................................................................................... 8..........

Data Protection using Storage Features..................................................................................................... 9..........

Data Redundancy....................................................................................................................................... 9..........

Storage Volumes........................................................................................................................................ 9..........

Storage Volume Groups............................................................................................................................. 11........

Resources............................................................................................................................................ 12

About the authors............................................................................................................................... 12

Trademarks and special notices........................................................................................................ 13



3/13

Abstract

This paper is intended to give an understanding of technical matters when setting up high availabilitySAP environments using IBM System Storage DS and Power Systems on AIX. While there are good

papers about how to set up storage and making the databases well performing in general, we focushere particularly on high availability (HA) installations and disaster recovery (DR) aspects in the SAPcontext.

For those skilled in the art it is obvious that IBM Power Systems technology including AIX, VIOSconcepts and the used Software namely PowerHA and SAP Software require a correspondingstorage layout to work.

This document is written on base of the following levels/technologies:

IBM PowerHA for AIX Version 5.5 (HACMP)

IBM System p5 Server with POWER5+ technology

IBM AIX Version 6.1

VIOS 2.1

SAP NetWeaver 7.00

IBM System Storage DS8300 Model 2107-9A2



4/13

IntroductionApplications, servers and storage are often administered by different divisions or persons, what often leads to anisolated view. Hence the intention is to show the end-to-end concept from a storage point of view for HA (highavailability) enabled SAP landscapes.

The Application:Today, SAP supports a special HA installation option. This option separates the database, the SAP CentralServices (enqueue and message server), the Enqueue Replication Server and the Instances into independentcomponents. This design allows each group to move independently between nodes, not requiring to move entireSAP systems.To enable a move of an application the management of these resources is required. One possibility is to usePowerHA (HACMP).

The Server:The virtualization technologies provided when using AIX on Power Systems became key for SAP landscapes byproviding more flexibility and better utilization. One item that will be later discussed in more detail is the dual VIOSconcept, allowing the virtualization of disk and network and sharingof few physical resources by many partitionswhile maintaining redundancy.

The following chapters will discuss each of the topics mentioned above in an end-to-end view. The focus is todescribe what needs to be accomplished on the storage side to increase the robustness of the concept. This isachieved by synchronization of the concept with the available technologies and software and elimination of singlepoint of failures (SPoF).

Dual VIOS

A Virtual IO Server is an appliance that provides virtual storage and shared Ethernet attachment. Today there ismore than one way to set it up. In general, the main benefit is to attach fewer physical adapters for Ethernet andstorage to the VIOS servers and virtualize them for many LPARS sharing the same physical server. This reducesthe amount of required physical adapter substantially. Whereas earlier, at least 2 Ethernet and 2 fibre channeladapters per partition were required to eliminate single point of failures (SPoF). With just 4 adapters of each typeand a dual VIOS concept we can fulfill this requirement for an entire server. Of course, for performanceconsiderations there might be more physical adapters required.

Due to the high value of virtualization and its benefits for cost reduction, a dual VIOS concept is highlyrecommended to be part of every SAP landscape.

The requirement for the storage layout in combination with PowerHA is to create a 1:1 association betweenstorage volume and AIX volume group on VIOS level to mange the disks by two VIOS Servers.

Rule 1:Do not split or merge storage volumes on the VIOS level.

To learn more about the different options to use and to setup a VIOS please see the Advanced POWERVirtualization on IBM System p: Virtual I/O Server Deployment Examples document at http://www.redbooks.ibm.com/abstracts/redp4224.html.

AIX Operating System

The AIX operating system is required to run the applications. If the operating system fails a fall-over needs to be

initiated. Although it can be entirely automated by PowerHA, storage redundancy can help to reduce the likelihood

of a fall-over. The recommendation is to mirror the rootvg to at least two storage volumes on different storage

boxes using a LVM mirror. This provides the benefit of un-disruptive operation of the operating system in case of a

http://www.redbooks.ibm.com/abstracts/redp4224.htmlhttp://www.redbooks.ibm.com/abstracts/redp4224.htmlhttp://www.redbooks.ibm.com/abstracts/redp4224.htmlhttp://www.redbooks.ibm.com/abstracts/redp4224.html


5/13

disk, cabling or storage subsystem failure within one site. The LVM mirror today supports up to three mirror

copies.

Rule 2:Create a LVM mirror volume for the rootvg on a separate storage subsystem.

A good reference for the different mirroring options can be found in AIX Logical Volume Manager, from A to Z:Introduction and Concepts in Chapter 2 at http://www.redbooks.ibm.com/redbooks/pdfs/sg245432.pdf

The rootvg has a slightly different procedure in comparison to mirroring non bootable disks. Step-by-step

instructions can be found here: http://publib.boulder.ibm.com/infocenter/aix/v6r1/index.jsp?topic=/

com.ibm.aix.baseadmn/doc/baseadmndita/mirrorootvg.htm

PowerHA

The PowerHA product is a piece of software that manages applications (not the OS) and the correspondingresources. This cluster software monitors all nodes to detect failures in the application, the operating system, thenetwork and storage connectivity to react on failures. It starts, stops, and moves applications and their resourceson the defined cluster nodes based on policies and scripts.

To make it work the executables and the data needs to be accessible on the node where the application iscurrently running. This has two aspects in regards to the layout of the storage:

a) the resources such as disks need to move between nodes to make the application available (HA).b) the data needs to be kept redundant to protect from a disaster (DR).

In order to have the data and executables accessible PowerHA can use two different approaches. If concurrentwrite is handled by the application a clustered filesystem (GPFS) can be used to have all data on all nodes at anytime. The recommended approach today for SAP is to use enhanced concurrent volume groups in non-concurrentmode which are created on shared storage volumes. Shared storage volumes have a path to all nodes theapplication can potentially run on. Now PowerHA can varyon and varyoff the volume groups on the node wherethe volume is required.

Rule 3:

PowerHA with enhanced concurrent volume groups require a minimum of oneshared storage volume foreach moving application.

A well designed HA concept also requires a DR concept. Meaning the duplication of data to recover from adisaster. On the one hand this protects the business relevant data and on the other the files required for anapplication files to restart it on the fall-over node. The scop of this concept reaches from Tape Backup, RAIDlevels, storage mirroring to LVM mirroring. Depending on the PowerHA release LVM mirroring, GLVM mirroring,storage mirroring (Metro Mirror and Global Mirror) are integrated into PowerHA, and by that are cluster-aware.For a one-site-HA-solution each volume used for a SAP component should be mirrored to a second storagesubsystem using LVM. The advantage of LVM mirroring is the seamless switch to a synchronous copy. A two sitesolution needs to transfer the data to a remote location where LVM usually is not suitable any more. Here storagebased mirroring technologies can be used. Details are discussed in the section Data Protection using StorageFeatures.

The other protection mechanisms are not directly linked to a SAP storage layout, but a necessity to protect thedata.

Rule 4:Rule 4:Each storage volume used for a SAP component needs at least one additional volume on a differentEach storage volume used for a SAP component needs at least one additional volume on a differentstorage subsystem to mirror the data and the SAP system files.storage subsystem to mirror the data and the SAP system files.

An overview of the different DR and HA concepts including the mirroring options can be found in the Resourcesection.

http://www.redbooks.ibm.com/redbooks/pdfs/sg245432.pdfhttp://www.redbooks.ibm.com/redbooks/pdfs/sg245432.pdfhttp://www.redbooks.ibm.com/redbooks/pdfs/sg245432.pdf


6/13

SAP

The SAP installation consists of the SAP executable and a database. First we look at the SAP executable,

followed by the database aspects.

SAP provides the standalone enqueue and the enqueue replication server by a new HA installation option. The

benefit is to split the SAP application into different components which: can be moved between cluster nodes using PowerHA. Thus they have to be installed in their own volume

groups see the PowerHA chapter before.

are required on each node or

are bound to a single node.

The benefit is the flexibility of not being forced to move the entire SAP system when performing administrative fall-

overs. Also, if one component dies a selective action can take place. Finally the replicated enqueue server is a

backup of the enqueue table on a second node in order to protectthe state in a fall-over situation.

The following tables summarizes SAP components per stack. As discussed before, each component needs at

least one storage volume and by that the minimum amount of storage volumes (without counting the ones formirroring) is described in these tables.

1. ABAP / JAVA Stack

Instance Description Node behavior

CentralServices ABAP central Services: ASCS

JAVA central Services: SCS

Moves between nodes each resides on a shared storage volume.

ERSHot standby enqueue table

Can be implemented to move between nodes or isinstalled on each node into a local filesystem. a shared volume for option one, one dedicatedvolume per node for option two.

SYSThe /usr/sap//SYS

directory

One copy on all nodes one dedicated volume pernode (often mounted under /usr/sap).

SAP globalincludes /sapmnt and /usr/sap/

trans

This can be either normal NFS mount and by thattransparent to this setup. When using a NFS cross-mount a shared volume is required.

Instance/usr/sap//

Instances are bound to a single node one dedicated volume per instance is required (min.two instances are required).

2. Double Stack


ASCSABAP central services

Moves between nodes resides on a shared storage volume.

SCSJAVA central services

Moves between nodes resides on a shared storage volume.



7/13


2 x ERSHot standby enqueue table, one

for the JAVA and one for the

ABAP enqueue.

Can be implemented to move between nodes or isinstalled on each node into a local filesystem. two shared volumes for option one, two dedicatedvolumes per node for option two.

SYS The /usr/sap//SYS

directory

One copy on all nodes one dedicated volume per node (often mountedunder /usr/sap).

SAP globalincludes /sapmnt and /usr/sap/

trans

See above.

Instance/usr/sap//

See above.

Rule 5:Each moving SAP component requires a minimum of one shared storage volume. The local componentsare recommended to have their own storage volumes too. It is recommended to mirror all of them to asecond storage subsystem which doubles the amount of required storage volumes.

The database availability can be achieved in different ways, depending on the used database technology.Basically there are the following possibilities:

Shared disk installation as a cold standby solution Clustered database (DB2 PureScale, Oracle RAC) Second DB instance as hot stand-by solution (DB2 HA/DR, Oracle Data Guard) covering the DR aspect

for the database.

Please check the support status of your combination of PowerHA, SAP, DB and OS.

In general data, log , and other files are put onto different AIX volume groups, having their own volumes on the

storage subsystem. The common approach for a SAP DB layout is to create storage volumes for each //

/sapdata* file system, at least one for the log-files and a last one for the rest. Other tools as for example the

Tivoli Storage FlashCopy Manager might have additional requirements for the volume group layout, so it isrecommendedto use a least in three different volume groups fordata, log and exe.

Please check the references in the Resources Section for recommendations on how to implement a storage

layout for a well performing database.

Rule 6:For a shared disk installation one shared storage volume might technically be sufficient for a shareddisk database installation. A better, but still basic approach is to use a minimum of six shared storagevolumes. Assign four storage volumes to the data volume group and create four filesystems for the data,followed by the log-files and the rest.

The size of the SAP volumes depends on many factors. Please refer to the SAP installation guides you can obtainfrom your SAP representative orwww.sap.com. Be aware that the installation guides describe the absolute

minimum requirements to run the installation. Usually you will need more.

Storage SetupThe task of the storage is to protect the data and the SAP executables. The Recovery Time Objective (RTO) is avariable describing the amount of lost data in case of a disaster. The RTO goes directly along with the usedmirroring technology, the distance to mirror and the quality of the network.

Also the connectivity between the server and the storage and volume layout influences the robustness.

http://www.sap.com/http://www.sap.com/http://www.sap.com/http://www.sap.com/


8/13

We discuss here a one site HA solution with full HA capabilities, meaning a HA concept also taking DR intoaccount. More details and the transfer to a more complex setup can be found in IBM Storage and SAP evaluation of HA and DR concepts which is listed in the Resource section.

Eliminate Hardware SPoF

The basic equipment for a one site full HA solution consists of a minimum of:

two Power Servers

two SAN Switches and

two Storage Units

as shown in the picture on the right.

The servers are required to be protected against a server outage to let the application switch over to the second

node.

Also a SAN or network switch can become a SPoF if it is not redundant.

Finally the storage subsystems are physically duplicated to protect data by mirroring and enable the application to

get undisrupted data access.

Eliminate SPoF in the Path

The next step is to ensure that beginning from the application running in a LPAR down to the storage volumes all

paths are kept redundant. From a Power server view the path for the fiber channel can look as illustrated in the

following picture:

Power Server

LPAR

Dual VIOS concept

Min. two physical adapter for each VIOS

Each VIOS is attached to min. two switches.

Switch

Each switch is to be zoned that each VIOS sees both storage systems through each switch it is

attached to.

Storage

Use at least two I/O enclosures to get to a volume.

For Ethernet the same rule of redundancy has to be applied accordingly to the technology.

LPAR

Dual VIOS concept

Min. two physical Ethernet adapter for each VIOS.

Each VIOS is attached to min. two different networks.



9/13

Data Protection using Storage Features

In order to protect data IBM storage provides a set of technologies suitable for SAP landscapes. The availability of

the feature depends on the model and the purchased licenses.

Full Disk Encryption (FDE):

This storage security feature is hardware based, well performing encryption of the physical disks. It is based onspecial FDE-disks and key servers holding the keys for encryption and decryption. This protects the content on

the disks for example during relocation or on the way to recycling as long the key servers are kept safe.

RAID:

RAID protects from disk failures within a box. The DS8000 models provides RAID level 5 (Single disk failure)

up to level 10 (full duplication and striping). The chosen level has an direct effect on the amount of required

disk space. It is best practice not to mix the RAID level for a SAP landscape.

Storage Mirroring:

Point to Point Remote Copy (PPRC) comes in two flavors. Metro Mirror is a synchronous mirroring technology

often used to connect data centers within city boundaries where LVM reaches its limits. Global Mirror isasynchronous and by that suitable for long, unlimited distances. These technologies are used if a secondary

site is to be connected into a HA/DR setup. More details can be found in IBM Storage and SAP evaluation of

HA and DR concepts listed in the Resource section.

Data Redundancy

To protect the environment from a loss of any kind of data, several mechanisms need to go hand in hand. First of

all the data needs to be protected from disk failures. This is the first mandatory step when initializing the storage

arrays: specifying a RAID level. With RAID 5 effectively 6 out of 8 disks can be used for data. The other two disks

in the rank are used for spare to replace the broken disk and parity to recalculate the lost data. RAID 6 and RAID

10 increase the robustness against disk failures within one rank. On the other hand it is not recommended for

SSD drives and reduces the net capacity by requiring more disks for spare and parity. The RAID level to choose

for SAP applications is RAID 5 for most cases.

The next step is to protect against the connectivity lost of one storage system. This can be caused by broken

cables, loss of power, or any kind of site disaster. Hence the recommendation is to mirror the data to at least one

additional, independent storage subsystem within one site and to a secondary remote site. Within one site LVM

mirroring is widely used to mirror all SAP, DB and rootvg storage volumes. The copy within one site minimizes the

RPO in case of a single hardware or connectivity failure. To provide additional robustness a second site protects

against an entire data center outage. For that purpose storage based mirroring (PPRC) of the SAP and DB related

volumes is recommended. The RTO value increases depending on the chosen technology (synchronous or

asynchronous), the distance and the bandwidth.

Storage Volumes

First of all the mirroring requires at least to duplicate the storage layout to a second storage subsystem. This

means duplicate the physical resources, power supply, , the volumes and their corresponding volume groups. If

the quorum check on OS level is used a Quorum Disk on an third storage box is recommended to avoid a failing

fall-over of the SAP application in case one volume is lost. Furthermore, the zoning of the switches need to allow

the servers to share volumes as described earlier in more detail.



10/13

Here a summary of the minimum recommended storage volume requirement:

AIX

Min. one volume per node for the rootvg

One volume for paging space

SAP, PowerHA and dual VIOS

Min. one (shared) volume for the database logfiles

Min. one (shared) volume for the / file system

Min. one (shared) volume per ///sapdata* file system, combined into one volume group on

AIX.

Min. one shared volume for each moving component: Central Services, ERS (optional), NFS

Min. one local volume for each instance

Min. one local volume per node for the /usr/sap file system

Data Redundancy

Mirror the volumes listed before to at least one second storage system

duplicate amount of volumes

RAID 5 or RAID 6

Backup solution (for example Tivoli Storage Manager for ERP)

The overall minimum amount of storage volumes is 29 for one SAP full HA installation. A SAP landscape has at

least three installations (test, quality assurance and production). Now we are counting already around 90 volumes.

In order not to loose track of the volumes a concept of assigning the LUNS can help. A LUN is a 4 digit hex

number to identify a storage volume within one storage box. The original meaning of the LUN needs to reflect the

original purpose. LUNs are having the following Format:

a b cc (each letter represents one hex number).

The original meaning is:

a

is the group identifier. Best practice is to use a value of 0 for System z, all other values for

other Systems.

ab

defines the LSS. Volumes where this value is even are handled by the first managing

Power Server in the storage system, uneven by the second. If one power Server fails all are handled

by the second. Hence the value ensures that the workload is spread between the two Power servers.

cc

specifies the device number.



11/13

One possible concept amongst lots of different possibilities to add some additional information to it can be as

followed:

a

is assigned to a SAP SID. PRD=volumes starting with 1, QAS= volumes starting with 2,

abuse it to spread workload evenly.

cc

identify the purpose.

On all storage systems, the volumes that are mirrored to each other, have the same number. This method does

not remove the necessity to document it, but helps to find mistakes in the end. Also for the VIOS setup it helps to

pre-order the disks for the PowerHA setup on AIX on all nodes in the same way to avoid mistakes in the setup.

NOTE: Planning for an SAN Mirror integration the LSS has additional meaning. Consistency groups are defined

on base of LSS, meaning for example the Database data for each database needs to have an unique LSS

number to avoid bottlenecks.

Storage Volume Groups

The storage volume groups are grouping storage volumes. The concept becomes clear when looking at a native

storage to server attachment (without using storage virtualization technoligies like SAN Volume Controller ). This

also means that each server sided fiber channel port can be attached to one storage volume group, not more. The

other way round: a storage volume group can be attached to multiple server ports. This means a 1:n relation

between fiber channel port on the server and storage volume groups.

Volumes for SAP applications with PowerHA are classified from an application view to be available on node1, on

node2, or shared between node1 and node2.

Recalling the information of the previous chapters describing the elimination of SPoF and the 1:n relation between

fibre-channel port and storage volume groups, we can assign one storage volume group to each server.

Otherwise the path redundancy is not fulfilled anymore. When using more adapters, more storage volume groups

can be created if desired. For the described hardware setup the storage volume group A must contain all node1

volumes and the shared volumes. The node2 storage volumes and the shared volumes are assigned to volume

group B as illustrated in the picture above.



12/13

Resources

These Web sites provide useful references to supplement the information contained in this document:

Logical configuration concepts for IBM System Storage DS8000 series in large data centersftp://service.boulder.ibm.com/storage/isv/LogicalConfigConceptsDS8000.pdf

DS8000 Performance guidewww.ibm.com/support/techdocs WP101602

IBM Redbookshttp://www.redbooks.ibm.com/

IBM Storage and SAP evaluation of HA and DR conceptshttp://www.ibm.com/support/techdocs WP101616

About the authorsKatharina Probst - probst @ de.ibm.com

Walter Orb - [email protected]

ftp://service.boulder.ibm.com/storage/isv/LogicalConfigConceptsDS8000.pdfhttp://www.ibm.com/support/techdocshttp://www.ibm.com/support/techdocshttp://www.redbooks.ibm.com/http://www.ibm.com/support/techdocsmailto:[email protected]:[email protected]://service.boulder.ibm.com/storage/isv/LogicalConfigConceptsDS8000.pdfhttp://www.ibm.com/support/techdocshttp://www.redbooks.ibm.com/http://www.ibm.com/support/techdocsmailto:[email protected]


13/13

Trademarks and special notices

Copyright IBM Corporation 2008. All rights Reserved.

References in this document to IBM products or services do not imply that IBM intends to make them available in every country.

IBM and the IBM logo are trademarks of International Business Machines Corporation in the United States, other countries, or both:http://www.ibm.com/legal/

copytrade.shtml

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

SET and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC.

ABAP, SAP EarlyWatch, SAP GoingLive, SAP NetWeaver, SAP, and SAP logos are trademarks or registered trademarks of SAP AG in Germany and in severalother countries.

Other company, product, or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual

environmental costs and performance characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and

does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available

information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of

performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of

those products.

All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your

local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction.

Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of

performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information

is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that

any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the

storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance

improvements equivalent to the ratios stated here.

Photographs shown are of engineering prototypes. Changes may be incorporated in production models.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites.

The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

http://www.ibm.com/legal/copytrade.shtmlhttp://www.ibm.com/legal/copytrade.shtmlhttp://www.ibm.com/legal/copytrade.shtmlhttp://www.ibm.com/legal/copytrade.shtmlhttp://www.ibm.com/legal/copytrade.shtml

ibmsystemstorageds layout for sap ha instllations v3

Documents