tivoli system automation for multiplatforms v4.1: high ...€¦ · availability solution fr om sa...

Tivoli System Automation for MultiplatformsVersion 4 Release 1

High Availability Policies Guide

SC34-2660-04

IBM

Note!

Before using this information and the product it supports, read the information in “Notices” on page 165.

Before using this information and the product it supports, read the information in “Notices” on page 165.

This edition of System Automation for Multiplatforms High Availability Policies Guide applies to Version 4, Release1, Modification 0 of IBM Tivoli System Automation for Multiplatforms, program number 5724–M00, and to allsubsequent releases and modifications of this product until otherwise indicated in new editions.

This edition replaces SC34-2660-01.

IBM welcomes your comments. A form for readers' comments may be provided at the back of this publication, oryou may address your comments to the following address:

IBM Deutschland Research and Development GmbH

Department 3282

Schoenaicher Str. 220

D-71032 Boeblingen

Federal Republic of Germany

FAX (Germany): 07031+16-3456

FAX (Other Countries): (+49)+7031-16-3456

Internet e-mail: [email protected]

If you would like a reply, be sure to include your name, address, telephone number, or FAX number.

Make sure to include the following in your comment or note:v Title and order number of this book

v Page number or topic related to your comment

When you send information to IBM, you grant IBM a nonexclusive right to use or distribute the information in anyway it believes appropriate without incurring any obligation to you.

© Copyright IBM Corporation 2014, 2016.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.

Contents

Figures . . . . . . . . . . . . . . . v

Tables . . . . . . . . . . . . . . . vii

About this guide . . . . . . . . . . . ixWho should use this guide . . . . . . . . . ixWhere to find more information . . . . . . . ixConventions . . . . . . . . . . . . . . xISO 9000. . . . . . . . . . . . . . . . xRelated RSCT information . . . . . . . . . . xHow to obtain publications . . . . . . . . . xiHow to reach us by e-mail . . . . . . . . . xi

What's new in this release . . . . . . xiii

Chapter 1. SAP Central Services highavailability policy. . . . . . . . . . . 1Planning. . . . . . . . . . . . . . . . 1

Single Point of Failure (SPOF) . . . . . . . 1Example of a two-node setup . . . . . . . . 3SAP Central Services high availability setupoptions . . . . . . . . . . . . . . . 4Location of SAP instance directories . . . . . 10Database high availability installation setup . . 10NFS high-availability installation setup . . . . 11High availability impact . . . . . . . . . 12

Installing and setting up . . . . . . . . . . 17Installing a new ASCS, SCS, or Double Stackhigh availability SAP system . . . . . . . 18Installing and setting up System Automation forMultiplatforms . . . . . . . . . . . . 24Installing and setting up the SAP highavailability policy feature . . . . . . . . . 26Migrating the SAP Central Services highavailability solution from SA MP V3.2 to V4.1 . . 28

Configuring and activating . . . . . . . . . 30SAP Central Services high availability policy . . 30Using the wizard to configure and activate theSAP Central Services high availability policy . . 42SAP Central Services high availability policyparameters . . . . . . . . . . . . . 43

Verifying . . . . . . . . . . . . . . . 78Starting and stopping the SAP Central Serviceshigh availability solution . . . . . . . . . 78Failover scenarios . . . . . . . . . . . 80

SAP Central Services high availability solutionchecklist . . . . . . . . . . . . . . . 82

Chapter 2. SAP HANA SystemReplication high availability policy. . . 85Planning . . . . . . . . . . . . . . . 85

Single Point of Failure (SPOF) . . . . . . . 85Example of a two-node setup . . . . . . . 85SAP HANA high availability setup options . . . 86

Database high availability installation setup . . 87NFS high availability installation setup . . . . 87High availability impact . . . . . . . . . 87SAP HANA documentation references . . . . 87

Installing and setting up . . . . . . . . . . 88Installing a new HANA database highavailability SAP system . . . . . . . . . 88Installing and setting up System Automation forMultiplatforms . . . . . . . . . . . . 89Installing and setting up the SAP highavailability policy feature . . . . . . . . . 89Installing a new ASCS, SCS, or Double Stackhigh availability SAP system . . . . . . . 89

Configuring and activating . . . . . . . . . 90SAP HANA System Replication automation . . 90Using the wizard to configure and activate theSAP HANA high availability policy . . . . . 93SAP HANA high availability policy parameters 94

Verifying . . . . . . . . . . . . . . . 96Starting and stopping the SAP HANA highavailability solution. . . . . . . . . . . 96Testing SAP HANA System Replication failoverscenarios . . . . . . . . . . . . . . 98

Chapter 3. NFS high availability policy 101Planning the NFS server automation . . . . . 102Installing and setting up the NFS server forautomation . . . . . . . . . . . . . . 103

Configuring the NFS server components . . . 103NFS server processes . . . . . . . . . . 104Using a shared disk for configuration data . . 105Using a shared disk for exported data . . . . 107Configuring file systems within the samevolume group on AIX . . . . . . . . . 108NFS server virtual IP address . . . . . . . 108

Configuring and activating the NFS server highavailability policy . . . . . . . . . . . . 109

NFS server high availability policy . . . . . 109NFS server policy parameters . . . . . . . 114

Using the wizard to configure and activate the NFSserver high availability policy . . . . . . . . 123Verifying the NFS server high availability policy 124

Chapter 4. Oracle high availabilitypolicy. . . . . . . . . . . . . . . 127Planning the Oracle database automation and highavailability . . . . . . . . . . . . . . 128Installing and setting up the Oracle database . . . 129

Installing the Oracle database software . . . . 129Configuring the individual Oracle databasecomponents . . . . . . . . . . . . . 131

Configuring and activating the Oracle databasehigh availability policy . . . . . . . . . . 132

Oracle database high availability policy . . . 132

© Copyright IBM Corp. 2014, 2016 iii

Oracle policy high availability and automationpreparation . . . . . . . . . . . . . 135Oracle database high availability policyparameters . . . . . . . . . . . . . 135

Using the wizard to configure and activate theOracle database high availability policy . . . . 138Verifying the Oracle database high availabilitypolicy . . . . . . . . . . . . . . . . 139

Chapter 5. Using the wizard toconfigure and activate the policy . . . 141Starting the Policy Setup Wizard . . . . . . . 141Migrating between an old and a new template . . 142Overview dialog . . . . . . . . . . . . 143

Dialog elements . . . . . . . . . . . 144Skipped parameters . . . . . . . . . . 145

Parameter dialog . . . . . . . . . . . . 145Dialog elements . . . . . . . . . . . 146

Entering values. . . . . . . . . . . . . 147Single value . . . . . . . . . . . . . 148Multiple values. . . . . . . . . . . . 148Value selection . . . . . . . . . . . . 149Decision . . . . . . . . . . . . . . 150

Option parameters . . . . . . . . . . . 151Wizard dialog flow . . . . . . . . . . . 153Parameter help information . . . . . . . . 154

Enable and disable help information for allparameter dialogs . . . . . . . . . . . 154Display help information in selected parameterdialogs . . . . . . . . . . . . . . 155

Closing the wizard . . . . . . . . . . . 155Activating the policy . . . . . . . . . . . 156

Chapter 6. Troubleshooting the SAPhigh availability solution . . . . . . 157No valid license available when activating SAPCentral Services high availability policy . . . . 157

Chapter 7. Troubleshooting the Oraclehigh availability policy . . . . . . . 159Oracle database listener does not start up . . . . 159

Problem . . . . . . . . . . . . . . 159Explanation . . . . . . . . . . . . . 159Solution . . . . . . . . . . . . . . 159

Oracle database process is monitored unknown 159Explanation . . . . . . . . . . . . . 159Solution . . . . . . . . . . . . . . 160

Chapter 8. Troubleshooting the NFStiebreaker . . . . . . . . . . . . . 161NFS tiebreaker on Linux. . . . . . . . . . 161NFS tiebreaker on AIX . . . . . . . . . . 161

Using IBM Support Assistant . . . . 163Installing IBM Support Assistant and the TivoliSystem Automation for Multiplatforms plug-in . . 163

Notices . . . . . . . . . . . . . . 165Trademarks . . . . . . . . . . . . . . 166

Index . . . . . . . . . . . . . . . 167

iv Tivoli System Automation for Multiplatforms V4.1: High Availability Policies Guides

Figures

1. Symbols used in this guide. . . . . . . . x2. Components of a distributed SAP system 23. Example of a two-node setup . . . . . . . 44. ABAP (ASCS) high availability setup . . . . 55. Java (SCS) high availability setup . . . . . 76. Double Stack high availability setup . . . . 87. Initial startup of (A)SCS . . . . . . . . 158. Failure of (A)SCS and recovery of the enqueue

table . . . . . . . . . . . . . . . 169. Movement of the Enqueue Replication Server 16

10. ASCS high availability policy . . . . . . 3111. SCS high availability policy for Java . . . . 3412. Resources of the Double Stack (ABAP and Java

SCS) high availability policy . . . . . . . 3813. Relationships between the Enqueue and

Message Servers and the Enqueue ReplicationServer . . . . . . . . . . . . . . 39

14. Example of a two-node setup . . . . . . 8615. System Replication overview with single nodes 8716. Resources of the SAP HANA high availability

policy . . . . . . . . . . . . . . 9017. NFS server parts . . . . . . . . . . 101

18. Components to be configured whenautomating an NFS server . . . . . . . 104

19. NFS server high availability policy . . . . 11020. NFS server high availability policy sample

scenario . . . . . . . . . . . . . 11321. High availability architecture for an Oracle

database . . . . . . . . . . . . . 12722. Oracle database high availability policy 13323. Overview dialog . . . . . . . . . . 14324. Parameter dialog #3 without value definition 14525. Parameter dialog #3 with value definition 14626. Single value prompt . . . . . . . . . 14827. Multiple values prompt . . . . . . . . 14928. Value selection mode . . . . . . . . . 15029. Decision prompt . . . . . . . . . . 15130. Option parameter in overview dialog 15231. Option parameter dialog . . . . . . . . 15232. Nested parameter dialog . . . . . . . . 15333. Wizard dialog flow . . . . . . . . . 15434. Help information in parameter dialogs 15535. Policy activation in the wizard . . . . . . 156

© Copyright IBM Corp. 2014, 2016 v

vi Tivoli System Automation for Multiplatforms V4.1: High Availability Policies Guides

Tables

1. Highlighting conventions used in this book x2. ABAP resources and the corresponding

components . . . . . . . . . . . . . 63. Java resources and the corresponding

components . . . . . . . . . . . . . 84. Double-Stack resources and the corresponding

components . . . . . . . . . . . . . 95. Fully implemented high availability solution

for SAP . . . . . . . . . . . . . . 136. Start_Program_<NR> and

Restart_Program_<NR> values of SAP profiles . 217. Resources and resource groups of the ASCS

high availability policy . . . . . . . . . 318. Description and examples of placeholders for

ABAP resource names . . . . . . . . . 339. Resources and resource groups of the Java SCS

high availability policy . . . . . . . . . 3410. Description and examples of placeholders for

JAVA resource names . . . . . . . . . 3611. Location for the SAP high availability XML

template files . . . . . . . . . . . . 4212. ABAP policy parameters . . . . . . . . 4313. Java policy parameters . . . . . . . . . 5314. Double-Stack resources and the corresponding

components . . . . . . . . . . . . 6315. Planned Outages . . . . . . . . . . . 80

16. Unplanned outages . . . . . . . . . . 8117. SAP high availability solution checklist 8218. Documentation reference . . . . . . . . 8819. SAP HANA System Replication parts to

automate . . . . . . . . . . . . . 9020. Location for the SAP HANA high availability

XML template file . . . . . . . . . . 9321. SAP HANA System Replication HA policy

parameters . . . . . . . . . . . . . 9422. Planned Outages . . . . . . . . . . . 9823. Unplanned outages . . . . . . . . . . 9924. NFS server parts which are automated by the

NFS server high availability policy . . . . 11025. NFS server high availability policy

relationships . . . . . . . . . . . . 11126. NFS server high availability policy

parameters (AIX) . . . . . . . . . . 11427. NFS server high availability policy

parameters (Linux) . . . . . . . . . . 11828. Oracle database high availability resources 13329. Oracle database high availability policy

relationships . . . . . . . . . . . . 13430. Oracle database high availability policy

parameters . . . . . . . . . . . . 13531. Help information text of the Setup Policy

Wizard. . . . . . . . . . . . . . 154

© Copyright IBM Corp. 2014, 2016 vii

viii Tivoli System Automation for Multiplatforms V4.1: High Availability Policies Guides

About this guide

This guide explains how to implement and use the policy-based automatedrecovery capabilities that are provided by IBM Tivoli System Automation forMultiplatforms (System Automation for Multiplatforms).

System Automation for Multiplatforms provides high-availability for resources onAIX® clusters (on IBM® System p), Linux clusters (on IBM System x, System z®,System i®, and System p), and Windows clusters (on IBM System x).

Who should use this guideThis guide is intended for system administrators and operators who want to usethe automation and failover capabilities of System Automation for Multiplatforms.

Where to find more informationThe Tivoli System Automation library comprises the following books, includingthis publication, describing Tivoli System Automation for Multiplatforms:v System Automation for Multiplatforms Administrator's and User's Guide,

SC34-2698-01v Tivoli System Automation for Multiplatforms Installation and Configuration Guide,

SC34-2699-01v Tivoli System Automation for Multiplatforms Reference Guide, SC34-2700-01v Tivoli System Automation for Multiplatforms High Availability Policies Guide,

SC34-2660-01

You can download the complete documentation athttp://www.ibm.com/support/knowledgecenter/SSRM2X/welcome

The Tivoli System Automation library contains the following books, including thisone, describing System Automation Application Manager:v System Automation Application Manager Administrator's and User's Guide,

SC34-2701-00v System Automation Application Manager Installation and Configuration Guide,

SC34-2702-00v System Automation Application Manager Reference and Problem Determination Guide,

SC34-2703-00

You can download the books at:http://www.ibm.com/support/knowledgecenter/SSPQ7D/welcome

The IBM Tivoli System Automation home page contains useful up-to-dateinformation, including support links and downloads for maintenance packages.You will find the IBM Tivoli System Automation home page at:www.ibm.com/software/tivoli/products/sys-auto-multi/

© Copyright IBM Corp. 2014, 2016 ix

http://www.ibm.com/support/knowledgecenter/SSRM2X/welcome

http://www.ibm.com/support/knowledgecenter/SSPQ7D/welcome

http://www.ibm.com/software/tivoli/products/sys-auto-multi/

ConventionsThe following highlighting conventions are used in this book:

Table 1. Highlighting conventions used in this book

Bold Identifies commands, subroutines, keywords, files, structures, directories,and other items whose names are predefined by the system. Also identifiesgraphical objects such as buttons, labels, and icons that the user selects.

Italic Identifies parameters whose actual names or values are to be supplied bythe user.

monospace Identifies examples of specific data values, examples of text similar towhat you might see displayed, examples of portions of program codesimilar to what you might write as a programmer, messages from thesystem, or information you should actually type.

This manual uses symbols to show resources, resource groups, equivalencies, andrelationships. The symbols used are as follows:

ISO 9000ISO 9000 registered quality systems were used in the development andmanufacturing of this product.

Related RSCT informationThe following IBM Reliable Scalable Cluster Technology (RSCT) publications areavailable on the System Automation for Multiplatforms CD:v RSCT Administration Guide

v RSCT for AIX 5L™: Technical Reference

v RSCT for Multiplatforms: Technical Reference

v RSCT Messages

v RSCT Diagnosis Guide

For more information about RSCT, see IBM Cluster systems.

For more information, see the Linux on IBM zSeries and S/390®: High Availabilityfor z/VM® and Linux IBM Redpaper.

Figure 1. Symbols used in this guide

x Tivoli System Automation for Multiplatforms V4.1: High Availability Policies Guides

http://www.ibm.com/systems/clusters/

http://publib-b.boulder.ibm.com/Redbooks.nsf/RedpaperAbstracts/redp0220.html

http://publib-b.boulder.ibm.com/Redbooks.nsf/RedpaperAbstracts/redp0220.html

How to obtain publicationsThe System Automation for Multiplatforms publications are also available (valid atthe time of release) at these Web sites:www.ibm.com/servers/eserver/clusters/library/www.ibm.com/servers/eserver/zseries/software/sa/www.ibm.com/software/sysmgmt/products/support/

How to reach us by e-mailIf you would like to contact us by e-mail, send your comments [email protected]

About this guide xi

xii Tivoli System Automation for Multiplatforms V4.1: High Availability Policies Guides

What's new in this release

Get a quick overview about the new features of System Automation forMultiplatforms version 4.1.0.

Improved operations on the command line with new samcc command

System Automation for Multiplatforms version 4.1.0.2 adds a newcommand samcc, which can be used as operations console on the commandline interface. For more information, see .

Additional platforms support

System Automation for Multiplatforms version 4.1.0.1 supports thefollowing new platforms:v SUSE SLES 12 (64-bit)v Red Hat RHEL 7 (64-bit)v Ubuntu 14.04 (64-bit): System x, Power Systems (Little Endian only)

System Automation for Multiplatforms version 4.1.0.2 supports thefollowing new platforms:v Red Hat RHEL 7.1 on Power Systems Little Endian (64-bit)

System Automation for Multiplatforms version 4.1.0.3 supports thefollowing new platforms:v AIX 7.2

System Automation for Multiplatforms version 4.1.0.4 supports thefollowing new platforms:v Ubuntu 16.04 (64-bit): System x, Power Systems (Little Endian only).

For more information, see System Automation for Multiplatforms Installationand Configuration Guide.

Improved high availability policy for SAP

The SAP Central Services high availability policy is available as SystemAutomation for Multiplatforms optional feature, which is priced separately.This SAP Central Services high availability policy is now adapted to SAPNetweaver technology.

The user can start and stop the SAP Netweaver stack by using the SAPuser interface without interfering with the System Automation policy. TheSAP Software Update Manager is able to update the Netweaver solutionwithout the need to disable System Automation during the update process.

Supported SAP configuration options: Java, ABAP, and DUAL stacksupport for SAP Central Services failover. Additionally, the followingconfiguration options are supported:v Application server (restart in place of primary and additional application

server)v SAP router failoverv SAP Web Dispatcher failoverv Start after dependency support to database

System Automation for Multiplatforms version 4.1.0.2 adds support for:v SAP HANA System Replication failover

The supported SAP kernel version is 7.20 or higher.

© Copyright IBM Corp. 2014, 2016 xiii

For more information, see System Automation for Multiplatforms HighAvailability Policies Guide.

Gathering information about application failures

The samwhy program is a simple and easy-to-use tool that offers thedetection of application failures and their analysis for applications that arecontrolled by System Automation. samwhy helps the operator to understandwhat happened and provides an explanation why System Automationreacted the way it did.

For more information, see System Automation for Multiplatforms ReferenceGuide.

High availability of the end-to-end automation adapter is simplified

An extra automation policy or virtual IP address is not required any more.

For more information, see System Automation for MultiplatformsInstallation and Configuration Guide.

Run the end-to-end automation adapter with a non-root user

By default, the end-to-end automation adapter runs with a root user. Nowthe adapter can also be set up to run with a non-root user.

For more information, see System Automation for MultiplatformsInstallation and Configuration Guide.

xiv Tivoli System Automation for Multiplatforms V4.1: High Availability Policies Guides

Chapter 1. SAP Central Services high availability policy

You can define SAP Central Services high availability policies for your SAPinstallation. The topics in this section describe the process.

PlanningThis topic explains the high availability concepts for a SAP system. It providesinformation about which SAP Central Services high availability policy to choosedepending on the planned SAP installation. The main reason for this highavailability setup is to reduce the downtime of an SAP system in case of softwareor hardware failures.

The high availability solution for SAP uses System Automation for Multiplatformsto automate all SAP components. System Automation for Multiplatforms detectsfailed components and restarts or initiates a failover. This setup will also help toreduce the operational complexity of an SAP environment and to avoid operatorerrors resulting from this complexity.

Single Point of Failure (SPOF)In a distributed or standard SAP installation the SAP Central Services, the databaseserver, and the NFS server are single points of failures (SPOFs).

To minimize the impact of SPOF services outages, it is necessary to setupredundancy. Run one or more standby servers to which each of the SPOF servicescan be failed over and restarted independently. Each SPOF service must beassociated with its own virtual host name, which is started where the service runs.Clients reconnect to the same host name independently of where the SPOF serviceruns.

The following SAP components are available for a distributed SAP system.

© Copyright IBM Corp. 2014, 2016 1

(A)SCS nodeThe (A)SCS node consists of the stand-alone components Enqueue Server(ES) and Message Server (MS) that operate as SAP Central Servicesinstance. Depending on the SAP solution the (A)SCS node contains theABAP, Java, or both components.

An SAP Instance Agent is running for each instance.

SAP uses the following terms and abbreviations:

ASCS SAP Central Services for ABAP instances.

SCS SAP Central Services for Java™ instances.

(A)SCSSAP Central Services for ABAP or Java instances.

Primary AS nodeThe Primary AS node consists of the Primary Application Server (PAS)instance that is running the SAP Services Dialog, Update, Batch, Gateway,and Spool. An Instance Agent accompanies the Primary Application Server.

Note: In a standard non-distributed SAP system all the main instances(SCS, PAS, and database) run on the same node.

Additional Application Server nodeAdditional AS nodes are optional. They host the Additional ApplicationServer (AAS) instances, which were called Dialog Instance (DI) in releasesprior then SAP kernel 7.1. You can have one or more AdditionalApplication Servers. Again an Instance Agent accompanies each AdditionalApplication Server.

/saprouttab

/usr/sap/trans

Primary AS node

/sapmnt/SID

(A)SCS

(A)SCS node

Primary applicationserver instance

Additional AS node

Additional applicationserver instance

Web Dispatcher node*

SAP Web Dispatcher

Database node

Database instance

NFS node

Client

Data connections

NFS export

SAProuter node*

SAProuter

Process

Node

* Optional

Host agents

Host agent

Figure 2. Components of a distributed SAP system

2 Tivoli System Automation for Multiplatforms V4.1: High Availability Policies Guides

Web Dispatcher and SAProuter nodeThe optional Web Dispatcher and SAProuter nodes run the SAP WebDispatcher (WD) and SAProuter, which are used as proxies to access theother SAP instances.

An Instance Agent is running for the Web Dispatcher.

Host AgentsOne SAP Host Agent runs on each node that hosts SAP-providedcomponents.

Database nodeThe database node holds the database instance. The database product canbe IBM DB2 or another SAP supported database.

NFS nodeThe NFS node runs the NFS server. It can also be a NAS device, whichexports the NFS file systems.

SAP clientsThe SAP clients connect directly to the Application Servers or to anoptional Web Dispatcher.

For more information about the SAP components, see “High availability impact”on page 12.

Example of a two-node setupThe minimum hardware setup consists of a two-node System Automation forMultiplatforms domain. The two nodes are either two physical machines or twoLPARs running on different physical machines. The systems must be connected vianetwork and need to access the database and SAP data. Data can be provided by aSAN attached disk subsystem, which is connected to each node using fiber channel(FC).

Figure 3 on page 4 shows an example of a two-node System Automation forMultiplatforms domain. It shows all the main SAP instances and the correspondingfailover groups of a SAP ABAP system.

Chapter 1. SAP Central Services high availability policy 3

Each machine or LPAR must be capable to run all instances. These are the mainSAP instances, which must be made highly available by System Automation forMultiplatforms. High availability for application servers is achieved by having atleast two application server instances (PAS and AAS) as fixed resources.

If your setup contains the database server and SAProuter or SAP Web Dispatcher,make them highly available as well.

SAP Central Services high availability setup options

System Automation for Multiplatforms supports three different SAP CentralServices high availability installation setups:v ABAP (ASCS) high availability installationv Java (SCS) high availability installationv Double-stack (ASCS and SCS) high availability installation

Select the high availability installation that matches your SAP installation.

Note: To transform your existing SAP into a highly available SAP, keep yourexisting SAP up and running until the newly installed SAP Central Services highavailability setup is tested.

Storage A Storage B

ASCS

ERS

ASCS

ERS

Host A Host B

Web Dispatcher Web Dispatcher

SAProuter SAProuter

DB

Host agent

DB

Host agent

Instance agent Instance agent

Instance agent Instance agentInstance agent


Figure 3. Example of a two-node setup


ABAP (ASCS) high availability setupThis ABAP (ASCS) high availability setup is used for ABAP only SAP solutions.

ABAP (ASCS) high availability setupThe ASCS high availability setup consists of at least two SCS nodes whichrun the ASCS and ERS instances. Under regular conditions the ERS willalways be started on the node where the ASCS is not running. Thisfailover setup has no downtime due to fast failure detection andin-memory data exchange between ES and ERS in case the ASCS must bemoved to the failover node. During the failover, the virtual IP address forthe ASCS is moved to the failover node too, so its addressing remainsunchanged.

Primary and Additional AS nodeThe Primary AS node and the Additional AS nodes consist of ABAPApplication Servers which will be restarted in place in case of softwarefailures. Protection against hardware outages is done by setting upmultiple Application Servers on different hardware. Therefore the SystemAutomation for Multiplatforms concept for SAP high availability does notconsider failovers of Application Servers to other nodes becauseApplication Server restarts take a lot of time. Other Application Serversmust be sized to take the additional workload from failing servers.

ASCS

ERS

ASCS node

Instance agent

Instance agent

Instance agent

Instance agent



Process

Data connections

NFS export

Node

Primary AS node

/sapmnt/SID

ASCS

ERS

ASCS node

Primary application server

Additional AS node

Add. application server

Database node

Database instance

Web Dispatcher node

SAP Web Dispatcher

Web Dispatcher node

SAP Web Dispatcher

SAP Web Dispatcher HA setup*

Database node

Database instance

Database HA setup

NFS node

NFS node

NFS HA setup

/usr/sap/trans

Client

ASCS HA setup

/sapmnt/SID

/usr/sap/trans

Failover * optional

SAProuter node

SAProuter

SAProuter node

SAProuter

SAProuter HA setup*

Host agentHost A

Host agents

Figure 4. ABAP (ASCS) high availability setup


Web Dispatcher and SAProuter node (optional)The Web Dispatcher and SAP router high availability setups consists of atleast two nodes which run the instances in a failover setup. Thecomponents are key for the clients to access the Application Servers, so incase of a failure the instance failover to standby nodes includes theirvirtual IP addresses too.

Database high availability setupThe database high availability setup is explained in “Database highavailability installation setup” on page 10.

NFS high availability setupThe NFS high availability setup is explained in “NFS high-availabilityinstallation setup” on page 11.

Setup a ASCS SAP high availability solution with the following components.

Table 2. ABAP resources and the corresponding components

ABAP Resources SAP component

ABAP resources v ABAP SAP Central Services (ASCS) instances using anown virtual host name on two nodes.

v Enqueue Replication Server using an own virtual hostname on two nodes.

v Database Server instances using an own virtual hostname on two nodes.

v Primary Application Server for ABAP instance on firstnode.

v Additional Application Server for ABAP instances onother nodes.

ABAP independent resources(optional)

v Host Agent

v Web Dispatcher instances using an own virtual host nameon two nodes.

v SAProuter setup on two nodes.

Java (SCS) high availability setupThis Java (SCS) high availability setup is used for Java only SAP solutions.


Java (SCS) high availability setupThe Java (SCS) high availability setup consists of at least two SCS nodes,which run the SCS and ERS instances. Under regular conditions, the ERS isalways started on the node where the SCS is not running. This failoversetup has no downtime due to fast failure detection and in-memory dataexchange between ES and ERS in case the SCS must be moved to thefailover node. During the failover, the virtual IP address for the SCS ismoved to the failover node too, so its addressing remains unchanged.

Primary and Additional AS nodeThe Primary AS node and the Additional AS nodes consist of JavaApplication Servers, which are restarted in place in case of softwarefailures. Protection against hardware failures is done by setting up multipleApplication Servers on different hardware. Therefore, the SystemAutomation for Multiplatforms concept for SAP high availability does notconsider failovers of Application Servers to other nodes becauseApplication Server restarts take a lot of time. Other Application Serversmust be sized to take the additional workload from failing servers.

Web Dispatcher and SAProuter node (optional)The Web Dispatcher and SAP router high availability setups consists of atleast two nodes, which run the instances in a failover setup. Thecomponents are key for the clients to access the Application Servers, so incase of a failure the instance failover to standby nodes includes theirvirtual IP addresses too.

SCS

ERS

SCS node

Process

Data connections

NFS export

Node

Primary AS node

/sapmnt/SID

SCS

ERS

SCS node


Additional AS node


Database node

Database instance

Web Dispatcher node

SAP Web Dispatcher

Web Dispatcher node

SAP Web Dispatcher


Database node

Database instance

Database HA setup

NFS node

NFS node

NFS HA setup

/usr/sap/trans

Client

SCS HA setup

/sapmnt/SID

/usr/sap/trans

Failover * optional

SAProuter node

SAProuter

SAProuter node

SAProuter

SAProuter HA setup*

Instance agent

Instance agent

Instance agent

Instance agent



Host agentHost A

Host agents

Figure 5. Java (SCS) high availability setup




Set up a Java SCS SAP high availability solution with the following components.

Table 3. Java resources and the corresponding components

Java Resources SAP component

Java resources v Java SAP Central Services (SCS) instances, which use anown virtual host name on two nodes.

v Enqueue Replication Server, which uses an own virtualhost name on two nodes.

v Database server instances, which uses an own virtual hostname on two nodes.

v Primary Application Server for Java instance on first node.

v Additional application server for Java instances on othernodes.

Java independent resources(optional)

v Host Agent

v Web Dispatcher instances, which use an own virtual hostname on two nodes.


Double-Stack (ASCS and SCS) high availability setupThis Double-Stack high availability setup is used for Double-Stack SAP solutions.

SCS

ERS

SCS node

Instance agent

Instance agent

Instance agent

Instance agent



Process

Data connections

NFS export

Node

Primary AS node

/sapmnt/SID

ASCS

ERS

ASCS node


Additional AS node


Database node

Database instance

Web Dispatcher node

SAP Web Dispatcher

Web Dispatcher node

SAP Web Dispatcher


Database node

Database instance

Database HA setup

NFS node

NFS node

NFS HA setup

/usr/sap/trans

Client

(A)SCS HA setup

/sapmnt/SID

/usr/sap/trans

Failover * optional

SAProuter node

SAProuter

SAProuter node

SAProuter

SAProuter HA setup*

Host agentHost A

Host agents

SCS

ERS

SCS node

Instance agent

Instance agent

Instance agent

Instance agent

ASCS

ERS

ASCS node

Figure 6. Double Stack high availability setup


(A)SCS high availability setupThe Double-Stack installation contains all the components of both theABAP and Java installation. A Double-Stack system has two SAP CentralServices instances in parallel: an ABAP SCS and a Java SCS. Each of themworks the same way as the ABAP only and Java only setups. During thefailover, the corresponding two virtual IP addresses for the ASCS and SCSare moved to the failover node as well.

Primary and Additional AS nodeA Double-Stack Application Server comprises one SAP instance that runsan ABAP and a Java Application Server in parallel. Within a SystemAutomation for Multiplatforms domain, the Double Stack ApplicationServer instance is automated as two Application Server resources. TheStartAfter relationship is defined between the Java Application Server andthe ABAP Application Server. This StartAfter relationship causes thatstarting the Java Application Server automatically triggers the prior start ofthe ABAP Application Server.

Web Dispatcher and SAProuter node (optional)The Web Dispatcher and SAP router high availability setups consist of atleast two nodes, which run the Web Dispatcher and SAP Router instancesin a failover setup. Both components are essential for the clients to accessthe Application Servers. Therefore, the failover of one instance to standbynodes includes the corresponding virtual IP address as well.



Table 4. Double-Stack resources and the corresponding components

Double Stack resources SAP component

ABAP resource ABAP SAP Central Services (ASCS)instances use an own virtual host name ontwo nodes.

Java resources Java SAP Central Services (SCS) instancesuse an own virtual host name on two nodes.

ABAP and Java common resourcesv Both Enqueue Replication Servers have

their own virtual host name on twonodes.

v Database server instances use an ownvirtual host name on two nodes.

v Primary Application Server for ABAP andJava instance on first node.

v Additional Application Server for ABAPand Java instances on other nodes.

Independent resources (optional)v Host Agent

v Web Dispatcher instances use an ownvirtual host name on two nodes.



Location of SAP instance directoriesThe SAP instance directories /usr/sap/<sapsid>/<instancename> must be locatedon a local file system.

NFS or other distributed file systems are not allowed or accepted.

Database high availability installation setupYou can choose between a database installation within the same SystemAutomation for Multiplatforms cluster together with the SAP installation or adatabase installation on a separate cluster of its own.

SAP Central Services and database in the same high availabilityclusterIf you choose to install the database on the same System Automation forMultiplatforms cluster as SAP, you can run them on the same or on differentsystems.

(A)SCS and database running on the same nodes (2 or 3 nodes):Use this setup only if your workload is small enough, so that the databaseand the SAP Enqueue Server can run on one system and the SAP PrimaryApplication Server on the other system.

(A)SCS and database running on different nodes in same cluster (4 or morenodes):

Use this setup to separate the workload of the SAP installation and thedatabase server to different systems. Additional nodes can be joined intothis cluster to host Additional Application Servers within this highavailability setup.

When using the policy wizard to define your SAP high availability policy, it isrecommended to select a "StartAfter" relationship between the SAP ApplicationServers and the database server. This relationship will let the database server startbefore the SAP Application Servers are started. This helps to avoid the problem ofhaving an Application Server started without a database running, which wouldrequire a restart of the Application Server.

SAP Central Services and database on different high availabilityclustersYou can have the SAP installation and the database in different System Automationfor Multiplatforms high availability clusters.

Advantages:1. The setup and maintenance of the database high availability cluster is

independent from the SAP high availability cluster.2. Separate non-root user authorizations for cluster commands against DB2 and

SAP resources can be better applied when using separate System Automationfor Multiplatforms clusters.

Note: In case you have a DB2 database you can use the SAP installer to installDB2 and System Automation for Multiplatforms together on the database nodes.For more information, refer to “High availability for DB2 and other databases” onpage 11.

As a drawback, although there are functional dependencies between SAP and itsdatabase, they cannot be mapped to high availability relationships across twoclusters with System Automation for Multiplatforms. Such cross-cluster


relationships can be implemented with the System Automation ApplicationManager product.

High availability for DB2 and other databasesYou can either use DB2® for your high availability setup or use other databases.

DB2 high availability setup

In a high availability environment a DB2 high availability setup isrequired. You can use the System Automation for Multiplatforms softwarewhich is bundled with DB2.

Note: If the DB2 database runs in the same cluster together with the SAPsoftware, the DB2-bundled System Automation for Multiplatforms licensecannot be used, because it allows the automation of DB2 only. So you mustensure that you have a full System Automation for Multiplatforms licensefor each cluster node.

Use the DB2 high availability setup wizard which is shipped with the SAPproduct.

You can find the DB2 high availability setup installation information in thefollowing chapter of the SAP Installation & Upgrade Guide. An SAP userID and password are required to access the SAP documentation.v SAP NetWeaver

– SAP NetWeaver 7.0 (2004s) Installation (2 -Installation -SAPNetWeaver Systems)

– SAP NetWeaver PI 7.1 Installation (2 -Installation -SAP NetWeaverSystems)System Automation for Multiplatforms-DB2 for LUW

– SAP NetWeaver 7.3 Installation (2 -Installation -SAP NetWeaverSystems)System Automation for Multiplatforms-DB2 for LUW

– SAP NetWeaver 7.4 Installation (2 -Installation -SAP NetWeaverSystems)System Automation for Multiplatforms-DB2 for LUW

Further links:

v How to setup and run a DB2 LUW database server or client see SAPNote No. 960843: DB6: Installation SA MP

v For more information about high availability with SAP on DB2, refer toIBM DB2 for LUW Cluster Using IBM Tivoli® SA MP SAP CommunityNetwork link: High Availability with SAP on DB2: IBM DB2 for LUWCluster Using IBM Tivoli SAMP

High availability with other databases

System Automation for Multiplatforms can be used to automate the start,stop, monitor, restart and failover of database servers other than DB2.v Chapter 2, “SAP HANA System Replication high availability policy,” on

page 85v Chapter 4, “Oracle high availability policy,” on page 127

NFS high-availability installation setup

Two central NFS shares are essential for an SAP system:


http://service.sap.com/instguides

http://scn.sap.com/docs/DOC-14508


v /sapmnt/<SID>: Required to share binary files and configuration data for theapplication servers.

v /usr/sap/trans: Must be shared in a logical transport landscape so that thesource SAP system can write the transport files to the share and the targetsystems can pick them up from this location.

You have the following three options for an NFS high availability setup:1. Use System Automation for Multiplatforms for the NFS high-availability

cluster.2. Use an existing NFS high-availability cluster.3. Use a Network Attached Storage (NAS) device with integrated high availability

capabilities.

It is preferred to install the NFS server HA setup outside of the SAP HA cluster toreduce the complexity and prevent setup problems. Starting with version 4.1.0.1you can run the NFS server on the same cluster nodes as the SAP HA solution ifthe following conditions apply:v If the NFS server is installed with an HA setup, it is required to use the NFS HA

policy which is bundled with System Automation for Multiplatforms.v Configure the SAPCPE tool to copy all binary files required for local SAP instance

monitoring (sapcontrol / sapstartsrv) to the local SAP instance directories. Formore information, see SAP documentation.

v The SAP setup and NFS setup must pass a manual failover test without SystemAutomation for Multiplatforms policy being activated. This manual failover testincludes:– Fail over all SAP Central Services resources without NFS server failover. For

more information, see “Verifying the initial installation” on page 21.– Fail over all NFS server resources without SAP Central Services failover:

shared disk, IP address, and all NFS server processes.– Use the SAP GUI to check whether the Central Services are working correct

before and after the manual failover.

The start of SAP by System Automation for Multiplatforms is not successful if theNFS server is not up and running. In case the NFS server runs in a separatehigh-availability cluster, it is not possible to define a StartAfter relationship acrosstwo clusters with System Automation for Multiplatforms. Such a cross-clusterrelationship can be implemented with the System Automation ApplicationManager.

Make sure to configure all nodes in the SAP high availability solution cluster asNFS clients. Use the automounter to configure the NFS clients. The SAP CentralServices high availability policy does not keep the NFS mount points highavailable, therefore it is required to use the automounter for keeping the NFSmount points high available.

An NFS high availability policy is part of the SAP high availability solution that isprovided by System Automation for Multiplatforms. For more information, seeChapter 3, “NFS high availability policy,” on page 101.

High availability impactThis topic discusses the impact of various failure scenarios of the SAP systemcomponents when using System Automation for Multiplatforms to automate


https://support.sap.com/documentation.html

recovery. Manual recovery actions are minimized which otherwise would causeSAP transactions to timeout and roll back.

Note: There is no single point of failure anymore if System Automation forMultiplatforms is used for automation. The impact of a failure has a local scope; itis limited to the transactions currently using the failing resource. The SAP systemremains available. Without System Automation the impact on the SAP systemwould be much worse from what is shown in the Impact column in the tablebelow. Without System Automation all recovery actions would have to be donemanually. Manual recovery actions usually take longer and are error prone underthe pressure of a system outage.

In Table 5 the following abbreviations are used:v SA: actions taken automatically and instantaneously by IBM Tivoli System

Automation for Multiplatformsv User: actions taken by the user

Table 5. Fully implemented high availability solution for SAP

Failing resource Impact Actions

Database v Rollback of transactions

v Remote application servers failover automatically to the otherdatabase node

SA: Restart or Failover Database(Optional setup in same highavailability cluster)

User: Restart transactions

Enqueue ServerNo impact.

Refer to “Enqueue Server andEnqueue Replication Server” onpage 14.

SA: Failover Enqueue Server

Remote application serversautomatically reconnect to theother database node.

Enqueue ReplicationServer No impact.

Refer to “Enqueue Server andEnqueue Replication Server” onpage 14.

SA: Restart Enqueue ReplicationServer

Message Server v No impact on mosttransactions

v Certain transactions inhibited(for example, SM66)

v Update/batch workloadbalancing inhibited

v Group logon inhibited

SA: Restart Message Server

SA: failover optional

SAP restart feature enabled.

Application Serverinstance

v Transactions on this instanceare lost

v Rollback of database updates

v User sessions on this instanceare lost

User: Connect to anotherApplication Server instance

User: Restart transactions

SA: Restart Application Serverinstance


Table 5. Fully implemented high availability solution for SAP (continued)

Failing resource Impact Actions

SAP-Gateway (ABAPonly)

v No impact on mosttransactions

v Connections to registeredservers inhibited until theyhave reconnected to theSAP-Gateway

SA: Restart SAP-Gateway

Web Dispatcher v User sessions (via HTTP) lost

v Re-connection inhibited

SA: Restart or Failover WebDispatcher

User: Reconnect

SAProuter v User sessions (via SAProuter)lost

v Re-connection inhibited

SA: Restart or Failover SAProuter

User: Reconnect

NFS server, NASdevice

v Refer to “NFS server” on page17.

v If data was written to file, lastwritten data is in doubt

SA: Restart or Failover NFSserver

Enqueue Server and Enqueue Replication ServerThe availability of the Enqueue Server is key for a SAP system. If the EnqueueServer cannot be reached, the SAP system is not operational since mosttransactions fail to run.

The Enqueue Server is a standalone component and does not require access to thedatabase. An application server instance connects directly to the Enqueue Server byusing a virtual host name.

When connected, the Enqueue Server on system 1 transmits replication data to theEnqueue Replication Server on system 2 which stores the data in a shadowenqueue table residing in shared memory. In case the Enqueue Server fails, theshadow enqueue table on the Enqueue Replication Server is used to rebuild thetables and data structures for the recovered Enqueue Server that is started on thesame node. The Enqueue Replication Server stops after transferring the data to therecovered Enqueue Server.

If the Enqueue Replication Server is unavailable, the SAP system continues to beup and running, but no shadow enqueue table is maintained as a backup. TheEnqueue Replication Server runs on a different system than the Enqueue Server tokeep the enqueue table and its shadow backup apart.

The multithreaded architecture of the standalone Enqueue Server allows parallelprocessing and replication. The I/O processing for the TCP/IP communication isdistributed over several I/O threads which allows a very high throughput.

The following figure shows the principal TCP/IP communication paths betweenthe Application Server instances, the Enqueue Server, and Message Servers.


If system 1 fails, system 2 takes over the role of the first one as shown in Figure 8on page 16:1. The virtual IP address related to the Enqueue and Message Servers is taken

over to system 2.2. Enqueue and Message Servers are restarted on system 2.3. The enqueue table is rebuilt from the shadow table hosted by the Enqueue

Replication Server.4. The Enqueue Replication Server stops after the Enqueue Server has rebuild the

enqueue table.5. The Application Servers reconnect to the Enqueue Server and the Message

Server.

System Automation for Multiplatforms will handle this complete failover process.The failover is fully transparent to the application. Enqueue locks are preservedand transactions continue to run.

switch

ES shadowtable

SAP Application Server

rebuild

ES table

connect

connect

ES table

ESStandaloneEnqueue Server

MSMessage Server


System 1 System 2

MSMessage Server

ERSEnqueue Replication Server

virutalIPaddress

virutalIPaddress

replicateERSEnqueue Replication Server

VirtualIPaddress

VirtualIPaddress

Figure 7. Initial startup of (A)SCS


After a successful failover of the Enqueue Server, the Enqueue Replication Server isno longer needed on system 2 and therefore stops itself. If another system isavailable, the Enqueue Replication Server is started by System Automation forMultiplatforms on that system and a new shadow enqueue table is established.This is shown in Figure 9

Database hostIf the database server is not available, the entire SAP system becomes unavailable.The database host maintains the persistent storage for the entire SAP system. Oncethe database is available again, all uncommitted transactions are rolled back andthe SAP system continues to run.

switch


reconnect

reconnect

ES shadowtable

rebuild

ES tableES table


MSMessage Server


System 1 System 2

MSMessage Server


virtualIPaddress

virtualIPaddress


VirtualIPaddress

VirtualIPaddress

Figure 8. Failure of (A)SCS and recovery of the enqueue table

switch


ES shadowtable

ES table


MSMessage Server


System 1 System 2

MSMessage Server

virtualIPaddress

virtualIPaddress



VirtualIPaddress

VirtualIPaddress

Figure 9. Movement of the Enqueue Replication Server


System Automation for Multiplatforms can be used to automate the start, stop,monitor, restart, and failover of the database server. For more information, refer to“High availability for DB2 and other databases” on page 11.

NFS serverA SAP high availability setup requires shared access for directories like global,profile, and trans.

On UNIX and Linux systems you need NFS to share files. As a result, theavailability of the file systems together with the NFS server is critical.

Note: NFS file access is not transactional. There is no commit or rollback logic. Incase of system failure or communication loss there is no guarantee that the lastwritten data has been stored on disk.

System Automation for Multiplatforms can be used to automate the start, stop,monitor, restart and failover of the NFS server. For more information, refer to “NFShigh-availability installation setup” on page 11.

SAP Host AgentThe SAP Host Agent is a tool that you can use for monitoring and controlling SAPand non-SAP instances, operating systems, and databases. It is installedautomatically during the installation of new SAP instances with SAP kernel 7.20 orhigher.

The SAP Host Agent provides features for SAP instance discovery and inventory,instance control, database monitoring and management, and operating systemmonitoring using saposcol. It aids in system or instance provisioning by hostingthe infrastructure of SAP NetWeaver Landscape Virtualization Management (LVM).

SAP Web DispatcherThe Web Dispatcher is the entry point for all external HTTP requests and theinterface between all HTTP clients and the SAP system. The Web Dispatcher canwork as load balancer for incoming requests which are distributed among allavailable application servers.

When the SAP Web Dispatcher fails, clients cannot connect to the SAP subsystemsusing HTTP request.

SAProuterThe SAProuter controls the access between the external network and the SAPsubsystem.

In case of a failure of the SAProuter, no connections can be processed using thisproxy.

Installing and setting upBefore you set up your SAP environment for high availability, you must plan forthe installation option which best suits your system.

The following three installation options apply if you want to setup a SAP highavailability solution:


Installing a new high availability SAP systemInstall a new system using the sapinst installation optionHigh-Availability System. This option is described in detail in thefollowing topic:

“Installing a new ASCS, SCS, or Double Stack high availability SAPsystem.”

Exporting an existing non-high availability SAP systemExport an existing non-high availability SAP system and import it into anew high availability SAP system using the system copy feature ofsapinst. See the SAP documentation for assistance.

Post-installation tasks

v Setup your NFS high availability server. For more information, see “NFShigh-availability installation setup” on page 11. You can also performthis task before you start the SAP installation.

v Setup your database high availability server. For more information, see“Database high availability installation setup” on page 10.

Installing a new ASCS, SCS, or Double Stack high availabilitySAP system

This topic describes a new installation of an ASCS, SCS, or Double Stack highavailability SAP system.

SAP NetWeaver 7.0 or higher with a kernel version of at least 7.20 is required. Thisdescription is based on a two node cluster architecture with a primary and afailover node.

For more information about the ASCS, SCS, or Double Stack high availability SAPsystem setups, refer to “SAP Central Services high availability setup options” onpage 4.

PrerequisitesFollow the prerequisites before you start the installation of the ASCS, SCS, orDouble Stack high availability SAP system.

Before you begin, make sure that you follow the guidelines:1. Use unique instance numbers for every instance you install on a single host for

one SAPSID. The SAP installation does not work if you did not use uniqueinstance numbers.

2. The installation tasks for the primary and failover node contain ASCS and SCSinstallation tasks. Depending on your application stack (ABAP only, Java only)you might skip the ASCS or SCS installation tasks. For a Double Stack system,install ASCS and SCS.

3. The SAP product documentation installation guide documents to install theswitchover software after you installed the Central Services Instance. If you useSystem Automation for Multiplatforms, skip this step until you installed andverified the complete SAP system.

4. For more information, see SAP Note 1704753 - Installing Systems based onNetWeaver version 7.3 and higher: UNIX.

Prerequisite installation tasks:


Note: The SAP installation tool sapinst is also called Software ProvisioningManager in the SAP documentation.v Create a separate directory for every installation task to store installation logs

and traces. Switch into this directory before you start sapinst.v Ensure to set the correct ulimits and umask values for your environment as it is

documented in the SAP product installation guide.v Ensure to configure the automounter on all nodes to connect to the NFS server

and automatically mount the default SAP directory /sapmnt before you startsapinst. For more information about NFS, see “NFS high-availability installationsetup” on page 11.

v Register permanent entries for the virtual host names in the DNS server.v Ensure that the network interfaces that you want to use have the same name on

each system. For each virtual IP address defined in the high availability policy,an equivalency of network interfaces is created. Only network interfaces withthe same name on each node can be part of each equivalency.

v Temporarily define and activate all required virtual IP addresses on the physicalhost where you want to start the installation before you start sapinst. Be sure toremove the virtual IP addresses after the installation is completed. Incorrectbehavior of the SAP high availability solution occurs if you leave the virtual IPaddresses permanently defined.

Initial installation on primary nodeUse the sapinst command to execute the following tasks for the installation optionSAP Systems > High-Availability System. For some installation tasks its requiredto start sapinst with a virtual host name.1. Activate all virtual IP addresses corresponding to the virtual host names before

starting the installations.2. Central Services Instance for ABAP (ASCS) or Java (SCS):

sapinst SAPINST_USE_HOSTNAME=<virtual (A)SCS host name>

3. Enqueue Replication Server instances (ERS) for ASCS or SCSsapinst SAPINST_USE_HOSTNAME=<virtual ERS host name>

4. Database instancesapinst SAPINST_USE_HOSTNAME=<virtual DB host name>

5. Primary Application Server instancesapinst

6. Web Dispatcher instance (optional)sapinst SAPINST_USE_HOSTNAME=<virtual Web Dispatcher host name>

7. Remove all virtual IP addresses that have been activated before.

Initial installation on failover nodeTo install SAP on the failover node, perform the following steps:1. Activate all virtual IP addresses corresponding to the virtual host names before

starting the installation.2. Use the sapinst command to execute the following tasks with the installation

option System Copy – Target System - High-Availability System.v Central Services Instance for ABAP (ASCS) or Java (SCS):

sapinst SAPINST_USE_HOSTNAME=<virtual (A)SCS host name>

v Enqueue Replication Server instances (ERS) for ASCS or SCSsapinst SAPINST_USE_HOSTNAME=<virtual ERS host name>

v Database instance


sapinst SAPINST_USE_HOSTNAME=<virtual DB host name>

3. Use the sapinst command to execute the following installation tasks using theinstallation option SAP-System > High-Availability System:v Additional Application Server instance (old name: Dialog Instance)

sapinst

v Web Dispatcher instance (optional): Remove the existing Web DispatcherSAPSID directory on /sapmnt and install with the same SAPSID and instanceID on the second node again.sapinst SAPINST_USE_HOSTNAME=<virtual Web Dispatcher host name>

4. Remove all virtual IP addresses that have been activated before.

Configuring SAP profilesConfigure the SAP profiles to comply with the high availability solution providedby System Automation for Multiplatforms.v

Disable autostart of all SAP instances in all their profiles by commenting theline Autostart = 1.

vTo share the enqueue backup file within the Linux or AIX cluster, store the filein the NFS-mounted /sapmnt/<SID>/global directory. Iit can be accessed from allnodes in the cluster where the Enqueue Server can start. Add the followingparameter to the (A)SCS profile for sharing the enqueue backup files betweennodes:enque/backup_file = $(DIR_GLOBAL)/ENQBCK(A)SCS

vAll SAP ABAP services that run on the Primary Application Server installationmust also be manually configured in the instance profiles of the AdditionalApplication Server. SAP ABAP services:– Batch service– Update/Update two service– Spool service

Through this setup, all SAP ABAP services are running on each ApplicationServer and are no single points of failure (SPOF) any more.

vThe SAP high availability connector must be enabled in the default profile. Thisstep requires that System Automation for Multiplatforms is already installed onthe cluster nodes, so it probably must be done later. For more information abouthow to enable the SAP high availability connector for your platform, see“Enabling the SAP high availability Connector” on page 27.

v It is required to disable the SAP restart capability for the Enqueue Server andthe Enqueue Replication Server in the appropriate profiles. Otherwise, theautomatic restart of SAP by using the command startsapsrv mismatches withSystem Automation for Multiplatforms start function and causes problems withthe automation. Set the SAP profile parameter for EN and ERS toStart_Program_<NR> in the EN and ERS profiles in the /sapmnt/<SID>/profiledirectory.For all servers other than EN and ERS, set the SAP profile parametersRestart_Program_<NR>.If Start_Program is defined, then– Initial start is started by startsapsrv framework.


– Recovery start is started by System Automation for Multiplatforms. Resourcescan start either in place or resources can fail over.

If Restart_Program is defined, then– Initial start is started by startsapsrv framework– Recovery start is started by startsapsrv framework. Resources can start in

place.

Table 6. Start_Program_<NR> and Restart_Program_<NR> values of SAP profiles.

Components Start_Program_<NR> Restart_Program_<NR>

Enqueue Server X

Message Server X

Enqueue Replication Server X

Application Server X

ABAP Gateway (inDVEBMGS profile)

X

Java Gateway (in SCSprofile)

X

Web dispatcher X

vOptional: Faster message server logon group handling during restart.When the SAP messsage server is restarted, for example as part of amaintenance operation, it takes some time until the messages server recoveredthe information about SAP logon groups. Users and RFC connections usingthese logon groups can not log on during this time. By implementing SAP Note1787163 - Message Server: save logon groups feature you can shorten this delay.

Verifying the initial installationThe following steps verify the correct setup of the ERS replication.

The commands listed with each verification step assume an ASCS setup. If youverify a Java setup replace the following instance names:v ASCS with SCSv DVEBMGS with Jv D with J

The syntax of the ifconfig commands shown in the samples below applies to theAIX operating system.

For Linux, replace the AIX commands with the following Linux commands:

Add IP alias

v AIX: ifconfig <interface_name> <IP_alias> netmask <IP_netmask>alias up

v Linux: ifconfig <interface_name>:<unique_number> <IP_alias>netmask <IP_netmask> up

Delete IP alias

v AIX: ifconfig <interface_name> <IP_alias> delete

v Linux: ip addr del <IP_alias> dev <interface_name>

Prerequisites:


http://service.sap.com/sap/support/notes/1787163


v All instances are stopped.v Execute all steps as <sid>adm user.

For the verification the SAP system is started and a manual failover of the ASCSand ERS is executed twice. First a failover of the ASCS instance from the first nodeto the second node and then a failover of the ASCS instance from the second nodeback to the first node.1. Verification step: Initial start - ASCS on first node and ERS on second node

a. Start the ERS instance on the second node:ifconfig <interface_name> <ERS_IP_alias> netmask <IP_netmask> alias upstartsap r3 ERS<ID>

b. Start ASCS and the Primary Application Server instance on the first node:ifconfig <interface_name> <ASCS_IP_alias> netmask <IP_netmask> alias upstartsap r3 ASCS<ID>startsap r3 DVEBMGS<ID>

c. Start the Additional Application Server instance on the second node:startsap r3 D<ID>

d. Check replication status for each ERS instance on the second node usingensmon utility:ensmon pf=/usr/sap/<SID>/ERS<ID>/profile/<SID>_ERS<ID>_<node2>

Select task Get replication information. The output looks like this...Replication is enabled in server, replication server is connected.Replication is active...

e. Verify successful start of all Application Servers:v Logon to the Primary Application Server for ABAP using the SAP

graphical user interface (SAPGUI).v Logon to the Additional Application Server for ABAP using the SAP

graphical user interface (SAPGUI).v Logon to the Primary Application Server for Java using a web browser.

The default is:http://node1:5<ID>00/index.html

v Logon to the Additional Application Server for Java using a web browser.The default is:http://node2:5<ID>00/index.html

2. Verification step: Change replication direction - (A)SCS on second node andERS on first nodea. Stop Additional Application Server instance on the second node:

stopsap r3 ERS<ID>stopsap r3 D<ID>ifconfig <interface_name> <ERS_IP_alias> delete

b. Stop ASCS, ERS and Primary Application Servers instances on the firstnode:stopsap r3 DVEBMGS<ID>stopsap r3 ASCS<ID>ifconfig <interface_name> <ASCS_IP_alias> delete

c. Start ASCS IP instances on the second node:ifconfig <interface_name> <ASCS_IP_alias> netmask <IP_netmask> alias upstartsap r3 ASCS<ID>

d. Start ERS instances on the first node:ifconfig <interface_name> <ERS_IP_alias> netmask <IP_netmask> alias upstartsap r3 ERS<ID>


e. Check replication status for each ERS instance on first node using ensmonutility:ensmon pf=/usr/sap/<SID>/ERS<ID>/profile/<SID>_ERS<ID>_<node1>


f. Start the Primary Application Server instance on the first node:startsap r3 DVEBMGS<ID>

g. Start the Additional Application Server instance on the second node:startsap r3 D<ID>

h. Verify successful start of all Application Servers:v Logon to the Primary Application Server for ABAP using the SAP

graphical user interface (SAPGUI).v Logon to the Additional Application Server for ABAP using the SAP

graphical user interface (SAPGUI).v Logon to the Primary Application Server for Java using a web browser.

The default is:http://node1:5<ID>00/index.html


3. Verification step: Change replication direction: ASCS on first node and ERS onsecond node:a. Stop ERS and Primary Application Server instance on the first node:

stopsap r3 DVEBMGS<ID>stopsap r3 ERS<ID>ifconfig <interface_name> <ERS_IP_alias> delete

b. Stop ASCS and Additional Application Servers instances on the secondnode:stopsap r3 D<ID>stopsap r3 ASCS<ID> <virtual ASCS host name>ifconfig <interface_name> <ASCS_IP_alias> delete

c. Start ASCS instances on the first node:ifconfig <interface_name> <ASCS_IP_alias> netmask <IP_netmask> alias upstartsap r3 ASCS<ID>

d. Start ERS instances on the second node:ifconfig <interface_name> <ERS_IP_alias> netmask <IP_netmask> alias upstartsap r3 ERS<ID>

e. Check replication status for each ERS instance on the second node usingensmon utility:ensmon pf=/usr/sap/<SID>/ERS<ID>/profile/<SID>_ERS<ID>_<node2>


f. Start Primary Application Server instance on the first node:startsap r3 DVEBMGS<ID>

g. Start Additional Application Server instance on the second node:startsap r3 D<ID>

h. Verify successful start of all Application Servers:


v Logon to the Primary Application Server for ABAP using the SAPgraphical user interface (SAPGUI).

v Logon to the Additional Application Server for ABAP using the SAPgraphical user interface (SAPGUI).

v Logon to the Primary Application Server for Java using a web browser.The default is:http://node1:5<ID>00/index.html


After successful verification everything has to be stopped. All virtual IP addresseshave to be deactivated.

Installing and setting up System Automation forMultiplatforms

This topic describes the installation and setup of System Automation forMultiplatforms for an automated and highly available SAP system.

PrerequisitesFor a list of UNIX and Linux prerequisites, see Tivoli System Automation forMultiplatforms Installation and Configuration Guide.

Installing System Automation for Multiplatforms on all clusternodesSystem Automation for Multiplatforms must be installed on all nodes.

Proceed as follows for each node:1. Verify that your system has the necessary prerequisites for System Automation

for Multiplatforms. Enter the following command:./prereqSAM

This command writes a log to /tmp/prereqSAM.<number>.log. Check this log filebefore you proceed with the installation.

2. Install the System Automation for Multiplatforms software. Enter:./installSAM

The command writes a log to /tmp/installSAM.<number>.log. Check this logfile to verify the installation.

3. Set and export the environment variable CT_MANAGEMENT_SCOPE to 2 for all usersof System Automation for Multiplatforms. Verify your changed settings. Afteryou logged in, enter:env | grep $CT_MANAGEMENT_SCOPE

The output should beCT_MANAGEMENT_SCOPE=2

For more information about installing, see Tivoli System Automation forMultiplatforms Installation and Configuration Guide.


Granting read and write access for the <sapsid>admThe userid <sapsid>adm must have the authority to perform System Automationfor Multiplatforms commands which are launched by the SAP high availabilityinterface.

To setup the non-root security for the <sapsid>adm user ID, see Tivoli SystemAutomation for Multiplatforms Administrator's and User's Guide. It is sufficient to usethe role sa_operator.

Setting up the domainTo set up the IBM Tivoli System Automation for Multiplatforms domain performthe following steps:1. Execute the following command on each node:

preprpnode <node1> <node2>

2. Create the SAP domain for IBM Tivoli System Automation for Multiplatforms,where sap is the domain name. You can use another name of your choice:mkrpdomain sap <node1> <node2>

3. Start the domain:startrpdomain sap

4. Query the domain until it is displayed as online:lsrpdomain

You should see output similar toName OpState RSCTActiveVersion MixedVersions TSPort GSPortsap Online 3.1.5.2 No 12347 12348

5. Ensure that all nodes in the domain are online too:lsrpdomain

You should see output similar toName OpState RSCTVersionnode1 Online 3.1.5.3node2 Online 3.1.5.3

For more information about how to set up a domain, refer to Tivoli SystemAutomation for Multiplatforms Administrator's and User's Guide.

Setting up the tiebreakerIf the number of nodes in the System Automation domain is even, a tiebreakerneeds to be defined to solve situations, where the nodes cannot communicate witheach other.

For more information about how to set up a network tie breaker, refer to TivoliSystem Automation for Multiplatforms Administrator's and User's Guide.

Enabling syslog daemon on AIXAIX only:

Activate syslog daemon to write into /tmp/syslog.out files. The syslog will containmessages written by the System Automation for Multiplatforms scripts.1. In /etc/syslog.conf activate or uncomment the entry:

*.debug /tmp/syslog.out rotate size 100k files 4

2. Create the log file in case it does not exist:touch /tmp/syslog.out


3. Restart the syslog daemon.refresh -s syslogd

Installing and setting up the SAP high availability policyfeature

Find out how to install and set up the SAP high availability policy feature.

PackagingThe code of the SAP high availability policy feature is shipped as part of theSystem Automation for Multiplatforms product, but you need a separate license toenable the code

You get the license by ordering the SAP high availability policy feature. The nameof the license file is sam41SAP.lic, which is stored in the following locations of theSAP high availability policy feature deliverable:

DVD Install the SAP high availability policy feature license from the DVD IBMTivoli System Automation for Multiplatforms v4.1 – IBM Tivoli SystemAutomation for Multiplatforms SAP Policy. The license file is located inthe directory SAM4100FeatSAP/license.

Electronic deliverableIf you obtain the SAP high availability policy feature license throughelectronic distribution, the license file is named CIVG8ML.txt. Rename orcopy the electronic distribution file to sam41SAP.lic.

Installing the SAP high availability policy feature license

Before you can install the SAP high availability policy feature license, install theSystem Automation for Multiplatforms base product as described in “InstallingSystem Automation for Multiplatforms on all cluster nodes” on page 24.

Use the samlicm command to install the SAP high availability policy feature licenseon all nodes in the cluster.

Execute the following command to install the license on all nodes:samlicm -i <license file location> /sam41SAP.lic

To verify that the feature license has been successfully installed, issue the followingcommand:samlicm -s

The name of the SAP high availability policy feature should appear as value of theProduct Annotation field in the output of the command. For example:...Product Annotation: SA for MP - SAP Central Services high availability policyCreation date: Fri Dec 6 00:00:01 MET 2013Expiration date: Thu Dec 31 00:00:01 MET 2037...

For more information about the samlicm command, see System Automation forMultiplatforms Reference Guide.

RestrictionsIf you edit the start, stop, and monitor scripts used in the SAP high availabilitypolicy feature, no support is provided for the modified scripts.


The scripts will still be used to automate your SAP installation, but you will findthe message Modified script not supported in the syslog for each modifiedscript.

Enabling the SAP high availability ConnectorAfter System Automation for Multiplatforms is installed on all cluster nodes, theSAP high availability Connector must be configured in the SAP profiles. It issufficient to enter the required entries into the default profile.

Enable the SAP high availability connector. Otherwise, all start or stop commandsof all SAP tools, for example sapcontrol or SAP MC, are reversed by SystemAutomation for Multiplatforms.

Depending on whether you have an AIX or Linux platform, or you are runningSAP HANA on Power Linux, add the following entries to the default profile ofyour SAP system. Replace <SAPSID> with the SAPSID of your SAP System:

AIX#-----------------------------------------------------------------------# SAP high availability connector#----------------------------------------------------------------------service/halib = /usr/sap/<SAPSID>/SYS/exe/uc/rs6000_64/saphascriptco.oservice/halib_cluster_connector = /usr/sbin/rsct/sapolicies/sap/bin/sap_tsamp_cluster_connector

LINUXReplace <your platform> with the appropriate directory name:

#-----------------------------------------------------------------------# SAP high availability connector#-----------------------------------------------------------------------service/halib = /usr/sap/<SAPSID>/SYS/exe/uc/<your platform>/saphascriptco.soservice/halib_cluster_connector = /usr/sbin/rsct/sapolicies/sap/bin/sap_tsamp_cluster_connector

SAP HANA on Power Linux#-----------------------------------------------------------------------# SAP high availability connector#-----------------------------------------------------------------------service/halib = /usr/sap/<SAPSID>/SYS/exe/hdb/saphascriptco.soservice/halib_cluster_connector = /usr/sbin/rsct/sapolicies/sap/bin/sap_tsamp_cluster_connector

Refer to SAP Note 1693245 - "SAP high availability Script Connector Library" fordetails about the latest patch level for the SAP high availability Script ConnectorLibrary.

Note: For SAP HANA instances: If the file /usr/sap/<SAPSID>/SYS/exe/hdb/saphascriptco.so is not included in the HANA installation, extract the file from aPower LINUX SAP Kernel Part I package manually:1. Identify SAP Kernel version (00 - default HANA instance number), enter

sapcontrol -nr 00 -function GetVersionInfo

2. Download SAP Kernel Part I package. Use the highest patch levelSAPEXE_<patch_level>-<number>.SAR. You can find the patch on the SAPSupport Portal.

3. Extract saphascriptco.so from SAR file, enterSAPCAR -xvf SAPEXE_<patch_level>-<number>.SAR saphascriptco.so

4. Copy saphascriptco.so to HANA installation, entercp saphascriptco.so /usr/sap/<SAPSID>/SYS/exe/hdb/saphascriptco.so

5. Use the HANA installation path in the profile, enterservice/halib = /usr/sap/<SAPSID>/SYS/exe/hdb/saphascriptco.so


https://support.sap.com/software/patches.html

https://support.sap.com/software/patches.html

6. Add the halib_cluster_connector parameter in the profile, enterservice/halib_cluster_connector = /usr/sbin/rsct/sapolicies/sap/bin/sap_tsamp_cluster_connector

Note: It is required to setup non-root user security for the <SID>adm user. Thesapstartsrv process calls the HA connector by using the <SID>adm user.

Migrating the SAP Central Services high availability solutionfrom SA MP V3.2 to V4.1

The SAP Central Services high availability automation policy shipped with SystemAutomation for Multiplatforms version 4.1 incorporates major changes in the waythe components of a SAP solution are controlled and managed. The new SAPCentral Services high availability policy makes use of new interfaces and conformsto advisories made by SAP.

Because of these changes, a SAP Central Services high availability policy generatedwith System Automation for Multiplatforms 4.1 will show major differencescompared to a SAP Central Services high availability policy generated with version3.2 Thus the migration process is essentially a way of a smoothly replacing theformer policy with a new policy that conforms to the changed methods ofcontrolling a SAP solution:v With System Automation for Multiplatforms version 4.1 all existing SAP Central

Services high availability version 3.2 policies can still be used and activated withthe following restrictions:– Policies are stored as complete policy file in the policy pool.– The complete policy file was created with System Automation for

Multiplatforms version 3.2 from a SAP Central Services high availabilitypolicy template of the same version.

These existing SAP Central Services high availability policies are usually storedin the System Automation for Multiplatforms policy pool and are not erased oroverwritten when a new release is installed. The start, stop, monitor scripts usedby version 3.2 SAP Central Services high availability policies are still deliveredwith version 4.1, so they do not have to be saved before migrating.

v The former SAP Central Services high availability policy template files and theirrelated snippets are replaced by new versions when System Automation forMultiplatforms version 4.1 is installed. The new SAP Central Services highavailability policy wizard contained in System Automation for Multiplatformsversion 4.1 will not work with the former policy templates. So you cannot usethe old template files to generate or change SAP Central Services highavailability policies version 3.2, even if you saved these template files to anotherlocation before installing System Automation for Multiplatforms version 4.1.

v Only SAP NetWeaver versions 7.0 or higher with kernel versions 7.20 or higherare supported by the SAP Central Services high availability solution provided bySystem Automation for Multiplatforms version 4.1.

v SAP solutions that are based on a SAP Central Instance (CI) implementation areno longer supported by the SAP Central Services high availability solution. Youneed to have a SAP Central Services (CS) setup.

Upgrading the SAP cluster to System Automation forMultiplatforms version 4.1

To upgrade your cluster to the new version of System Automation forMultiplatforms, follow the steps described in System Automation for


Multiplatforms Installation and Configuration Guide. You might either migrate theentire domain or use the node-by-node approach.

The System Automation for Multiplatforms upgrade will not interfere with thecurrently running SAP Central Services high availability policy, the policy willremain loaded. But for safety reasons be sure to have the complete XMLautomation policy file for your SAP Central Services high availability setupavailable as a backup, either stored in the System Automation for Multiplatformspolicy pool(s) or in another location. The former complete XML automation policyis also working with the new System Automation for Multiplatforms version.

Granting access for the <sapsid>adm user IDThe user ID <sapsid>adm must have the authority to run System Automation forMultiplatforms commands, which are started by the SAP Central Services highavailability interface.

Refer to System Automation for Multiplatforms Administrator's and User's Guideto set up the non-root security for the <sapsid>adm user ID. It is sufficient to usethe role sa_operator.

Creating a new SAP high availability automation policy

Use the policy wizard contained in System Automation for Multiplatforms version4.1 to create the SAP high availability automation policy version 4.1 as described inChapter 5, “Using the wizard to configure and activate the policy,” on page 141.

The policy wizard will not automatically extract configuration information fromyour former SAP Central Services high availability policy (template) files. You haveto enter again all values manually, but you are assisted by the harvesting functionsof the new policy wizard. In addition you can refer to the parameter and valuesummary, that was created in HTML format at the time you configured yourformer version 3.2 SAP Central Services high availability policy. The file shouldhave been saved into the policy pool with the same name as the former templatefile and the added extension .html.

Modifying SAP profile parameters

Before the new SAP Central Services high availability policy can be activated, youhave to adjust some settings in your SAP profiles:v Set the SAP profile parameter for EN and ERS to Start_Program_<NR>. Do NOT

set the profile parameter to Restart_Program, otherwise proper recovery bySystem Automation for Multiplatforms will not work.

v Set the SAP profile parameters for all other servers to Restart_Program_<NR>.

Enabling the SAP high availability Connector

After System Automation for Multiplatforms has been installed on all clusternodes, the SAP high availability Connector must be configured in the SAP profiles.It is sufficient to put the required entries into the default profile.

Refer to “Enabling the SAP high availability Connector” on page 27 for moreinformation about how to add the SAP high availability Connector for yourplatform.


Activating and verifying the SAP Central Services highavailability policy version 4.1

Activate and verify your new SAP Central Services high availability policy:v Deactivate your existing policy using the sampolicy -d command.v Activate the new SAP Central Services high availability policy as described in

“Using the wizard to configure and activate the SAP Central Services highavailability policy” on page 42.

v Verify the policy as described in “Verifying” on page 78.

Configuring and activatingThe following topics describe how to configure the SAP Central Services highavailability policy with the sampolicy wizard. When you defined the policy withthe correct parameter values, you can then activate your SAP Central Services highavailability solution.

SAP Central Services high availability policyThe SAP Central Services high availability policy defines all SAP components asresources and starts and stops them in a well defined sequence to provide highavailability for your SAP system.

For each setup described in “Planning” on page 1 there is a separate policy. SeeTable 11 on page 42 for a list of the policy files depending on the SAP CentralServices high availability setup.

ABAP Central Services (ASCS) high availability policyThe ABAP Central Services (ASCS) high availability policy consists ofequivalencies, resource groups, floating and fixed resources, that are connected toeach other with various relationships.

Figure 10 on page 31 provides an overview of all resources that can be part of aABAP Central Services (ASCS) high availability policy.


Table 7 provides an overview of all resources.

See Table 8 on page 33 for a description of the tags used for the resource names aswell as examples.

Table 7. Resources and resource groups of the ASCS high availability policy

Name Resource name (according to policy naming conventions) Resource Type

ABAP NetworkEquivalency

<A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ASCS>_NETIF Equivalency

Network Interface <INTERFACENAME>:<NODENAME_1> Fixed


ERS Network Equivalency <A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS>_NETIF Equivalency



SAPRouter NetworkEquivalency

<ROUT_PREFIX>_<SAPSID>_SYS_ROUTER_NETIF Equivalency



ASCSInstance Agent

Network Interface(one per cluster node)





SAProuter NetworkEquicalency*

Enqueue Server

Message Server

ABAP SAP Central Services group

SAProuter Service IP

SAProuter

SAProuter group*

Enqueue Replication Server group

ASCS Service IPERS Service IP


SAP Web DispatcherNetwork Equivalency

SAP Web Dispatcher group

SAP Web DispatcherService IP

SAP Web Dispatcher


ERS Instance Agent

SA

SO

SA

SO

SA

SO

DO

Web DispatcherInstance Agent

SA

SO

SA

SO

DO

SA

SO

DO

Enqueue ReplicationServer

SA

SO

SA

SO Network Interface(one per cluster node)


Enqueue Repl. ServerNetwork Equivalency

DO

CO (IfIWasOnline)SA (IfPossible)AC(IfIWasNotOnline)

AC, IS

CO(IfIWasOnline)

Primary Application Server Group

PrimaryApplication Server

SO

SA

PrimaryApplication ServerInstance Agent

Additional Application Server Group

AdditionalApplication Server

SO

SA

AdditionalApplication ServerInstance Agent

SA

Host Agent Group

Host Agent

RelationshipsAC AntiCollocatedCO CollocatedIS IsStartableDO DependsOnSA StartAfterSO StopAfter

SA

Figure 10. ASCS high availability policy


Table 7. Resources and resource groups of the ASCS high availability policy (continued)


Web Dispatcher NetworkEquivalency

<WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD>_NETIF Equivalency



ABAP SAP CentralServices group top-levelgroup

<A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ASCS> Group

ABAP SAP Instance Agentgroup

<A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ASCS>_SRV Group

ASCS Service IP <A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ASCS>_ip Floating

Instance Agent <A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ASCS>_sapstartsrv Floating

ABAP SAP Central Servicesgroup

<A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ASCS>_ASCS Group

Enqueue Server <A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ASCS>_es Floating

Message Server <A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ASCS>_ms Floating

Enqueue ReplicationServer group

<A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS> Group

ERS Instance Agent group <A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS>_SRV Group

Enqueue Replication ServerServiceIP

<A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS>_ip Floating

Instance Agent <A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS>_sapstartsrv Floating

Enqueue Replication Servergroup

<A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS>_ERS Group

Enqueue Replication Server <A_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS>_ers Floating

Primary Application Servertop-level group

<A_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_PRIMARY> Group

Primary Application ServerInstance Agent group

<A_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_PRIMARY>_SRV Group

Instance Agent <A_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_PRIMARY>_sapstartsrv

Fixed

Primary Application Servergroup

<A_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_PRIMARY>_AS Group

Primary Application Server <A_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_PRIMARY>_as Fixed

Additional ApplicationServer top-level group

<A_PREFIX>_<SAPSID>__<INSTANCE_NAME>_<NODENAME_ADDITIONAL> Group

Additional ApplicationServer Instance Agentgroup

<A_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_ADDITIONAL>_SRV Group

Instance Agent <A_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_ADDITIONAL>_sapstartsrv

Fixed

Additional ApplicationServer group

<A_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_ADDITIONAL>_AS Group

Additional ApplicationServer

<A_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_ADDITIONAL>_as Fixed

SAProuter group <ROUT_PREFIX>_<SAPSID>_SYS_ROUTER Group


Table 7. Resources and resource groups of the ASCS high availability policy (continued)


SAProuter <ROUT_PREFIX>_<SAPSID>_SYS_ROUTER_saprouter Floating

SAP router Service IP <ROUT_PREFIX>_<SAPSID>_SYS_ROUTER_ip Floating

SAP WEB Dispatchertop-level group

<WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD> Group

SAP WEB DispatcherInstance Agent group

<WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD>_SRV Group

SAP Web Dispatcher ServiceIP

<WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD>_ip Floating

Instance Agent <WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD>_sapstartsrv Floating

SAP WEB Dispatcher group <WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD>_WD Group

SAP WEB Dispatcher <WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD>_sapwebdisp Floating

Host Agent Group SAP_HOST_AGENT_<NODENAME> Group

Host Agent SAP_HOST_AGENT_<NODENAME>_ha Fixed

Table 8. Description and examples of placeholders for ABAP resource names

Tag Description Example

<A_PREFIX>,

<ROUT_PREFIX>,

<WD_PREFIX>

Prefix for resources, has to be specified in policy.Use a meaningful value to easily identifyresources later.

ABAP, SAPROUTER

<SAPSID>,

<SAPWEBDISP_SID>

SAP System SID LOP

<INTERFACENAME> Name of a network interface eth0

<NODENAME_PRIMARY>,

<NODENAME_ADDITIONAL>

Hostname of the node on which the AS isrunning

sapnode01

<NODENAME_1>,

<NODENAME_2>

Hostnames of the node on which the ASCSCentral Services and the Enqueue ReplicationServer are allowed to run

sapnode01

<INSTANCE_NAME> SAP Application Server Instance Name DVEBMGS01

<INSTANCE_NAME_ERS> SAP Enqueue Replication Server Instance Name ERS10

<INSTANCE_NAME_WD> SAP Web Dispatcher Instance Name WD00

The ABAP SAP Central Services (ASCS) group:The ABAP SAP Central Services (ASCS) group contains four floating resources: aServiceIP, the Instance Agent resource, and the ABAP Enqueue and MessageServers. All are tied together by StartAfter (SA) and StopAfter (SO) relationships.When the ASCS group is started, the IP resource is started first. Once the IPresource is online, the Instance Agent resource is started next, followed by theEnqueue Server and the Message Server. All resources are contained in collocatedgroups, so they are always started on the same node. All resources are mandatorygroup members. No restart is attempted by System Automation for Multiplatformsif one of the resources fails, but a failover of the whole group is triggered instead.

The ABAP Enqueue Replication Server group:


The Enqueue Replication Server group contains three mandatory floating resources:a ServiceIP, the Instance Agent resource and the ABAP Enqueue Replication Server(ERS) itself. All are tied together by StartAfter (SA) and StopAfter (SO)relationships. When the ABAP Enqueue Replication Server group is started, theServiceIP is started first, followed by Instance Agent and the ABAP enqueuereplication server.

Java Central Services (SCS) high availability policyThe Java Central Services (SCS) high availability policy consists of equivalencies,resource groups, floating resources, and fixed resources that are connected to eachother with various relationships.

Table 9 provides an overview of all resources.

See Table 10 on page 36 for a description of the tags that are used for the resourcenames and examples.

Table 9. Resources and resource groups of the Java SCS high availability policy

Name Resource name (according to policy naming conventions) Resource type

Java Network Equivalency <J_PREFIX>_<SAPSID><INSTANCE_NAME_SCS>_NETIF Equivalency


SCSInstance Agent



Java NetworkEquivalency



SAProuter NetworkEquicalency

Enqueue Server

Message Server

Java SAP Central Services group

SAProuter

SAProuter group

Enqueue Replication Server group

SCS Service IPERS Service IP





SAP Web Dispatcher


ERS Instance Agent

SA

SO

SA

SO

SA

SO

DO


SA

SO

SA

SO

DO

SA

SO

DO


SA

SO

SA




DO


AC, IS

CO(IfIWasOnline)


PrimaryApplication Server

SO

SA



AdditionalApplication Server

SO

SA


SA

Host Agent Group

Host AgentRelationshipsAC AntiCollocatedCO CollocatedIS IsStartableDO DependsOnSA StartAfterSO StopAfter

SA

Figure 11. SCS high availability policy for Java


Table 9. Resources and resource groups of the Java SCS high availability policy (continued)




ERS Network Equivalency <J_PREFIX>_<SAPSID><INSTANCE_NAME_ERS>_NETIF Equivalency



SAPRouter NetworkEquivalency

<ROUT_PREFIX>_<SAPSID>_SYS_ROUTER_NETIF Equivalency



Web Dispatcher NetworkEquivalency

<WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD>_NETIF Equivalency



JAVA SAP Central Servicestop-level group

<J_PREFIX>_<SAPSID>_<INSTANCE_NAME_SCS> Group

Java SAP Instance Agent group <J_PREFIX>_<SAPSID>_<INSTANCE_NAME_SCS>_SRV Group

ASCS Service IP <J_PREFIX>_<SAPSID>_<INSTANCE_NAME_SCS>_ip Floating

Instance Agent <J_PREFIX>_<SAPSID>_<INSTANCE_NAME_SCS>_sapstartsrv Floating

Java SAP Central servicesgroup

<J_PREFIX>_<SAPSID>_<INSTANCE_NAME_SCS>_SCS Group

Enqueue Server <J_PREFIX>_<SAPSID>_<INSTANCE_NAME_SCS>_en Floating

Message Server <J_PREFIX>_<SAPSID>_<INSTANCE_NAME_SCS>_ms Floating

Enqueue Replication Servertop-level group

<J_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS> Group

Enqueue Replication ServerInstance Agent group

<J_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS>_SRV Group

Enqueue Replication ServerService ServiceIP

<J_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS>_ip Floating

Instance Agent <J_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS>_sapstartsrv Floating

Enqueue Replication Servergroup

<J_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS>_ERS Group

Enqueue Replication Server <J_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS>_ers Floating

Primary Application Servertop-level group

<J_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_PRIMARY>_ sapstartsrv

Group

Primary Application ServerInstance Agent group

<J_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_PRIMARY>_SRV Group

Instance Agent <J_PREFIX>_<SAPSID>_<INSTANCE_NAME_ERS>_sapstartsrv Floating

Primary Application Servergroup

<J_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_PRIMARY>_AS Group

Primary Application Server <J_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_PRIMARY>_as Fixed

Additional Application Servergroup

<J_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_ADDITIONAL> Group

Additional Application ServerInstance Agent group

<J_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_ADDITIONAL>_SRV Group


Table 9. Resources and resource groups of the Java SCS high availability policy (continued)


Instance Agent <J_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_ADDITIONAL>_ sapstartsrv

Floating

Additional Application Servergroup

<J_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_ADDITIONAL>_AS Group

Additional Application Server <J_PREFIX>_<SAPSID>_<INSTANCE_NAME>_<NODENAME_ADDITIONAL>_as Fixed

SAProuter group <ROUT_PREFIX>_<SAPSID>_SYS_ROUTER Group

SAProuter <ROUT_PREFIX>_<SAPSID>_SYS_ROUTER_saprouter Floating

SAProuter ServiceIP <ROUT_PREFIX>_<SAPSID>_SYS_ROUTER_ip Floating

SAP Web Dispatcher top-levelgroup

<WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD> Group

SAP Web Dispatcher InstanceAgent group

<WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD>_SRV Group

SAP Web Dispatcher ServiceIP <WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD>_ip Floating

Instance Agent <WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD>_sapstartsrv Floating

SAP Web Dispatcher group <WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD>_WD Group

SAP Web Dispatcher <WD_PREFIX>_<SAPWEBDISP_SID>_<INSTANCE_NAME_WD>_sapwebdisp Floating

Host Agent Group SAP_HOST_AGENT_<NODENAME> Group

Host Agent SAP_HOST_AGENT_<NODENAME>_ha Fixed

The namings of the SAProuter and the SAP WEB dispatcher are the same for theABAP and the JAVA policy. For more information, see Table 7 on page 31.

Table 10. Description and examples of placeholders for JAVA resource names

Tag Description Example

<J_PREFIX>

<ROUT_PREFIX>

<WD_PREFIX>

Prefix for resources, to be specified in policy. Use ameaningful value to easily identify resources later.

JAVA_XI

<SAPSID>,

<SAPWEBDISP_SID>

SAP System SID LOP

<INTERFACENAME> Name of a network interface eth0

<NODENAME_PRIMARY>,

<NODENAME_ADDITIONAL>

Host name of the node on which the AS is running sapnode01

<NODENAME_1>,

<NODENAME_2>

Host names of the node on which the ASCS CentralServices and the Enqueue Replication Server areallowed to run.

sapnode01

<INSTANCE_NAME> SAP Application Server Instance Name v DVEBMGS01

v D02

<INSTANCE_NAME_ERS> SAP Enqueue Replication Server Instance Name ERS10

<INSTANCE_NAME_WD> SAP Web Dispatcher Instance Name WD00


The Java SAP Central Services (SCS) group:

The Java SAP Central Services(SCS) group contains four floating resources: aService IP, the Instance Agent resource, and the Java Enqueue and Message Servers(ES and MS).

All are tied together by StartAfter (SA) and StopAfter (SO) relationships. When theSCS group is started, the IP resource is started first. When the IP resource is online,the Instance Agent resource is started next, followed by the Enqueue Server andthe Message Server. The SAP gateway process is monitored together with theMessage Server resource. All resources are contained in collocated groups, so theyare always started on the same node.

All resources are mandatory group members. No restart is attempted by SystemAutomation for Multiplatforms if one of the resources fail, but a failover of thewhole group is triggered instead.

The Java Enqueue Replication Server group:

The Java Enqueue Replication Server group contains three mandatory floatingresources: a ServiceIP, the Instance Agent resource and the Java EnqueueReplication Server (ERS) itself.

All are tied together by StartAfter (SA) and StopAfter (SO) relationships. When theJava Enqueue Replication Server group is started, the ServiceIP is started first,followed by Instance Agent and the Java Enqueue Replication Server.

Double Stack (ABAP and Java SCS) high availability policyThe ABAP and Java SCS HA policy consists of equivalencies, resource groups,floating resources, and fixed resources.

All components are described the table in “ABAP Central Services (ASCS) highavailability policy” on page 30 and “Java Central Services (SCS) high availabilitypolicy” on page 34.


Instead of two separate ABAP and Java Application Server groups, theDouble-Stack policy consists of only one Application Server group, which containsthe Java and ABAP Application Server resources.

The Double Stack application server is logically divided into the ABAP applicationserver and the Java application server. Within a System Automation forMultiplatforms domain, one Double Stack application server instance is automatedas two logical application server resources: an ABAP application server resourceand a Java application server resource. There is a close relationship between thesetwo logical application server resources:

The Java instance is always started after the ABAP instance. A correspondingStartAfter relationship causes that starting the Java instance initiates a precedingstart of the ABAP instance.


ABAPApplication Server

SO

SA


JavaApplication Server

SA

SCSInstance Agent



Java NetworkEquivalency



SAProuter NetworkEquicalency

Enqueue Server

Message Server

Java SAP Central Services group

SAProuter

SAProuter group

Java Enqueue Replication Server group

SCS Service IPERS Service IP





SAP Web Dispatcher


ERS Instance Agent

SA

SO

SA

SO

SA

SO

DO


SA

SO

SA

SO

DO

SA

SO

DO


SA

SO

SA




DO


AC, IS

CO(IfIWasOnline)


ABAPApplication Server

SO

SA



SA

Host Agent Group

Host AgentRelationshipsAC AntiCollocatedCO CollocatedIS IsStartableDO DependsOnSA StartAfterSO StopAfter

SA

ASCSInstance Agent




Enqueue Server

Message Server

ABAP SAP Central Services group ABAP Enqueue Replication Server group

ASCS Service IPERS Service IP

ERS Instance Agent

SA

SO

SA

SO

SA

SO

DO


SA

SO

SA




DO


AC, IS

CO(IfIWasOnline)

JavaApplication Server

SA

SA

SA

Figure 12. Resources of the Double Stack (ABAP and Java SCS) high availability policy


Parts of the ABAP and Java policies with the same behaviorThis topic describes parts of the ABAP and Java policy that show the samebehavior and are automated equally, independent of whether they are part of aABAP or a Java policy.

Interaction between Enqueue Server and Enqueue Replication Server:

A set of six relationships between the Enqueue and Message Servers and theEnqueue Replication Server provide the most important rules for the highavailability of the SAP Central Services.

Figure 13 shows an overview of these relationships, which are referred by theirnumbers as shown in the picture.

A set of example scenarios explain the functions of these relationships. For adetailed description of relationships and their properties, see System Automation forMultiplatforms Administrator's and User's Guide.

ES and MS always start collocated on the same node because of their commongroup constraint.

Initial Start: all nodes are availableERS starts first (because of [1] and [7]), followed by ES and MS insuccession. Since ES was not online before the initial start, ES/MS starts onanother node than ERS because of [2], relationship [3] and [5] are notapplicable in this situation. So the shadow enqueue table is maintained byERS on another node than then the one on which ES is running.

Initial Start: only one node is available in a two-node clusterBecause of [2] and [5] ES/MS and ERS cannot be started on the same node.The competitive situation is resolved by the priorities that are assigned tothe groups of ES/MS and ERS. The group that holds ES and MS has ahigher priority than the ERS group, thus it is started on the sole node. The

Enqueue Server(ES)

Messages Server(MS)

EnqueueReplicationServer (ERS)

RG Priority = 5 RG Priority = 0

AntiCollocated

IsStartable

Collocated (IfIWasOnline)

Anticollocated (IfIWasNotOnline)

StartAfter (IfPossible)

Collocated (IfWasOnline)

StopAfterStartAfter

1

2

3

4

5

6

7 8

Figure 13. Relationships between the Enqueue and Message Servers and the Enqueue Replication Server


“IfPossibe” condition relaxes relationship [1]. Thus the SAP CentralServices can be made available under the adverse conditions.

Failure of ES

When the Enqueue Server fails, it brings down all members of its grouptoo because ES is a mandatory member of the group. System Automationfor Multiplatforms recovers ES on the node where ERS is running becauseof [3]. So ES can rebuild its enqueue table from the shadow that wasmaintained by ERS. Relationship [5] does not apply to this situation nordoes relationship [2] since ES was Online before. MS and the other groupmembers follow ES to the node where it was restarted.

There is also an optional restart feature for the Enqueue Server in the SAPprofile, which is able to recover a failed ES on the same node. This restartfeature must be disabled. Otherwise, ES does not start on the node whereERS runs, hence the rebuild of the enqueue table is not possible. For moreinformation, see “Configuring SAP profiles” on page 20.

Failure of MS

In former SAP releases, System Automation for Multiplatforms can attempta restart in place on the same node for a failed Message Server. In the newSAP release, relationship [6] forces the restart of MS on the node whereERS runs, pulling all other group members, including ES to move to theERS node.

The restart feature for the Message Server in the SAP profile can beenabled to recover a failed MS on the same node. For more information,see “Configuring SAP profiles” on page 20. The described recovery actionby System Automation for Multiplatforms is run in case the SAP restartfeature was not able to restart MS on its former node.

ERS stop or relocation after ES or MS failure recovery

As described in the previous paragraphs, a failed ES is restarted on thenode where ERS is running. After ES recovered its enqueue table from theshadow table, ERS stops itself and is restarted by System Automation forMultiplatforms in succession. The restart of ERS takes place on the nodewhere the ES is running, for example anticollocated to ES on another nodebecause of [5]. Arrow number [4] allows only a node where theappropriate ES constituent is not Failed Offline, so ES would be startableon that node. All other relationships do not apply here.

There is also an optional restart feature for the Enqueue Replication Serverin the SAP profile, which is able to recover a failed ERS on the same node.This restart feature must be disabled. Otherwise, ERS does not start onanother node away from EN. For more information, see “Configuring SAPprofiles” on page 20.

ERS failure

If ERS fails for any reason in an otherwise up and running SAP system, itis restarted anticollocated to the ES node because of [5]. Arrow number [4]allows only a node where the appropriate ES constituent is not FailedOffline. All other relationships do not apply.

As already mentioned before there is also an optional restart feature for theEnqueue Replication Server in the SAP profile, which must be disabled.For more information, see “Configuring SAP profiles” on page 20.


The node where ES is running failsThis scenario is a similar to Failure of ES, followed by the ERS stop orrelocation after ES or MS failure recovery scenario. In a two-node cluster, ERScannot be restarted on another node as enforced by [5] if the failed ESnode is not recovered.

The node where ERS is running failsThis situation is similar to ERS stop or relocation after ES or MS failurerecovery. The restart of ERS is anticollocated on another node because of[5], but in a two-node cluster there is no other node left. ERS cannot berestarted on another node if the failed ERS node is not recovered.

The ABAP and Java application server (AS) groups:

The application servers are implemented as fixed resources in the automationpolicy, because moving an AS instance to another node might cause a longdowntime for the AS. Since the SAP architecture facilitates to run more than oneAS, it is far better to run at least two AS on different hardware to have thenecessary AS redundancy.

Each of the application server groups contains an Instance Agent resource and afixed application server (AS) resource as mandatory members. The applicationservers are in separate groups to not affect each other. All application serverresources have a StartAfter relationship to their Message Server because theMessage Server must be online during startup of an application server. Theapplication server must read the license key from the Message Server. Otherwise, alogon to the AS is not possible, which also inhibits to monitor the applicationserver.

It is not required to restart an application server, if a failure of the Enqueue Serveror Message Server occurs. The application server reconnects to the Message Serverautomatically.

The Primary Application Server group and the Additional Application Servergroups contain the application server (AS) resource as non-mandatory member.

The start and stop of application servers, especially the Java application servers,can take a long time. Therefore, the start and stop command timeouts are to be setto a value of 300 - 500 seconds, depending on ABAP or Java AS.

The ABAP and Java SAProuter group:

The SAProuter program is a SAP utility that controls access to SAP systems.

The SAProuter group contains two floating resources:v SAProuterv Service IP address

Both are tied together by StartAfter and StopAfter relationships If you start theSAProuter group, the IP resource is started first, followed by the SAProuterresource itsself. All resoures are contained in a collocated group, so they willalways be started on the same node.

The ABAP and Java SAP Web Dispatcher group:

The SAP Web Dispatcher program is an SAP utility that controls Web access forSAP ABAP and Java systems.


The SAP Web Dispatcher group contains three floating resources:v SAP Web Dispatcherv Service IP addressv SAP Web Dispatcher Instance Agent (sapstartsrv)

All are tied together by StartAfter and StopAfter relationships If you start the SAPWeb Dispatcher group, the IP resource is started first, followed by the InstanceAgent resource and the Web Dispatcher resource itself. All resources are containedin a collocated group, so they will always be started on the same node.

Host agent groups:

A SAP Host Agent is running on each cluster node that is able to host SAP serversand instances.

With System Automation for Multiplatforms each of these Host Agents is modelledas a fixed resources in its own resource group. No dependencies exist to other SAPservers and instances.

Using the wizard to configure and activate the SAP CentralServices high availability policy

Each SAP Central Services high availability policy consists of a policy template thatis tailored by using the sampolicy wizard.

To configure the template, run the following command:sampolicy –w templateFileName

Depending on the SAP Central Services high availability setup option that youchoose, specify one of the following fully qualified XML template file names:

Table 11. Location for the SAP high availability XML template files

SAP CentralServices highavailability setup XML template file

ABAP /usr/sbin/rsct/sapolicies/sap/sap_ABAP_v41.tmpl.xml

Java /usr/sbin/rsct/sapolicies/sap/sap_JAVA_v41.tmpl.xml

Double-Stack /usr/sbin/rsct/sapolicies/sap/sap_DoubleStack_v41.tmpl.xml

You must configure a policy pool before you can run the wizard. The wizard storesall modifications to the policy pool. Set your policy pool to /etc/opt/IBM/tsamp/sam/policyPool. Run the wizard by entering the following command:sampolicy -w /usr/sbin/rsct/sapolicies/sap/sap_ABAP_v41.tmpl.xml

The results are stored to/etc/opt/IBM/tsamp/sam/policyPool/sap_ABAP_v41.tmpl.xml

The next time you start the wizard, use the file that is stored in the policy pool. Ifyou want to start the wizard for the second time, enter the following command:sampolicy -w /etc/opt/IBM/tsamp/sam/policyPool/sap_ABAP_v41.tmpl.xml

For a detailed description of the sampolicy wizard, see Chapter 5, “Using thewizard to configure and activate the policy,” on page 141.


SAP Central Services high availability policy parametersThe following paragraphs list all parameters that have to be specified for thedifferent policy options.

A HTML file containing all parameter descriptions and all currently defined valuesis generated each time the wizard is started and finished using the 0 (zero)selection option. For more information, refer to “Closing the wizard” on page 155.

SAP ABAP Central Services (ASCS) - Enqueue ReplicationServer (ERS) high availability policy (ABAP)Find the ABAP policy parameter description, value type, and value.

Table 12. ABAP policy parameters

# Parameter description Value type Value

1Enter the name of your SA MP domain.Note: Value harvesting is provided for this parameter.

Provide the name of an existing SA MP domain. The SA MPdomain will host the SAP resources that will be configured withthis template.

String

2Select the IP version used in the SAP environment.

Depending on the IP version, either a NetMask for IPv4 or aNetPrefix for IPv6 has to be specified.

One of the followingvalues:

IPv4

IPv6

IPv4

3Specify the existing SAP system ID (SID).

The SAP system ID consists of 3 characters and is configuredduring the SAP installation.

String

Minimum number ofcharacters: 3, maximumnumber of characters: 3(plus additional valuechecking)

4Specify your SAP admin user name.Note: Value harvesting is provided for this parameter.

The default SAP admin user's name is composed of the SAP SID(in lower case) and the suffix adm.

String (plus additionalvalue checking)

5Enter your desired prefix for all ABAP resources.

This prefix will be used as a prefix for all SA MP resources thatcover ABAP, for example "SAP_ABAP". For later operational tasks,the prefix can be used to start and stop resources with the sameprefix with one single command. You may consider to encode theSAP solution name, e.g. PI, ECC or SCM, which would result in aprefix like "PI_ABAP"..

String SAP_ABAP

6Enter the nodes where you want to automate your SAP CentralServices Instance for ABAP (ASCS).Note: Value harvesting is provided for this parameter.

These nodes must be listed by the SA MP command lsrpnode forthe specified domain. You can use either the long or the shortname for a node. An ASCS resource will be created for each of thespecified nodes.

List of values. Value typefor each value:

Hostname or IP version 4address (plus additionalvalue checking)


Table 12. ABAP policy parameters (continued)


7Specify the instance name of the SAP Central Services Instance forABAP (ASCS).Note: Value harvesting is provided for this parameter.

This instance name is used for the instance directory that containsall necessary files for the ASCS instance. A sample instance name isASCS00.

String

Minimum number ofcharacters: 6, maximumnumber of characters: 6)

8Specify the virtual host name for the Central Services Instance forABAP (ASCS).Note: Value harvesting is provided for this parameter.

This host name will be used as a virtual host name for the CentralServices Instance for ABAP (ASCS). Enter the same virtual hostname that was used for sapinst SAPINST_USE_HOSTNAME=-virt_hostname- during the ASCS installation.

Hostname

9This parameter will be ignored unless all following conditions arefulfilled:

- Parameter # 2 has the value "IPv4"Note: Value harvesting is provided for this parameter.

Specify the virtual IPv4 address for the SAP Central ServicesInstance for ABAP (ASCS).

This IPv4 address will be used as a virtual IP address for thefloating ASCS instance.

IP version 4 address


- Parameter # 2 has the value "IPv6"

Specify the virtual IPv6 address for the SAP Central ServicesInstance for ABAP (ASCS).

This IPv6 address will be used as a virtual IP address for thefloating ASCS instance.




Specify the netmask for the virtual ASCS instance IP address.

Enter the netmask for the subnet of the virtual ASCS instance IPaddress. An example for a netmask is 255.255.255.0

IP version 4 address 255.255.255.0






Enter the NetPrefix for the ASCS instance virtual IP address.

Enter the NetPrefix for the ASCS instance virtual IP address. Anexample for a NetPrefix is 80.

Numeric

Minimum value: 0,maximum value: 128

13Specify the network interface name where your ASCS instancevirtual IP address is activated on each node as alias. The followingnetwork interfaces are available on your local system:

(remaining part of description is harvested from running system)

The network interface specifies to which network interface on eachnode the virtual ASCS instance IP address can be bound, for AIXan example is "en0", for Linux, an example is "eth0". The samenetwork interface name needs to be available on all nodes wherethe ASCS instance will be automated.


14 Specify the instance name of the ABAP ERS instance.Note: Value harvesting is provided for this parameter.

This instance name is used for the instance directory that containsall necessary files for the ABAP ERS instance. A sample instancename is 'ERS12'.

String


15 Specify the virtual host name of the ABAP ERS instance.Note: Value harvesting is provided for this parameter.

This host name will be used as a virtual host name for the ABAPERS instance. Enter the same virtual host name that was used forsapinst SAPINST_USE_HOSTNAME=-virt_hostname- during the ABAPERS installation. If the ERS instance was not installed using avirtual host name, then specify the host name of a unique IPaddress that will be moved together with the ERS. This host nameallows the SAP tools to connect to the node currently running theERS.

String

16 This parameter will be ignored unless all following conditions arefulfilled:


Specify the virtual IPv4 address for the ABAP enqueue replicationserver (ABAP ERS).

This IPv4 address will be used as a virtual IP address for theABAP enqueue replication server (ABAP ERS) instance.






Parameter # 2 has the value "IPv6"Note: Value harvesting is provided for this parameter.

Specify the virtual IPv6 address for the ABAP enqueue replicationserver (ABAP ERS).

This IPv6 address will be used as a virtual IP address for theABAP enqueue replication server (ABAP ERS) instance.



Parameter # 2 has the value "IPv4"Note: Value harvesting is provided for this parameter.

Specify the netmask for the virtual ABAP ERS instance IP address.Enter the netmask for the subnet of the virtual ABAP ERS instanceIP address. An example for a netmask is "255.255.255.0".



Parameter # 2 has the value "IPv6"

Enter the NetPrefix for the ABAP ERS instance virtual IP address.

Enter the NetPrefix for the ABAP ERS instance virtual IP address.An example for a NetPrefix is 80.

Numeric


20 Specify the network interface name where your ABAP ERSinstance virtual IP address is activated on each node as alias. Thefollowing network interfaces are available on your local system:


The network interface specifies to which network interface oneeach node the virtual ABAP ERS instance IP address can be bound,for AIX an example is "en0", for Linux, an example is "eth0". Thesame network interface name needs to be available on all nodeswhere the ABAP ERS instance will be automated.


21Do you want to automate the ABAP application servers?

The ABAP application servers host the applications and serve theuser requests. Automation of the ABAP application servers isrecommended, but optional. Choose yes if you want to automatethe ABAP application servers.

{yes|no}




21.1Optional; a value for this parameter is only required if parameter #21 hasthe value "yes".

Enter the nodes where you want to automate the applicationservers.Note: Value harvesting is provided for this parameter.

These nodes must be listed by the SA MP command "lsrpnode" forthe specified domain. You can use either the long or the shortname for a node. An SA MP application server resource will becreated for each of the specified nodes.

This parameter must have the same number of values as the followingparameters:

- Parameter # 21, nested parameter 2

List of values, value typefor each value:

Hostname or IP version 4address


Specify all instance names of your application servers. Use thesame order as for the nodes in one of the previous questions.Note: Value harvesting is provided for this parameter.

In this policy, the instance names are used to identify the instancedirectory that contains all necessary files for the application server.The naming syntax is DVEBMGS-InstanceID- or D-InstanceID-. Usethe same order as for the nodes in one of the previous questions,i.e. if you specified node01 first, then you now have to specify theinstance directory for your application server on node01 first.






Enter the start timeout value for your ABAP application servers.The start timeout attribute determines the maximum run time inseconds of the StartCommand. If the StartCommand does notreturn within the timeout period, System Automation forMultiplatforms kills the StartCommand with the SIGKILLcommand and logs a message to the system log of the node. Thedefault value for the ABAP application servers is 300.

Numeric 300

21.4 Optional; a value for this parameter is only required if parameter #21 hasthe value "yes".

Enter the stop timeout value for your ABAP application servers.With the stop timeout attribute you specify the amount of time inseconds the stop command for your application servers allowed torun before it is killed by Tivoli System Automation. The defaultvalue for the ABAP application servers is 300 seconds.

Numeric 300





- Parameter # 21 has the value "yes"

Did you configure a virtual host name during installation for atleast one of the application servers specified in the previousquestion? Choose yes if at least one of the application servers hasbeen installed with a virtual host name.

{yes|no}


Specify the virtual host name for each application server. Use thesame order as for the nodes in one of the previous questions. Ifyou installed one of the application servers without a virtual hostname, specify the system host name instead.Note: Value harvesting is provided for this parameter.

This host name will be used as a virtual host name for anapplication server. Enter the same virtual host name that was usedfor sapinst SAPINST_USE_HOSTNAME=-virt_hostname- during theSAP installation. Use the same order as for the nodes in one of theprevious questions, i.e. if you specified node01 first, then you nowhave to specify the virtual host name for your application serveron node01 first.


Hostname

23 Do you want to automate the SAP Host Agent?

SAP Host Agent can be used for monitoring and control of SAPinstances and non-SAP instances, operating systems, anddatabases.

{yes|no}


Enter the nodes where you want to automate the SAP Host Agent.Note: Value harvesting is provided for this parameter.

These nodes must be listed by the SA MP command lsrpnode forthe specified domain. You can use either the long or the shortname for a node. An SA MP host agent resource will be created foreach of the specified nodes.



24 Do you want SA MP to automate your SAP router?

SAP router serves as a proxy in a network connection betweenSAP systems or between SAP systems and external networks. Ifyou answer this question with yes, SA MP will create automationresources for the SAP router.

{yes|no}


Enter the desired prefix for the SAP router resources.

You are allowed to use the same prefix as for other resources, likeJAVA or ABAP.

String SAP_ROUTER





Enter the nodes where you want to automate the SAP router.Note: Value harvesting is provided for this parameter.

For each of the nodes specified, SA MP will create SAP routerresources.



24.3 This parameter will be ignored unless all following conditions arefulfilled:


Optional; a value for this parameter is only required if parameter #24 hasthe value "yes".

Specify the virtual IPv4 address that clients will use to connect tothe SAP router.

This virtual IPv4 address is used to reach the SAP router,independent of the system it is currently running on.











Enter the NetPrefix for the SAP router virtual IP address.

Enter the NetPrefix for the SAP router virtual IP address. Anexample for a NetPrefix is 80.





Specify the netmask for the SAP router virtual IP address.

Enter the netmask for the subnet of the SAP router virtual IPaddress. An example for a netmask is "255.255.255.0".

Numeric






Enter the network interface for the SAP router IP address. Thefollowing network interfaces are available on your local system:


The available network interface specifies to which networkinterfaces the SAP router virtual IP address can be bound, for AIXan example is "en0", for Linux, an example is "eth0". The samenetwork interface name needs to be available on all nodes wherethe SAP router will be automated.



Specify the fully qualified SAP router routing table filename.


Use the location of the NFS device that contains the routing tablefile. An example for the fully qualified SAP router routing tablefilename is /usr/sap/-SAPSID-/SYS/global/saprouttab.

String

25 Do you want SA MP to automate the SAP Web Dispatcher?

The SAP Web Dispatcher receives HTTP(s) requests from theinternet that are targeted for your SAP system. If you answer thisquestion with yes, SA MP will create automation resources for theSAP Web Dispatcher.

{yes|no}


Enter the desired prefix for the SAP Web Dispatcher resources.


String SAP_WDISP


Enter the nodes where you want to automate the SAP WebDispatcher.

You are allowed to use the same prefix as for other resources, likeJAVA or ABAP.Note: Value harvesting is provided for this parameter.

Nodes where the SAP Web Dispatcher is configured and where itwill be automated.




Specify the SAP system ID (SAPSID) for the SAP Web Dispatcher.The SAP system ID (SAPSID) for the SAP Web Dispatcher isrequired to identify the SAP Web Dispatcher correctly.Note: Value harvesting is provided for this parameter.






Specify the instance owner username that will be used to executethe start, stop and monitor commands for SAP Web Dispatcherresources.Note: Value harvesting is provided for this parameter.

The instance owner username is composed of the SAP WebDispatcher SID and the suffix "adm".

String


Specify the instance name of the SAP Web Dispatcher instance, i.e.'W00'.

This instance name is used for the instance directory that containsall necessary files for the SAP Web Dispatcher instance.Note: Value harvesting is provided for this parameter.

String


This host name will be used as a virtual host name for the SAPWeb Dispatcher. Enter the same virtual host name that was usedfor sapinst SAPINST_USE_HOSTNAME=-virt_hostname- during theSAP installation. Use the same order as for the nodes in one of theprevious questions, i.e. if you specified node01 first, then you nowhave to specify the virtual host name for your application serveron node01 first.Note: Value harvesting is provided for this parameter.

Hostname




Specify the virtual IPv4 address that clients will use to connect tothe SAP Web Dispatcher.Note: Value harvesting is provided for this parameter.

This virtual IPv4 address is used to reach the SAP Web Dispatcher,independent of the system it is currently running on.





Specify the virtual IPv6 address that clients will use to connect tothe SAP Web Dispatcher.

This virtual IPv6 address is used to reach the SAP Web Dispatcher,independent of the system it is currently running on.








Specify the netmask for the SAP Web Dispatcher virtual IP address.

Enter the netmask for the subnet of the SAP Web Dispatchervirtual IP address. An example for a netmask is "255.255.255.0".





Enter the NetPrefix for the SAP Web Dispatcher IP address.

Enter the NetPrefix for the SAP Web Dispatcher IP address. Anexample for a NetPrefix is 80.

Numeric



Specify the network interface on which SAP Web Dispatchervirtual IP address is activated on each node as alias. The followingnetwork interfaces are available on your local system:


The available network interface specifies to which networkinterfaces the SAP web dispatcher virtual IP address can be bound,for AIX an example is "en0", for Linux, an example is "eth0". Thesame network interface name needs to be available on all nodeswhere the SAP Web dispatcher instance will be automated.




If your database is automated with SA MP in the same cluster, doyou want to create startAfter relationships for your applicationservers?

A startAfter relationship will be created for each ApplicationServer. If you want to create startAfter relationships to a database,the database needs to be automated in the same cluster as SAP.

{yes|no}


Enter the name of your floating SA MP database resource.

This is the name of your floating SA MP database resource, i.e.db2_db2ax0_0-rs.

String


SAP JAVA Central Services (SCS) - Enqueue Replication Server(ERS) high availability policy (Java)Find the Java policy parameter description, value type, and value.

Table 13. Java policy parameters




String




IPv4

IPv6

IPv4

3Specify the existing SAP system ID (SID).Note: Value harvesting is provided for this parameter.


String


4Specify your SAP instance owner user name.Note: Value harvesting is provided for this parameter.

The default SAP instance owner user's name is composed of theSAP SID (in lower case) and the suffix "adm".


5Enter your desired prefix for all JAVA resources.

This prefix will be used as a prefix for all SA MP resources thatcover SAP JAVA components. For later operational tasks, theprefix can be used to start and stop resources with the sameprefix with one single command. You may consider to encode theSAP solution name, e.g. EP (Enterprise Portal), which wouldresult in a prefix like "PE_JAVA".

String SAP_JAVA

6Enter the nodes where you want to automate your SAP CentralServices Instance for JAVA (SCS).Note: Value harvesting is provided for this parameter.

These nodes must be listed by the SA MP command lsrpnode forthe specified domain. You can use either the long or the shortname for a node. A SCS resource will be created for each of thespecified nodes.



7Specify the instance name of the SAP Central Services Instance forJAVA (SCS).Note: Value harvesting is provided for this parameter.

This instance name is used for the instance directory that containsall necessary files for the SCS instance. A sample instance name is'SCS01'.

String

Minimum number ofcharacters: 5, maximumnumber of characters: 5


Table 13. Java policy parameters (continued)


8Specify the virtual host name for the Central Services Instance forJAVA (SCS). Enter the same virtual host name that was used forsapinst SAPINST_USE_HOSTNAME=-virt_hostname- during the SAPinstallation.Note: Value harvesting is provided for this parameter.

This host name will be used as a virtual host name for theCentral Services Instance for JAVA (SCS).

Hostname



Specify the virtual IPv4 address for the SAP Central ServicesInstance for JAVA (SCS).




Specify the virtual IPv6 address for the SAP Central ServicesInstance for JAVA (SCS).

This IPv6 address will be used as a virtual IP address for thefloating SAP Central Services Instance for JAVA (SCS).




Specify the netmask for the virtual JAVA SCS instance IP address.

Enter the netmask for the subnet of the JAVA SCS instance IPaddress. An example for a netmask is "255.255.255.0".




Enter the NetPrefix for the JAVA SCS instance virtual IP address.

Enter the NetPrefix for the JAVA SCS instance virtual IP address.An example for a NetPrefix is 80.

Numeric





13 Specify the network interface name where your JAVA SCSinstance virtual IP address is activated on each node as alias. Thefollowing network interfaces are available on your local system:


The network interface specifies on which network interface oneach node the virtual JAVA SCS instance IP address can be boundto, for AIX an example is "en0", for Linux, an example is "eth0".The same network interface name needs to be available on allnodes where the JAVA SCS instance will be automated.


14 Specify the instance name of the JAVA ERS instance.Note: Value harvesting is provided for this parameter.

This instance name is used for the instance directory that containsall necessary files for the JAVA ERS instance. A sample instancename is 'ERS11'.

String


15Specify the virtual host name of SAP JAVA enqueue replicationserver (JAVA ERS).Note: Value harvesting is provided for this parameter.

This host name will be used as a virtual host name for the JAVAERS instance. Enter the same virtual host name that was used forsapinst SAPINST_USE_HOSTNAME=-virt_hostname- during the JAVAERS installation. If the ERS instance was not installed using avirtual host name, then specify the host name of a unique IPaddress that will be moved together with the ERS. This hostname allows the SAP tools to connect to the node currentlyrunning the ERS.

String



Specify the virtual IPv4 address for the JAVA enqueue replicationserver (JAVA ERS).Note: Value harvesting is provided for this parameter.

This IPv4 address will be used as a virtual IP address for theJAVA enqueue replication server (JAVA ERS) instance.




Specify the virtual IPv6 address for the JAVA enqueue replicationserver (JAVA ERS).

This IPv6 address will be used as a virtual IP address for theJAVA enqueue replication server (JAVA ERS) instance.







Specify the netmask for the virtual JAVA ERS instance IP address.

Enter the netmask for the subnet of the virtual JAVA ERS instanceIP address. An example for a netmask is "255.255.255.0".




Enter the NetPrefix for the JAVA ERS instance virtual IP address.

Enter the NetPrefix for the JAVA ERS instance virtual IP address.An example for a NetPrefix is 80.

Numeric


20Specify the network interface name where your JAVA ERSinstance virtual IP address is activated on each node as alias. Thefollowing network interfaces are available on your local system:


The network interface specifies to which network interface oneeach node the virtual JAVA ERS instance IP address can bebound, for AIX an example is "en0", for Linux, an example is"eth0". The same network interface name needs to be available onall nodes where the JAVA ERS instance will be automated.


21Do you want to automate the JAVA application servers?

Automation of the JAVA application servers is recommended, butoptional. Choose yes if you want to automate the JAVAapplication servers.

{yes|no}

21.1Optional; a value for this parameter is only required if parameter #21has the value "yes".

Enter the nodes where you want to automate the applicationservers.Note: Value harvesting is provided for this parameter.

These nodes must be listed by the SA MP command "lsrpnode"for the specified domain. You can use either the long or the shortname for a node. An SA MP application server resource will becreated for each of the specified nodes.









Specify all instance names of your application servers. Use thesame order as for the nodes in one of the previous questions.Note: Value harvesting is provided for this parameter.

In this policy, the instance names are used to identify the instancedirectory that contains all necessary files for the applicationserver. The naming syntax is J-InstanceID- or JC-InstanceID-. Usethe same order as for the nodes in one of the previous questions,i.e. if you specified node01 first, then you now have to specify theinstance directory for your application server on node01 first.






Enter the start timeout value for your JAVA application servers.

The start timeout attribute determines the maximum run time inseconds of the StartCommand. If the StartCommand does notreturn within the timeout period, System Automation forMultiplatforms kills the StartCommand with the SIGKILLcommand and logs a message to the system log of the node. Thedefault value for the JAVA application servers is 500.

Numeric 500


Enter the stop timeout value for your JAVA application servers.

With the stop timeout attribute you specify the amount of time inseconds the stop command for your application servers allowedto run before it is killed by Tivoli System Automation. The defaultvalue for the JAVA application servers is 360 seconds.

Numeric 360



Did you configure a virtual host name during installation for atleast one of the application servers specified in the previousquestion?

Choose yes if at least one of the application servers has beeninstalled with a virtual host name.

{yes|no}





Specify the virtual host name for each application server. Use thesame order as for the nodes in one of the previous questions. Ifyou installed one of the application servers without a virtual hostname, specify the system host name instead.Note: Value harvesting is provided for this parameter.

This host name will be used as a virtual host name for anapplication server. Enter the same virtual host name that wasused for sapinst SAPINST_USE_HOSTNAME=-virt_hostname- duringthe SAP installation. Use the same order as for the nodes in oneof the previous questions, i.e. if you specified node01 first, thenyou now have to specify the virtual host name for yourapplication server on node01 first.


Hostname

23 Do you want to automate the SAP Host Agent?

SAP Host Agent can be used for monitoring and control of SAPinstances and non-SAP instances, operating systems, anddatabases.

{yes|no}


Enter the nodes where you want to automate the SAP HostAgent.Note: Value harvesting is provided for this parameter.

These nodes must be listed by the SA MP command "lsrpnode"for the specified domain. You can use either the long or the shortname for a node. An SA MP host agent resource will be createdfor each of the specified nodes.



no

24 Do you want SA MP to automate your SAP router?

SAP router serves as a proxy in a network connection betweenSAP systems or between SAP systems and external networks. Ifyou answer this question with yes, SA MP will create automationresources for the SAP router.

{yes|no}




String SAP_ROUTER


Enter the nodes where you want to automate the SAP router.Note: Value harvesting is provided for this parameter.

For each of the nodes specified, SA MP will create SAP routerresources.






24.3This parameter will be ignored unless all following conditions arefulfilled:


Optional; a value for this parameter is only required if parameter #24has the value "yes".













Specify the netmask for the SAP router virtual IP address.

Enter the netmask for the subnet of the SAP router virtual IPaddress. An example for a netmask is "255.255.255.0".


24.6Do you want SA MP to automate the SAP Web dispatcher?




Enter the NetPrefix for the SAP router virtual IP address. Anexample for a NetPrefix is 80.

Numeric






Enter the network interface for the SAP router IP address. Thefollowing network interfaces are available on your local system:


The available network interface specifies to which networkinterfaces the SAP router virtual IP address can be bound, for AIXan example is "en0", for Linux, an example is "eth0". The samenetwork interface name needs to be available on all nodes wherethe SAP router will be automated.




Use the location of the NFS device that contains the routing tablefile. An example for the fully qualified SAP router routing tablefilename is /usr/sap/-SAPSID-/SYS/global/saprouttab.

String

25Do you want SA MP to automate the SAP Web Dispatcher?

The SAP Web dispatcher receives HTTP(s) requests from theinternet that are targeted for your SAP system. If you answer thisquestion with yes, SA MP will create automation resources for theSAP Web dispatcher.

{yes|no}


Enter the desired prefix for the SAP Web Dispatcher resources.You are allowed to use the same prefix as for other resources, likeJAVA or ABAP.

String SAP_WDISP


Enter the nodes where you want to automate the SAP WebDispatcher.Note: Value harvesting is provided for this parameter.

Nodes where the SAP Web Dispatcher is configured and where itwill be automated.




Specify the SAP system ID (SAPSID) for the SAP Web Dispatcher.

The SAP system ID (SAPSID) for the SAP Web Dispatcher isrequired to identify the SAP Web Dispatcher correctly.Note: Value harvesting is provided for this parameter.






Specify the instance owner username that will be used to executethe start, stop and monitor commands for SAP Web Dispatcherresources.Note: Value harvesting is provided for this parameter.

The instance owner username is composed of the SAP WebDispatcher SID and the suffix "adm".



Specify the instance name of the SAP web dispatcher instance, i.e.'W00'.

This instance name is used for the instance directory that containsall necessary files for the SAP web dispatcher instance.Note: Value harvesting is provided for this parameter.

String


Specify the virtual host name for the SAP Web Dispatcher.

This host name will be used as a virtual host name for the SAPWeb dispatcher. Enter the same virtual host name that was usedfor sapinst SAPINST_USE_HOSTNAME=-virt_hostname- during theSAP installation. Use the same order as for the nodes in one ofthe previous questions, i.e. if you specified node01 first, then younow have to specify the virtual host name for your applicationserver on node01 first.Note: Value harvesting is provided for this parameter.

Hostname




Specify the virtual IPv4 address that clients will use to connect tothe SAP Web Dispatcher.Note: Value harvesting is provided for this parameter.

This virtual IPv4 address is used to reach the SAP WebDispatcher, independent of the system it is currently running on.








Specify the virtual IPv6 address that clients will use to connect tothe SAP Web Dispatcher.

This virtual IPv6 address is used to reach the SAP WebDispatcher, independent of the system it is currently running on.





Specify the netmask for the SAP Web Dispatcher virtual IPaddress

Enter the netmask for the subnet of the SAP Web Dispatchervirtual IP address. An example for a netmask is "255.255.255.0".





Enter the NetPrefix for the SAP Web Dispatcher IP address.

Enter the NetPrefix for the SAP Web Dispatcher IP address. Anexample for a NetPrefix is 80.

Numeric



Specify the network interface on which SAP Web Dispatchervirtual IP address is activated on each node as alias. Thefollowing network interfaces are available on your local system:


The available network interface specifies to which networkinterfaces the SAP web dispatcher virtual IP address can bebound, for AIX an example is "en0", for Linux, an example is"eth0". The same network interface name needs to be available onall nodes where the SAP Web dispatcher instance will beautomated.







If your database is automated with SA MP in the same cluster, doyou want to create startAfter relationships for your applicationservers?

A startAfter relationship will be created for each ApplicationServer. If you want to create startAfter relationships to a database,the database needs to be automated in the same cluster as SAP.

{yes|no}


Enter the name of your floating SA MP database resource.

This is the name of your floating SA MP database resource, i.e."db2_db2ax0_0-rs".

String

SAP ABAP and Java Central Services (ASCS + SCS) - EnqueueReplication Server (ERS) high availability policy (Double Stack)Find a parameter description, value type, and value of the Double-Stack resourcesand the corresponding components.

Table 14. Double-Stack resources and the corresponding components


1Enter the name of your System Automation forMultiplatforms domain.Note: Value harvesting is provided for this parameter.

Provide the name of an existing System Automation forMultiplatforms domain. The System Automation forMultiplatforms domain hosts the SAP resources that areconfigured with this template.

String

2Select the IP version that is used in the SAPenvironment.

Depending on the IP version, either a NetMask for IPv4or a NetPrefix for IPv6 must be specified.


IPv4

IPv6

IPv4


The SAP system ID consists of 3 characters and isconfigured during the SAP installation.

StringMinimumnumber ofcharacters: 3

Maximumnumber ofcharacters: 3(plus extravaluechecking)


Table 14. Double-Stack resources and the corresponding components (continued)



The default SAP instance owner user's name is composedof the SAP SID (in lowercase) and the suffix adm.

String (plus extra valuechecking)

5Enter your desired prefix for all ABAP resources.

This prefix is used as a prefix for all resources that coverABAP, for example SAP_ABAP. For operational tasks, theprefix can be used to start and stop resources with thesame prefix with one single command. You mightconsider encoding the SAP solution name, for example,PI, ECC, or SCM, which results in a prefix like PI_ABAP.

String SAP_ABAP

6Enter the nodes where you want to automate your SAPCentral Services Instance for ABAP (ASCS).Note: Value harvesting is provided for this parameter.

These nodes must be listed by the command lsrpnodefor the specified domain. You can use either the long orthe short name for a node. An ASCS resource is createdfor each of the specified nodes.

List of values, value typefor each value: Host nameor IP version 4 address(plus extra value checking)

7Specify the instance name of the SAP Central ServicesInstance for ABAP (ASCS).Note: Value harvesting is provided for this parameter.

This instance name is used for the instance directory thatcontains all necessary files for the ASCS instance. Asample instance name is 'ASCS00'.

String Minimumnumber ofcharacters: 6,maximumnumber ofcharacters: 6(plus extravaluechecking)

8Specify the virtual host name for the Central ServicesInstance for ABAP (ASCS).Note: Value harvesting isprovided for this parameter.

This host name is used as a virtual host name for theCentral Services Instance for ABAP (ASCS). Enter thesame virtual host name that was used for sapinst

SAPINST_USE_HOSTNAME=-virt_hostname- during the ASCSinstallation.

Host name

9This parameter is ignored unless all following conditionsare fulfilled: - Parameter # 2 has the value "IPv4"

Specify the virtual IPv4 address for the SAP CentralServices Instance for ABAP (ASCS).Note: Value harvesting is provided for this parameter.

This IPv4 address is used as a virtual IP address for thefloating ASCS instance.





10This parameter will be ignored unless all followingconditions are fulfilled: - Parameter # 2 has the value"IPv6"

Specify the virtual IPv6 address for the SAP CentralServices Instance for ABAP (ASCS).

This IPv6 address is used as a virtual IP address for thefloating ASCS instance.



Specify the netmask for the virtual ASCS instance IPaddress.

Enter the netmask for the subnet of the virtual ASCSinstance IP address. An example for a netmask is"255.255.255.0".



Enter the NetPrefix for the ASCS instance virtual IPaddress.

Enter the NetPrefix for the ASCS instance virtual IPaddress. An example for a NetPrefix is 80.

Numeric Minimumvalue: 0,maximumvalue: 128

13Specify the network interface name where your ASCSinstance virtual IP address is activated on each node asalias. The following network interfaces are available onyour local system: (remaining part of description isharvested from running system)

The network interface specifies to which networkinterface on each node the virtual ASCS instance IPaddress can be bound. For AIX an example is en0. ForLinux, an example is eth0. The same network interfacename needs to be available on all nodes where the ASCSinstance is automated.

String plus extra valuechecking

14Specify the instance name of the ABAP ERS instance.Note: Value harvesting is provided for this parameter.

This instance name is used for the instance directory thatcontains all necessary files for the ABAP ERS instance. Asample instance name is 'ERS12'.





15Specify the virtual host name of the ABAP ERS instance.Note: Value harvesting is provided for this parameter.

This host name is used as a virtual host name for theABAP ERS instance. Enter the same virtual host namethat was used for sapinst

SAPINST_USE_HOSTNAME=-virt_hostname- during theABAP ERS installation. If the ERS instance was notinstalled by using a virtual host name, then specify thehost name of a unique IP address that is moved togetherwith the ERS. This host name allows the SAP tools toconnect to the node currently running the ERS.

String

16This parameter is ignored unless all following conditionsare fulfilled: Parameter # 2 has the value "IPv4"

Specify the virtual IPv4 address for the ABAP enqueuereplication server (ABAP ERS).Note: Value harvesting is provided for this parameter.

This IPv4 address is used as a virtual IP address for theABAP enqueue replication server (ABAP ERS) instance.



Specify the virtual IPv6 address for the ABAP enqueuereplication server (ABAP ERS).

This IPv6 address is used as a virtual IP address for theABAP enqueue replication server (ABAP ERS) instance.



Specify the netmask for the virtual ABAP ERS instanceIP address.

Enter the netmask for the subnet of the virtual ABAPERS instance IP address. An example for a netmask is"255.255.255.0".



Enter the NetPrefix for the ABAP ERS instance virtual IPaddress.

Enter the NetPrefix for the ABAP ERS instance virtual IPaddress. An example for a NetPrefix is 80.





20Specify the network interface name where your ABAPERS instance virtual IP address is activated on each nodeas alias. The following network interfaces are availableon your local system: remaining part of description isharvested from running system

The network interface specifies to which networkinterface one each node the virtual ABAP ERS instanceIP address can be bound. For AIX an example is en0. ForLinux, an example is eth0. The same network interfacename needs to be available on all nodes where the ABAPERS instance is automated.

String plus extra valuechecking

21Do you want to automate the ABAP and Java applicationservers?

Automation of the ABAP/Java application servers isrecommended, but optional. Choose yes if you want toautomate the ABAP application servers.

{yes|no}yes

21.1Optional; a value for this parameter is only required ifparameter # 21 has the value "yes".

Enter the nodes where you want to automate theapplication servers.Note: Value harvesting is provided for this parameter.

These nodes must be listed by the System Automationfor Multiplatforms command lsrpnode for the specifieddomain. You can use either the long or the short namefor a node. A System Automation for Multiplatformsapplication server resource is created for each of thespecified nodes.

This parameter must have the same number of values asthe following parameters: Parameter # 21, nestedparameter 2

List of values, value typefor each value: Host nameor IP version 4 address





Specify all instance names of your application servers.Use the same order as for the nodes in one of theprevious questions.Note: Value harvesting is provided for this parameter.

In this policy, the instance names are used to identify theinstance directory that contains all necessary files for theapplication server. The naming syntax isDVEBMGS-InstanceID- or D-InstanceID-. Use the sameorder as for the nodes in one of the previous questions.For example if you specified node01 first, then you mustspecify the instance directory for your application serveron node01 first.

This parameter must have the same number of values asthe following parameters: Parameter # 21, nestedparameter 1

List of values, value typefor each value: String (plusextra value checking)


Enter the start timeout value for your ABAP applicationservers.

The start timeout attribute determines the maximum runtime in seconds of the StartCommand. If theStartCommand does not return within the timeoutperiod, System Automation for Multiplatforms stops theStartCommand with the SIGKILL command and logs amessage to the system log of the node. The default valuefor the ABAP application servers is 300.

Numeric300


Enter the stop timeout value for your ABAP applicationservers.

With the stop timeouts attribute you specify the amountof time in seconds the stop command for yourapplication servers that are allowed to run before it isstopped by System Automation. The default value for theABAP application servers is 300 seconds.

Numeric 300





Enter the start timeout value for your Java applicationservers.

The start timeout attribute determines the maximum runtime in seconds of the StartCommand. If theStartCommand does not return within the timeoutperiod, System Automation for Multiplatforms kills theStartCommand with the SIGKILL command and logs amessage to the system log of the node. The default valuefor the Java application servers is 500.

Numeric 500


Enter the stop timeout value for your Java applicationservers.

With the stop timeouts attribute, you specify the amountof time in seconds the stop command for yourapplication servers that are allowed to run before it isstopped by System Automation. The default value for theJava application servers is 360 seconds.

Numeric 360

22This parameter is ignored unless all following conditionsare fulfilled: - Parameter # 21 has the value "yes"

Did you configure a virtual host name during installationfor at least one of the application servers that arespecified in the previous question?

Choose yes if at least one of the application servers areinstalled with a virtual host name.

{yes|no}no


Specify the virtual host name for each application server.Use the same order as for the nodes in one of theprevious questions. If you installed one of the applicationservers without a virtual host name, specify the systemhost name instead.Note: Value harvesting is provided for this parameter.

This host name is used as a virtual host name for anapplication server. Enter the same virtual host name thatwas used for sapinst

SAPINST_USE_HOSTNAME=-virt_hostname- during the SAPinstallation. Use the same order as for the nodes in oneof the previous questions. That is, if you specifiednode01 first, then you must specify the virtual host namefor your application server on node01 first.

List of values, value typefor each value: Host name




23Enter your desired prefix for all Java resources.

This prefix is used as a prefix for all System Automationfor Multiplatforms resources that cover SAP Javacomponents. For operational tasks, the prefix can be usedto start and stop resources with the same prefix with onesingle command. You might consider encoding the SAPsolution name, for example, EP (Enterprise Portal), whichwould result in a prefix like PE_JAVA.

String SAP_JAVA

24Enter the nodes where you want to automate your SAPCentral Services Instance for Java (SCS).Note: Value harvesting is provided for this parameter.

These nodes must be listed by the System Automationfor Multiplatforms command lsrpnode for the specifieddomain. You can use either the long or the short namefor a node. An SCS resource is created for each of thespecified nodes.


25Specify the instance name of the SAP Central ServicesInstance for Java (SCS).Note: Value harvesting is provided for this parameter.

This instance name is used for the instance directory thatcontains all necessary files for the SCS instance. A sampleinstance name is 'SCS01'.


26Specify the virtual host name for the Central ServicesInstance for Java (SCS). Enter the same virtual host namethat was used for sapinst

SAPINST_USE_HOSTNAME=-virt_hostname- during the SAPinstallation.Note: Value harvesting is provided for this parameter.

This host name is used as a virtual host name for theCentral Services Instance for Java (SCS).

Host name


Specify the virtual IPv4 address for the SAP CentralServices Instance for Java (SCS).Note: Value harvesting is provided for this parameter.

This IPv4 address is used as a virtual IP address for thefloating SAP Central Services Instance for Java (SCS).






Specify the virtual IPv6 address for the SAP CentralServices Instance for Java (SCS).

This IPv6 address is used as a virtual IP address for thefloating SAP Central Services Instance for Java (SCS).



Specify the netmask for the virtual Java SCS instance IPaddress.

Enter the netmask for the subnet of the Java SCS instanceIP address. An example for a netmask is "255.255.255.0".



Enter the NetPrefix for the Java SCS instance virtual IPaddress.

Enter the NetPrefix for the Java SCS instance virtual IPaddress. An example for a NetPrefix is 80.

Numeric


31Specify the network interface name where your Java SCSinstance virtual IP address is activated on each node asalias. The following network interfaces are available onyour local system: (remaining part of description isharvested from running system)

The network interface specifies on which networkinterface on each node the virtual Java SCS instance IPaddress can be bound to. For AIX, an example is en0. ForLinux, an example is eth0. The same network interfacename needs to be available on all nodes where the JavaSCS instance is automated.


32Specify the instance name of the Java ERS instance.Note: Value harvesting is provided for this parameter.

This instance name is used for the instance directory thatcontains all necessary files for the Java ERS instance. Asample instance name is 'ERS11'.

String

Minimum number ofcharacters: 5, maximumnumber of characters: 5(plus extra value checking)




33Specify the virtual host name of the SAP Java enqueuereplication server (Java ERS).Note: Value harvesting is provided for this parameter.

This host name is used as a virtual host name for theJava ERS instance. Enter the same virtual host name thatwas used for sapinst

SAPINST_USE_HOSTNAME=-virt_hostname- during the JavaERS installation. If the ERS instance was not installed byusing a virtual host name, then specify the host name ofa unique IP address that is moved together with the ERS.This host name allows the SAP tools to connect to thenode currently running the ERS.

String


Specify the virtual IPv4 address for the Java enqueuereplication server (Java ERS).Note: Value harvesting is provided for this parameter.

This IPv4 address is used as a virtual IP address for theJava enqueue replication server (Java ERS) instance.



Specify the virtual IPv6 address for the Java enqueuereplication server (Java ERS).

This IPv6 address is used as a virtual IP address for theJava enqueue replication server (Java ERS) instance.



Specify the netmask for the virtual Java ERS instance IPaddress.

Enter the netmask for the subnet of the virtual Java ERSinstance IP address. An example for a netmask is"255.255.255.0".

Value type: IP version 4address

255.255.255.0


Enter the NetPrefix for the Java ERS instance virtual IPaddress.

Enter the NetPrefix for the Java ERS instance virtual IPaddress. An example for a NetPrefix is 80.

Numeric





38Specify the network interface name where your Java ERSinstance virtual IP address is activated on each node asalias. The following network interfaces are available onyour local system: remaining part of description isharvested from running system

The network interface specifies to which networkinterface one each node the virtual Java ERS instance IPaddress can be bound. For AIX, an example is en0. ForLinux, an example is eth0. The same network interfacename needs to be available on all nodes where the JavaERS instance is automated.


39Do you want to automate the SAP Host Agent?

SAP Host Agent can be used for monitoring and controlof SAP instances and non-SAP instances, operatingsystems, and databases.

{yes|no}yes


Enter the nodes where you want to automate the SAPHost Agent.Note: Value harvesting is provided for this parameter.

These nodes must be listed by the System Automationfor Multiplatforms command lsrpnode for the specifieddomain. You can use either the long or the short namefor a node. A System Automation for Multiplatforms hostagent resource is created for each of the specified nodes.

List of values, value typefor each value: Host nameor IP version 4 address

40Do you want System Automation for Multiplatforms toautomate your SAP router?

SAP router serves as a proxy in a network connectionbetween SAP systems or between SAP systems andexternal networks. If you answer this question with yes,System Automation for Multiplatforms createsautomation resources for the SAP router.

{yes|no}no



You are allowed to use the same prefix as for otherresources, like Java or ABAP.

String SAP_ROUTER





Enter the nodes where you want to automate the SAProuter.Note: Value harvesting is provided for this parameter.

For each of the nodes that are specified, SystemAutomation for Multiplatforms creates SAP routerresources.


40.3This parameter is ignored unless all following conditionsare fulfilled: - Parameter # 2 has the value "IPv4"

Optional; a value for this parameter is only required ifparameter # 40 has the value "yes".

Specify the virtual IPv4 address that clients use toconnect to the SAP router.

This virtual IPv4 address is used to reach the SAP router,independent of the system it is running on.




Specify the virtual IPv6 address that clients use toconnect to the SAP router.

This virtual IPv6 address is used to reach the SAP router,independent of the system it is running on.




Specify the netmask for the SAP router virtual IPaddress.

Enter the netmask for the subnet of the SAP routervirtual IP address. An example for a netmask is"255.255.255.0".





Enter the NetPrefix for the SAP router virtual IP address.An example for a NetPrefix is 80.

Numeric Minimum value:0, maximum value: 128





Enter the network interface for the SAP router IPaddress. The following network interfaces are availableon your local system: remaining part of description isharvested from running system

The available network interface specifies to whichnetwork interfaces the SAP router virtual IP address canbe bound. For AIX, an example on a network interface isen0, for Linux, an example of a network interface is eth0.The same network interface name needs to be availableon all nodes where the SAP router is automated.




Use the location of the NFS device that contains therouting table file. An example for the fully qualified SAProuter routing table file name is /usr/sap/-SAPSID-/SYS/global/saprouttab.

String

41Do you want System Automation for Multiplatforms toautomate the SAP Web Dispatcher?

The SAP Web Dispatcher receives HTTP requests fromthe internet that are targeted for your SAP system. If youanswer this question with yes, System Automation forMultiplatforms creates automation resources for the SAPWeb Dispatcher.

{yes|no}no


Enter the desired prefix for the SAP Web Dispatcherresources.

You are allowed to use the same prefix as for otherresources, like Java or ABAP.

StringSAP_WDISP


Enter the nodes where you want to automate the SAPWeb Dispatcher.Note: Value harvesting is provided for this parameter.

Nodes where the SAP Web Dispatcher is configured andwhere it is automated.






Specify the SAP system ID (SAPSID) for the SAP WebDispatcher.Note: Value harvesting is provided for this parameter.

The SAP system ID (SAPSID) for the SAP WebDispatcher is required to identify the SAP WebDispatcher correctly.



Specify the instance owner user name that is used to runthe start, stop, and monitor commands for SAP WebDispatcher resources.Note: Value harvesting is provided for this parameter.

The instance owner user name is composed of the SAPWeb Dispatcher SID and the suffix adm.



Specify the instance name of the SAP Web Dispatcherinstance, for example 'W00'.Note: Value harvesting is provided for this parameter.

This instance name is used for the instance directory thatcontains all necessary files for the SAP Web Dispatcherinstance.

String


Specify the virtual host name for the SAP WebDispatcher.Note: Value harvesting is provided for this parameter.

This host name is used as a virtual host name for theSAP Web Dispatcher. Enter the same virtual host namethat was used for sapinst

SAPINST_USE_HOSTNAME=-virt_hostname- during the SAPinstallation. Use the same order as for the nodes in oneof the previous questions. For example, if you specifiednode01 first, then you must specify the virtual host namefor your application server on node01 first.

Host name





Optional: a value for this parameter is only required ifparameter # 41 has the value "yes".

Specify the virtual IPv4 address that clients use toconnect to the SAP web dispatcher.Note: Value harvesting is provided for this parameter.

This virtual IPv4 address is used to reach the SAP WebDispatcher, independent of the system it is running on.




Specify the virtual IPv6 address that clients use toconnect to the SAP Web Dispatcher.

This virtual IPv6 address is used to reach the SAP WebDispatcher, independent of the system it is running on.




Specify the netmask for the SAP Web Dispatcher virtualIP address.

Enter the netmask for the subnet of the SAP WebDispatcher virtual IP address. An example for a netmaskis "255.255.255.0".



Optional: a value for this parameter is only required ifparameter # 41 has the value "yes".

Enter the NetPrefix for the SAP Web Dispatcher IPaddress.

Enter the NetPrefix for the SAP Web Dispatcher IPaddress. An example for a NetPrefix is 80.






Specify the network interface on which SAP WebDispatcher virtual IP address is activated on each nodeas alias. The following network interfaces are availableon your local system: remaining part of description isharvested from running system

The available network interface specifies to whichnetwork interfaces the SAP web dispatcher virtual IPaddress can be bound. For AIX, an example is en0. ForLinux, an example is eth0. The same network interfacename needs to be available on all nodes where the SAPWeb Dispatcher instance is automated.


42This parameter is ignored unless all following conditionsare fulfilled: - Parameter # 21 has the value "yes"

If your database is automated with System Automationfor Multiplatforms in the same cluster, do you want tocreate startAfter relationships for your applicationservers?

A startAfter relationship is created for each applicationserver. If you want to create startAfter relationships to adatabase, the database needs to be automated in thesame cluster as SAP.

{yes|no}no


Enter the name of your floating System Automation forMultiplatforms database resource, for exampledb2_db2ax0_0-rs.

String

VerifyingStart and stop your SAP Central Services high availability solution to verify if yourinstallation run successfully. Verify the failover scenarios for planned andunplanned outages.

Starting and stopping the SAP Central Services highavailability solution

You can start your entire SAP system by issuing the command:chrg -o online -s "Name like ’%’"

Enter the following command to display your sample SAP ABAP and Java highavailability policy:lssam -noequ

Output:


root@node1 ~# lssam -noequOffline IBM.ResourceGroup:SAP_ABAP_AX6_ASCS00 Nominal=Offline

|- Offline IBM.ResourceGroup:SAP_ABAP_AX6_ASCS00_ASCS Nominal=Offline|- Offline IBM.Application:SAP_ABAP_AX6_ASCS00_en

|- Offline IBM.Application:SAP_ABAP_AX6_ASCS00_en:node1’- Offline IBM.Application:SAP_ABAP_AX6_ASCS00_en:node2

’- Offline IBM.Application:SAP_ABAP_AX6_ASCS00_ms|- Offline IBM.Application:SAP_ABAP_AX6_ASCS00_ms:node1’- Offline IBM.Application:SAP_ABAP_AX6_ASCS00_ms:node2

’- Offline IBM.ResourceGroup:SAP_ABAP_AX6_ASCS00_SRV Nominal=Offline|- Offline IBM.Application:SAP_ABAP_AX6_ASCS00_sapstartsrv

|- Offline IBM.Application:SAP_ABAP_AX6_ASCS00_sapstartsrv:node1’- Offline IBM.Application:SAP_ABAP_AX6_ASCS00_sapstartsrv:node2

’- Offline IBM.ServiceIP:SAP_ABAP_AX6_ASCS00_ip IP=9.152.135.234|- Offline IBM.ServiceIP:SAP_ABAP_AX6_ASCS00_ip:node1’- Offline IBM.ServiceIP:SAP_ABAP_AX6_ASCS00_ip:node2

Offline IBM.ResourceGroup:SAP_ABAP_AX6_D03_node2 Nominal=Offline|- Offline IBM.ResourceGroup:SAP_ABAP_AX6_D03_node2_AS Nominal=Offline

’- Offline IBM.Application:SAP_ABAP_AX6_D03_node2_as:node2’- Offline IBM.ResourceGroup:SAP_ABAP_AX6_D03_node2_SRV Nominal=Offline

’- Offline IBM.Application:SAP_ABAP_AX6_D03_node2_sapstartsrv:node2Offline IBM.ResourceGroup:SAP_ABAP_AX6_DVEBMGS02_node1 Nominal=Offline

|- Offline IBM.ResourceGroup:SAP_ABAP_AX6_DVEBMGS02_node1_AS Nominal=Offline’- Offline IBM.Application:SAP_ABAP_AX6_DVEBMGS02_node1_as:node1

’- Offline IBM.ResourceGroup:SAP_ABAP_AX6_DVEBMGS02_node1_SRV Nominal=Offline’- Offline IBM.Application:SAP_ABAP_AX6_DVEBMGS02_node1_sapstartsrv:node1

Offline IBM.ResourceGroup:SAP_ABAP_AX6_ERS10 Nominal=Offline|- Offline IBM.ResourceGroup:SAP_ABAP_AX6_ERS10_AERS Nominal=Offline

’- Offline IBM.Application:SAP_ABAP_AX6_ERS10_ers|- Offline IBM.Application:SAP_ABAP_AX6_ERS10_ers:node1’- Offline IBM.Application:SAP_ABAP_AX6_ERS10_ers:node2

’- Offline IBM.ResourceGroup:SAP_ABAP_AX6_ERS10_SRV Nominal=Offline|- Offline IBM.Application:SAP_ABAP_AX6_ERS10_sapstartsrv

|- Offline IBM.Application:SAP_ABAP_AX6_ERS10_sapstartsrv:node1’- Offline IBM.Application:SAP_ABAP_AX6_ERS10_sapstartsrv:node2

’- Offline IBM.ServiceIP:SAP_ABAP_AX6_ERS10_ip IP=9.152.135.235|- Offline IBM.ServiceIP:SAP_ABAP_AX6_ERS10_ip:node1’- Offline IBM.ServiceIP:SAP_ABAP_AX6_ERS10_ip:node2

Offline IBM.ResourceGroup:SAP_HOST_AGENT_node1 Nominal=Offline’- Offline IBM.Application:SAP_HOST_AGENT_node1_ha:node1

Offline IBM.ResourceGroup:SAP_HOST_AGENT_node2 Nominal=Offline’- Offline IBM.Application:SAP_HOST_AGENT_node2_ha:node2

Offline IBM.ResourceGroup:SAP_ROUTER_AX6_SYS_ROUTER Nominal=Offline|- Offline IBM.Application:SAP_ROUTER_AX6_SYS_ROUTER_saprouter

|- Offline IBM.Application:SAP_ROUTER_AX6_SYS_ROUTER_saprouter:node1’- Offline IBM.Application:SAP_ROUTER_AX6_SYS_ROUTER_saprouter:node2

’- Offline IBM.ServiceIP:SAP_ROUTER_AX6_SYS_ROUTER_ip IP=9.152.135.227|- Offline IBM.ServiceIP:SAP_ROUTER_AX6_SYS_ROUTER_ip:node1’- Offline IBM.ServiceIP:SAP_ROUTER_AX6_SYS_ROUTER_ip:node2

Offline IBM.ResourceGroup:SAP_WDISP_AW6_W00 Nominal=Offline|- Offline IBM.ResourceGroup:SAP_WDISP_AW6_W00_WD Nominal=Offline

|- Offline IBM.Application:SAP_WDISP_AW6_W00_sapwebdisp|- Offline IBM.Application:SAP_WDISP_AW6_W00_sapwebdisp:node1’- Offline IBM.Application:SAP_WDISP_AW6_W00_sapwebdisp:node2

’- Offline IBM.ResourceGroup:SAP_WDISP_AW6_W00_SRV Nominal=Offline|- Offline IBM.Application:SAP_WDISP_AW6_W00_sapstartsrv

|- Offline IBM.Application:SAP_WDISP_AW6_W00_sapstartsrv:node1’- Offline IBM.Application:SAP_WDISP_AW6_W00_sapstartsrv:node2

’- Offline IBM.ServiceIP:SAP_WDISP_AW6_W00_ip IP=9.152.135.228|- Offline IBM.ServiceIP:SAP_WDISP_AW6_W00_ip:node1’- Offline IBM.ServiceIP:SAP_WDISP_AW6_W00_ip:node2

Your high availability SAP system is now ready for use.

To stop your entire SAP system use the command:chrg -o offline -s "Name like ’%’"


Failover scenariosThe scenarios cover both planned outages (normal operation, maintenance) andunplanned outages (failures). Each scenario should be verified for properoperation.

Test setup

The following scenarios expect the topology, as defined in the sample policy, to bea cluster with two nodes (node1, node2). We have floating groups for theSAProuter, Web Dispatcher and the Enqueue and Enqueue Replication servers, andfixed groups for one application server on each node.

You can use the lssam command to monitor the reaction of the system to theactions taken.

Scenarios for planned and unplanned outages

In Table 15 and Table 16 on page 81 list the important scenarios for planned andunplanned outages. The preconditions for executing the scenarios are listed abovethe Action, Command and Expected result columns. Each scenario is divided intosteps, where each steps precondition is the successful completion of the precedingaction. The commands to be executed are listed in the Command column. If youhave different naming conventions, you have to adapt the commands accordingly.The last column of the tables lists the expected result.

Refer to “ABAP Central Services (ASCS) high availability policy” on page 30, “JavaCentral Services (SCS) high availability policy” on page 34, and Chapter 5, “Usingthe wizard to configure and activate the policy,” on page 141 for a detailedexplanation of the <placeholders> for the resource names within the commandcolumn.

For the command examples Table 15, replace the ABAP or Java prefix dependingon whether you have ABAB or Java setup. Also replace DVEBMSG and D forABAP application servers with J for Java application servers.

Table 15. Planned Outages

Scenario Action Command Expected result

Normaloperation

Precondition: All groups offline

Start an SAP system chrg -o online -s"Name like ’<PREFIX>_%’"

v ROUTER, WEBDISP, (A)SCSand DVEBMGS/J groups starton node1.

v ERS and D/J groups start onnode2.

Stop SAP system AX6 chrg -o offline -s "Name like’<PREFIX>_AX6_%'"

(A)SCS, ERS, DVEBMGS/J andD/J groups stop.

(Re-)Start SAP systemAX6

chrg -o online -s "Name like’<PREFIX>_AX6_%'"

v (A)SCS, and ABAPDVEBMGS/J groups start onnode1.

v ERS and D/J groups start onnode2.

Stop an SAP system chrg -o offline -s"Name like ’<PREFIX>_%’"

All groups stop.


Table 15. Planned Outages (continued)


Maintenance Precondition:

v ROUTER, WEBDISP, (A)SCS, and DVEBMGS/J groups are online on node1

v ERS and D/J online on node2

Move all resourcesaway from node1 inorder to applyoperating system orhardware maintenance.

samctrl -u a node1 v ROUTER, WEBDISP, (A)SCS,and DVEBMGS/J groups stop.

v DVEBMGS/J groups havestatus “Failed Offline”.

v ROUTER and WEBDISP groupstart on node2.

v (A)SCS group starts on node2.

v ERS terminates.

v ERS groups sacrificed.

Apply maintenance, reboot, etc.

samctrl -u d node1 DVEBMGS/J groups and ERSgroups start on node1.

Stop and restartEnqueue ReplicationServer in order to applySAP maintenance (codeor profile changes).

rgreq -o stop<PREFIX>_<SAPSID>_

<INSTANCE_NAME_ERS>

ERS groups stop.

rgreq -o cancel<PREFIX>_<SAPSID>_

<INSTANCE_NAME_ERS>

ERS groups start on node1.

Move Enqueue Serverin order to apply SAPmaintenance (code orprofile changes).

rgreq -o move<PREFIX>_<SAPSID>_

<INSTANCE_NAME_(A)SCS>

v (A)SCS group stops on node2and restarts on node1.

v ERS on node1 stops after someseconds and is restarted onnode2.

Stop and restartPrimary ApplicationServer in order to applySAP maintenance (codeor profile changes).

rgreq -o stop<PREFIX>_<SAPSID>_

<INSTANCE_NAME>_<NODENAME_PRIMARY>

DVEBMGS/J groups stop.

rgreq -o cancel<PREFIX>_<SAPSID>_

<INSTANCE_NAME>_<NODENAME_PRIMARY>

DVEBMGS/J groups restart onnode1.

Table 16. Unplanned outages

Scenario Simulation action/command Expected result

Precondition:

v ROUTER, WEBDISP, (A)SCS and DVEBMGS/J groups online on node1

v ERS and D groups online on node2

To simulate a software failure create a script using the name killscript. Add the following content:

kill $1 `ps -ef | grep $2 | grep -v grep | awk ’{print $2}’`

Failure of the (A)SCSEnqueue Server

node1:

killscript -9 en.sap<SAPSID>_<INSTANCE_NAME_(A)SCS>

v (A)SCS group stops and restarts onnode2.

v ERS ends after some seconds.

v ERS group stops and restarts onnode1.


Table 16. Unplanned outages (continued)


Failure of the EnqueueReplication Server

node1:

killscript -9 er.sap<SAPSID>_<INSTANCE_NAME_ERS>

ERS group stops and restarts onnode1.

Failure of the MessageServer

node2:

killscript -9 ms.sap<SAPSID>_<INSTANCE_NAME_(A)SCS>

v (A)SCS MS restarts on node2 ifrestart is configured in MS profile.

v (A)SCS MS restarts on node1 ifrestart is not configured in MSprofile.

Failure of an (A)SCSapplication server

node1, ASCS:

killscript -9 dw.sap<SAPSID>_DVEBMGS<ID>

node1, SCS:

killscript -2jc.sap<SAPSID>_J<ID>

DVEBMGS/J application serversrestart on node1.

Failure of the node whereES is running

node2:

rebootv (A)SCS groups are started on

node1.

v ERS on node1 stops after someseconds and is restarted on node2as soon as node2 is available in thecluster again.

SAP Central Services high availability solution checklistThe following table lists all steps that are required to implement the SAP CentralServices high availability solution. You can print the table and check the steps thatyou already completed in the column "Done".

Table 17. SAP high availability solution checklist

# Tasks Reference Done

1.1 Configure automounter forNFS SAP data directory.

“NFS high-availability installationsetup” on page 11

1.2 Setup IP configuration andvirtual host names.

“Prerequisites” on page 18 in“Installing a new ASCS, SCS, orDouble Stack high availability SAPsystem” on page 18

1.3 Set virtual host namesmanually (= IP-AliasAddresses).

“Installing a new ASCS, SCS, orDouble Stack high availability SAPsystem” on page 18

2.1 Install SAP ASCS with virtualhost name.

“Initial installation on primarynode” on page 19 in “Installing anew ASCS, SCS, or Double Stackhigh availability SAP system” onpage 18

2.2 Install SAP Java SCS withvirtual host name.

2.3 Install SAP EnqueueReplication Server usingvirtual host name.


Table 17. SAP high availability solution checklist (continued)

# Tasks Reference Done

2.4 Install SAP database instancewith virtual host name.


2.5 Install SAP PrimaryApplication Server.

2.6 Install SAP AdditionalApplication Servers.

“Initial installation on failovernode” on page 19 in “Installing anew ASCS, SCS, or Double Stackhigh availability SAP system” onpage 18

2.7 Install SAP web Dispatcher(optional)


2.8 Verify SAP installation. “Verifying the initial installation”on page 21 in “Installing a newASCS, SCS, or Double Stack highavailability SAP system” on page18

2.9 Stop SAP.

2.10 Remove IP addressesmanually.

3.1 Install SA MP 4.1 or later. Tivoli System Automation forMultiplatforms Installation andConfiguration Guide

3.2 Create and setup SA MPdomain.

“Setting up the domain” on page25

3.3 Activate SA MP domain. “Setting up the domain” on page25

3.4 Create network tiebreaker. “Setting up the tiebreaker” on page25

3.5 Granting System Automationfor Multiplatforms access.

“Granting read and write access forthe <sapsid>adm” on page 25

3.6 Enabling syslog daemon. “Enabling syslog daemon on AIX”on page 25

3.7 Activate SAP policy featurelicense.

System Automation forMultiplatforms Installation andConfiguration Guide

4.1 Configure database highavailability policies.

“Database high availabilityinstallation setup” on page 10

4.2 Provide the policyparameters.

“SAP Central Services highavailability policy parameters” onpage 43

4.3 Configure SAP policies. “Using the wizard to configure andactivate the SAP Central Serviceshigh availability policy” on page 42

4.4 Start SAP high availability. “Starting and stopping the SAPCentral Services high availabilitysolution” on page 78

4.5 Verify the SAP highavailability solution.

“Verifying” on page 78


Chapter 2. SAP HANA System Replication high availabilitypolicy

You can define SAP HANA System Replication policies for your SAP HANAinstallation. The topics in this section describe the process.

PlanningLearn more about the high availability concepts for an SAP HANA SystemReplication system. The main reason for this high availability setup is to reduce thedowntime of an SAP HANA system in case of software or hardware failures.

The following terms are used:

SAP HANA takeoverA takeover makes sure that an SAP HANA System Replication secondaryinstance can operate as SAP HANA System Replication primary instance.

System Automation failoverAutomatic movement of applications from one node to another nodewithin the same cluster. System Automation uses the term node for onespecific operating system image.

SAP host versus SAP systemSAP uses the term host for one specific operating system image and SAPsystem in general for one or multiple hosts by using the same SAP SystemID (SAPSID).

The high availability solution for SAP HANA System Replication uses SystemAutomation for Multiplatforms to automate all SAP components. SystemAutomation for Multiplatforms detects failed components and restarts them orinitiates a failover. This setup helps to reduce the operational complexity of an SAPenvironment and to avoid operator errors, which result from this complexity.

Single Point of Failure (SPOF)In a distributed or standard SAP installation, the SAP Central Services, thedatabase server, and the NFS server are single points of failures (SPOFs).

For more information, see “Single Point of Failure (SPOF)” on page 1 in the SAPCentral Services high availability policy chapter.

Example of a two-node setupThe minimum hardware setup consists of a two-node System Automation forMultiplatforms domain. The two nodes are either two physical computers or twoLPARs that run on different physical computers.

System Automation can automate the takeover of the SAP HANA SystemReplication setup.


System Automation is installed and a high-availability cluster is set up on theprimary and secondary host. If the primary host has an outage, then SystemAutomation triggers the failover by using the SAP HANA command hdbnsutil-sr_takeover on the secondary host. Then, the IP address is moved to the newprimary host after the old secondary host becomes the new primary host.

When the old primary host comes back online, System Automation integrates theold primary host as new secondary host again by using the SAP HANA commandhdbnsutil -sr_register. The log replication starts from the newprimary host to the new secondary host.

SAP HANA high availability setup optionsA high availability (HA) solution is required to automate the failover of theprimary node, including IP address, to the secondary node.

The following picture shows the SAP HANA System Replication setup with singlenodes.

Data

Primary host Secondary host

SAProuter

SystemAutomation

SystemAutomation

HANA IP

HANA Instance HANA Instance

HANA IP(inactive)

HANA SR Primary(inactive)

HANA SR Secondary(inactive)

HANA SRPrimary

HANA SPSecondary

Data LOGLOG

HA cluster

Virtual IP

LOG replication

LOG replication

same SID+InstanceID

SAP HANA System Replication

Client

Figure 14. Example of a two-node setup


Database high availability installation setupYou can choose between an SAP HANA database installation within the sameSystem Automation for Multiplatforms cluster together with the SAP CentralServices installation or a database installation on a separate cluster.

For more information, see “Database high availability installation setup” on page10 in the Chapter 1, “SAP Central Services high availability policy,” on page 1chapter.

NFS high availability installation setup

For more information, see “NFS high-availability installation setup” on page 11 inthe Chapter 1, “SAP Central Services high availability policy,” on page 1 chapter.

High availability impactLearn more about the impact of various failure scenarios of the SAP systemcomponents, when you use System Automation for Multiplatforms to automaterecovery. Manual recovery actions are minimized which otherwise would causeSAP transactions to timeout and roll back.

For more information, see “High availability impact” on page 12 in the Chapter 1,“SAP Central Services high availability policy,” on page 1 chapter.

SAP HANA documentation referencesThe following table lists references to documentation that are used in thisdocument. Use these links, if you need further details about a specific topic.

Primary host(active)

Secondary host(active, data pre-loaded)

Name Server Name Server

Index Server Index Server

DataDisks

LogDisks

Data Center 1

Clients

Data Center 2

Application Servers

Internal DisksInternal Disks

DataDisks

LogDisks

Transferred byHANA database kernel

Operating System: DNS, hostnames, virtual IPs

HA

So

luti

on

Part

ner

HA

So

lutio

n P

artn

er

Figure 15. System Replication overview with single nodes

Chapter 2. SAP HANA System Replication high availability policy 87

Table 18. Documentation reference

Title Location

SAP HANA Administration Guide http://scn.sap.com/docs/DOC-60368

SAP HANA Server Installation and UpdateGuide

http://help.sap.com/hana/sap_hana_server_installation_guide_en.pdf

SAP HANA in Data Centers - Introduction http://scn.sap.com/docs/DOC-60341

How to Perform System Replication for SAPHANA


Network Recommendations for SAP HANASystem Replication


Introduction to High Availability for SAPHANA


SAP HANA SPS 08 - What’s New?Scale-Out, High Availability and DisasterRecovery


SAP Note: FAQ SAP HANA http://service.sap.com/sap/support/notes/2000003

SAP Note: FAQ SAP HANA SystemReplication


Installing and setting upBefore you set up your SAP HANA environment for high availability, you mustplan for the installation option which best suits your system.

Installing a new HANA database high availability SAP systemInstall a new HANA database high availability SAP system.

This description is based on a two node cluster architecture with a primary and afailover node.

For more information about HANA database high availability SAP system setups,see “SAP HANA high availability setup options” on page 86.

PrerequisitesObserve the following prerequisites before you start the installation of the HANAdatabase high availability SAP system.

The System Automation for Multiplatforms high availability policy manages theSAP HANA System Replication Single-Node setup. SAP HANA on Power isrequired. The SAP HANA Scale-Out (Multi-Node Primary) setup and multitierSystem Replication is not supported.

The following SAP restrictions exist for an SAP HANA System Replication setup:v Host names for primary instance and secondary instance cannot be the same.v SAPSID and Instance numbers for primary and secondary instance must be the

same.v Primary instance needs to have a full backup before you can set up a secondary

instance system replication.

The restrictions are documented in SAP Note: 1834153 - HANA high availabilitydisaster tolerance config.














https://websmp230.sap-ag.de/sap/support/notes/1834153

https://websmp230.sap-ag.de/sap/support/notes/1834153

Initial installation on primary and secondary node

Follow the SAP documentation SAP HANA Server Installation and Update Guideto install the SAP HANA database on the primary and secondary node. Ensure tofollow the prerequisites.

Setup System Replication between primary and secondary nodeFollow the SAP documentation How to Perform System Replication for SAPHANA, Chapter “Enable system replication” to setup the SAP HANA SystemReplication on the primary and secondary node.

Configuring SAP profilesConfigure the SAP profiles to comply with the high availability solution providedby System Automation for Multiplatforms.v Disable autostart of all SAP instances in all their profiles by commenting the

line Autostart = 1.v The SAP high availability connector must be enabled in the default profile. This

step requires that System Automation for Multiplatforms is already installed onthe cluster nodes, so it probably must be done later. For more information abouthow to enable the SAP high availability connector for your platform, see“Enabling the SAP high availability Connector” on page 29.

Verifying the initial installation

Follow the SAP documentation How to Perform System Replication for SAPHANA, Chapter “System Replication status checks” to verify the correct setup ofthe SAP HANA System Replication.

Installing and setting up System Automation forMultiplatforms

Find out how to install and set up System Automation for Multiplatforms for anautomated and highly available SAP system.

For more information, see “Installing and setting up System Automation forMultiplatforms” on page 24 in the chapter Chapter 1, “SAP Central Services highavailability policy,” on page 1.

Installing and setting up the SAP high availability policyfeature

Find out how to install and set up the SAP high availability policy feature.

For more information, see “Installing and setting up the SAP high availabilitypolicy feature” on page 26 in the Chapter 1, “SAP Central Services high availabilitypolicy,” on page 1.

Installing a new ASCS, SCS, or Double Stack high availabilitySAP system

Find out how to run a new installation of an ASCS, SCS, or Double Stack highavailability SAP system.

For more information, see “Installing a new ASCS, SCS, or Double Stack highavailability SAP system” on page 18 in the Chapter 1, “SAP Central Services highavailability policy,” on page 1 chapter.







Configuring and activatingThe following topics describe how to configure the SAP high availability policywith the sampolicy wizard. You can activate your SAP high availability solutionafter defining the policy with the correct parameter values.

About this task

For more information about how to invoke the sampolicy wizard, see SystemAutomation for Multiplatforms Reference Guide.

SAP HANA System Replication automationThe SAP HANA high availability policy defines all SAP components as resourcesand starts and stops them in a well-defined sequence to provide high availabilityfor your SAP HANA system.

These components must be specified as automated resources in System Automationfor Multiplatforms.

The following tables show each part, which is to be automated and thecorresponding System Automation resource class and resource name. It alsoindicates dependencies among the resources and how each component is connectedto the automated resource.

Table 19. SAP HANA System Replication parts to automate

Component Resource class and name* Description

HDB Primary instance Resource class

IBM.Application

Resource name

SAP_HDB_PH1_HDB00_sr_primary_hdb

SAP HANA SR Primary Instance that isrepresented by System Automation forMultiplatforms floating resource.

Resource dependency:

StartAfter sapstartsrv

HDB Secondary Instance

HDB Primary Instance

Network Interface

Network Interface

HDB Network Equivalency HDB SR primary group HDB Instance Agent group

HDB ServiceIP HDB Instance Agent

SA

AD

SA

DO

Status Dashboard

HDB SR secondary group

SA

SO

SA

SO

hdb instance status

primary configured

primary last online

primary replication active

primary replication syncing

secondary configured

secondary register in progress

secondary takeover in progress

Figure 16. Resources of the SAP HANA high availability policy


Table 19. SAP HANA System Replication parts to automate (continued)


HDB Secondaryinstance

Resource class

IBM.Application

Resource name

SAP_HDB_PH1_HDB00_sr_secondary_hdb

SAP HANA SR Secondary Instance that isrepresented by System Automation forMultiplatforms floating resource.


StartAfter sapstartsrv

StartAfter sr_primary_hdb

HDB ServiceIP Resource class

IBM.ServiceIP

Resource name

SAP_HDB_PH1_HDB00_sr_primary_ip

IP address to access Primary Instance fromthe SAP clients (SAP Application Servers)instance that is represented by SystemAutomation floating resource.


StartAfter sr_primary_hdb

DependsOn NETIF

HDB Instance Agent Resource class

IBM.Application

Resource name

SAP_HDB_PH1_HDB00_sapstartsrv

SAP web service framework that is usedto start, stop, and monitorhdbdaemoninstance that is represented by SystemAutomation concurrent resource.


StopAfter sr_primary_hdb

StopAfter sr_secondary_hdb

HDB NetworkEquivalency

Resource class

IBM.Equivalency

Resource name

SAP_HDB_PH1_HDB00_NETIF

All members of IBM.Equivalency are resourcesof the class IBM.NetworkInterface, for exampleeth0.

Selecting the network adapters to be usedfor IBM.ServiceIP resource.

HDB instance status Resource class

IBM.Test

Resource name

SAP_HDB_PH1_HDB00_sr_hdb

Status indicates the HDB instance statusreported by the sapcontrol utility.

Primary configured Resource class

IBM.Test

Resource name

SAP_HDB_PH1_HDB00_sr_primary_configured

Status Online indicates the currentconfigured primary node.


Table 19. SAP HANA System Replication parts to automate (continued)


Primary last online Resource class

IBM.Test

Resource name

SAP_HDB_PH1_HDB00_sr_primary_last_online

Status Online indicates the last configuredprimary node. Only used if both instancesare Offline.

Primary replicationactive

Resource class

IBM.Test

Resource name

SAP_HDB_PH1_HDB00_

sr_primary_replication_active

Status Online indicates the replication iscurrently active between primary andsecondary instance.

Primary replicationsyncing

Resource class

IBM.Test

Resource name

SAP_HDB_PH1_HDB00_

sr_primary_replication_syncing

Status Online indicates the replication iscurrently in status syncing (Savepointpending).

Secondary configured Resource class

IBM.Test

Resource name

SAP_HDB_PH1_HDB00_

sr_secondary_configured

Status Online indicates the currentconfigured secondary node.

Secondary register inprogress

Resource class

IBM.Test

Resource name

SAP_HDB_PH1_HDB00_

sr_secondary_register_in_progress

Status Online indicates the hdb commandsr_register is currently running.

Secondary takeover inprogress

Resource class

IBM.Test

Resource name

SAP_HDB_PH1_HDB00_

sr_secondary_takeover_in_progress

Status Online indicates the hdb commandsr_takeover is currently running.

Note: Resource names are examples. You can choose the names when youconfigure the resources.

The following components of the software stack in an SAP HANA SystemReplication installation need to be automated:


v Primary and Secondary Host with the following processes on each host:hdbdaemon manages the following subprocesses:– hdbindexserver

– hdbnameserver

– hdbxsengine

– hdbwebdispatcher

– hdbcompileserver

– hdbpreprocessor

v sapstartsrv start, stop, monitor for hdbdaemonv IP address to access Primary Host

Using the wizard to configure and activate the SAP HANAhigh availability policy

This SAP HANA high availability policy consists of a policy template that istailored by using the sampolicy wizard.

To configure the template, run the following command:sampolicy –w templateFileName

Specify the following fully qualified XML template file name:

Table 20. Location for the SAP HANA high availability XML template file

SAP high availability setup XML template file

HANA System Replication /usr/sbin/rsct/sapolicies/sap/sap_HDB_SR_v41.tmpl.xml

The wizard stores all modifications to the policy pool. The default location of yourpolicy pool is /etc/opt/IBM/tsamp/sam/policyPool. Run the wizard by entering thefollowing command:sampolicy -w /usr/sbin/rsct/sapolicies/sap/sap_HDB_SR_v41.tmpl.xml

The results are stored to/etc/opt/IBM/tsamp/sam/policyPool/sap_HDB_SR_v41.tmpl.xml

Next time you start the wizard, use the file that is stored in the policy pool. If youwant to start the wizard for the second time, enter the following command:sampolicy -w /etc/opt/IBM/tsamp/sam/policyPool/sap_HDB_SR_v41.tmpl.xml

For a detailed description of the sampolicy wizard, see Chapter 5, “Using thewizard to configure and activate the policy,” on page 141.

Note:

v The SAP HANA database should be offline before the policy is activated.Otherwise, the database gets stopped with the policy activation.

v Ensure to authorize the <sid>adm user to run System Automation forMultiplatforms cluster commands. For more information, see “Granting read andwrite access for the <sapsid>adm” on page 25.

v Ensure to have the IP address that is used for the IBM.ServiceIP deactivated, ifit was manually added for testing, before the new policy is activated.


SAP HANA high availability policy parametersThis section provides a list of all parameters that have to be specified for the SAPHANA high availability policy.

An HTML file that contains all parameter descriptions and all currently definedvalues is generated each time the wizard is started and finished using the 0 (zero)selection option. For more information, see “Closing the wizard” on page 155.

Table 21. SAP HANA System Replication HA policy parameters




String

2Enter the nodes where you want to automate SAP HANA.Note: Value harvesting is provided for this parameter.

These nodes must be listed by the SA MP command lsrpnode forthe specified domain. You can use either the long or the shortname for a node. An SA MP SAP HANA resource will be createdfor each of the specified nodes. The first node in list will be usedas primary instance node. The node name is also used for theremoteHost parameter within the hdbnsutil command to enableSystem Replication for the secondary node.






IPv4

IPv6

IPv4



Specify the IPv4 address that will be used as tiebreaker.

This IPv4 address is used to resolve cluster quorum in case of acluster split.




Specify the virtual IPv4 address that clients will use to connect to SAPHANA.

This virtual IPv4 address is used to reach SAP HANA,independent of the system it is currently running on.



Table 21. SAP HANA System Replication HA policy parameters (continued)




Specify the virtual IPv6 address that clients will use to connect to SAPHANA.

This virtual IPv6 address is used to reach SAP HANA,independent of the system it is currently running on.




Specify the netmask for the SAP HANA virtual IP address.

Enter the netmask for the subnet of the SAP HANA virtual IPaddress. An example for a netmask is "255.255.255.0".


255.255.255.0



Enter the NetPrefix for the SAP HANA virtual IP address. An examplefor a NetPrefix is 80.

Value type: NumericMinimum value: 0,maximum value: 128

9Enter the network interface for the SAP HANA IP address.(remainingpart of description is harvested from running system)

The available network interface specifies to which networkinterfaces the SAP HANA virtual IP address can be bound, for AIXan example is en0, for Linux, an example is eth0. The samenetwork interface name needs to be available on all nodes wherethe SAP HANA will be automated.

Value type: String (plusadditional value checking)



Value type: StringMinimum number ofcharacters: 3, maximumnumber of characters: 3(plus additional valuechecking)


The default SAP instance owner user's name is composed of theSAP SID (in lower case) and the suffix adm.



Table 21. SAP HANA System Replication HA policy parameters (continued)


12Enter your desired prefix for all SAP HANA resources.

This prefix will be used as a prefix for all SA MP resources thatcover SAP HANA, for example SAP_HDB. For later operationaltasks, the prefix can be used to start and stop resources with thesame prefix with one single command. You may consider toencode the SAP solution name, e.g. PI, ECC or SCM, which wouldresult in a prefix like PI_HDB.

Value type: StringSAP_HDB

13Specify the instance name of the SAP HANA instance.Note: Value harvesting is provided for this parameter.

This instance name identifies the SAP HANA instance on allcluster nodes. A sample instance name is HDB00.

Value type: StringMinimum number ofcharacters: 5, maximumnumber of characters: 5(plus additional valuechecking)

14Specify all site names of your SAP HANA nodes. (remaining part ofdescription is harvested from running system)Note: Value harvesting is provided for this parameter.

The site names are used to enable System Replication for the SAPHANA instance. Use the same order as for the list of nodes. Asample site name is dcsite1.

List of values, value typefor each value: String

15Select the log replication mode for SAP HANA System Replication.Note: Value harvesting is provided for this parameter.

The SAP HANA log replication mode specifies how to send loginformation to the secondary instance. The modes synchronous(sync), synchronous in memory (syncmem) or asynchronous (async)can be set.

Value type: String

VerifyingStart and stop your SAP high availability solution to verify if your installation runssuccessfully. Verify the failover scenarios for planned and unplanned outages.

Starting and stopping the SAP HANA high availability solutionStart your entire SAP system by entering the command:chrg -o online -s "Name like ’%’"

Enter the following command to display your sample SAP HANA availabilitypolicy:lssam

Output:[root@p6sa47]# lssamOnline IBM.ResourceGroup:SAP_HDB_PH1_HDB00_sapstartsrv_rg Nominal=Online

’- Online IBM.Application:SAP_HDB_PH1_HDB00_sapstartsrv|- Online IBM.Application:SAP_HDB_PH1_HDB00_sapstartsrv:p6sa47’- Online IBM.Application:SAP_HDB_PH1_HDB00_sapstartsrv:p6sa48

Online IBM.ResourceGroup:SAP_HDB_PH1_HDB00_sr_primary_rg Nominal=Online|- Online IBM.Application:SAP_HDB_PH1_HDB00_sr_primary_hdb

|- Offline IBM.Application:SAP_HDB_PH1_HDB00_sr_primary_hdb:p6sa47’- Online IBM.Application:SAP_HDB_PH1_HDB00_sr_primary_hdb:p6sa48


’- Online IBM.ServiceIP:SAP_HDB_PH1_HDB00_sr_primary_ip|- Offline IBM.ServiceIP:SAP_HDB_PH1_HDB00_sr_primary_ip:p6sa47’- Online IBM.ServiceIP:SAP_HDB_PH1_HDB00_sr_primary_ip:p6sa48

Online IBM.ResourceGroup:SAP_HDB_PH1_HDB00_sr_secondary_rg Nominal=Online’- Online IBM.Application:SAP_HDB_PH1_HDB00_sr_secondary_hdb

|- Online IBM.Application:SAP_HDB_PH1_HDB00_sr_secondary_hdb:p6sa47’- Ineligible IBM.Application:SAP_HDB_PH1_HDB00_sr_secondary_hdb:p6sa48

Online IBM.Equivalency:SAP_HDB_PH1_HDB00_NETIF|- Online IBM.NetworkInterface:eth0:p6sa48’- Online IBM.NetworkInterface:eth0:p6sa47

Online IBM.Equivalency:SAP_HDB_PH1_HDB00_STATUS_DASHBOARD|- Online IBM.Test:SAP_HDB_PH1_HDB00_sr_hdb:p6sa47|- Online IBM.Test:SAP_HDB_PH1_HDB00_sr_hdb:p6sa48|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_configured:p6sa47|- Online IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_configured:p6sa48|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_last_online:p6sa47|- Online IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_last_online:p6sa48|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_replication_active:p6sa47|- Online IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_replication_active:p6sa48|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_replication_syncing:p6sa47|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_replication_syncing:p6sa48|- Online IBM.Test:SAP_HDB_PH1_HDB00_sr_secondary_configured:p6sa47|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_secondary_configured:p6sa48|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_secondary_register_in_progress:p6sa47|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_secondary_register_in_progress:p6sa48|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_secondary_takeover_in_progress:p6sa47’- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_secondary_takeover_in_progress:p6sa48

Your high availability SAP system is now ready for use.

To stop your entire SAP system, use the command:chrg -o offline -s "Name like ’%’"

Output:[root@p6sa47]# lssamOffline IBM.ResourceGroup:SAP_HDB_PH1_HDB00_sapstartsrv_rg Nominal=Offline

’- Offline IBM.Application:SAP_HDB_PH1_HDB00_sapstartsrv|- Offline IBM.Application:SAP_HDB_PH1_HDB00_sapstartsrv:p6sa47’- Offline IBM.Application:SAP_HDB_PH1_HDB00_sapstartsrv:p6sa48

Offline IBM.ResourceGroup:SAP_HDB_PH1_HDB00_sr_primary_rg Nominal=Offline|- Ineligible IBM.Application:SAP_HDB_PH1_HDB00_sr_primary_hdb

|- Ineligible IBM.Application:SAP_HDB_PH1_HDB00_sr_primary_hdb:p6sa47’- Offline IBM.Application:SAP_HDB_PH1_HDB00_sr_primary_hdb:p6sa48

’- Offline IBM.ServiceIP:SAP_HDB_PH1_HDB00_sr_primary_ip|- Offline IBM.ServiceIP:SAP_HDB_PH1_HDB00_sr_primary_ip:p6sa47’- Offline IBM.ServiceIP:SAP_HDB_PH1_HDB00_sr_primary_ip:p6sa48

Offline IBM.ResourceGroup:SAP_HDB_PH1_HDB00_sr_secondary_rg Nominal=Offline’- Ineligible IBM.Application:SAP_HDB_PH1_HDB00_sr_secondary_hdb

|- Offline IBM.Application:SAP_HDB_PH1_HDB00_sr_secondary_hdb:p6sa47’- Ineligible IBM.Application:SAP_HDB_PH1_HDB00_sr_secondary_hdb:p6sa48

Online IBM.Equivalency:SAP_HDB_PH1_HDB00_NETIF|- Online IBM.NetworkInterface:eth0:p6sa48’- Online IBM.NetworkInterface:eth0:p6sa47

Online IBM.Equivalency:SAP_HDB_PH1_HDB00_STATUS_DASHBOARD|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_hdb:p6sa47|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_hdb:p6sa48|- Online IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_configured:p6sa47|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_configured:p6sa48|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_replication_active:p6sa47|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_replication_active:p6sa48|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_replication_syncing:p6sa47|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_primary_replication_syncing:p6sa48|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_secondary_configured:p6sa47|- Online IBM.Test:SAP_HDB_PH1_HDB00_sr_secondary_configured:p6sa48|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_secondary_register_in_progress:p6sa47


|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_secondary_register_in_progress:p6sa48|- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_secondary_takeover_in_progress:p6sa47’- Offline IBM.Test:SAP_HDB_PH1_HDB00_sr_secondary_takeover_in_progress:p6sa48

Testing SAP HANA System Replication failover scenariosThe scenarios cover both planned outages (normal operation, maintenance) andunplanned outages (failures). Verify each scenario for proper operation.

Test setup

The following scenarios expect the topology, as defined in the sample policy, to bea cluster with two nodes (node1, node2). We have floating groups for the primaryand secondary hdb instance, and one fixed group for the sapstartsrv process oneach node.

You can use the lssam command to monitor the reaction of the system to theactions taken.

Scenarios for planned and unplanned outages

Table 22 and Table 23 on page 99 list the important scenarios for planned andunplanned outages. The preconditions for executing the scenarios are listed abovethe Action, Command and Expected result columns. Each scenario is divided intosteps, where each steps precondition is the successful completion of the precedingaction. The commands to be executed are listed in the Command column. If youhave different naming conventions, you have to adapt the commands accordingly.The last column of the tables lists the expected result.

In the command examples in Table 22, replace the <PREFIX>, <SAPSID>, and<INSTANCE_NAME_HDB> depending on your environment.

Table 22. Planned Outages


Normaloperation

Precondition: All groups offline

Start an SAP HANAsystem

chrg -o online -s"Name like ’<PREFIX>_%’"

v sr_primary_rg group start onnode1

v sr_secondary_rg group starton node2.

Stop SAP HANAsystem <SAPSID>

chrg -o offline -s"Name like ’<PREFIX>_<SAPSID>_%'"

Primary and secondary groupsstop.

(Re-)Start SAP HANAsystem <SAPSID>

chrg -o online -s "Namelike’<PREFIX>_<SAPSID>_%'"

v sr_primary_rg group start onnode1.

v sr_secondary_rg group starton node2.

Stop an SAP HANAsystem

chrg -o offline -s"Name like ’<PREFIX>_%’"

All groups stop.


Table 22. Planned Outages (continued)


Maintenance Precondition:

v sr_primary_rg group is online on node1

v sr_secondary_rg group is online on node2

Move all resourcesaway from node1 toapply operating systemor hardwaremaintenance.

samctrl -u a node1 v sr_primary_rg group stop onnode1.

v sr_primary_rg group start onnode2.

v sr_secondary_rg group getssacrificed.

Apply maintenance, restart, and so on.

samctrl -u d node1 sr_secondary_rg group start onnode1.

Stop and restartsecondary hdb instanceto apply SAP

rgreq -ostop <PREFIX>_<SAPSID>_

Secondary hdb instance groupsstop.

Maintenance (code orprofile changes).

<INSTANCE_NAME_HDB>_ sr_secondary_rg

rgreq -o cancel <PREFIX>_<SAPSID>_<INSTANCE_NAME_HDB>_ sr_secondary_rg

Secondary hdb instance groupsstart on node1.

Move primary hdbinstance to apply SAPmaintenance (code orprofile changes).

rgreq -omove <PREFIX>_<SAPSID>_<INSTANCE_NAME_HDB>_ sr_primary_rg

v Primary hdb instance groupstops on node2 and restarts onnode1.

v Secondary hdb instance onnode1 stops after someseconds and is restarted onnode2.

Table 23. Unplanned outages


Precondition:

v sr_primary_rg group is online on node1

v sr_secondary_rg group is online on node2

To simulate a software failure, create a script by using the name killscript. Add the following content:

kill $1 `ps -ef | grep $2 | grep -v grep | awk ’{print $2}’`

Failure of the primary hdbinstance.

node1:

killscript -9 hdb.sap<SAPSID>v Primary hdb instance group stops

and restarts on node2.

v Secondary hdb instance group stopsand restarts on node1.

Failure of thesecondary hdb instance.

node1:

killscript -9 hdb.sap<SAPSID>

Secondary hdb instance group stopsand restarts on node1.

Failure of the node whereprimary hdb instance isrunning.

node2:

reboot -fnv Primary hdb instance groups are

started on node1.

v Secondary hdb instance on node1stops after some seconds and isrestarted on node2 as soon asnode2 is available in the clusteragain.


Table 23. Unplanned outages (continued)


Failure of the node wheresecondary hdb instance isrunning.

node2:

reboot -fnv Node2 restarts and secondary hdb

instance is restarted on node2 assoon as node2 is available in thecluster again.

Note: In any case, after failed over to the other node, the HANA clients reconnectto the SAP HANA primary instance and run any task.


Chapter 3. NFS high availability policy

You can define high availability policies for your NFS server. The topics in thissection describe the process.

The Network File System (NFS) is an application to access files exported by anNFS server to an NFS client as if they existed locally.

System Automation can automate a failover from one NFS server to another NFSserver. The following must be automated to make an NFS server highly available:v NFS server processesv Virtual IP address used to connect to the NFS server by remote clients.v Shared disk for the data to be exported by the NFS server.v Shared disk for the NFS server specific configuration files, located in the

following directory:– Linux: /var/lib/nfs– AIX: /etc and /var/statmon

These shared disk(s) must be accessible from all the nodes that potentially host theNFS server in the cluster. These parts must be specified as automated resources inSystem Automation. You can find an overview on the parts that belong to thepolicy in “Configuring and activating the NFS server high availability policy” onpage 109.

The figure below shows the basic application parts involved in running an NFSserver:

An NFS server runs on node-1 in a cluster. It listens to an IP address and a port forrequests from NFS clients and exports data to those clients in NFS format. IfNode-1 breaks down, operations can be restored by starting the NFS server onNode-2. It must listen to the same IP address to serve the clients without anyinterruption. Shared data that was accessible for NFS clients exclusively fromNode-1 must now be exclusively available from Node-2.

NFS clients

Node 1

NFS server

IP address

Node 2

NFS server

IP address

shared data

Figure 17. NFS server parts


Planning the NFS server automationBefore you implement NFS server automation, you must fulfill specificprerequisites and infrastructure requirements.

Prerequisites

Virtual IP addressFor each virtual IP address that is defined in the high availability policy, anequivalency of network interfaces will be defined. Only network interfaceswith the same network interface name can be part of the equivalency.Ensure that the network interfaces you want to use have the same name oneach system.

Infrastructure requirements

To allow an NFS server to failover between different systems the followingrequirements against the system infrastructure have to be fulfilled.

At least one shared disk is required for the NFS server high availabilityconfiguration

One or more disks contain the data that is exported by the NFS server.They contain configuration files that the NFS server requires to runproperly and behave identically on all systems.

On a Linux machine the NFS server uses files located in /var/lib/nfs tostore information about any NFS clients which have mounted a file systemthat is exported by this NFS server and information about file locks.

On AIX this information is contained in the files /etc/xtab, /etc/exports,/etc/rmtab and the directory /var/statmon.

This information must persist after a node failover to the NFS server,otherwise the NFS clients lose their connections to the NFS server. TheNFS client information is stored in a small file system on a shared disk tokeep this information in sync on all nodes. These disk(s) must be accessibleby all nodes that are allowed to host the NFS server, for example if thenodes are connected to a storage device. Each shared disk has a uniquedevice name and number. The unique device name and number must bethe same on all nodes. Depending on your operating system, theserequirements are:v Linux: To transparently fail over the NFS server from one host to

another on a Linux machine, the shared disk(s) must be connected toeach server with the same unique major and minor number. SystemAutomation for Multiplatforms can only identify the shared disk if allnodes identify the shared disk with the same unique number. Theshared disk must have the same device name and number on all nodes,for example /dev/sdc1. You can determine the major and minor numberas follows:#cd /dev#ls –l sd*The last command will return an output like this<systemname>:/dev # ls -l sd*brw-rw---- 1 root disk 8, 0 2011-01-19 14:25 sdabrw-rw---- 1 root disk 8, 1 2011-01-19 14:25 sda1brw-rw---- 1 root disk 8, 2 2011-01-19 14:25 sda2brw-rw---- 1 root disk 8, 3 2011-01-19 14:25 sda3brw-rw---- 1 root disk 8, 4 2011-01-19 14:25 sda4


For sda, the major number is 8 and the minor number is 0.For each file system that resides on such a shared disk, a mount pointmust be defined within /etc/fstab. For each mount point, anIBM.AgFileSystem resource must exist, either harvested by the StorageRMresource manager or user-defined. It is recommended to specify a labelfor the file system, because this label is taken as resource name forharvested IBM.AgFileSystem resources. For more information about theStorageRM, see System Automation for Multiplatforms Administrator's andUser's Guide.

v AIX: On an AIX machine the volume group MAJOR NUMBER must beidentical to the volume group containing the shared data on all of thenodes. For details about the MAJOR NUMBER, see the lsvg commandman pages. Make sure the setup is done accordingly and keep in mindthat this is also important for later changes of the configuration of theservers. This applies for every shared disk hosting data to be exportedby the NFS server.

NFS version 3 and 4 only are supported for the NFS high availability policyNFS version 4 authentication using Kerberos is not supported.

A single node can run only one NFS serverIf you want to run consolidation projects, consolidate the exports of severalNFS servers to one NFS server instead of placing several NFS servers on asingle machine. If there is more than one NFS server resource within aSystem Automation cluster, then these resources must have anAntiCollocated relationship to ensure that these NFS servers are neverstarted on the same node at a time.

Installing and setting up the NFS server for automationInstalling and setting up the NFS server for automation is a two step process. First,you must install the NFS server software and then configure the individual NFSserver components for automation.

The NFS server software is delivered as part of the operating system. Make surethat all required NFS server packages are installed on all of the systems where youwant to automate the NFS server.

For information about configuring and activating the NFS server high availabilitypolicy, see Chapter 5, “Using the wizard to configure and activate the policy,” onpage 141.

Configuring the NFS server componentsTo automate an NFS server, you must configure the:v “NFS server processes” on page 104v Configuration data required to manage the connection between the NFS server

and clients: “Using a shared disk for configuration data” on page 105v Data that will be exported to NFS clients: “Using a shared disk for exported

data” on page 107v “Configuring file systems within the same volume group on AIX” on page 108v “NFS server virtual IP address” on page 108

Figure 18 on page 104 shows the components you must configure in order toautomate an NFS server.

Chapter 3. NFS high availability policy 103

As shown in Figure 18, shared disks are required for that configuration data that isrequired to manage the:v Connection between the NFS server and clients.v Data to be exported to NFS clients.

NFS server processesNFS server processes are a part of the NFS server high availability policy that arerequired to get automated.

Linux systems

NFS server daemons are controlled by an entry in the /etc/init.d or /etc/rc.confto be started after booting the host. To automate the NFS server, SystemAutomation must be the only one in control of starting and stopping the NFSserver. Therefore, remove the respective references to the daemons from/etc/init.d or /etc/rc.conf:v SuSE distribution: Use the runlevel editor within yast.v RedHat distribution: Use the service command.

This autostart disablement of NFS daemons must be performed on each node thatwill host the high availability NFS server instance.

NFS clients

Node 1

NFS server

IP address

Node 2

NFS server

IP address

shared disk to store configuration data

shared disk to store data to be exported

Virtual IP address

NFS server processes

Figure 18. Components to be configured when automating an NFS server


For the NFS server processes, the NFS server high availability policy uses a controlscript in an IBM.Application resource to start, stop, and monitor the NFS serverprocesses. This script executes the following existing NFS server scripts:1. Start /etc/init.d/nfslock. It will only be executed if it exists.2. Start /etc/init.d/nfsserver for SuSE distributions and /etc/init.d/nfs for

RedHat distributions.

For NFS version 4, it will also be ensured that the idmapd daemon is started andstopped, and the pipefs file system is mounted and unmounted.

The control script creates a modified version of the file /etc/init.d/nfslock,which is used to start and stop the rpc.statd. The original version will be savedunder /etc/init.d/nfslock.SA_Linux.org.save. The only modification performedis to add to the start of rpc.statd the option –n followed by the host name for theNFS server IP address. This modification is performed on start-up of the NFSserver resource and will be removed when the NFS server is stopped, where theoriginal file will be copied back.

Note: This works only if the host name for the NFS server is found in the DNSserver or in /etc/hosts, otherwise the script is not able to determine the host nameand the rpc.statd can't be started correctly. This is not a concern if an IP addressis used rather than a host name.

AIX systems

The NFS server sub systems are controlled by the System Resource Controller(SRC), and the start-up behavior can be adjusted using smit(ty). To automate theNFS server, System Automation must be the only one in control of starting andstopping the NFS server. Use smit(ty) to prevent the following NFS server subsystems from being start-up at boot time:v nfsd

v rpc.mountd

For those NFS server processes, the NFS server HA policy uses a script in anIBM.Application resource to start, stop, and monitor the NFS server processes.

Using a shared disk for configuration dataThe NFS server relies on a number of configuration files to behave identically oneach node and maintain the connection to its clients. These configuration filesdepend on the operating system.

Linux systems

Data that is required for the management of the NFS server is normally storedlocally under /var/lib/nfs on each NFS server node. To make an NFS serverhighly available requires this data to be stored on a small file system, that is alsolocated on a shared disk accessible by all the nodes potentially hosting the NFSserver.

The following steps must be performed to put the data from /var/lib/nfs on a filesystem on a shared disk. Perform the following steps on one node only:1. Create a partition of about 10 MB on a shared disk using the appropriate

partitioning tool for your platform, for example fdisk for Linux on xSeries.2. Format the disk with the file system type of your choice, for example ext3.


3. Perform this step on every node: Create the mount point for this file system onevery node, for example /varlibnfs.#mkdir /varlibnfs

4. Perform the following step on one node only: Mount the newly created filesystem on the node, copy all files from /var/lib/nfs to /varlibnfs, andunmount the /varlibnfs file system.#mount -t ext3 /varlibnfs#cp /var/lib/nfs/* /varlibnfs#umount /varlibnfs

5. Perform this step on every node: Change to the /var/lib directory and movethe nfs sub directory to nfs.original for example. Create a link to the/varlibnfs directory.#cd /var/lib#mv nfs nfs.original#ln –s /varlibnfs nfs

System Automation for Multiplatforms does not keep the configurations of theexported directories for the NFS server in sync on all nodes in the domain wherethe NFS server can run. The configuration of the exported directories is stored inthe file /etc/exports. To ensure that this file stays in sync between all nodes thathost the NFS server, there are two options.v Keep /etc/exports in sync by copying it to each node after a change.v Store the /etc/exports file also in the file system on the shared disk for the

configuration data. This requires the creation of a link from the /etc/exports fileto the file in on the shared disk, similar as for the /var/lib/nfs directory asdescribed in 3.

Note: When creating file systems on the shared disk, specify mount optionmount=false to prevent the file system from being mounted during system start.

AIX systems

Data that is required for the management of the NFS server is stored locally in thedirectory /etc and /var/statmon on each NFS server node. To make an NFS serverhighly available this data has to be stored on a small file system, that is alsolocated on a shared disk accessible by all the nodes potentially hosting the NFSserver. The following steps must be performed to put the data from /etc and/var/statmon on a file system on a shared disk. Perform the following steps ononly one node:1. Create a file system of about 10 MB on a shared disk.2. Create the mount point for this file system, for example /nfsctrl # mkdir

/nfsctrl

3. Mount the newly created file system on one node.4. Copy the files /etc/xtab, /etc/rmtab, /var/statmon and optionally

/etc/exports to the directory /nfsctrl.5. Perform this step on every node: Change to the /etc directory and create

backups of the NFS controlling files by renaming them to for example*.original and create links to the files in the /nfsctrl directory, enter:# cd /etc# mv xtab xtab.original# mv rmtab rmtab.original# ln –s /nfsctrl/xtab xtab


# ln –s /nfsctrl/rmtab rmtab# cd /var# mv statmon statmon.original# ln -s /nfsctrl/statmon statmon

If the NFS server was not started at least once, the files rmtab and xtab do notexist yet.

Ensure that the copied files have the same access rights, the same owner, andbelong to the same group as the source files. Make sure that the symbolic linkshave the same user ID and group settings as the source files using the chown -hcommand to change the link. System Automation does not keep the configurationsof the exported directories for the NFS server in sync on the different nodes in thedomain where the NFS server can run. The configuration of the exporteddirectories is stored in the file /etc/exports. To ensure that this file stays in syncbetween all nodes that host the NFS server, there are two options.v Keep /etc/exports in sync by copying it to each node after a change.v Store the /etc/exports file also in the file system on the shared disk for the

configuration data. This requires the creation of a link from the /etc/exports fileto the file in on the shared disk, similar as for the /nfsctrl directory asdescribed in 5 on page 106

Note: When creating mount points, specify mount option noauto to prevent thefile system from being mounted during system start. In the smitty, command,answer “No” to the question “Mount AUTOMATICALLY at system restart?”

For NFS V4, there is an additional file that contains the local NFS domain of thesystem /etc/nfs/local_domain. You either need to ensure that the local NFSdomain is configured on all nodes hosting the NFS server, or you need to movethe /etc/nfs/local_domain file to the shared disk and create a symbolic link oneach node as it is described above for the files xtab and rmtab.

Using a shared disk for exported dataData that is exported to clients is defined in the /etc/exports file. This data needsto be accessible from each node where the NFS server potentially runs, thereforethese data must be on shared disks.v For each file system on a shared disk, a mount point must be defined, for

example on AIX in the file /etc/filesystems.v For each mount point, an IBM.AgFileSystem resource must exist, either harvested

by the StorageRM resource manager or user-defined. For more information aboutthe StorageRM, see System Automation for MultiplatformsAdministrator's and User'sGuide.

v Ensure that only file systems on shared disks are listed in /etc/exports.

Note:

1. If a shared disk configuration is changed, the StorageRM might not immediatelyremap these changes into the harvested storage resource information. Forexample, attaching a new device to use as a shared disk or adding andchanging mount points. Run the command refrsrc IBM.Disk or wait at leastthe configured harvest interval of IBM.Disk. The HarvestInterval is an attributeof IBM.Disk. The default value is 1800 seconds.

2. When mount points are created, specify mount option noauto or Mount=false toprevent that the file system is mounted during system start.


Configuring file systems within the same volume group onAIX

On AIX, file systems that are contained in the same volume group must bemounted sequentially to ensure a successful start when controlled by SystemAutomation for Multiplatforms.

Configure the NFS high availability policy as follows for those file systems:v Specify all file systems that are contained in the same volume group as one row

for parameter #12. This adds all those file systems to one resource group.v Specify startAfter/stopAfter relationships between these file systems within

parameter #14. Define a meaningful sequence of startAfter/stopAfterrelationships.

For example, your system has a volume group nfsvg. On nfsvg, there are threedefined file systems with three different mountpoints:Filesystem 1 with mountpoint /mnt/nfs1Filesystem 2 with mountpoint /mnt/nfs2Filesystem 3 with mountpoint /mnt/nfs3

In this example, you must specify:v The file systems as one row in parameter #12 of the wizard.v Two startAfter/stopAfter relationships for parameter #14 (for example, between

Filesystem 1 and Filesystem 2, and between Filesystem 2 and Filesystem 3).

NFS server virtual IP addressA virtual IP address is required to connect NFS clients to an NFS server bymounting an NFS file system from the NFS server using the virtual IP address.

NFS clients connect to an NFS server by mounting an NFS file system from theNFS server, which is determined by the host name or the IP address of the host.When the NFS server resource is restarted on a different node, then the virtual IPaddress (represented by class IBM.ServiceIP) chooses a network adapter on theother node based on its attributes IPAddress and NetMask, so that the client alwaysconnects to the same IP address.

Examplenode-1

network adapter eth0 192.168.2.2 netmask 255.255.255.0network adapter eth1 10.0.10.2 netmask 255.255.255.0

node-2network adapter eth0 192.168.2.3 netmask 255.255.255.0network adapter eth1 10.0.10.3 netmask 255.255.255.0

In the above configuration, suppose that you only want NFS clients to access thenodes through IP addresses of 192.168.2.2 and 192.168.2.3. Then you must use eth0of node-1 and node-2 in the IBM.Equivalency of IBM.NetworkInterface. TheIBM.ServiceIP that depends on this equivalency can have any free IP address inthe subnet. Check with your network administrator for an IP address that isavailable, for example 192.168.2.4:nfsserver-ip attributes:

IPAddress 192.168.2.4NetMask 255.255.255.0NodeNameList {"node-1","node-2"}


System Automation for Multiplatforms knows that IBM.ServiceIP depends onnetwork interfaces eth0 of node-1 and node-2, chooses a network adapter on thenode on which the NFS server runs, and passes it to IBM.ServiceIP, which createsan IP alias on that network adapter.

Note: For the setup of the high availability policy, you should:v Gather the network adapters of the nodes, for example using ifconfig and

select the ones which should be used by NFS clients to access data through theNFS server.

v Get a free IP address and the corresponding netmask from your networkadministrator.

Configuring and activating the NFS server high availability policySystem Automation for Multiplatforms provides a predefined NFS server highavailability policy to automate your NFS server and keep it highly available. Usethe sampolicy wizard to tailor the policy to your specific NFS server configurationand finally activate this policy.

The policy manages all NFS server components as described in “Configuring theNFS server components” on page 103.

NFS server high availability policySystem Automation for Multiplatforms provides a predefined NFS server highavailability policy that helps you to automate your NFS server.

The NFS server high availability policy contains two types of top-level groups.v nfsserver-rg: Contains NFS server processes, NFS server configuration data,

and virtual IP address. This group exists only once in an NFS server highavailability policy.

v filesys-data: Contains the file system that is exported to NFS clients. The grouphas one or more occurrences in an NFS server high availability policy.

The separation of the file systems from the NFS server itself and the separationinto more than one resource group prevents an outage of the NFS server in case ofan unrecoverable failure of one individual file system.


The table lists all resources contained in the policy:

Table 24. NFS server parts which are automated by the NFS server high availability policy

Name Resource name Description

NFS server group ▌1▐ IBM.ResourceGroup/<PREFIX>-rg

Group for all relevant NFS serverparts.

NFS server processes▌2▐

IBM.Application/<PREFIX>-server

NFS server process,start/stopped/monitored via script.

NFS serverconfiguration data ▌3▐

IBM.AgFileSystem /RSCT-harvested name

Filesystem containing all NFS serverconfiguration data. Started, stopped,and monitored using StorageRM.

Virtual IP address ▌4▐ IBM.ServiceIP<PREFIX>-ip-1

Virtual IP address that defines an IPalias. Started, stopped, and monitoredusing IBM.GblResRM.

NFS serverconfigurationdata

NFS servervirtual IP

NFS serverprocesses

NFS server group

NFS exported datacontainerresource

NFS exported dataexportedresource

NFS exported data group

NFS exported datacontainer resource


NFS exported dataexported resource

NFS exported dataexported resource

dependsOn dependsOn

dependsOndependsOn dependsOn

startAfter (AIX only) startAfter (AIX only)

dependsOn

2

3

5

1

4

6

8

10

7

9

11

12

Figure 19. NFS server high availability policy


Table 24. NFS server parts which are automated by the NFS server high availabilitypolicy (continued)


NFS exported datagroup ▌5▐

IBM.ResourceGroup/<PREFIX>-data<NUMBER>-filesys-rg

Group containing the data to beexported to clients. Multiple instancespossible.

NFS exported datacontainer resource ▌6▐

IBM.AgFileSystem/customer defined name

File system containing the data to beexported to clients. Started, stopped,and monitored using StorageRM.Multiple instances within a NFSexported data group possible

NFS exported dataexport resource ▌7▐

IBM.Application/<PREFIX>-<FILESYSNAME>-export-rs

One or more resources to control theexport of the file systems. Oneinstance per NFS exported datacontainer resource.

Network equivalency(not shown inFigure 19 on page 110)

IBM.Equivalency /<PREFIX>-nieq-1

Defines which network adapters willbe used to bind the virtual IP addressto. Started, stopped, and monitoredusing IBM.GblResRM.

Network interfaces(not shown inFigure 19 on page 110)

IBM.NetworkInterface /RSCT harvested name

Network adapters to bind the virtualIP address to.

The NFS server processes, the NFS server configuration data, and the virtual IPaddress belong to one resource group (nfsserver-rg). Your configuration mightcontain sets of mount points that have to be exported together. The policy willgroup each set of mount points in one single resource group.

The NFS server high availability policy contains the following relationships:

Table 25. NFS server high availability policy relationships

Relationship name Source Target Type

<PREFIX>data<NUMBER>-filesys-rg-DependsOnnfsserver- rg▌8▐

NFS exporteddata group

NFS server group dependsOnrelationship: Canoccur multiple timesdepending onnumber of NFSexported datagroups.

<PREFIX>data<NUMBER>-<FILESYSNAME>- export-DependsOn-<FILESYSNAME> ▌9▐

NFS exporteddata exportresource


dependsOn

<PREFIX>-server-DependsOn-nfsctrlfs▌10▐

NFS serverprocess

NFS serverconfiguration data

dependsOn

<PREFIX>-ip-onnieq-1-dyn (not shown inFigure 19 on page 110)

NFS servervirtual IPaddress

Network interfaces dependsOn


Table 25. NFS server high availability policy relationships (continued)


<PREFIX>-<FILESYSNAME_SOURCE>-stopAfter-<FILESYSNAME_TARGET> onAIX only ▌11▐

NFS exporteddata containerresource


stopAfter may occurmultiple timesdepending onnumber of exporteddata containerresources within oneNFS exported datagroup on AIX

<PREFIX>-server-DependsOn-nfsip ▌12▐

NFS serverprocess

NFS server virtual IPaddress

dependsOn

Since nfsserver-server cannot run without nfsserver-ip andnfsserver-data-varlibnfs being available, and the data exported data filesys-rgcannot be exported without the NFS server running, dependsOn relationshipsbetween the affected components are defined.

StartAfter or StopAfter relationships are created for file systems on AIX thatbelong to the same group. This ensures that mounting the file system is executedin an well defined order and consecutively.

All resources of the policy are defined as floating resources, so that they can bemove between the nodes of their domain. They are configured to always run onthe same node within the domain due to the collocated behavior of the defineddependsOn relationships. The equivalency nfsserver-nieq of network interfaces isnecessary to determine the network interfaces which will potentially host thefloating IP address (nfsserver-ip). A depends on relationship is required betweenthe floating IP address and the equivalency.

Example

In this sample scenario, the following setup is automated using the NFS Serverpolicy:v NFS server is automated using the NFS server high availability policy. Its

configuration data is located on shared disk hdisk1.v A set of filesystems (data1-fs1, data1-fs2 and data1-fs3) is located on shared disk

hdisk2.v A set of filesystems (data2-fs1) is located on shared disk hdisk3.v On AIX, all involved shared disks (hdisk1,hdisk2 and hdisk3) are contained in

different volume groups (vg1,vg2 and vg3).

Below is a figure of resources created for this sample scenarios:


The output of lssam for this setup is shown below:Online IBM.ResourceGroup:nfsserver-data1-filesys-rg Nominal=Online -.

|- Online IBM.AgFileSystem:fsa1 MNT=/data1/fsa1 | <. <. <.|- Offline IBM.AgFileSystem:fsa1:node-1 | | | |’- Online IBM.AgFileSystem:fsa1:node-2 | | SA |

|- Online IBM.AgFileSystem:fsa2 MNT=/data1/fsa1/fsa2 | | <. -’ ||- Offline IBM.AgFileSystem:fsa2:node-1 | | | |’- Online IBM.AgFileSystem:fsa2:node-2 | | | SA

|- Online IBM.AgFileSystem:fsa3 MNT=/data1/fsa1/fsa3 | | | -’|- Offline IBM.AgFileSystem:fsa3:node-1 | | | <.’- Online IBM.AgFileSystem:fsa3:node-2 | DO | |

|- Online IBM.Application:nfsserver-data1-fsa1-export-rs | -’ | ||- Offline IBM.Application:nfsserver-data1-fsa1-export-rs:node-1 | | |’- Online IBM.Application:nfsserver-data1-fsa1-export-rs:node-2 | DO |

’- Online IBM.Application:nfsserver-data1-fsa2-export-rs | -’ ||- Offline IBM.Application:nfsserver-data1-fsa2-export-rs:node-1 | |’- Online IBM.Application:nfsserver-data1-fsa2-export-rs:node-2 | DO

|- Online IBM.Application:nfsserver-data1-fsa3-export-rs | -’|- Offline IBM.Application:nfsserver-data1-fsa3-export-rs:node-1 |’- Online IBM.Application:nfsserver-data1-fsa3-export-rs:node-2 |

Online IBM.ResourceGroup:nfsserver-data2-filesys-rg Nominal=Online | -.|- Online IBM.AgFileSystem:fsb1 MNT=/data2/fsb1 | | <.

|- Offline IBM.AgFileSystem:fsb1:node-1 | | |’- Online IBM.AgFileSystem:fsb1:node-2 | | DO

|- Online IBM.Application:nfsserver-data2-fsb1-export-rs | | -’|- Offline IBM.Application:nfsserver-data2-fsb1-export-rs:node-1 | |’- Online IBM.Application:nfsserver-data2-fsb1-export-rs:node-2 DO DO

Online IBM.ResourceGroup:nfsserver-rg Nominal=Online <’ <’|- Online IBM.AgFileSystem:varlibnfs MNT=/varlibnfs <.

|- Offline IBM.AgFileSystem:varlibnfs:node-1 |’- Online IBM.AgFileSystem:varlibnfs:node-2 DO

|- Online IBM.Application:nfsserver-server -’ -.|- Offline IBM.Application:nfsserver-server:node-1 |’- Online IBM.Application:nfsserver-server:node-2 DO

IBM.AgFileSystemserver-data

IBM.ServiceIPserver-ip

IBM.Applicationserver

IBM.ResourceGroup: nfsserver-rg

IBM.AgFileSystemdata1-fs1

IBM.Applicationdata1-export-fs1

IBM.ResourceGroup: nfsserver-data1-filesys-rg





dependsOn dependsOn

dependsOn dependsOn dependsOn

startAfter (AIX only) startAfter (AIX only)

dependsOn



dependsOn

dependsOn

IBM.ResourceGroup: nfsserver-data2-filesys-rg

Figure 20. NFS server high availability policy sample scenario


’- Online IBM.ServiceIP:nfsserver-ip-1 IP=9.152.135.128 <’|- Offline IBM.ServiceIP:nfsserver-ip-1:node-1’- Online IBM.ServiceIP:nfsserver-ip-1:node-2

NFS server policy parametersThe NFS server high availability policy requires a number of parameters to bespecified in order to tailor the NFS server high availability policy for your specificenvironment, like the file system to be exported. The following parameter listingsconsist of a parameter description, a value type, and a default value. Theparameters vary by the operating system (AIX, Linux), where the NFS server isinstalled.

Policy parameters for AIXTable 26. NFS server high availability policy parameters (AIX)

# Parameter description

Value Type

Value

1 Enter the name of your SA MP domain.

Provide the name of an existing SA MP domain. The SA MP domain will hostthe NFS resources that will be configured with this template.

Value type: String

2 Enter your desired prefix for all NFS server resources.

This prefix will be used as a prefix for all SA MP resources that cover the NFSserver. For later operational tasks, the prefix can be used to start and stopresources with the same prefix with one single command.

Value type: String

nfsserver

3 Enter the nodes where you want to automate the NFS server.

These nodes must be listed when you enter the SA MP command lsrpnode forthe specified domain. You can use either the long or the short name for a node.NFS server resources will be created for each of the specified nodes.

List of values, value typefor each value: Hostnameor IP version 4 address

4 Do you want to monitor the NFS daemon nfsrgyd? The nfsrgyd daemon onlyexists for NFS v4 server on AIX.

The nfsrgyd daemon provides a name translation service for NFS servers andclients. It only needs to run if the NFS server is setup for NFS v4 on AIXoperating systems.

Value type: {yes|no}

no

5 Select the IP version used in your cluster.

Depending on the IP version you must specify either NetMask for IPv4, orNetPrefix for IPv6.

One of these values:v IPv4v IPv6

IPv4

6 This parameter will be ignored unless the following condition is fulfilled:

Parameter # 5 has the value IPv4.

Specify a virtual IPv4 address for the NFS server instance.

This IPv4 address will be used as a virtual IP address for the floating NFS serverinstance.



v Parameter # 5 has the value “IPv6”.





Table 26. NFS server high availability policy parameters (AIX) (continued)


Value Type

Value



Enter the netmask for the subnet of the virtual NFS server IP address.

Enter the netmask for the subnet of the virtual NFS server IP address. Anexample for a netmask is 255.255.255.0


255.255.255.0



Enter the NetPrefix for the virtual NFS server IP address.


Value type: numeric.


10 Enter the available network interface for the NFS Server. The following networkinterfaces are available on your local system: (remaining part of description isharvested from running system).

The available network interface specifies to which network interface the virtualNFS server IP address can be bound, for AIX an example is "en0", for Linux, anexample is "eth0". The same network interface name needs to be available on allnodes where the NFS server will be automated.

Value type: String

11 Specify the name of the IBM.AgFileSystem resource that represents the mountpoint of the NFS server configuration data directory. The followingIBM.AgFileSystem is available in your domain ("Name" "SysMountPoint"):(remaining part of description is harvested from running system)

SA MP will automatically mount the directory with NFS server configurationdata prior to starting the NFS server processes, and will automatically unmountthe directory after stopping the NFS server processes. You can determine thename of the IBM.AgFileSystem by issuing the command:

lsrsrc -s "ResourceType==1" IBM.AgFileSystem Name SysMountPoint

This command will return a list of IBM.AgFileSystem resources. Use the name ofthe resource that represents the mount point of the NFS server configurationdata directory.

Value type: String




Value Type

Value

12 Specify the names of the IBM.AgFileSystem resources that represent the mountpoints of the directories to be exported. The IBM.AgFileSystem resources can becombined in multiple IBM.ResourceGroups. For each group, the IBM.AgFileSystemresource names must be specified separately. The followingIBM.AgFileSystem isavailable in your domain ("Name" "SysMountPoint"): the remaining part of thedescription is harvested from the running system.

The values are specified in table form:

v Each table row (prompted as "value") consists of variable number of columns(each column prompted as "sub-value").

v The variable number of columns is prompted for each row separately.

v Each table row will result in a separate IBM.ResourceGroup.

v Each column within a row represents a single IBM.AgFileSystem resourcename, that will be added to one IBM.ResourceGroup.

SA MP will automatically mount and export each directory if the correspondingresource group is started, and will automatically unmount and unexport thedirectory if the corresponding resource group is stopped. You can determine thename of the IBM.AgFileSystem by issuing the command:

lsrsrc -s "ResourceType==1" IBM.AgFileSystem Name SysMountPoint

This command will return a list of IBM.AgFileSystem resources. Use the name ofthe resources that represent the mount points of the exported file system.

Example:

IBM.ResourceGroup:data1-filesys-rg- IBM.AgFileSystem:FS1 MNT=/mnt/test_a- IBM.AgFileSystem:FS2 MNT=/mnt/test_a/one- IBM.AgFileSystem:FS3 MNT=/mnt/test_a/twoIBM.ResourceGroup:data2-filesys-rg- IBM.AgFileSystem:FS4 MNT=/mnt/test_b- IBM.AgFileSystem:FS5 MNT=/mnt/test_b/one

Specify the values in the following form:Number of columns for value 1: 3Value 1, sub-value 1: "FS1"Value 1, sub-value 2: "FS2"Value 1, sub-value 3: "FS3"Number of columns for value 2: 2Value 2, sub-value 1: "FS4"Value 2, sub-value 2: "FS5"Number of columns for value 3: Just press ENTER to complete the input.


v Parameter # 13

List of values, each valueconsisting of sub-values,value type for eachsub-value: String




Value Type

Value

13 Specify the mountpoints of the directories to be exported. These mountpointsmust be specified in the same structure as you have specified theIBM.AgFileSystem names in the previous question.

Specify the mountpoints of the directories to be exported. For each directory,only one export can be configured. The values are specified in table form:



v Each table row belongs to an IBM.ResourceGroup specified in the previousquestion.

v Each column within a row represents a mount point name.

Example:


Specify the values in the following form:Number of columns for value 1: 3Value 1, sub-value 1: "/mnt/test_a"Value 1, sub-value 2: "/mnt/test_a/one"Value 1, sub-value 3: "/mnt/test_a/two"Number of columns for value 2: 2Value 2, sub-value 1: "/mnt/test_b"Value 2, sub-value 2: "/mnt/test_b/one"Number of columns for value 3: Just press ENTER to complete the input.


v Parameter # 12

List of values, each valueconsisting of sub-values,value type for eachsub-value: String

14 Are there any nested mountpoints between mounted file systems within one filesystem set?

Nested mountpoints are mountpoints where one mountpoint contains anothermountpoint, i.e. /mnt/test/one is a nested mountpoint to /mnt/test.


no




Value Type

Value

14.1 Optional; a value for this parameter is only required if parameter # 14 has thevalue "yes".

Specify the nested mountpoints that are contained in one file system set. Foreach nesting level, sub-value number 1 is the nested mountpoint, sub-valuenumber 2 is the mountpoint one level above. You need to specify the names ofthe IBM.AgFileSystem resources for each mountpoint.

Specify the names of the IBM.AgFileSystem of the nested mountpoints that arecontained in one file system set. The values are specified in table form. Eachtable row (prompted as "value") consists of two columns (each column promptedas "sub-value"). The first column value is the nested mountpoint and the secondcolumn value is the mountpoint at the higher directory level. Each row will beused to create a StartAfter relationship from first column value (source) tosecond column value (target).

Example:

IBM.ResourceGroup:Group1- IBM.AgFileSystem:FS1 MNT=/mnt/test_a- IBM.AgFileSystem:FS2 MNT=/mnt/test_a/one- IBM.AgFileSystem:FS3 MNT=/mnt/test_a/twoIBM.ResourceGroup:Group2- IBM.AgFileSystem:FS4 MNT=/mnt/test_b- IBM.AgFileSystem:FS5 MNT=/mnt/test_b/one

Specify the values in the following form:Value 1, sub-value 1: "FS2"Value 1, sub-value 2: "FS1"Value 2, sub-value 1: "FS3"Value 2, sub-value 2: "FS1"Value 3, sub-value 1: "FS5"Value 3, sub-value 2: "FS4"Value 4, sub-value 1: just press Enter to complete input

List of values, each valueconsisting of the samenumber of sub-values,value type for eachsub-value: String

15 Specify the desired state of your NFS server resources.

If you specify Online, the NFS server resources will be started on policyactivation, if not already started. If you specify Offline, the NFS server resourceswill not be started on policy activation, but stopped if they are online.


Online

Offline

Offline

Policy parameters for LinuxTable 27. NFS server high availability policy parameters (Linux)


Value Type

Value

1 Enter the name of your SA MP domain.

Provide the name of an existing SA MP domain. The SA MP domain will hostthe NFS resources that will be configured with this template.

Value type: String

2 Enter your desired prefix for all NFS server resources.

This prefix will be used as a prefix for all SA MP resources that cover the NFSserver. For later operational tasks, the prefix can be used to start and stopresources with the same prefix with one single command.

Value type: String

nfsserver


Table 27. NFS server high availability policy parameters (Linux) (continued)


Value Type

Value

3 Enter the nodes where you want to automate the NFS server.

These nodes must be listed when you enter the SA MP command lsrpnode forthe specified domain. You can use either the long or the short name for a node.NFS server resources will be created for each of the specified nodes

List of values, value typefor each value: Hostnameor IP version 4 address(plus additional valuechecking)

4 Select the IP version used in your cluster.

Depending on the IP version you must specify either NetMask for IPv4, orNetPrefix for IPv6.

One of these values:v IPv4v IPv6

IPv4













Enter the netmask for the subnet of the virtual NFS server IP address.

Enter the netmask for the subnet of the virtual NFS server IP address. Anexample for a netmask is 255.255.255.0


255.255.255.0





Value type: numeric.


9 Enter the available network interface for the NFS Server. The following networkinterfaces are available on your local system: (remaining part of description isharvested from running system)

The available network interface specifies to which network interface the virtualNFS server IP address can be bound, for AIX an example is "en0", for Linux, anexample is "eth0". The same network interface name needs to be available on allnodes where the NFS server will be automated.





Value Type

Value

10 Specify the name of the IBM.AgFileSystem resource that represents the mountpoint of the NFS server configuration data directory. The followingIBM.AgFileSystems are available in your domain ("Name" "SysMountPoint"):(remaining part of description is harvested from running system)

SA MP will automatically mount the directory with NFS server configurationdata prior to starting the NFS server processes, and will automatically unmountthe directory after stopping the NFS server processes. You can determine thename of the IBM.AgFileSystem by issuing the command 'lsrsrc -s"ResourceType==1" IBM.AgFileSystem Name SysMountPoint'. This command willreturn a list of IBM.AgFileSystem resources. Use the name of the resource thatrepresents the mount point of the NFS server configuration data directory.





Value Type

Value

11 Specify the names of the IBM.AgFileSystem resources that represent the mountpoints of the directories to be exported. The IBM.AgFileSystem resources can becombined in multiple IBM.ResourceGroups. For each group, theIBM.AgFileSystem resource names have to be specified separately. The followingIBM.AgFileSystems are available in your domain ("Name" "SysMountPoint"): theremaining part of description is harvested from the running system.

The values are specified in table form:



v Each table row will result in a separate IBM.ResourceGroups.

v Each column within a row represents a single IBM.AgFileSystem resourcename, that will be added to one IBM.ResourceGroups.

SA MP will automatically mount and export each directory if the correspondingresource group is started, and will automatically unmount and unexport thedirectory if the corresponding resource group is stopped. You can determine thename of the IBM.AgFileSystem by issuing the command:

lsrsrc -s "ResourceType==1" IBM.AgFileSystem Name SysMountPoint’.

This command will return a list of IBM.AgFileSystem resources. Use the name ofthe resources that represent the mount points of the exported file systems.

Example:


Specify the values in the following form:Number of columns for value 1: 3Value 1, sub-value 1: "FS1"Value 1, sub-value 2: "FS2"Value 1, sub-value 3: "FS3"Number of columns for value 2: 2Value 2, sub-value 1: "FS4"Value 2, sub-value 2: "FS5"Number of columns for value 3: Just press ENTER to complete the input.


v Parameter # 12

List of values, each valueconsisting of a variablenumber of sub-values,value type for eachsub-value: String (plusadditional value checking)




Value Type

Value

12 Specify the mount points of the directories to be exported. These mount pointsmust be specified in the same structure as you have specified theIBM.AgFileSystem names in the previous question.

Specify the mount points of the directories to be exported. For each directory,only one export can be configured. The values are specified in table form. Eachtable row (prompted as "value") consists of variable number of columns (eachcolumn prompted as "sub-value"). The variable number of columns is promptedfor each row separately. Each table row belongs to an IBM.ResourceGroupspecified in the previous question. Each column within a row represents amount point name.

Example:


Specify the values in the following form:Number of columns for value 1: 3Value 1, sub-value 1: "/mnt/test_a"Value 1, sub-value 2: "/mnt/test_a/one"Value 1, sub-value 3: "/mnt/test_a/two"Number of columns for value 2: 2Value 2, sub-value 1: "/mnt/test_b"Value 2, sub-value 2: "/mnt/test_b/one"Number of columns for value 3: Just press ENTER to complete the input.


v Parameter # 11

List of values, each valueconsisting of a variablenumber of sub-values,value type for eachsub-value: String (plusadditional value checking)

13 Are you running with NFS version 4?

If the NFS server is running with NFSv4, a separate daemon process, calledidmapd has to be started, stopped and monitored. Specify 'yes' if you are runningwith NFSv4.


14 This parameter will be ignored unless all following conditions are fulfilled:

v Parameter # 13 has the value "yes"

Specify the path of the RPC pipefs directory.

The RPC pipefs is a required directory of the rpc.idmapd daemon for NFSv4.The idmapd is the NFSv4 ID to name mapping daemon. The default path for thepipefs is /var/lib/nfs/rpc_pipefs.

Value type: String

/var/lib/nfs/rpc_pipefs

15 Are there any nested mount points between mounted file systems within one filesystem set?

Nested mount points are mount points where one mount point contains anothermount point, that is, /mnt/test/ one is a nested mount point to /mnt/test.


no




Value Type

Value

15.1 Optional; a value for this parameter is only required if parameter # 15 has thevalue "yes".

Specify the nested mount points that are contained in one file system set. Foreach nesting level, sub-value number 1 is the nested mount point, sub-valuenumber 2 is the mount point one level above. You need to specify the names ofthe IBM.AgFileSystem resources for each mount point.

Specify the names of the IBM.AgFileSystems of the nested mount points that arecontained in one file system set. The values are specified in table form. Eachtable row (prompted as "value") consists of two columns (each column promptedas "sub-value"). The first column value is the nested mount point and the secondcolumn value is the mount point at the higher directory level. Each row will beused to create a StartAfter relationship from first column value (source) tosecond column value (target).

Example:

IBM.ResourceGroup:Group1- IBM.AgFileSystem:FS1 MNT=/mnt/test_a- IBM.AgFileSystem:FS2 MNT=/mnt/test_a/one- IBM.AgFileSystem:FS3 MNT=/mnt/test_a/twoIBM.ResourceGroup:Group2- IBM.AgFileSystem:FS4 MNT=/mnt/test_b- IBM.AgFileSystem:FS5 MNT=/mnt/test_b/one

Specify the values in the following form:Value 1, sub-value 1: "FS2"Value 1, sub-value 2: "FS1"Value 2, sub-value 1: "FS3"Value 2, sub-value 2: "FS1"Value 3, sub-value 1: "FS5"Value 3, sub-value 2: "FS4"Value 4, sub-value 1: just press Enter to complete input

List of values, each valueconsisting of the samenumber of sub-values,value type for eachsub-value: String

16 Specify the desired state of your NFS server resources.

If you specify Online, the NFS server resources will be started on policyactivation, if not already started. If you specify Offline, the NFS server resourceswill not be started on policy activation, but stopped if they are online.


Online

Offline

Offline

Using the wizard to configure and activate the NFS server highavailability policy

The NFS server high availability policy consists of a policy template that is tailoredby using the sampolicy wizard.

The policy template samnfs_templateNFSServer.tmpl.xml is located in theinstallation directory of the NFS server high availability policy/usr/sbin/rsct/sapolicies/nfsserver. To configure the template, run thefollowing command:sampolicy –w /usr/sbin/rsct/sapolicies/nfsserver/samnfs_templateNFSServer.tmpl.xml


You must configure a policy pool before running the wizard. The wizard will thenstore all modifications to the policy pool. If you set your policy pool to/etc/myPolicyPool, a wizard run usingsampolicy -w /usr/sbin/rsct/sapolicies/nfsserver/samnfs_templateNFSServer.tmpl.xml

stores the results to/etc/myPolicyPool/samnfs_templateNFSServer.tmpl.xml

The next time you invoke the wizard, use the file stored in the policy pool. Withthe example above, invoking the wizard for the second time would look like this:sampolicy -w /etc/myPolicyPool/samnfs_templateNFSServer.tmpl.xml

“Using the wizard to configure and activate the SAP Central Services highavailability policy” on page 42Each SAP Central Services high availability policy consists of a policy templatethat is tailored by using the sampolicy wizard.“Using the wizard to configure and activate the Oracle database highavailability policy” on page 138The Oracle database server high availability policy consists of a policy templatethat is tailored by using the sampolicy wizard.Chapter 5, “Using the wizard to configure and activate the policy,” on page 141Use the policy setup wizard to configure and activate your IBM Tivoli SystemAutomation for Multiplatforms policies.

Verifying the NFS server high availability policyAfter you installed and configured your NFS server, verify that your setup iscorrect.

To verify your installation and configuration to make the NFS server highlyavailable, proceed with the following steps, using the following example setup:1. Check if the NFS server is ready to use.

An NFS server has been made highly available using the IP address:192.168.2.4. The directory to be exported is /work. Before trying to mount anexported file system from an NFS client, you need to check if the NFS server isready for NFS client connections and if all file systems defined in /etc/exportsare really exported to the specified hosts. Applying the example setup settings,entershowmount –e 192.168.2.4exportfs

The output of both commands should correspond to the content of/etc/exports:#cat /etc/exports/work node-3(rw,sync)#showmount –e 192.168.2.4xport list for 192.168.2.4:/work node-3#exportfs/work node-3

If any errors occur, solve them first before you continue with the next steps.2. Access automated NFS server from an NFS client.

Using the example above, check if an NFS client is able to mount the NFS filesystem exported from 192.168.2.4. The mount point on the NFS client is be/mnt/work. From a command prompt on the NFS client, run the followingcommand as user root:


#mount 192.168.2.4:/work /mnt/work

If no errors occur, check the result using the mount command.#mount

For this example, this command reports a list of all mounted file system withthe following entry:192.168.2.4:/work on /mnt/work type nfs (rw,soft,addr=192.168.2.4)

3. Testing NFS server failover:After you have verified that your automated NFS server works on one node,you can try a failover. In a two node cluster with node-1 and node-2, excludethe node from automation on which the NFS server currently runs and see if itis started on the other node. Assuming the NFS server runs on node-2, enter:samctrl -u a node-2

This adds node-2 to the list of nodes excluded from automation. Wait until thefailover starts. Then inspect the resources in nfsserver-rg using the lssamcommand:Online IBM.ResourceGroup:nfsserver-rg Nominal=Online

|- Online IBM.AgFileSystem: nfsserver-data-varlibnfs|- Online IBM.AgFileSystem:nfsserver-data-varlibnfs:node-1’- Offline IBM.AgFileSystem:nfsserver-data-varlibnfs:node-2 Node=Excluded

|- Online IBM.Application:nfsserver-server|- Online IBM.Application:nfsserver-server:node-1’- Offline IBM.Application:nfsserver-server:node-2 Node=Excluded

’- Online IBM.ServiceIP:nfsserver-ip-1|- Online IBM.ServiceIP:nfsserver-ip-1:node-1’- Offline IBM.ServiceIP:nfsserver-ip-1:node-2 Node=Excluded

Note: The groups hosting the exported file system will also move to node-1.You can always inspect which nodes are excluded using the lssamctrlcommand to display SAMControl information:SAMControl:

TimeOut = 60RetryCount = 3Automation = AutoExcludedNodes = {node-2}ResourceRestartTimeOut = 5

Ensure that the NFS server mounted on your NFS clients are still working, forexample by accessing the directory of the mount.You can now include node-2into automation again, enter:samctrl -u d node-2

The parameter d is used to delete the node from the list of excluded nodes.After the NFS server is online on node node-1, you should be able to accessand modify the NFS data of a NFS client.After having successfully completed this test scenario using samctrl, a moreadvanced scenario like a node failure can be run. Before you start, ensure thatthe following prerequisites are met:v Make sure that no node is on the list of excluded nodes. You can use the

command lssamctrl to verify.v If you have only a 2-node-cluster as in the previous example be sure you

have correctly setup a tiebreaker. For more information about the tiebreaker,see System Automation for Multiplatforms Administrator's and User's Guide.

If these prerequisites are satisfied you can test a failover by either rebooting thenode currently running the NFS server or pressing the power button. The NFSserver should be failed over to the other node, and the NFS clients are still ableto connect the mounted NFS data.


Chapter 4. Oracle high availability policy

You can define high availability policies for your Oracle database. The topics inthis section describe the process.

The high availability concept offered by System Automation for Multiplatformskeeps an Oracle database highly available.

Besides DB2, the Oracle database is the most frequently used database systems forSAP. There are different high availability concepts for an Oracle database, forexample Active Data Guard and the Real Application Cluster (RAC) feature. Thesolution offered by System Automation for Multiplatforms is an additional highavailability solution, using the concept of placing the database files on a shareddisk that is mounted to the database server exclusively.

The following figure depicts the high availability architecture for an Oracledatabase that is required for the Oracle database high availability policy:

The database control software, for example the Oracle database server and thedatabase listener are installed on node-1 and node-2. Both nodes are connected to ashared disk where the database is installed, but the disk is always mounted to onesingle node. The database listener is accessible from database clients using thevirtual IP address and is waiting for requests from a database client. The listenerqueries the database and returns results to the database client.

Assume that the Oracle database server and the database listener are running onnode-1. The shared disk with the database files is mounted on node-1, the virtualIP address is bound to a network interface on node-1. If node-1 fails, the shareddisk is mounted to node-2, the Oracle database server and the listener on node-2are started and the virtual IP address is bound to a network interface on node-2.The database clients remain connected to the same virtual IP all the time.

Database clients

IP address

Oracle database

listener

Oracle database

server

IP address

Oracle database

listener

Oracle database

server

node-1 node-2

shared disk

Figure 21. High availability architecture for an Oracle database


Planning the Oracle database automation and high availabilityBefore you automate your Oracle database with System Automation forMultiplatforms, check that the version of Oracle installed in your environment issupported, and that you have the required hardware and network infrastructure.

Supported Oracle versions

The following Oracle versions are supported for the Oracle database highavailability policy:v Oracle 10g Release 2 (10.2.0.4) and higher patchesv Oracle 11g

Prerequisites

For each virtual IP address that is defined in the high availability policy, anequivalency of network interfaces is defined. Only network interfaces with thesame network interface name can be part of the equivalency. Ensure that thenetwork interfaces you want to use have the same name on each system.

Infrastructure requirements

For an Oracle database to failover between different systems, the followingrequirements for the system infrastructure have to be fulfilled.v Ensure that your operating system meets all prerequisites of the Oracle database

release which you install. Refer to the Oracle database documentation for furtherinformation.

v At least one shared disk is required for the Oracle database high availabilityconfiguration. One or more disks contain the data that is exported by the Oracledatabase. They contain configuration files that the Oracle database requires torun properly and behave identically on all systems. On a Linux machine theOracle database uses files located in /var/lib/nfs to store information aboutany Oracle database which have mounted a file system that is exported by thisOracle database and information about file locks. On AIX this information iscontained in the files /etc/xtab, /etc/exports, /etc/rmtab and the directory/var/statmon. This information must persist after a node failover to the Oracledatabase, otherwise the Oracle databases lose their connections to the Oracledatabase. The Oracle database information is stored in a small file system on ashared disk to keep this information in sync on all nodes. These disk(s) must beaccessible by all nodes that are allowed to host the Oracle database, for exampleif the nodes are connected to a storage device. Each shared disk has a uniquedevice name and number. The unique device name and number must be thesame on all nodes. Depending on your operating system, these requirements are:

Linux To transparently fail over the Oracle database from one host to anotheron a Linux machine, the shared disk(s) must be connected to each serverwith the same unique major and minor number. System Automation forMultiplatforms can only identify the shared disk if all nodes identify theshared disk with the same unique number. The shared disk must havethe same device name and number on all nodes, for example /dev/sdc1.You can determine the major and minor number as follows:#cd /dev#ls –l sd*

The last command will return an output like this:


<systemname>:/dev # ls -l sd*brw-rw---- 1 root disk 8, 0 2011-01-19 14:25 sdabrw-rw---- 1 root disk 8, 1 2011-01-19 14:25 sda1brw-rw---- 1 root disk 8, 2 2011-01-19 14:25 sda2brw-rw---- 1 root disk 8, 3 2011-01-19 14:25 sda3brw-rw---- 1 root disk 8, 4 2011-01-19 14:25 sda4

For sda, the major number is 8 and the minor number is 0.

AIX On an AIX machine the volume group MAJOR NUMBER must beidentical to the volume group containing the shared data on all of thenodes. For details about the MAJOR NUMBER, see the lsvg commandman pages. Make sure your setup fulfills the MAJOR NUMBERrequirements and keep in mind that this is also important for laterchanges of the configuration of the servers. This applies for every shareddisk hosting data to be exported by the Oracle database.

Keep in mind that it is not possible to have more than one Oracle databaserunning on a single node. If you want to run consolidation projects, consolidate theexports of several Oracle databases to one Oracle database instead of placingseveral Oracle databases on a single machine. If there is more than one Oracledatabase resource within a System Automation for Multiplatforms cluster, thenthese resources must have an AntiCollocated relationship to ensure that theseOracle databases are never started on the same node at a time.

Restrictions

The Oracle database listener must not be started directly using the commandtnslsnr. System Automation for Multiplatforms will not be able to detect such arunning listener, because it is not registered properly with the listeners name.

Installing and setting up the Oracle databaseInstall the Oracle database, create a database on a shared disk, and then configurethe individual Oracle database components.

If you want to configure and activate the Oracle database high availability policy,see “Configuring and activating the Oracle database high availability policy” onpage 132.

Installing the Oracle database softwareTo install a database using the shared disk approach, you need to adhere to thefollowing installation requirements. Oracle supplies detailed and comprehensivedocumentation for the database installation. Carefully read the Oracledocumentation for your database version and the follow the steps described below.Prepare each system where you want to install the Oracle database, select onesystem to install the database software, create the database on a shared disk andfinally replicate the database software to all other systems you have prepared.

For all nodes

On each system where you want to install the Oracle database, perform thefollowing steps:1. Ensure that a shared disk is available on each of the systems, fulfilling the

requirements that are described in “Planning the Oracle database automationand high availability” on page 128. This disk contains the database files.

Chapter 4. Oracle high availability policy 129

2. Ensure that all preinstallation requirements to install the database are met.These requirements include hardware and software, examples for suchrequirements are sufficient disk space and memory or certain patches for anoperating system. Preinstallation requirements include mandatorypreinstallation tasks. These tasks include the creation of users and groups,directories, and the configuration of shell limits. In the Oracle documentation,the requirements tasks are listed in the chapters “Preinstallation tasks” forOracle 10.2 and “Oracle Database Preinstallation Requirements” for Oracle 11.You must create the directory that contains the database files on the shared diskyou designated for this purpose.

For a single node

Select a node where you want to install the Oracle database software and followthese steps:1. Mount the shared disk.2. Install the database software by following the instructions in the Oracle

documentation. Ensure that you install the database software only. Theinstaller has an option like Install database software only. Select this option.

3. For Oracle 11g only, configure a database listener with the command netca asOracle user. Oracle 10g contains the database listener configuration in thestandard installer.

4. Create a database with the command dbca as Oracle user. As storagemechanism, select file system and use common location for all databasefiles. As a location, select the directory you created on the shared disk.

5. Start the database and the listener.6. Verify your installation, for example by connecting a database client to the

database.7. Stop the database and the listener.8. Create an archive of the database software. For example, if you installed the

database in the directory /opt/oracle, run the following command:tar cvf MP1_oracle10g.tar /opt/oracle

.9. Copy the tar file to all systems you prepared to run the Oracle database

software. Unpack the archive that you created. For example, run the followingcommand:tar xvf MP1_oracle10g.tar /opt/oracle

.10. Test whether the replication step was successful:

v Unmount the shared disk on the system where you installed the database.v Mount the shared disk on the system where you unpacked the archive.v Start the database.v Start the database listener.v Ensure that the database works properly by connecting a database client to

the database.

If you completed all the steps successfully, you can proceed with the configuration.See “Configuring and activating the Oracle database high availability policy” onpage 132.


Configuring the individual Oracle database componentsTo automate an Oracle database, configure the following subcomponents of adatabase installation.

Oracle database server

To automate the database server, System Automation must be able to start, stop,and monitor the specified database server. For more information on how toautomate the database server using System Automation, refer to “Oracle databasehigh availability policy” on page 132.

If Oracle is configured to be started automatically by the operating system, forexample it is listed in /etc/rc.d/rc*.d (Linux) and /etc/inittab (AIX), you needto ensure that these listings are removed. This is not a default configurationperformed by the Oracle installation, but is required if your system administratorhas configured the operating system to automatically start Oracle.

Oracle database listener

To automate the database listener, System Automation must be able to start, stopand monitor the specified database listener. For more information on how toautomate the database listener using System Automation, refer to “Oracle databasehigh availability policy” on page 132.

Oracle database files

The Oracle database files need to be accessible from each node where the Oracledatabase server runs. Therefore, data must be located on shared disks. For each filesystem that resides on such a shared disk, a mount point must be defined withinthe operating system, for example in /etc/fstab on Linux. For each mount point,an IBM.AgFileSystem resource must exist, either harvested by the StorageRMresource manager or user-defined. On Linux it is recommended to specify a labelfor the file system, because the label is taken as resource name for harvestedIBM.AgFileSystem resources. For more information about StorageRM, see SystemAutomation for Multiplatforms Administrator's and User's Guide.

Note:

vIf a shared disk configuration is changed, for example attaching a new device touse as a shared disk or adding and changing mount points, the StorageRM maynot immediately remap these changes into the harvested storage resourceinformation. You need to run the command refrsrc IBM.Disk or wait at leastthe configured harvest interval of IBM.Disk. The default value is 1800 seconds.

v If you create mount points, specify mount option noauto to prevent that the filesystem is mounted during system start.

Oracle virtual IP address

Database clients connect to a database server by contacting a listener of thedatabase server, which is determined by its host name or the IP address of thehost. When the listener is restarted on a different node, then the virtual IP addresswill be bound to a network adapter on the same node, so that the client alwaysconnects to the same IP address.

Example for Linux network adapters:


node-1 network adapter eth0 192.168.2.2 netmask 255.255.255.0network adapter eth1 10.0.10.2 netmask 255.255.255.0

node-2 network adapter eth0 192.168.2.3 netmask 255.255.255.0network adapter eth1 10.0.10.3 netmask 255.255.255.0

Example for AIX network adapters:node-1 network adapter en0 192.168.2.2 netmask 255.255.255.0

network adapter en1 10.0.10.2 netmask 255.255.255.0node-2 network adapter en0 192.168.2.3 netmask 255.255.255.0

network adapter en1 10.0.10.3 netmask 255.255.255.0

For example, you only want database clients to access the nodes through the IPaddresses 192.168.2.2/3. Then you must specify eth0 on Linux or en0 on AIX ofnode-1 and node-2 as network interfaces in the Oracle policy. The virtual IPaddress can have any free IP address in the subnet, for example 192.168.2.4.

Oracle database clients

You need to configure your Oracle database clients to connect to the virtual IPaddress defined in the policy. Configure the host parameter of the filetnsnames.ora.

The database client is installed in /opt/oracle/product/10.2.0/db_1, and thedatabase alias to connect to a database is MPDBALIAS. The virtual IP address has theDNS name virtualIP105. The port 1521 must be available and the same on alldatabase servers. The database SID which the listener connects to is MPDB./opt/oracle/product/10.2.0/db_1/network/admin/tnsnames.ora

MPDBALIAS =(DESCRIPTION =(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = virtualIP105)(PORT = 1521)))(CONNECT_DATA =(SERVICE_NAME = MPDB)))

Configuring and activating the Oracle database high availability policySystem Automation for Multiplatforms provides a predefined Oracle database highavailability policy to automate your Oracle database and to keep it highlyavailable.

The policy takes care of all Oracle database components as described in“Configuring the individual Oracle database components” on page 131. Use thesampolicy wizard to configure the policy to your specific Oracle databaseconfiguration and activate this policy.

Oracle database high availability policyThe Oracle database high availability policy consists of one group containing theOracle database server, the Oracle database listener, the virtual IP address and thefile system containing the Oracle database.


The following table lists all resources contained in the policy:

Table 28. Oracle database high availability resources


Oracle resource group IBM.ResourceGroup/<PREFIX>-rg Resource group containingall Oracle resources.

Oracle database server IBM.Application/<PREFIX>- db Oracle database processes,which are started, stopped,and monitored using scripts.

Oracle database listener IBM.Application/<PREFIX>- lsn Oracle database listenerprocess, which is started,stopped, and monitoredusing scripts.

Virtual IP address IBM.ServiceIP/<PREFIX>-ip Virtual IP address to connectOracle database clients toOracle database listener.

Oracle database files IBM.AgFileSystem/<ORACLE_FILESYSNAME>

File system containing theOracle database files, mayoccur multiple times.

Network interfaces IBM.Equivalency andIBM.NetworkInterface/<PREFIX>-nieq-1-dyn

Selects network adapters tobind virtual IP to. Started,stopped, and monitoredusing IBM.GblResRM.

The Oracle database server and the Oracle database listener are resources of classIBM.Application. For IBM.Application, a script is required to start, stop, andmonitor. The samoraclectrl-db script that controls the Oracle database server hasthe following arguments:Action <start|stop|status>OracleHome <path to the oracle installation>OracleSid <instance name of the database to start>OracleOwner <user ID of the database owner>

start Starts the database processes.

stop Stops the database processes.

status Checks if a process for the specified instance runs

The samoracle-lsn script which controls the Oracle database listener has thefollowing arguments:

Oracle databaseserver

Oracle databasefiles

Oracle databaselistener

Virtual IPaddress

Oracle resource group

eth0:node-1

eth0:node-2

Network interfaces

*dependsOn *dependsOn *dependsOn

dependsOn

*: IBM.ManagedRelationship

Figure 22. Oracle database high availability policy


Action <start|stop|status>OracleHome <path to the oracle installation>ListenerName <name of the listener instance>OracleOwner <user ID of the database owner>

start Starts the listener processes.

stop Stops the listener processes.

status Checks if the listener processes for the specified listener instance runs.

The following table shows all relationships that are defined:

Table 29. Oracle database high availability policy relationships


<PREFIX>-lsn-on-ip-1 Oracle databaselistener

Virtual IPaddress

DependsOn

<PREFIX>-ip-on-nieq-1 Virtual IP address Networkequivalency

DependsOn

<PREFIX>-lsn-DependsOn-<ORACLE_FILESYSNAME>

Oracle databaselistener


DependsOn may occurmultiple timesdepending on numberof file systems.

<PREFIX>-db-DependsOn-ORACLE_FILESYSNAME>

Oracle databaseserver


DependsOn may occurmultiple timesdepending on numberof file systems.

The Oracle database listener and the Oracle database server cannot be startedwithout the file system being mounted that contains the Oracle database files.DependsOn relationships are defined between the listener and the database filesand the listener and the database server. The Oracle database clients connect to theOracle database listener using a virtual IP address, therefore there is also adependsOn relationship defined between the listener and the IP address. Thevirtual IP address needs to be bound to a network interface, which leads to adependsOn relationship between the virtual IP and the network equivalency.

In this sample scenario an Oracle database installation was automated using theOracle database policy:v The Oracle database was automated using the policy described above.v A mount point oracledata is defined on a shared disk.v A virtual IP address 10.15.13.241 is reserved for the database listener and

specified to be bound to the en0 network interfaces on both nodes.

Output of lssam for this setup:Offline IBM.ResourceGroup:oracle-rg Nominal=Offline

|- Offline IBM.AgFileSystem:oradata MNT=/oracledata <. <.|- Offline IBM.AgFileSystem:oracledata:node-1 | |’- Offline IBM.AgFileSystem:oracledata:node-2 | DO

|- Offline IBM.Application:oracle-db | -’|- Offline IBM.Application:oracle-db:node-1 |’- Offline IBM.Application:oracle-db:node-2 DO

|- Offline IBM.Application:oracle-lsn -’ -.|- Offline IBM.Application:oracle-lsn:node-1 |’- Offline IBM.Application:oracle-lsn:node-2 DO

’- Offline IBM.ServiceIP:oracle-ip-1 IP=10.15.13.241 -. <’|- Offline IBM.ServiceIP:oracle-ip-1:node-1 |’- Offline IBM.ServiceIP:oracle-ip-1:node-2 DO


Online IBM.Equivalency:oracle-nieq-1-dyn <’|- Online IBM.NetworkInterface:en0:node-1’- Online IBM.NetworkInterface:en0:node-2

Oracle policy high availability and automation preparationBefore you run the sampolicy wizard, to configure and activate the Oracle databasehigh availability policy, complete all of the following required installation andconfiguration tasks:v All Oracle installation and configuration steps must be completed successfully.

See “Installing the Oracle database software” on page 129 and “Configuring theindividual Oracle database components” on page 131.

v System Automation for Multiplatforms is installed and an automation domain isconfigured. For a description of this setup for the SAP high availability policy,see “Installing System Automation for Multiplatforms on all cluster nodes” onpage 24. Skip the SAP-specific steps in that section. Make sure that:– System Automation for Multiplatforms version 3.2.2 or higher is installed on

all nodes where you want the Oracle database server to run. Issue thesamversion command to check that the correct version of System Automationfor Multiplatforms is installed.

– Domain has quorum: lssrc -ls IBM.RecoveryRM– SAP Central Services high availability policy feature license is installed as

described in “Installing the SAP high availability policy feature license” onpage 26.

v Before you automate an Oracle database, ensure that the Oracle database worksproperly on all systems. For all necessary steps to test a manual failover fromone system to the other, see “Installing the Oracle database software” on page129.

Oracle database high availability policy parametersThe Oracle database high availability policy requires a number of parameters to bespecified in order to configure the Oracle database high availability policy for yourspecific environment, like the Oracle home directory or the file system on a shareddisks that hosts the database instance. The following parameter listings consist of aparameter description, a value type, and a default value.

Table 30. Oracle database high availability policy parameters.

# Parameter description Value Type

Value

1Enter the name of your SA MP domain.

Provide the name of an existing SA MPdomain. The SA MP domain will hostthe Oracle database resources that willbe configured with this template.

Value type: String

oracle


Table 30. Oracle database high availability policy parameters (continued).


Value

2Enter your desired prefix for all Oraclepolicy resources.

This prefix will be used as a prefix forall SA MP resources that cover theOracle database policy. For lateroperational tasks, the prefix can be usedto start and stop resources with the sameprefix with one single command.

Value type: String

SA-oracle

3Enter the nodes where you want toautomate the Oracle database.

These nodes must be listed when youenter the SA MP command lsrpnode forthe specified domain. You can use eitherthe long or the short name for a node.Oracle database resources will be createdfor each of the specified nodes.

List of values, value type for each value:Host name or IP version 4 address (plusadditional value checking)

4Specify a virtual IP address for theOracle database instance.

This IP address will be used as a virtualIP address for the floating Oracledatabase instance.

Value type: IP version 4 address

5Enter the netmask for the subnet of thevirtual Oracle database IP address.

Enter the netmask for the subnet of thevirtual Oracle database IP address. Anexample for a netmask is 255.255.255.0

Value type: IP version 4 address

255.255.255.0

6Enter the available network interface forthe Oracle database.

The following network interfaces areavailable on your local system: Theavailable network interface specifies towhich network interface the virtualOracle database IP address can bebound, for AIX an example is "en0", forLinux, an example is "eth0". The samenetwork interface name needs to beavailable on all nodes where the Oracledatabase will be automated.

Value type: String (plus additional valuechecking)




Value

7Specify the names of theIBM.AgFileSystem resources thatrepresent the mount points of the Oracledatabase instance directories. Thefollowing resource of classIBM.AgFileSystem are available in yourdomain ("Name" "SysMountPoint"):

SA MP will automatically mount thedirectories with Oracle database instancedata prior to starting the Oracle databaseprocesses, and will automaticallyunmount the directories after stoppingthe Oracle database processes. You candetermine the name of theIBM.AgFileSystem by issuing thefollowing command lsrsrc -s"ResourceType==1 IBM.AgFileSystemName SysMountPoint. This commandwill return a list of IBM.AgFileSystemresources. Use the name of the resourcesthat represent the mount point of theOracle database instance data directories.

List of values, value type for each value:String (plus additional value checking)

8Specify the path to the Oracle homedirectory, that was set during the Oracledatabase installation.

The Oracle home directory contains allOracle software files. For an Oracle 11ginstallation, the home directory is called'Software Location' in the Oracleinstaller, for an Oracle 10g installation,'Oracle Home'.

Value type: String

9Specify the Oracle System Identifier(SID) for the database that will be kepthighly available.

The Oracle System Identifier (SID)uniquely identifies a database instanceand is set during the Oracle databaseconfiguration or installation.

Value type: String

10Specify the name of the Oracle softwareowner user, for example "oracle".

The Oracle software owner user wascreated before the Oracle installation isstarted. This user is required to start andstop the database.

Value type: String




Value

11Specify the name of the Oracle Netlistener.

An Oracle Net listener is required foreach Oracle installation in order toestablish a remote connection to adatabase.

Value type: String

12Specify the desired state of your Oracleresources.

If you specify Online, the Oracleresources will be started on policyactivation, if not already started. If youspecify Offline, the Oracle resources willnot be started on policy activation, butstopped if they are online.

One of the following values:

Online

Offline

Offline

Using the wizard to configure and activate the Oracle database highavailability policy

The Oracle database server high availability policy consists of a policy templatethat is tailored by using the sampolicy wizard.

The policy template samora_templateOracle.tmpl.xml, is located in the installationdirectory of the Oracle database high availability policy, /usr/sbin/rsct/sapolicies/oracle. To configure the template, run the following command:sampolicy –w /usr/sbin/rsct/sapolicies/oracle/samora_templateOracle.tmpl.xml

You must configure a policy pool before running the wizard. The wizard will thenstore all modifications to the policy pool. If you set your policy pool to/etc/myPolicyPool, a wizard run usingsampolicy -w /usr/sbin/rsct/sapolicies/oracle/samora_templateOracle.tmpl.xml

stores the results to/etc/myPolicyPool/samora_templateOracle.tmpl.xml

The next time you invoke the wizard, use the file stored in the policy pool. Withthe example above, invoking the wizard for the second time would look like this:sampolicy -w /etc/myPolicyPool/samora_templateOracle.tmpl.xml

“Using the wizard to configure and activate the SAP Central Services highavailability policy” on page 42Each SAP Central Services high availability policy consists of a policy templatethat is tailored by using the sampolicy wizard.“Using the wizard to configure and activate the NFS server high availabilitypolicy” on page 123The NFS server high availability policy consists of a policy template that istailored by using the sampolicy wizard.


Chapter 5, “Using the wizard to configure and activate the policy,” on page 141Use the policy setup wizard to configure and activate your IBM Tivoli SystemAutomation for Multiplatforms policies.

Verifying the Oracle database high availability policyAfter you installed and configured your Oracle database, verify that your settingsare correct.

Verifying the settings

To verify the installation and configuration of your Oracle highlyavailability policy, proceed with the following steps:v Start the database policy. Enter the command chrg –o online

<database_group_name>

v Connect a database client to the database. If the connection is successful,continue with testing Oracle database failover.

Testing Oracle database failover

After you verified that your automated Oracle database works on onenode, initiate a failover. In a two node cluster with node-1 and node-2,exclude the node from automation on which the Oracle database currentlyruns and check if it is started on the other node. Assuming the Oracledatabase runs on node-2, enter:samctrl -u a node-2

Node-2 is added to the list of nodes that are excluded from automation.Wait until the failover starts. Then inspect the resourcesin SA-nfsserver-rgusing the lssam command:Offline IBM.ResourceGroup:oracle-rg Nominal=Offline

|- Online IBM.AgFileSystem:oradata MNT=/oracledata|- Online IBM.AgFileSystem:oracledata:node-1’- Offline IBM.AgFileSystem:oracledata:node-2 Node=Excluded

|- Offline IBM.Application:oracle-db|- Online IBM.Application:oracle-db:node-1’- Offline IBM.Application:oracle-db:node-2 Node=Excluded

|- Offline IBM.Application:oracle-lsn|- Online IBM.Application:oracle-lsn:node-1’- Offline IBM.Application:oracle-lsn:node-2 Node=Excluded

’- Offline IBM.ServiceIP:oracle-ip-1 IP=10.15.13.241|- Online IBM.ServiceIP:oracle-ip-1:node-1’- Offline IBM.ServiceIP:oracle-ip-1:node-2 Node=Excluded

You can always inspect which nodes are excluded using the lssamctrlcommand:SAMControl informationSAMControl:TimeOut = 60RetryCount = 3Automation = AutoExcludedNodes = {node-2}ResourceRestartTimeOut = 5

Ensure that the NFS server mount points on your NFS clients are stillworking, for example by accessing the directory of the mount points.

You can now include node-2 to the automation again, enter:samctrl -u d node-2


The parameter d is used to delete the node from the list of excluded nodes.After the Oracle database is online on node-1 you should be able to accessand modify the Oracle database from a database client.

After having successfully completed this test scenario using samctrl, run anode failure test scenario. Before you start, ensure that the followingprerequisites are met:v Make sure that the list of excluded nodes is empty. Use the command

lssamctrl to verify.v If you have a two node cluster as in the previous example make sure

you have correctly setup a tiebreaker. For more information about thetiebreaker, see System Automation for Multiplatforms Administrator's andUser's Guide .

If these prerequisites are satisfied you can test a failover by eitherrebooting the node currently running the Oracle database server orpressing the power button. The Oracle database server should be failedover to the other node, and the Oracle clients are still able to connect to thedatabase.


Chapter 5. Using the wizard to configure and activate thepolicy

Use the policy setup wizard to configure and activate your IBM Tivoli SystemAutomation for Multiplatforms policies.

You can perform the following tasks using the System Automation forMultiplatforms Policy Setup Wizard:v Start the Policy Setup Wizard. Select the policy that you want to configure. You

can find one policy XML file for each of the SAP high availability setup optionsthat are described in “Starting the Policy Setup Wizard.”

v Specify or modify the parameters to configure the policy and adapt it to yourenvironment. You can find a description of the parameters in “SAP CentralServices high availability policy parameters” on page 43 for each SAP highavailability setup to help you configuring the policy and choosing the rightvalues.

v Save the policy XML file. You may want to quit the wizard at any point in time,even if your input is still incomplete or incorrect. If you later resume the policysetup task, the Policy Setup Wizard will pick up the values that you specified sofar.

v Activate the policy. Once you have created a complete and correct version of thepolicy, you can activate the policy using the Policy Setup Wizard or you enterthe sampolicy command from the command line.“Using the wizard to configure and activate the SAP Central Services highavailability policy” on page 42Each SAP Central Services high availability policy consists of a policy templatethat is tailored by using the sampolicy wizard.“Using the wizard to configure and activate the NFS server high availabilitypolicy” on page 123The NFS server high availability policy consists of a policy template that istailored by using the sampolicy wizard.“Using the wizard to configure and activate the Oracle database highavailability policy” on page 138The Oracle database server high availability policy consists of a policy templatethat is tailored by using the sampolicy wizard.

Starting the Policy Setup WizardUse the sampolicy command to start the Policy Setup Wizard. The followingdescribes a subset of the sampolicy syntax that is required to start the Policy SetupWizard. Refer to Tivoli System Automation for Multiplatforms Reference for adescription of the complete set of options and parameters for this command.

Synopsissampolicy –w Templatefilename

Options-w Starts the Policy Setup Wizard.

ParametersTemplatefilename


If you start the Policy Setup Wizard, an xml template is used as input file. Definevalues for the variables contained in the xml template before you can activate thecorresponding policy. The Policy Setup Wizard will fill all values based on yourinput. Please observe the following when specifying the xml template file as inputparameter:v After installing System Automation for Multiplatforms, all template files are

located in one of the subdirectories of /usr/sbin/rsct/sapolicies. If you startthe Policy Setup Wizard for a particular xml template for the first time, specifyone of those template files, for example /usr/sbin/rsct/sapolicies/sap/samsap_templateABAP.tmpl.xml.

v The Policy Setup Wizard modifies the xml file and stores the modified version inthe System Automation for Multiplatforms policy pool using the same file name.You must configure the policy pool before you can start the wizard. For moreinformation, see System Automation for Multiplatforms Installation and ConfigurationGuide.

v If you want to change a policy again, specify the template file that is located inthe policy pool when you invoke the Policy Setup Wizard. Your changes areagain stored in the policy pool. Whenever an xml template in the policy pool ismodified, a new backup copy of the original template file is created in the policypool before overwriting the original file with the latest changes. The backup filehas the same name as the original file with a time-stamp appended. This allowsyou to fall back to any previous version of the policy.

v If you want to restore one of the backup versions from the policy pool, invokethe Policy Setup Wizard specifying the corresponding backup file. If you want togo back to the initial version that was delivered with System Automation forMultiplatforms, invoke the Policy Setup Wizard with the corresponding templatefile in /usr/sbin/rsct/sapolicies/<policy>.

v Regardless of whether a template file is located in a product install directory orin the policy pool, there are two ways you can specify the template file as aninput parameter:1. By providing the fully qualified name.2. By using the directory where the template is located as working directory

and specifying just the file name.

Migrating between an old and a new templateIf a new template file is shipped with a new System Automation forMultiplatforms version, the new template file can contain changes. If you enter thecommand sampolicy with the existing template file you receive the information if amigration is necessary. Values that are already specified in an existing template filecan then be migrated to the new template file using the wizard migrate option:

Synopsissampolicy –w -m <TemplatefilenameNew> <TemplateFileNameOld>

Options-w -m: Starts the Policy Setup Wizard in migration mode.

Parameters

<TemplatefilenameNew>: The latest XML template file, installed in/usr/sbin/rsct/sapolicies/<policyname>, containing the phrase tmpl inits filename.

<TemplatefilenameOld>: Any template file that already contains values youwish to migrate to the new policy template.


Using this option, all values of the old template file are copied to the new templatefile and the wizard proceeds based on the new file with the copied values. If it isrequired to specify additional values in the new template they are indicated in theparameter overview with the flag Missing besides a value question.

Overview dialogWhen the Policy Setup Wizard is started, the overview dialog is displayed. It listsall parameters and their current value status. From the overview dialog you canswitch directly to a parameter dialog for each of the parameters. The following isan example for an overview dialog:

Figure 23. Overview dialog

Chapter 5. Using the wizard to configure and activate the policy 143

Dialog elementsDialog header

The dialog header provides a short description of the policy that youcurrently configure. If you configure multiple policies, the dialog headerdescription clearly identifies with which policy you are working. Below thedescription you see the accumulated overall parameter status: OK orIncorrect.

Parameter listThe parameter list provides an entry for each parameter that you canconfigure for this policy. The list consists of three columns with thefollowing values:v # : Contains a sequence number that uniquely identifies the parameter.

Enter this number to switch to the corresponding parameter dialog.v Parameter: Contains a descriptive text for the parameter. If the

parameter represents a yes or no decision, the text may also be thecorresponding question.

v Value:Shows the current value status of the parameter. This can be oneof the following:– OK: A valid value is defined for this parameter.– Missing: Currently no valid value is defined for this parameter.

Although a value might have been previously entered into thetemplate file, this value can become invalid if the system environmentchanges, for example if a NetworkInterface is no longer available on asystem. The wizard will not display invalid values.

– Incorrect: A value is specified for this parameter, but this value is notvalid due to unfulfilled dependencies to other parameters.

The overall parameter status that is shown in the dialog header is OK if thevalue status for all parameters is OK.

OptionsThe options that you can select in the overview dialog are:v ?: Enables or disables to display parameter help information in all

parameter dialogs. Refer to “Parameter help information” on page 154for a description how you can control the display of help information inthe overview dialog as well as in parameter dialogs.

v H: Enables or disables to automatically retrieve the currently definedvalue in all parameter dialogs that support the value harvesting option.A command to retrieve the value may be defined for one or multipleparameters in the XML template file that you have specified whenstarting the wizard. If no harvesting command is defined for any of theparameters in the template, this option is not available. You mayconsider to enable automatic value harvesting before you step throughthe parameter dialogs for the first time. Refer to “Dialog elements” onpage 146 for a description of the value harvesting option in a parameterdialog.

v 0: Saves your changes and quits the Policy Setup Wizard. Changes thatyou have applied to any of the parameter values are saved.

v X: Cancels the Policy Setup Wizard. No changes are saved.

Refer to “Closing the wizard” on page 155 for more details on how to finish thewizard using Finish or Cancel in the overview dialog as well as in parameterdialogs.


Interactions

Type either one of the numbers that are listed in the # column of the parameter listor one of the options and press ENTER.

Skipped parametersGaps of missing numbers in the parameter list indicate that parameters areskipped. In the overview dialog shown in Figure 23 on page 143 you can see thatparameters 15 and 18 are skipped. Which parameters are currently skippeddepends on values that are specified for related parameters. The set of skippedparameters changes depending on your input. An example for a parameter thatdetermines which related parameters are to be skipped is the product version forwhich you configure a policy template. For different product versions different setsof parameters are required. If you select "Version A" for the product versionparameter, all parameters that are required for version B are skipped and viceversa.

Note: The last parameter(s) in the sequence may be skipped. This is not as obviousas actual gaps in the parameter number sequence.

Parameter dialogInvoke a parameter dialog by entering the assigned number in the # column on theoverview dialog. The following are two examples for the parameter dialog # 3. Thefirst one is an example for the case where no value is specified yet.

In the second example, a value is already defined.

Figure 24. Parameter dialog #3 without value definition


Dialog elementsDialog header

The dialog header provides a short description of the policy that youcurrently configure. If you configure multiple policies, the dialog headerdescription clearly identifies with which policy you are working. Below thedescription you see the accumulated overall parameter status: OK orIncorrect.

Parameter informationBelow the dialog header you find some information that helps you toconfigure this particular parameter:v Parameter n of m: The parameter sequence number is located on the

right hand side. n is the number of this parameter that matches thecorresponding number from the # column in the parameter list of theoverview dialog. m is the total number of parameters that need to beconfigured for this policy.

v Parameter description: This is the text that matches the text from theParameter column for this parameter in the parameter list of theoverview dialog.

v Parameter value: Reflects the configured value for this parameter.Figure 24 on page 145 shows a parameter that has no entered value.Figure 25 shows a parameter with a defined value of 7.0.

OptionsThe options that you can select in a parameter dialog are:v 1: Enter a value for this parameter as shown in Figure 24 on page 145 or

edit an existing value as shown in Figure 25. Selecting this option startsthe prompt mode for this parameter which is described in “Enteringvalues” on page 147.

Figure 25. Parameter dialog #3 with value definition


v 2: Return to the previous parameter dialog in the sequence as listed inthe overview dialog in Figure 23 on page 143. This option is notavailable for the first parameter.

Note: This option may also be not available although the number of thecurrent parameter is larger than "1". In this case all proceedingparameters are skipped. Refer to “Skipped parameters” on page 145.

v 3: Display the next parameter dialog in the sequence as listed in theoverview dialog in Figure 23 on page 143. This option is not availablefor the last parameter.

Note: This option may also be not available although the number of thecurrent parameter is smaller than the total number of parameters. In thiscase all following parameters are skipped. Refer to “Skippedparameters” on page 145.

v 4: Retrieve the currently defined value from the running system. Theparameter value in the parameter information section is filled with theharvested value. This option is only available if a command is defined toactually retrieve the value in the XML template file that you havespecified when starting the wizard. If a value is already shown in theparameter information section, it will be overwritten with the harvestedvalue. You can overwrite again a harvested value by using option 1. Ifyou have enabled automatic value harvesting in the overview dialog andno value is specified yet, the harvesting function is performedautomatically as soon as you open the dialog.

v ?: Display help information for this parameter. Refer to “Parameter helpinformation” on page 154 for a description how you can control thedisplay of help information in the overview dialog as well as inparameter dialogs.

v +: Return to the overview dialog.v 0: Save and exit the Policy Setup Wizard. Changes that you have applied

to any of the parameter values are saved.v X: Cancel the Policy Setup Wizard. No changes are saved.

Refer to “Closing the wizard” on page 155 for more details on how to quit thewizard using Finish or Cancel in the overview dialog as well as in parameterdialogs.

Interactions

Type one of the options and press ENTER.

Entering valuesEach parameter dialog lets you select an option. If you select option 1, the dialogswitches to the value prompt mode where you can select or enter a value.Depending on the kind of value that you specify, different prompt modes are used.The following value prompt modes apply:v Single value: Enter a value.v Multiple values: Enter a set of different values. For the multiple value prompt

mode, you can enter one of the following types:– List of values: Enter multiple values, where each value is one row in the list.


– Table of values with a fixed number of table columns: Enter multiple values,where each value is one cell in the table.

– Table of values with a variable number of table columns: Enter multiplevalues, where each value is one cell in the table and for each table row enterthe number of columns.

v Value section: Select one of a distinct set of valuesv Decision: Select yes or no

In the value prompt mode the Policy Setup Wizard executes syntactical verificationof the input values that you specify.

Examples:

v Enter a numeric value within a certain minimum or maximum range.v Enter an IP address.v Select a value listed in the value selection mode.

Single valueThis prompt mode is used for the most of the parameters to enter a single value.The following is an example for this mode:

Type the requested value and press ENTER. If you want to cancel the valueprompt and return to the option selection mode, press ENTER without typing anyvalue.

Multiple valuesPrompts for a set of different values. The following is an example for this mode inwhich the dialog prompts for a list of values:

Figure 26. Single value prompt


For each value that you want to specify, enter the value and press ENTER. Afteryou have specified the last value, press ENTER without typing any value.

If the set of multiple values that are prompted is a table of values instead ofsimply a list, enter the value for each table cell and press ENTER.v In this case, each table row is referred to in the prompt text as "value" and each

table column is referred to as "sub-value".v If the set of input values is a table with a variable number of columns per row,

the dialog prompts also for the number of values in each row before promptingfor the actual sub-values for that row.

If you want to cancel the value prompt and return to the option selection mode,press ENTER when the dialog asks for the very first value.

Value selectionSelect one value from a set of predefined values. The following is an example forthis mode:

Figure 27. Multiple values prompt


Type the option for the value that you want to choose and press ENTER. If youwant to cancel the value prompt and return to the option selection mode, just pressENTER without typing any value.

Note: This is also an example for how to modify an existing value.

DecisionPrompts for a yes or no decision. The following is an example for this mode:

Figure 28. Value selection mode


Type the option that matches your decision and press ENTER. If you want tocancel the value prompt and return to the option selection mode, press ENTERwithout typing any value.

Note: This is at the same time an example for how to modify an existing value.

Option parametersOption parameters represent a yes or no option. Your decision has the followingimpact:v yes: Configure one or more additional parameters.v no: Additional parameters are obsolete and will be ignored.

As long as no decision is made or no is selected, the additional parameters are notshown in the overview dialog.

Parameter # 17 in the overview dialog in Figure 23 on page 143 is an example foran option parameter that has currently the value no. If you select yes, theparameters are shown in the overview dialog as follows:

Figure 29. Decision prompt


You can find five additional parameters that are nested under the optionparameter. The nested parameters need to be configured as well. Proceed asfollows:1. Switch to the option parameter dialog.2. Navigate from one parameter to the next parameter using the Next option.

The following is the parameter dialog for this option parameter:

Select Next to configure the first nested parameter. If you have multiple nestedparameters to configure for the same option parameter, Next navigates youthrough the list of nested parameters. The number of nested parameters that youhave to define is added to the parameter information on the right hand side.

Figure 30. Option parameter in overview dialog

Figure 31. Option parameter dialog


Wizard dialog flowFigure 33 on page 154 shows the flow between the overview dialog and parameterdialogs, including option parameters and the corresponding nested parameterdialogs.

Figure 32. Nested parameter dialog


Parameter help informationIn addition to the descriptive text, there is also a more detailed help text availablefor each parameter. You can control the display of the help at two differentlocations:1. Overview dialog2. Parameter dialog

Enable and disable help information for all parameter dialogsIn the overview dialog you can enable or disable parameter help information in allparameter dialogs by using the ? option. The text that is associated with the ?option indicates whether help information is currently enabled or disabled. Usingthe ? option toggles the status of help information display.

Table 31. Help information text of the Setup Policy Wizard

Help information text Description

? Enable help for all parameters Indicates that help information is currentlydisabled. Selecting the ? option will thenenable help information in all parameterdialogs.

? Disable help for all parameters Indicates that help information is currentlyenabled. Selecting the ? option will thendisable help information in all parameterdialogs.

Option value = yes?

yes

false

Nested parameter i dialog

Nested parameter i+1 dialog

Nested parameter j dialog

... ...

... ...

... ...

... ...

Overview dialog

Parameter 1

Parameter n

Parameter n+1

Parameter m

Parameter 1 dialog

Parameter n dialog

Parameter n+1 dialog

Parameter m+1 dialog

Nested parameter 1 dialog

Figure 33. Wizard dialog flow


Display help information in selected parameter dialogsIn each parameter dialog you can use the ? option to display help information forthe respective parameter. When you select the ? option, the parameter informationsection of the dialog is extended by adding the help text. The ? option is removedfrom the option list of this dialog.

You can remove the help text only by switching to the overview dialog and disablehelp information for all parameter dialogs. The following is an example for aparameter dialog with help information being shown.

Closing the wizardYou have the following two options to close the wizard:1. Use the 0 option to close the wizard. The following actions are performed:

a. The content of the xml policy template file that you have specified with thesampolicy command to start the wizard is modified. The modified versionis stored in the System Automation for Multiplatforms policy pool. If youwant to change a modified policy again, refer to “Starting the Policy SetupWizard” on page 141. This section describes the different options andconsiderations for modifying policies.

b. A parameter and value summary is created in HTML format. The generatedHTML table is similar to the tables that are included in “SAP CentralServices high availability policy parameters” on page 43. The file is alsosaved in the policy pool with the same name as the template and theextension .html.

Figure 34. Help information in parameter dialogs


c. If all parameter values are correctly specified which is indicated by theoverall parameter status OK, then you are prompted for policy activation.Refer to “Activating the policy.”

2. Use the X option to cancel the wizard. Confirm your cancel request to quit thewizard without any saved changes. No parameter summary is created and youdo not have the opportunity to activate the policy.

The Finish and Cancel options are available in the overview dialog as well as ineach parameter dialog. The actions described are executed independently from thetype of dialog in which the option is selected.

Activating the policyIf you used the 0 option to end the wizard and all parameter values are correctlyspecified with overall parameter status OK, you can activate the policy. Thefollowing is an example for the dialog asking to activate the policy:

Depending on the option that you select, one of the following actions is performed:1. Yes, activate as new policy: The policy is activated as a new policy. This is

equivalent to invoking the commandsampolicy –a –t templatefilename

2. Yes, activate by updating currently active policy: The policy is activated byupdating the currently active policy. This is equivalent to invoking thecommandsampolicy –u –t templatefilename

3. No, save modifications and exit: No policy activation is performed. Yourmodifications are saved and the wizard is closed.

4. No, return to parameter overview: No policy activation is performed.Modifications are not saved and the wizard returns to the overview dialog.

Note: A policy template can only be activated if you have installed the license forthe SAP high availability feature of System Automation for Multiplatforms. Thisapplies for both activation within the wizard and activation via the sampolicycommand.

Figure 35. Policy activation in the wizard


Chapter 6. Troubleshooting the SAP high availability solution

No valid license available when activating SAP Central Services highavailability policy

If you activate the SAP Central Services high availability policy and the messageSAMP0096E was returned, make sure that the IBM Tivoli System Automation forMultiplatforms license for the SAP Central Services high availability policy isinstalled on all nodes in the cluster. If the license is not installed on all nodes, theSAP Central Services high availability policy cannot be activated.SAMP0096E: There is no valid license available for the policy you are trying

to activate.EXPLANATION: The policy you are trying to activate requires a separate license

that is currently not installed.USER ACTION: Install the license and resubmit the command.Policy has been verified.


Chapter 7. Troubleshooting the Oracle high availability policy

Oracle database listener does not start up

ProblemIf the Oracle database listener is not started by the IBM Tivoli System Automationfor Multiplatforms policy successfully, check the following:1. Start the database listener using Oracle commands:

#su - <oracle_user>#lsnrctl start

2. The following message should be displayed:The command completed successfully.

ExplanationIf the Oracle database listener is started manually using the command tnslsnr,ensure that the listener port is not blocked by another process. The default port is1521. This command is not supported to use the System Automation forMultiplatforms policy.

Solution1. Stop the tnslsnr process by using the following command:

#ps -ef | grep tnslsnr

If an entry like the one below is found, the Oracle listener was startedmanually using the command tnslsnr:#oracle 13749 13708 0 19:21 pts/2 00:00:00 tnslsnr

2. Terminate this process and reset the IBM Tivoli System Automation forMultiplatforms Oracle database listener resource if necessary.

3. Log on as Oracle user.4. Start the database listener manually, enter:

lsnctrl start

Such a start will be detected by IBM Tivoli System Automation forMultiplatforms.

If the listener does not start up after you ensured that the port is not blocked, referto the Oracle database documentation.

Oracle database process is monitored unknown

ExplanationThe Oracle database process is monitored as Unknown when the following situationis observed:

The MonitorCommandScript for the Oracle database searches for the processora_pmon_<Oracle_SID>. If this process is found, the script checks whether thedatabase is open for requests using the following commands on the node wherethe state Unknown is monitored:


#su - <Oracle_user>#sqlplus /nolog#connect / as sysdba#select status from v\\$instance;#exit

If the select statement returns the status OPEN, the MonitorCommandScriptreturns the status Online, in which the database allows transactions.

Example output for status OPEN:sqlplus /nologSQL*Plus: Release 11.2.0.1.0 Production on Fri Feb 4 17:56:37 2011Copyright (c) 1982, 2009, Oracle. All rights reserved.SQL> connect / as sysdbaConnected.SQL> select status from v$instance;STATUS------------OPEN

For any other state, for example STARTED, the MonitorCommandScript returns thestatus Unknown, since the database is neither Online nor Offline, but also not in anintermediate state like Starting or Stopping.

Example output for status STARTED:sqlplus /nologSQL*Plus: Release 11.2.0.1.0 Production on Fri Feb 4 17:53:39 2011Copyright (c) 1982, 2009, Oracle. All rights reserved.SQL> connect / as sysdbaConnected.SQL> select status from v$instance;STATUS------------STARTED

SolutionTo resolve the Unknown state, execute the following steps on the node whereUnknown is monitored:#su -<Oracle_user>#sqlplus /nolog#connect / as sysdba#shutdown immediate


Chapter 8. Troubleshooting the NFS tiebreaker

If the System Automation for Multiplatforms NFS tiebreaker is started with optionLog=2, additional trace information is written to syslog.

NFS tiebreaker on LinuxOn the NFS v4 server system, the output of the command rpcinfo –u localhostnfs needs to contain the lineprogram 100003 version 4 ready and waiting

The nfsstat –s command returns details on the current status of NFS. Thecommand dmesg|grep -i nfs displays NFS related kernel messages from the pastfew minutes.

NFS tiebreaker on AIXOn the NFS v4 server system, the output of the command rpcinfo –u localhostnfs needs to contain the lineprogram 100003 version 4 ready and waiting

The nfsstat –m command returns details on the current status of NFS. You mayrestart the NFS related daemons with the following commands:stopsrc –g nfsstartsrc –g nfs


Using IBM Support Assistant

IBM Support Assistant is a free, standalone application that you can install on anyworkstation. IBM Support Assistant saves you time searching product, support,and educational resources and helps you gather support information when youneed to open a problem management record (PMR) or Electronic Tracking Record(ETR), which you can then use to track the problem.

You can then enhance the application by installing product-specific plug-inmodules for the IBM products you use. The product-specific plug-in for TivoliSystem Automation for Multiplatforms provides you with the following resources:v Support linksv Education linksv Ability to submit problem management reportsv Capability to collect traces

Installing IBM Support Assistant and the Tivoli System Automation forMultiplatforms plug-in

To install the IBM Support Assistant V4.1, complete these steps:v Go to the IBM Support Assistant Web Site:

www.ibm.com/software/support/isa/

v Download the installation package for your platform. Note that you will need tosign in with an IBM user ID and password (for example, a MySupport ordeveloperWorks® user ID). If you do not already have an IBM user ID, you maycomplete the free registration process to obtain one.

v Uncompress the installation package to a temporary directory.v Follow the instructions in the Installation and Troubleshooting Guide, included in

the installation package, to install the IBM Support Assistant.

To install the plug-in for Tivoli System Automation for Multiplatforms, completethese steps:1. Start the IBM Support Assistant application. IBM Support Assistant is a Web

application that is displayed in the default, system configured Web-browser.2. Click the Updater tab within IBM Support Assistant.3. Click the New Products and Tools tab. The plug-in modules are listed by

product family.4. Select Tivoli > Tivoli Tivoli System Automation for Multiplatforms.5. Select the features you want to install and click Install. Be sure to read the

license information and the usage instructions.6. Restart IBM Support Assistant.


http://www.ibm.com/software/support/isa/

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document inother countries. Consult your local IBM representative for information on theproducts and services currently available in your area. Any reference to an IBMproduct, program, or service is not intended to state or imply that only that IBMproduct, program, or service may be used. Any functionally equivalent product,program, or service that does not infringe any IBM intellectual property right maybe used instead. However, it is the user's responsibility to evaluate and verify theoperation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matterdescribed in this document. The furnishing of this document does not give youany license to these patents. You can send license inquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 10504-1785U.S.A.

Licensees of this program who wish to have information about it for the purposeof enabling: (i) the exchange of information between independently createdprograms and other programs (including this one) and (ii) the mutual use of theinformation which has been exchanged, should contact:

IBM CorporationMail Station P3002455 South RoadPoughkeepsie New York 12601-5400U.S.A.

Such information may be available, subject to appropriate terms and conditions,including in some cases, payment of a fee.

The licensed program described in this document and all licensed materialavailable for it are provided by IBM under terms of the IBM Customer Agreement,IBM International Program License Agreement or any equivalent agreementbetween us.

For license inquiries regarding double-byte (DBCS) information, contact the IBMIntellectual Property Department in your country or send inquiries, in writing, to:

Intellectual Property LicensingLegal and Intellectual Property LawIBM Japan, Ltd.1623-14, Shimotsuruma, Yamato-shiKanagawa 242-8502 Japan

The following paragraph does not apply to the United Kingdom or any othercountry where such provisions are inconsistent with local law: INTERNATIONAL


BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS"WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OFNON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULARPURPOSE. Some states do not allow disclaimer of express or implied warranties incertain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors.Changes are periodically made to the information herein; these changes will beincorporated in new editions of the publication. IBM may make improvementsand/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM Web sites are provided forconvenience only and do not in any manner serve as an endorsement of those Websites. The materials at those Web sites are not part of the materials for this IBMproduct and use of those Web sites is at your own risk.

If you are viewing this information softcopy, the photographs and colorillustrations may not appear.

Trademarksv IBM, the IBM logo, ibm.com, AIX, DB2, developerWorks, HACMP, NetView,

Tivoli, Tivoli Enterprise, Tivoli Enterprise Console, WebSphere, and z/OS aretrademarks of International Business Machines Corporation in the United States,other countries, or both. IBM Redbooks and the IBM Redbooks logo areregistered trademarks of IBM.

v Adobe, Acrobat, Portable Document Format (PDF), and PostScript are eitherregistered trademarks or trademarks of Adobe Systems Incorporated in theUnited States, other countries, or both.

v Microsoft, Windows, and the Windows logo are trademarks of MicrosoftCorporation in the United States, other countries, or both.

v Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. inthe United States, other countries, or both.

v Linux is a registered trademark of Linus Torvalds in the United States, othercountries, or both.

v Red Hat and all Red Hat-based trademarks are trademarks or registeredtrademarks of Red Hat, Inc., in the United States and other countries.

v UNIX is a registered trademark of The Open Group in the United States andother countries.

v Other company, product, and service names may be trademarks or service marksof others.


Index

Special characters<sapsid>adm

granting access 29

AABAP

ERS 33ABAP high availability setup 5ABAP policyJava policy 39about this guide ixapplication server group 41ASCS 33

installing 18, 88, 89prerequisites 18

ASCS high availability policyresource overview 30

audience of this guide ix

Ddatabase

different high availability cluster 10high availability 11installation setup 10, 87same high availability cluster 10

database hostfailure scenario 17

domainsetting up 25

Double Stackhigh availability policy 63

Ee-mail address xienqueue replication server

failure scenario 14enqueue server

failure scenario 14ERS

ES 39ES

ERS 39example

two-node setup 3, 85

Ffailover node

installation 19failover scenarios

SAP high availability solution 80, 98

Ggroup

application server 41

group (continued)host agent 42SAP Web Dispatcher 42SAProuter 41

Hhigh availability

failure scenarios 13, 87high availability setup

Java (SCS) 6highlighting x

Iinstallation

verifying 21, 89installation setup

NFS high availability 11, 87installing

SAP policy feature 26, 89system automation 24, 89

installing system automationcluster nodes 24

ISO 9000 x

JJava

ERS 37Java (SCS)

high availability setup 6Java high availability policy

resource overview 34Java SAP Central Services 37

Mmigrating

SAP Central Services high availabilitypolicy 28

NNFS

policy parameter 114NFS policy

verify 124NFS server

activating 109AIX policy parameter 114configuring 103, 109failure scenario 17installing 103Linux policy parameter 118planning 102policy 101, 109setting up 103wizard 123

OOracle

activating 132automation preparation 135configuring 132policy 127, 132

verify 139policy parameter 135using the wizard 138

Oracle databaseinstalling 129planning 128setting up 129

overviewSAP Central Services high availability

policy 1, 85

Ppolicy feature license

installing 26policy parameter

ABAP 43java 53SAP HANA high availability 94

policy paramter 43policy wizard

option parameters 151prerequisite knowledge for this guide ixprimary node

installation 19publications ix

RRSCT

related information x

SSAP

high availability concepts 1, 85instance directories 10single point of failure 1

SAP Central Services high availabilitypolicy

activating 30creating 29migrating 28overview 1, 85setups 30verifying 30

SAP HANAsingle point of failure 85

SAP HANA high availability policyactivate 93configure 93

SAP HANA high availability solutioninstalling 88


SAP HANA system replicationautomation

setups 90SAP high availability Connector

enabling 27, 29SAP high availability solution

activate 42activating 30, 90checklist 82configure 42configuring 30, 90failover scenarios 80, 98planning 17starting 78, 96stopping 78, 96verifying 78, 96

SAP Host Agentfailure scenario 17

SAP policy featurepackaging 26restrictions 27

SAP profile parametermodifying 29

SAP profilesconfiguring 20, 89

SAP Web Dispatcherfailure scenario 17

SAP Web Dispatcher groupABAP 42Java 42

SAProuterfailure scenario 17

SAProuter group 41SCS

prerequisites 18SCS HANA high availability setup 86SCS high availability setup

ASCS 4Java 4

shared diskconfiguration data 105exported data 107

syslog daemonenabling 25

system automationprerequisites 24

Ttiebreaker

setting up 25trademarks 166

Uupgrading

SAP cluster 28

Vverifying

SAP high availability solution 78, 96volume group

AIX 108

Wwhat's new

4.1 xiiiwizard

activate policy 141configure policy 141SAP HANA high availability

policy 93SAP high availability solution 42


Readers’ Comments — We'd Like to Hear from You

Tivoli System Automation for MultiplatformsHigh Availability Policies GuideVersion 4 Release 1

Publication No. SC34-2660-04

We appreciate your comments about this publication. Please comment on specific errors or omissions, accuracy,organization, subject matter, or completeness of this book. The comments you send should pertain to only theinformation in this manual or product and the way in which the information is presented.

For technical questions and information about products and prices, please contact your IBM branch office, yourIBM business partner, or your authorized remarketer.

When you send comments to IBM, you grant IBM a nonexclusive right to use or distribute your comments in anyway it believes appropriate without incurring any obligation to you. IBM or any other organizations will only usethe personal information that you supply to contact you about the issues that you state on this form.

Comments:

Thank you for your support.

Submit your comments using one of these channels:v Send your comments to the address on the reverse side of this form.v Send a fax to the following number: (+49)+7031-16-3456v Send your comments via email to: [email protected] Send a note from the web page: http://www.ibm.com/support/knowledgecenter/SSRM2X/welcome

If you would like a response from IBM, please fill in the following information:

Name Address

Company or Organization

Phone No. Email address

Readers’ Comments — We'd Like to Hear from YouSC34-2660-04

SC34-2660-04

IBM®Cut or FoldAlong Line

Cut or FoldAlong Line

Fold and Tape Please do not staple Fold and Tape

Fold and Tape Please do not staple Fold and Tape

NO POSTAGENECESSARYIF MAILED IN THEUNITED STATES

BUSINESS REPLY MAILFIRST-CLASS MAIL PERMIT NO. 40 ARMONK, NEW YORK

POSTAGE WILL BE PAID BY ADDRESSEE

IBM Deutschland Research and Development GmbHDepartment 3248Schoenaicher Str. 22071032 BoeblingenGermany

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

__

_

IBM®

Product Number: 5724-M00

Printed in USA

SC34-2660-04

tivoli system automation for multiplatforms v4.1: high ...€¦ · availability solution fr om sa...

Documents