oracle and aix best practices v1.3 · • internal read/write caches (0,5 - 64 gbyte) • internal...

IBM Global Services

November 2003 © 2003 IBM Corporation

Oracle on AIX : Best Practices

Dr. Andreas GrubermySAP Technology Services

2

IBM Global Services

Oracle on AIX : Best Practices © 2003 IBM Corporation

Agenda

1 Introduction

2 Design of Disk Layout for SAP/Oracle Databases

3 Performance Tuning for the Journaled Filesystem

4 AIX 5L and the Enhanced Journaled Filesystem (JFS2)

5 Multiple SAP Databases on a Single System

3

IBM Global Services


Agenda

1 Introduction





4

IBM Global Services


Introduction

§ Knowledge transfer from SAP infrastructure projects based on • IBM pSeries, Enterprise Storage Server, EMC Symmetrix,

• Storage Area Networks

• Tivoli Storage Manager and Data Protection for SAP

§ Performance tuning as a holistic approach that involves all components of a system

§ Explanation and advertising of new features implemented by hardware and software components

"A best practice is a technique or methodology that, through experience and research, has proven to reliably lead to a desired result." http://searchVB.com

5

IBM Global Services


Oracle Performance Tuning

List of Priorities§ Business Rules

§ Data and Application Design

§ Logical Structure of the Database

§ Database Operations and Access Paths

§ Memory Allocation

§ I/O and Physical Structure

§ Resource Contention

§ Underlying Platform(s)

Introduction > Oracle Performance Tuning

AIX / Hardware

AIX / Hardware

AIX / Hardware

6

IBM Global Services


Agenda

1 Introduction





7

IBM Global Services


Old Style Disk Layout for SAP/Oracle Database

§ Use of single and separated disks • SCSI-Attachment

• SSA-Loops

§ Placement of files according to the following criteria• Type of Files (Redolog, Archives, Datafiles)

• Data / Index Tablespaces

• Master Data / Transaction Data

§ Elimination of arising hot spots via reorganization of tables into separate tablespaces

§ Problem• Wasting of disk resources (e.g. Redologs)

• Complex and difficult administration of scattered data files

Disk Layout > Old Style

8

IBM Global Services


New Paradigm for SAP/Oracle Database

§ Intelligent Disksubsystems • Internal Read/Write Caches (0,5 - 64 GByte)

• Internal RAID-5 or RAID-10 (or similar) implementations

• SCSI or Fibre Channel-Attachment

§ Oracle's S.A.M.E Strategy (Stripe And Mirror Everything)

Disk Layout > New Paradigm

IBM Enterprise Storage ServerEMC Symmetrix / DMX

§ Use "Middle" for ESS arraysto minimize seek

§ Use "Outside Edge" of disksPlacement Policy

§ Use RAID-5 or RAID-10§Mirror for data protectionMirroring

§ Stripe across ESS RAID Arrays§ Physical Partition striping (16/32 MB)

§ Stripe almost everything§ Don't subset by individual disks§ Hardware striping for Log Files§ Use 1 MB stripe size

Striping

9

IBM Global Services


Disk Layout Example for SAP/Oracle Database

Disk Layout > Example

10

IBM Global Services


Golden Rules for ESS (Shark)

§ Number of Disksubsystems • Should be based on throughput needs

• Should not be based on capacity needs (Terabytes)

§ Decision Support Systems (e.g. BW)• 100-200 GByte Data per 2GBit Fibre Channel Port

• Moderate Cache Requirements

§ OLTP Systems• 200-300 GB Data per 2GBit Fibre Channel Port

• Large Cache Requirements

§ Number of RAID Arrays according to throughput requirements• 50 MB/s sustained sequential Read/Write Performance

Disk Layout > Golden Rules for ESS (Shark)

11

IBM Global Services


Agenda

1 Introduction





12

IBM Global Services


Journaled Filesystem vs. Raw Devices

§ "Raw Devices" correspond to Logical Volumes in AIX• Disk devices (/dev/rhdisk10) are not allowed to be accessed directly

§ Filesystems are easier to manage for most administrators

§ Journaled Filesystems ensure the integrity of the filesystem structure and data

• Locking of filesystem structures prevents simultaneous access

§ JFS access to data always uses the AIX Virtual Memory Manager and the real memory as a buffer for files

• Improves performance when data is accessed multiple times

• Oracle data blocks are buffered twice and consume memory

Performance Tuning for JFS > JFS vs. Raw Devices

13

IBM Global Services


Areas of Tuning

ç Oracle Parameters init.ora

ç Asynchronous I/O

ç Read-Ahead

ç File Caching

ç Database Layout

ç Multipath and SAN

ç Cache and internal access

ç Number and speed of drives

Performance Tuning for JFS > Areas of Tuning

14

IBM Global Services


File Caching

§ Memory is categorised into two different types• Computational memory: Consists of the pages that belong to

working-storage or executable files.

• File memory: Consists of the remaining pages. These are usually pages from permanent data files in persistent storage.

§ AIX is using up all memory until only "minfree" pages are left• The percentage of memory that is used for file cache (numperm) can

be seen with the command /usr/samples/kernel/vmtune (AIX 4.3 + 5.1)/usr/bin/vmstat -v (AIX 5.2)

• minperm If numperm falls below this level, the page-replacement algorithm steals both file and computational pages, regardless of repage rates.

• maxperm If numperm rises above this level, the page-replacement algorithm steals only file pages.

Performance Tuning for JFS > File Caching

15

IBM Global Services


Virtual Memory Manager (VMM) Tuning

§ Tuning of the file cache will only improve performance if the system was memory constrained before

§ Tuning is meant to avoid duplicate buffering of Oracle data blocks which leads to a waste of physical memory

§ If the SAP R/3 database server has more than 2 GB memory and isonly used for SAP services

• minperm should be set equivalent to ~ 150 MB

• maxperm should be set equivalent to ~ 300 MB

• In case of 3 GB memory this corresponds to /usr/samples/kernel/vmtune -p 5 -P 10

§ The number of pages on the freelist• minfree = 120 * Number of CPUs

Performance Tuning for JFS > VMM Tuning

16

IBM Global Services


Tuning Read-Ahead

§ VMM tries to anticipate the future need for pages of a sequential file by observing the pattern a program uses to access the file.

• minpgahead The number of pages read ahead when the VMM first detects the sequential access pattern. If the program continues to access the file sequentially, the next read ahead will be doubled.

• maxpgahead The maximum number of pages the VMM will read ahead in a sequential file.

§ Recommended values for SAP database servers • minpgahead 2

maxpgahead 8 * #CPUs rounded to the next power of 2maxfree minfree + maxpgahead

• In case of a 6 CPU System this corresponds to ( < AIX 5.2)/usr/samples/kernel/vmtune -r 2 -R 64 -f 720 -F 784

Performance Tuning for JFS > Tuning Read-Ahead

17

IBM Global Services


Read-Ahead for backup with TSM and TDP

§ Backup of Oracle databases with TSM and TDP heavily rely on read-ahead

• Performance Improvements of 100% have been observed

§ Journaled Filesystem• maxpgahead = 256

§ Enhanced Journal Filesystem (JFS2)• j2_maxPageReadAhead = 256

§ High values of read-ahead might have negative effects on cache efficiency of disk subsystems for SAP production

• Switch to these parameters only during backup

Performance Tuning for JFS > Read-Ahead for backup

18

IBM Global Services


Asynchronous I/O § Asynchronous I/O is an AIX operating system feature consisting of

• Request Queue for I/O write requests

• Kernel processes (aioservers) that take requests from the queue

§ Programs must use the corresponding application programming interface in order to use asynchronous I/O

• no wait for the completion of a write request

• no blocking of application execution

§ Parameters for tuning asynchronous I/O• minservers The minimum number of aioservers that are started for

asynchronous disk I/O. The default value is 1.

• maxservers The maximum number of aioservers that are started for asynchronous disk I/O. The default value is 10. Since each aioserver uses memory, this number should not be much larger than the expected amount of simultaneous asynchronous disk I/O requests.

• maxreqs Maximum number of asynchronous disk I/O requests that can be stored in the queue. The default value is 4096.

Performance Tuning for JFS > Asynchronous I/O

19

IBM Global Services


AIO Tuning Recommendations

§ Asynchronous I/O is treated like a device and can be managed viasmit aio

• autoconfig should be switched on, to make it "available" after reboot

§ Recommended starting values are• minserver 2 or Number of CPUs -1, whatever is larger

• maxserver Two times the number of datafiles

• maxreqs 12288

§ The aioserver processes are started as oracle child processes• for large SAP systems this number may become very large and the

maximum number of processes per user has to be adoptedchdev -l sys0 -a maxuproc=2000

§ The number of active aioservers can best be monitored with the performance tool "nmon"

• http://www-106.ibm.com/developerworks/eserver/articles/analyze_aix/

Performance Tuning for JFS > AIO Tuning Recommendation

20

IBM Global Services


Agenda

1 Introduction





21

IBM Global Services


New features with AIX 5L

§ Support for POWER4 processors and LPAR technology

§ Workload Management for CPU, Memory and I/O

§ Alternatively 32-Bit or 64-Bit AIX Kernel• 64-Bit Kernel recommended for JFS2

§ Enhanced Journal Filesystem (JFS2) • Max. Filesize 16TB (vs. 64 GB)

• Max. Filesystemsize 16 TB (vs. 1 TB)

• Dynamic number of i-nodes

• B-Tree directory structure

• Higher performance for creation and deletion of files

§ AIX 5.2 implements new commands for performance tuning• Administration via smit

• Reboot safe

AIX 5L and JFS2 > New features

22

IBM Global Services


Direct I/O for JFS2 with AIX 5L

§ Reads under cached I/O - cache miss

AIX 5L and JFS2 > Direct I/O

§ Reads with direct I/O

23

IBM Global Services


Implications and restrictions of direct I/O

§ In order to activate direct I/O, the filesystem must be mounted with the option "mount -o dio"

§ Read-Ahead is disabled for these filesystem

§ The alignment of the read requests must be on 4K boundaries• In AIX 5.2 ML01 this alignment must be according to "agblksize",

a parameter which is specified during file system creation

• If the alignment criteria is not met, normal cached access takes place but the cache is discarded immediately

• Oracle Online-Redologs should be located in separate filesystems with an allocation group blocksize (agblksize) of 512 Byte

§ Applications that see performance benefits when using raw logical volumes for storage are likely to benefit from the use of Direct I/O

• e.g. Oracle

AIX 5L and JFS2 > Implications of direct I/O

24

IBM Global Services


Locking and serialization

§ Filesystem Inodes contain meta information about files or directories• if a file is changed, also the inode is changed

• if the inode is changed the file might be unchanged

§ Inode locks prevent concurrent modifications of files that lead to inconsistencies of data

• JFS2 uses a read-shared write-exclusive lock

• Read and write serialization granularity is only at file level

§ Database applications are more sophisticated and have a finer granularity concerning locking and serializaton of access to data

• Table locking

• Row-level locking

AIX 5L and JFS2 > Locking and serialization

25

IBM Global Services


Concurrent I/O

§ AIX 5.2 ML01 provides JFS2 with the concurrent I/O feature• Multiple threads can simultaneously perform read and write operations

on a shared file

• The option can either be used to open files or a whole filesystem can be mounted withmount -o cio

§ If only a subset of files in a filesystem should be opened in concurrent mode, the subdirectory can be mounted as namefs • mount –v namefs –o cio /somefs/subsomefs /somefs

§ Direct I/O is implicitly used with concurrent I/O

§ Extension or truncation of files results in a change of the inode lock from shared to exclusive during the time of modification

AIX 5L and JFS2 > Concurrent I/O

26

IBM Global Services


Benchmark results

AIX 5L and JFS2 > Benchmark results

27

IBM Global Services


Tuning Parameters for JFS2

§ New tuning commands with AIX 5.2 replace vmtune and schedtune• vmo, ioo, schedo

§ Values are preserved after reboot, if pre AIX 5.2 compatibility mode is switched off• chdev -l sys0 -a pre520tune=disable

§ JFS2 file cache does not use the minperm/maxperm values• maxclient Hard limit of file pages for JFS2 and NFS and must be set

smaller than or equal to maxperm

§ Read-Ahead parameters must be set separately for JFS2 via ioo• j2_minPageReadAhead

• j2_maxPageReadAhead

AIX 5L and JFS2 > Tuning Parameters for JFS2

28

IBM Global Services


Agenda

1 Introduction





29

IBM Global Services


Motivation for consolidation of databases

§ Servers have become more and more powerful• Provide more performance than can be used by one database

• Less operating system images to maintain

• Number of SAP systems and databases is exploding, budgets are shrinking

§ Workload Management enables balancing of resources• Better usage of computing resources, including I/O and network

• Peak usage can be satisfied

§ Dynamic Logical Partitioning enables on-demand resources• CPUs and memory can be moved around freely

• Storage technology enables flexible disk assignment

• Hot-swapping of adapters

Multiple SAP Databases > Motivation

30

IBM Global Services


Guidelines and hints for Oracle§ Naming and numbering conventions

• Enforce User-ID and Group-ID standards

• Use naming conventions for Networks, Volume Groups and Volumes

§ Oracle Installation Information• Work with symbolic links for each SID for the installation directories

/oracle/jre, /oracle/oui, /oracle/inventory(SAP OSS Note 350251)

§ AIX kernel and Oracle kernel extensions• Oracle 8 is only supported as 32-Bit version on AIX 5L

• Oracle 8 kernel extensions only work with the 32-Bit AIX 5L kernel

• Oracle 8 Patch 2896876 (Metalink note 231901.1) provides extension for 64-Bit AIX 5L Kernel

§ Oracle listeners• Separate listeners should be configured for each instance with unique

port numbers

Multiple SAP Databases > Guidelines and hints

31

IBM Global Services


Guidelines and hints for SAP

§ Transport Directory

• Each system environment should have a separate transport directory, e.g. /usr/sap/trans<SID>

• The following variables should be set in the SAP systemDIR_TRANS=/usr/sap/trans<SID>DIR_EPS_ROOT=/usr/sap/trans<SID>/EPS

§ Performance Collector saposcol

• Start the saposcol from an SID independent path such as /etc/sap/saposcol

• Create an entry in the inittab filemkitab –i strload saposcol:2:once:su – sidadm –c /etc/sap/saposcol –l

Multiple SAP Databases > Guidelines and hints SAP

32

IBM Global Services


Workload Management

§ Workload Management included in AIX at no additional costs

• Easy setup for SAP and Oracle (30 Minutes)

• Passive mode for testing, performance monitoring and accounting

§ Management of CPU Resources

• Management of memory easier with SAP and Oracle parameters

§ Soft limits and hard limits possible

• No wasting of resources

§ Classification via User-IDs or Tags

• orasid and sidadm are perfectly suited for easy classification

• Separation of classification of Dialog/Background workprocesses through direct assignment

Multiple SAP Databases > Workload Management

33

IBM Global Services


Workload Management Example

Multiple SAP Databases > Workload Management Example

Superclasses for 4 SAP systems I1A,IPC, KPC and IPB Subclasses SAP, Oracle DB,

Live Cache for APO system I1A

34

IBM Global Services


Summary

§ Modern disk subsystems facilitate database layouts through largecaches and internal striping and mirroring

§ Correct configuration of file cache in AIX prevents waste of memory

§ Sufficient asynchronous I/O server processes are key to good performance

§ JFS2 offers almost raw device performance with concurrent I/O

§ Workload Manager in passive mode ideal for monitoring

36

IBM Global Services


Agenda

1 Introduction





6 References

37

IBM Global Services


References

§ IBM Redbooks and Redpapers at www.redbook.ibm.com

• "A holistic approach to a reliable infrastructure for SAP R/3", SG24-5050

• "Consolidating multiple SAP systems on one pSeries", REDP-3626-00

§ SAP Market Place at service.sap.com

• "Database Layout for R/3 installation under Oracle"

§ Oracle Whitepapers at technet.oracle.com

• "Optimal Storage Configuration Made Easy"

§ Enterprise Storage Server white papers for Oracle and SAP

• http://www.storage.ibm.com/disk/ess/whitepaper.htm

§ AIX Whitepapers, e.g. on Concurrent I/O

• http://www-1.ibm.com/servers/eserver/pseries/library/wp_aix_lit.html

oracle and aix best practices v1.3 · • internal read/write caches (0,5 - 64 gbyte) • internal...

Documents