memory matters in 2011

© 2011 IBM Corporation

1

zZS29 Memory Matters in 2011Martin Packer

IBM System z Technical University – Vienna , Austria – May 2-6



NoticesThis information was developed for products and services offered in the U.S.A.

Note to U.S. Government Users Restricted Rights — Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.



Trademarks

This presentation contains trade-marked IBM products and technologies. Refer to the following Web site:

http://www.ibm.com/legal/copytrade.shtml






Abstract

Memory - hardware, z/OS and associated middleware - continues to be an exciting subject, with enhancements coming thick and fast. Meanwhile its usage and management is often poorly understood.

This presentation brings performance practitioners and capacity planners up to date with topics on both real and virtual memory, with DB2 discussed as a key memory-consuming product.

There will also be an opportunity to discuss what instrumentation and capabilities are needed for the future. (Martin is in active discussions with hardware and software development organisations on both the function and instrumentation fronts.)



Topics

System-Level– Paging Subsystem Design– Dump Space– 1 MB (Large) Pages– z/OS R.10 64-Bit Common– ADDR64– System Area Above 2GB

Coupling Facility

DB2

Conclusion



System-Level


IBM System z Technical University – Vienna , Austria – May 2-6 Systems That The “New” z/OS R.8 Algorithm Might Impact

Design point is for “zero paging” considerations– Most production systems

Systems likely to page:– Small production systems– Development and Test systems

• Is “Development” really so low priority these days?

A potential mitigator for paging systems:– Frequently-referenced pages generally “more important”

Pragmatic choice:– Whether to allow some memory constraint– Even more pragmatic choice:

• Whether to configure for some unusual peak– (See later foils on dumping)



Paging Subsystem Design

Zero paging is not a reason to skimp on your paging subsystem

– Even more memory to dump when you have to• At least be able to dump your big address spaces eg DBM1• It’s preferable to be able to dump the entire system

Rule of Thumb:

– Have your free paging space be at least 1.5 times your real storage• Gives you a chance at containing "big dump" cases

Another Rule of Thumb:

– Don't let your page data sets get above 30% full• Contiguous Slot Algorithm degrades above that point

These Two Rules of Thumb complement each other

– But consider them a starting point in your design discussions

Consider the value of PAV now that ASM supports it

– But still consider the importance of proper I/O design



Paging Subsystem Design …

It’s possible to have two page data sets on the same volume

– Optimism that performance is OK with PAVS enabled• IOSQ observed to be 0• Disc observed to be non-trivial but probably the case with 1

– Paging I/O not cached

– Motivation: To use large volumes exclusively for paging

APAR OA20749 allows larger page data sets

– Limit up from 4 GB to 44.9 GB• 64K cylinders

– Available with z/OS R.8 onwards

– 1 large page data set can have 2 PAVs• 2 smaller ones can have 4

In a not-so-recent customer case 2107 greatly outperformed 2105

– And ASM was observed to route paging I/O towards the 2107 data sets

HyperPAVs work properly for page data sets as of R.10



Dump Space

Dumping managed at system level by CHNGDUMP (CD) operator command

– CD SDUMP,BUFFERS=nnnM• Target value for real frames

– CD SDUMP,MAXSPACE=nnnnnnnnM• Maximum concurrent virtual space

– Defaults to 500MB

• If you run out dumps can’t be taken• Partial dumps a possibility

– Likely to impede diagnostics

Dumping much faster if all in memory than a combination of paging space and memory

– A trade off between availability and memory provisioning

Subsystems may be smart in what they dump and in what order–To maximise the likelihood of getting adequate diagnostics



Dumping Enhancements in z/OS R.12

Batch page-in I/O operations during SDUMP capture eliminates much of the I/O delay

Data captured will no longer look recently referenced–This data will be paged-out before other potentially more important

data

Certain components now exploit a more efficient capture method in their SDUMP exits

–For example GRS for SDATA GRSQ data, IOS for CTRACE data, and configuration dataspaces



1MB (Large) Pages●Issue: Translation Lookaside Buffer (TLB) Coverage shrinking as % of memory size

● 1 MB Pages allow for a single TLB entry to fulfill many more address translations than 4KB pages

● 1 MB Pages will provide exploiters with better TLB coverage

●Only fixed above the bar virtual can be backed by 1 MB pages● No paging

●Mixed 4 KB and 1 MB running the rule

●OA25485: NEW FUNCTION - FACILITY CLASS FOR RSM LARGE PAGE ACCESS● Allows non-Authorised applications to use IARV64 REQUEST=GETSTOR

PAGEFRAMESIZE=1MEG or MAX● Facility Class is IARRSM.LRGPAGES

●Exploitation by Websphere Application Server V7● Java heap● Java 6 SR 1 or later with -X1p1M flag

●Buffer Pools in DB2 10

●OA32001: LFAREA PARAMETER INCONSISTENT WITH VALIDITY CHECKS● Corrects calculation if LFAREA=nn% used to be % of >2GB real



z/OS R.12 Large Page Support

Nucleus–Note: Nucleus is in 31-bit storage–Expected to improve performance for z/OS itself

Large Page Coalesce support–During page storage shortage situations available large pages

converted to 4KB pages–Later 4KB pages will be coalesced to form 1MB pages–OA31116: VARIOUS LARGE PAGE COALESCE AND RECOVERY

FIXES• Implements this and fixes some other problems• For Release 10 and 11 also



z/OS R.10 64-Bit Common IARV64 REQUEST=GETCOMMON

Allows sharing above the bar across every address space– In a much simpler fashion than Shared 64-Bit

Controlled by HVCOMMON

Any address space can access storage without explicitly having to request access

To limit overlays only allowed if:– Authorized code – System keys 0-7

Storage can be DREF, Fixed– Unlike Shared

Storage needs to be explicitly freed

Requires explicit exploitation– To achieve 31-Bit Common VSCR



RMF Support For 64-Bit Common

Type 71:

– Number of 64-bit common memory objects allocated

– 64-bit common memory frames backed in real storage

– 64-bit fixed common memory frames

– 64-bit common memory auxiliary storage slots

– Additional information on shared memory objects:

• Number of 64-bit shared memory objects allocated

• High virtual shared memory frames backed in real storage



RMODE 64 For Non-Executable Modules

New with z/OS R.11

The LOAD macro supports new keywords to identify an area above 2G in virtual for a directed load:

– eg ADDR64 keyword as the analog of ADDR

The system does not verify that the module truly can be relocated above 2G– For example, it may have a 4-byte address constant to another part of itself– This is consistent with current LOAD with ADDR behavior

The system does not verify that the module is truly non-executable.– If attempt is made to transfer flow of execution to the module, results are

unpredictable• Also true if any executable code is placed above 2G by any mechanism



USEZOSV1R9RULES

Setting to NO changes GETMAIN / STORAGE OBTAIN behaviour–Generally a safe thing to do–However may result in ABEND0Cx, overlay of storage, or other

problems• Documented in APAR OA27291

Setting to NO has been observed to improve DB2 startup behaviour:–Particularly the opening of a large number of VSAM data sets–Note also DB2 APAR PM17542 and z/OS R.12 MEMDSENQMGMT:

• Uses in-memory structures to decrease ENQ time for large numbers of data sets

USEZOSV1R9RULES introduced with z/OS Release 10



System Area Above 2GB

A new system area will be created in the address space 64-bit map

– Analogue of (E)LSQA

Memory objects allocated in the System Area will start at x’8_00000000’

An authorized program can issue IARV64 REQUEST=GETSTOR, LOCALSYSAREA=NO|YES to indicate that the memory object should be allocated from the System Area above 2GB

In fork processing, during the 64-bit copy phase, memory objects that are allocated in the system area will not be copied to the child space



Coupling Facility


IBM System z Technical University – Vienna , Austria – May 2-6 Coupling Facility Memory

Much more static usage than most z/OS LPARs– Though varying traffic suggests different demands for memory

“White space” considerations apply– Duplexing obviously lowers the requirement

Main categories:– Dump– Structures

• Structure sizing and management is an important topic– Available

Instrumentation:– Type 74 Subtype 4– (Type 70 Subtype 1 just gives memory at ICF LPAR level)– Exploiter-level statistics



Coupling Facility Memory - Structures

Each structure type has different memory management considerations– e.g “False Lock Contention” avoidance in lock structures– Not just structure type but exploiter

• e.g how VSAM RLS Cache works vs DB2 Group Buffer Pools

Structures occasionally need to increase in size when upgrading to a later CFLEVEL

Use CFSIZER website to size CF structures– http://www.ibm.com/systems/z/cfsizer/ – Most IBM Product structures catered for

• Given a product name it tells you which structures you want– And suggests a size

• Produces an “initial” configuration

– For example DB2 structures are likely to need tuning

> In fact CFSIZER probably suggests to you too few GBPs• Has good Help

http://www.ibm.com/systems/z/cfsizer/



Coupling Facility Memory – Structures ...

4 Potential sizes:– Current size - R744SSIZ– Maximum size (MAXSIZE) - R744SMAS– Initial size (INITSIZE) - R744QSIZ– Minimum Size (MINSIZE) - R744SMIS

AUTOALTER can change current size– For all structure types– e.g Directory Entry / Data Element ratio for cache structures

• Increases the size of the portion that's constrained

AUTOALTER is NOT perceived as a “workload spike” handler– It's really for gradual workload growth

Current usage and limits for contents of structure in 74-4– Also Lock Structure Contention / False Contention numbers– True understanding requires exploiter instrumentation

• e.g DB2 Statistics and Accounting traces



Subsystems and Applications



DB2



DB2

DBM1 Address Space is biggest user of real memory

– IRLM, MSTR and DIST address spaces generally smaller• Rare to see virtual storage issues with these 3

Main DBM1 virtual storage usage in 2 areas:

– Buffer Pools• In 64-Bit Virtual in V8• WLM Automatic Buffer Pool Management in V9

– Thread-Related Storage• Used for callers of DB2 services i.e. applications• In 31-Bit Virtual in V8

Other areas generally smaller

– But not always • e.g. EDM Pool

– Sizes usually related to applications• But not linear with concurrent threads

– e.g. Prepared Statement Cache, RID Pool, Sort Pool



DB2 Memory Effectiveness Instrumentation

SMF 100 Records

– Give virtual memory view of buffer pools

• Virtual pools, dataspace pools (and hiperpools)– And thresholds

• All these are variable using ALTER BUFFERPOOL command• Effectiveness of buffer pools• Group Buffer Pools

– Reinforced by SMF 101 application-level reporting

– Some other areas

• e.g Logging, EDM Pool



DB2 Virtual Storage Instrumentation

IFCID 225 breaks down into eg– Long-Term used

• eg Buffer pools– Used unrelated to threads– Thread related storage

• All threads– Answers general questions about scalability– Actual groupings are GETMAINed, Variable, Stack and Fixed– Also gives REAL memory usage

IFCID 225 is enhanced significantly in DB2 Versions 8, 9 and 10– For example to show real memory usage

• Very recent APAR in V10 to show real memory above the Bar IFCID 217 allows you to break down thread-related storage into

– eg Blue, Red and Green threads– Potentially answers questions about scalability based on detailed thread growth "by

colour"

Lots of tuning knobs for DB2 virtual storage



Client H DB2 Virtual StorageSYSA Subsystem DB2A July 14, 2004

0

200

400

600

800

1000

1200

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Hour

MB

Virtual Pools EDM Pool Free EDM Pool Used GBP Castout Comp Dicts Other GETMAINed

Agt Local Non-Sys Ag Local Sys Pipe Mgr RDS Ops RID Pool PSC Control Blocks

PSC Statements Trace Tables Other Variable Fixed Stack



Client H DB2 DBM1 Thread Memory - Headline NumbersSYSA Subsystem DB2A

0

100

200

300

400

500

600

700

Hour

MB

0

100

200

300

400

500

600

700

800

Th

rea

ds

Thread Storage Threads



DB2 Version 8 Virtual Storage



64-Bit Virtual Storage Constraint Relief in DB2 Version 8

Most of thread storage stays below the 2GB bar

– Implies you still need to manage threads

– Experiences suggest some growth in thread-related storage

IRLM also is 64 bit

– Allows many more concurrent locks

– PC=NO is no longer supported• ECSA usage is no longer possible• PC instruction used instead to access IRLM address space

Expect to see significant growth in real memory usage

– As subsystems are now able to grow• Larger buffer pools• Larger areas such as Prepared Statement Cache• Perhaps more threads

– As long-term page fixing is available as an option• Pain if even a single page is not backed by real memory

– As DB2 Version 8 requires more real memory anyhow



64-Bit Virtual Storage Constraint Relief in DB2 Version 9 EDM Pool:

– SKPT and SKCT sections moved above the bar

– CT and PT sections split between above and below the bar

– Hash tables and fixed pools moved above the bar

– BUT what remains is not subject to LRU algorithm• So EDM Pool MUST be sized for the peak usage and then some

Dynamic Statement Cache:– Considerable movement of control blocks to above the bar

DBM1 Internal Monitor:– Reports with console messages when thresholds reached

DDF:– Uses 128GB Memory Object instead of ECSA

• Always created at DB2 startup• All DB2 address spaces are registered to use it• Eliminates most data moves• Reduces total storage usage as only 1 copy• Reduces ECSA usage• Eases use of larger communications buffers

Utilities use Shared Memory Objects– Avoids the CPU to move rows between the Batch Utility and DBM1 Address Spaces



DB2 Long-Term Page Fixing

For each I/O the buffers must be page-fixed and –freed

– Expensive in CPU terms

V8 allows LONG TERM page fixing by pool

– PGFIX(YES)

– Can save significant CPU• Most especially for high I/O rate buffer pools• At the other end of the scale – 100% hits – maybe LRU algorithm good

In V9 Group Buffer Pool castout buffers are ALWAYS long term page fixed

– Separate area from buffer pools

– If high castout I/O rate could be a significant saving

IMPORTANT: Long term page fixing MUST be considered in the light of system conditions:

– For example, don’t page fix to the point of causing other memory users to page to death• Especially important for z/OS Release 8 and beyond



DB2 10

Vast majority of thread storage moved above The Bar–Expectation of 5 to 10 times as many threads supportable–Might allow coalescing of DB2 subsystems

• But caution on the other reasons for maintaining the current number–Might allow other things

• Such as RELEASE(DEALLOCATE) to cut CPU–Some of working storage (some of stack, xproc) remains below The Bar

=> Focus shifts to REAL memory

Reworking of Dynamic SQL Prepared Statement Cache could alter the basis for its sizing

–Two statements which are identical but with DIFFERENT literal values are now treated as the same

• One copy in the cache• More hits per cached copy

–Might allow a reduced cache• Equally, might make Prepared Statement Cache more worth doing

–Still doesn't obviate the benefit of “keeping prepared statements beyond Commit”NOTE: MAXKEEPD could be bigger because area moved above The Bar



DB2 10 ...

In-Memory Pagesets–Data read into buffer pool at startup–Useful for small lookup tables

1 MB Page Size–Requires long-term page-fixing the buffer pools


IBM System z Technical University – Vienna , Austria – May 2-6 Conclusion Virtual and Real memory and the paging subsystem are very much worth managing

– For capacity planning

– For resource optimisation

– Because installations can still face constraints

– Because running out of space can cause an outage

• Whether real or virtual memory or paging space

Consider your stance on keeping memory free for e.g. dump capture

There’s lots of instrumentation to help you

Product capabilities are continuing to evolve

– 2007 saw DB2 Version 9, IMS Version 10 and CICS TS 3.2

– 2008 has seen System z10 EC, z/OS Release 10, WAS V7 and MQ V7

– 2009 saw IMS Version 11 and CICS TS 4.1 and z/OS Release 11

– 2010 saw DB2 10 and z/OS Release 12

It’s important to look at the system, the subsystem & the application all together

– And at the machine level wouldn’t be a bad idea, either

– Applications drive subsystem memory demand

• But supply comes from the system

memory matters in 2011

Documents

ibm corporation

nonibm web sites

local ibm representative

ibm director of licensing

products andor

products andcannot

sample application programs

noticesthis information