best practice for sap performance monitoring for sap scm

Performance Monitoring

for mySAP SCM (4.1 + 5.0) Best Practice for Solution Management

Version Date: September 2005 This version is valid for mySAP SCM 4.1 and mySAP SCM 5.0

The newest version of this Best Practice can always be obtained through the SAP Solution Manager

Contents

Applicability, Goals, and Requirements....................................................................................................2 Best Practice Procedure and Verification .................................................................................................4

Preliminary Information ......................................................................................................................4 1. The SCM System Landscape ..................................................................................................5 2. Basics of Optimal System Performance ..................................................................................6 3. Basics of Performance Analysis...............................................................................................7 1. Prerequisites ............................................................................................................................9 2. Daily System Monitoring ........................................................................................................10 3. General Steps.........................................................................................................................10 Performance analysis of APO / liveCache .................................................................................12

System Load Analysis (ST03N)........................................................................................................13 Parameter stat/dbprocrec...........................................................................................................14

Determining CPU-Intensive COM Routines .....................................................................................15 ABAP Trace (SE30)....................................................................................................................17 Accesses to the Data Cache......................................................................................................23 5. Performance Optimization......................................................................................................42 6. Monitoring Roadmaps ............................................................................................................47

Further Information .................................................................................................................................51

Best Practice: Performance Monitoring for mySAP SCM

© 2005 SAP AG

2

Applicability, Goals, and Requirements To ensure that this Best Practice is the one you need, consider the following goals and requirements.

Goal of Using this Service This Best Practice enables you to set up a concept (or template) for performance monitoring and tuning of SAP’s Supply Chain Management (SCM) solution using the Advanced Planner and Optimizer (APO). This performance monitoring concept aims to:

• Define procedures for regular performance monitoring of APO software and hardware components

• Define criteria for bottlenecks

• Define roadmaps for problem solution and roles and responsibilities for all persons involved in the customer’s support and monitoring organization as well as escalation procedures with respect to APO Basis performance requirements

These procedures ensure the system performance required in order to meet your business needs. These procedures describe APO-specific performance monitoring methods and do not include a detailed description of R/3 Basis performance monitoring technologies which are assumed to be known and implemented.

Alternative Practices The concepts listed below are covered in part by several different Solution Management Optimization (SMO) services, namely SAP System Administration, SAP Customer Program Optimization, and SAP SQL Statement Optimization. Development of a dedicated SAP Performance Optimization SMO service is planned. SAP experts can deliver this Best Practice on-site if you order the above mentioned services.

Additional Best Practices Several other Best Practice documents are required to make your SCM administration and monitoring concept complete:

• Backup and Recovery + High Availability – describes SCM (including liveCache) backup and recovery, disaster recovery scenarios and high availability concepts or requirements

• CIF / Integration – describes the configuration, administration and monitoring requirements for integration scenarios

• Data Consistency Checks – explains the usage of /SAPAPO/OM17 (internal consistency), /SAPAPO/DELTAREPORTx (external consistency) and several application specific consistency checks.

• Monitoring and Administration for SCM / APO – is focused on system monitoring and administration of your SCM landscape

• System Monitoring for mySAP SCM with SAP Solution Manager – is focused on the enabling of SAP APO system monitoring with the SAP Solution Manager

Staff and Skills Requirements To implement this Best Practice, you require the following teams:

Basis Management Team The SCM / APO performance monitoring concept (which this Best Practice aims to produce) should be created by the Basis Management Team. This team combines experts from your company:

System Administrators


© 2005 SAP AG

3

Hardware Administrators

Network Administrators

Necessary or Useful Training Courses:

See the SAP Service Marketplace (http://service.sap.com/education) for the most up to date information on SCM specific training courses.

Prerequisites • Existing experience with SAP Basis 4.x/6.x monitoring tools and functionality, including the

relevant SAP and Database training courses. • Existing experience in SAP Basis performance tuning, including the relevant SAP training

courses.

Duration and Timing Duration

Creating a concept for SCM performance monitoring can take several weeks. It can take months to implement it effectively.

Timing

SAP recommends that you consider the optimization measures listed below during the early project planning stage.

This Best Practice can be used whenever performance problems occur during productive system operation.


© 2005 SAP AG

4

Best Practice Procedure and Verification

Preliminary Information The SAP Advanced Planner and Optimizer (APO) is the planning component of mySAP SCM, the Supply Chain Management solution provided by SAP. SAP APO is used to make strategic, tactical, and operational decisions and supports you in performing the following planning activities:

• Demand Planning (DP)

• Supply Network Planning (SNP)

• Production Planning (PP)

• Detailed Scheduling (DS)

• Deployment

• Transport Planning and Vehicle Scheduling (TP/VS) / Transport Load Builder (TLB)

• Global Available-to-Promise (gATP)

• Sequencing

• Network Design

SAP APO is a planning tool. Execution functions, such as confirmations, goods receipt, purchasing, and so on are performed in the SAP R/3 OLTP system, which contains all functionality for Material Management MM, Sales and Distribution SD, Production Order Processing PP-SFC, Logistics Execution LES, and Controlling CO.

The online transaction processing (OLTP) system – provided by the SAP R/3 system – also provides relevant planning data (master data and transaction data) for the APO System. Products are planned in the APO System, and the planning results are transferred back to the OLTP system. If necessary, the planning can be completed in the OLTP system if not all components of a BOM structure are planned in the APO System.

In general, we recommend that you plan critical products and critical components in APO and non-critical products and components in the OLTP System. However, if you plan a component in APO, you must also plan all components in the Bill of Material (BOM) structure, including the BOM header product in APO. Alternatively, if you plan a component in the OLTP System, you also must plan all sub components in the OLTP System.

The various strategies for using R/3 and APO in combination are called integration scenarios, which utilize additional mySAP functionality – the Core Interface (CIF), which itself is part of the mySAP PlugIn.


© 2005 SAP AG

5

1. The SCM System Landscape The substantial components of an SAP SCM system landscape are summarized in the following table and shown schematically in the subsequent illustration.

SAP APO System The SAP Advanced Planner and Optimizer System facilitates the strategic, tactical, and operational planning processes.

APO consists of several software components. Firstly, as in any R/3 system, a relational database system (RDBMS) known as the APO-DB, an SAP R/3 Basis, and the APO application programs. In addition, there is a separate, very fast object-oriented SAPDB database, called liveCache, and a number of programs that execute elaborate optimization algorithms, called optimizers. These components can run on the same or on different servers.

OLTP System The Online Transaction Processing System covers functionality for sales and distribution, material and inventory management, controlling, shop floor control, logistic execution and so on.

OLAP System An Online Analysis Processing System such as SAP Business Warehouse provides cumulated historical data as a basis for future extrapolation purposes in APO Demand Planning.

Plug-In

OLTP System

RDBMS

Plug-In

OLTP System

RDBMS OLAP System

RDBMSRDBMS

SAP APO System

liveCache

liveCache

Optimizers


© 2005 SAP AG

6

The following diagram shows the relationship between APO application software components and the databases:

Optimizer Application

Server

liveCacheliveCache

COM RoutinesAPO DB

RFC:

call to Optimizer executable

DATA request:

RFC call to APO application server

APO Application

Server

SAPGate-Way

SAPGate-Way

SNP

PP/DS

CTM

SEQ

ND

VSR

Dialog WP

Batch WP

Batch WP

Dialog WP

Dialog WP

DATA request:

Direct to liveCache via PP/DS optimizer and database interface

DATA request:

ABAP Open SQL

DATA request:Calling COM Routine

2. Basics of Optimal System Performance The basis of the high system performance is established at the very beginning of the project. It is necessary to understand the key factors affecting performance during the project planning phase in order to design the system landscape, the core business processes and the hardware configuration in an optimal way to avoid future performance bottlenecks.

There are two general approaches to performance optimization. First, unnecessary workload should be avoided. This goal is reached by the optimal design of the core business processes. This aspect is the subject of analysis by the application team. Second, remaining workload should be distributed evenly over all system resources. The Technical administration team deals with this aspect.

Close cooperation between functional and technical departments is necessary in order to optimize system performance. Almost all performance problems can be helped through optimization potential which can be found in both functional and technical areas.

Definition of SCM/APO Performance Analysis

What is SCM/APO Performance Analysis SCM/APO performance analyses are detailed monitoring and tuning procedures performed on a system and used to provide optimal and balanced system performance.

When to Analyze Performance System performance should be monitored regularly according to the monitoring concept designed for each system. Performance analysis must be carried out when system performance indicators are over threshold values as well as if dangerous trends are discovered during routine system monitoring. Another indication for performance analysis is a feedback from end users who meet performance degradation.


© 2005 SAP AG

7

Goals of SCM/APO Performance Analysis The main goal of performance analysis is to find system resource bottlenecks and remove them in order to provide the required system performance.

Who does SCM/APO Performance Analysis System administrators in cooperation with application specialists should carry out performance analysis. Specialists from your hardware partner, administrators of other systems, and application developers can be involved in system performance analysis if necessary. If a customer cannot solve the problem in reasonable time without help from SAP, he should open an OSS message.

Tools for Performance Analysis SAP provides the integrated tools necessary for performance analysis. These are the general basis analysis transactions ST02, ST03N, ST04, OS07, DB02, SQL trace ST05, ABAP trace SE30, liveCache analysis transaction LC10, APO analysis transactions /SAPAPO/OM14, /SAPAPO/OM13, /SAPAPO/OM16, /SAPAPO/OM11 and others.

3. Basics of Performance Analysis Analysis of the system performance has to be constantly carried out by the system administrator or service engineer during system operation. A Handbook with all monitoring activities should be created for each system and landscape as whole (see also Best Practices “Monitoring and Administration for SCM/APO”).

When a service engineer starts the performance analysis, he needs to take many objective and subjective factors into consideration in order to be able to properly diagnose the problem. These factors are customer business processes, subjective user reaction, season peak workload, other application runs etc. If the end users face long response times, corrective action should take place. To avoid performance bottlenecks, proactive measures should be carried out.

General Criteria Performance analysis can be triggered by a message that business transactions are running unacceptably long or longer than before. These messages can come from end users, who have noticed runtime increase during everyday work or from system administrators during permanent planned system monitoring, or from planned proactive SAP remote services such as the EarlyWatch Alert. If the alarm comes from system administrators or proactive services (EWA), this is an indication of an effective monitoring strategy.

Performance analysis can also be triggered when daily system monitoring indicates dangerous trends or sudden problems.

Runtime increase can be caused by increased data volume, newly introduced applications and/or transactions, inadequately implemented modifications, increased volume of data stored in database or operation environment etc.


© 2005 SAP AG

8

The specific of an APO system is that there are no common runtime criteria for all business transactions. These runtime should be estimated on a benchmark during project implementation phase. In the situation when a message about runtime increase comes, the following variants are possible:

• Runtime exceeds the limits defined on the benchmark. The situation should be identified as bottleneck and actions should be carried out immediately.

• Runtime is still within the limits defined on the benchmark, but grows continuously. Before you perform the steps described below in this Best Practice, check the following points:

o Analyze business processes considering possible workload peaks (e.g. month closing), estimate peak duration (e.g. several days), and system capacity.

o Analyze business processes considering possible permanent workload increase and system capacity.

o Analyze last modifications in the system.

o Analyze other applications running on the same hardware.

o Consider system status.

System Status The approaches to performance analysis are different in productive, test and development system.

Productive System

Performance tuning in the productive system has the highest priority. There are very definite requirements for system availability. A possible solution strategy should be the use of a test landscape and/or establishment of an additional test landscape in order to test recommended changes, new configuration variants, new parameter settings, error corrections, Notes, new releases etc. The procedures listed below are intended for analysis of productive systems.

Test System

During performance analyses in a test system consider possible differences in the system configuration and workload between productive and test systems. It is recommended that the test system be configured as closely as possible to represent the productive system.

Development System

The specifics of a development system are often a small hardware configuration, sufficient workload caused by debugging, numerous expensive SQL statements on the database, many ABAP dumps etc.

End Users End users should be informed about benchmarking results at least for most frequently used transactions and trained to inform the system administrator should transactions run much longer than previously. The users should also be educated to avoid bottlenecks or inconsistencies caused by incorrect user actions.

In general, possible bottlenecks should be identified and corrected before end users can identify performance degradation. If users are reporting worse response times, that should be a reason for immediate performance analysis.

Procedure The performance analysis procedure is a multi-level detailed analysis with steps covering different aspects of system operation. Any conclusion about problem origin and possible solutions can be carried out after many sequential and interconnected analysis steps. The procedures described below cover only the minimal set of steps to be performed during performance analysis.


© 2005 SAP AG

9

1. Prerequisites

Monitoring of the Software Component Versions Before any step in the performance analysis is started, the service engineer should collect complete information about running software components and their releases (versions, support packages, etc.).

The information about APO, BW, SAP Basis, ABAP and application versions can be obtained from Main Menu -> System -> Status. For complete information about liveCache Kernel version and Build, use transaction LC10 (see detailed description below). To display the change list number of the currently installed SAP APO COM object, use transaction /SAPAPO/OM13 -> “Versions”, or, in case of unavailability, transaction /SAPAPO/OM04. The change list number identifies the COM object build. To check the assignment between change list numbers and COM object build numbers see SAP Note 326494. To view identifiers and versions of optimization programs available on optimization servers, use transaction /SAPAPO/OPT09.

The information about database, liveCache and kernel patches is displayed in transaction SM59 -> “Release information”. For the installed SAP GUI patch look in SAP Logon -> About SAP Logon.

The installed qRFC version can be monitored in transaction SMQ1 -> Execute (F8) -> Information -> Version.

This information should be used to restrict the search area while searching for known problems and solutions as well as entered into an OSS Message if one is opened.

Minimal System Runtime A sufficient part of the values displayed in the monitoring transactions described below are average values calculated over certain period (minutes, hours, days, weeks, months, since system start etc.). The periods immediately after system start generally do not reflect stable system operation conditions (for example, buffers are still empty and hit rates are low) and therefore are not representative. In order to collect valid statistics, analysis should be carried out after a minimal system runtime with representative user activity.

The minimal system runtime with representative user activity means a time period since the system start, when the most frequently used business transactions in different applications have been executed at least once. This can be few hours for large systems with many tens or hundreds of users or a working day for smaller systems. Statistics for COM Routines are worthless until at least 1.5 times as much data has been read from liveCache as the liveCache contains. LiveCache should reach a state with a nearly constant Data Cache hit rate.

As an objective criterion, the quantity of the COM Routines calls from applications can be checked in transaction LC10 -> “liveCache Monitoring” -> “Problem Analysis” -> “Performance” -> “Monitor” -> “External DBPROC calls” (or LC10 -> “liveCache Monitor” -> “SQL Stats” -> “External DBPROC calls” in the “old” transaction LC10, see below). The statistics are considered to be representative when the value is at least 50 000.

EarlyWatch Alert It is strongly recommended that remote EarlyWatch Alert services be activated. For detailed information, see SAP Notes 207223 and 215361. EarlyWatch Alert reports provide important input information for further performance analysis.


© 2005 SAP AG

10

Remote Connections It is strongly recommended that all available remote connections to the productive and test systems be configured and tested during the system tuning project phase, and before productive start in order to obtain fast and competent support from SAP in case of problems. These connections are:

• R/3 Support (SAP Note 35010)

• EarlyWatch (SAP Note 35010)

• TCP/IP Connection (SAP Note 35010)

• Telnet Connection (SAP Note 37001 for Unix systems)

• SAP-DB Connection (SAP Note 202344)

• PCanywhere Connection (SAP Note 100740 for Windows systems)

2. Daily System Monitoring Monitoring data obtained during routine daily system monitoring is a very important input for performance analysis. Collected historical data and statistics provide a basis for problem location and diagnosis.

A daily system monitoring handbook should be worked out taking into consideration SAP standard recommendations, recommendations given during remote and onsite service delivery, and system specifics. For detailed information regarding recommended daily monitoring activities see the Solution Manager Best Practice “Monitoring and Administration for SCM/APO”.

3. General Steps The following general aspects related to the project management and system landscape should be analyzed first If a problem cannot be detected after the first general steps, a detailed analysis of the system key performance indicators should be carried out.

A full analysis procedure includes steps relevant for SAP Basis, APO Basis, liveCache, Core Interface and Optimizers. SAP Basis performance analysis is described only in general terms in this Best Practice. For more detailed information see the book: Thomas Schneider, “SAP Performance Optimization”. The part describing Core Interface performance analysis procedures is described in a separate Best Practice “Manage APO Core Interface in SAP APO (3.x) / mySAP SCM (4.x)”.

Business Processes System capacity should be defined taking into consideration possible predictable workload peaks during the sizing procedure in the project implementation phase. Generally each business process has peaks of activity, but amount and frequency of peaks are different for different operations. If response time increases, it is very important to clarify with application specialists, whether a peak which has lead to response time increase, was considered during system set up. If not, the situation should be classified as a bottleneck and tuning and/or re-sizing actions taken immediately. If an unplanned permanent workload increase takes place, the situation should be also classified as a bottleneck.

Software Changes in the System Software changes in the system include new releases, versions, builds, support packages, patches, coding corrections, customer-written programs, customizing, parameter settings etc. Occasionally, these modifications can conflict with current system settings or can overwrite previously carried out customizing and therefore cause performance degradation. Therefore, all software modifications should be very carefully tested in a quality system. No modifications should be implemented in a productive environment without testing.


© 2005 SAP AG

11

If the problem appears immediately after the implementation of a modification, deactivate the changes, restore the system to the status from before the modification and analyze the problem in the test system.

Other Applications Running Depending on the own landscape strategy, a customer can install other applications on the same hardware as the APO system. During performance analysis, keep in mind that resource shortage can be caused by a bottleneck in another application. Check if any other application runs on the same hardware and tune or remove it if necessary.

Standard SAP Services The SAP onsite and remote services delivered in the recent past and planned for the near future should be checked. Service reports can contain important proposals, which can help in problem solution. If the services are scheduled soon, check whether it is necessary to re-schedule them for an earlier time in order to analyze problems in detail.


© 2005 SAP AG

12

Performance analysis of APO / liveCache If you see generally poor performance in the APO system, you have to find out what component is causing the bottlenecks.

Tasks to Perform for a general system check • Determine the share of response time, using transaction ST03N, Workload Analysis to get

analyse which component (ABAP, Database, liveCache) has performance problems • Check the operating system paging on the liveCache server. A maximum of 10% of main

memory should be paged in/out each hour. • Check the processor load of the liveCache server. The average idle time should be greater

than 20%. This check is especially important when components other than liveCache (APO, database) are also running on the liveCache server.

• Check the buffers in ST02 • Test the network performance using transaction /SAPAPO/OM13, but only if APO and

liveCache are running on different servers • Determine the OMS data cache hit rate in liveCache . The data cache hit rate should be at

least 99.8%. If the data cache hit rate is less than 99.8%:

Check the size of the APO demand planning versions, using transaction /SAPAP/OM16

Check whether the share of history pages in the data cache is greater than 40% of the OMS data pages If so, check the number of long-running transactional simulations

• Run transaction /SAPAPO/OM_PERFORMANCE in the APO system This mini-benchmark displays runtimes for various liveCache-intensive test runs. The runtime of each individual test series should be less than 15 seconds. Times of around 5 seconds are optimal.

• Check the output of the Database Analyzer

Workload Analysis with Transaction ST03N The Workload overview provides information about response times and its parts for different task types. This view allows identification of symptoms of CPU bottlenecks, non-optimal DB access, inadequate configuration of work processes, and network and front end bottlenecks. General principles and guidelines on how to use transaction ST03N and how to interpret the statistical information provided are discussed in SAP Training Course BC315 “Workload Analysis” and described in the SAP book: Thomas Schneider “SAP Performance Optimization”. During APO system analysis the following points have to be checked in Expert Mode.

In the first monitoring step, you have to find out where the most time is required. This will help you determine whether the liveCache is responsible for the performance problems at all or whether the APO server or the network, for example, are the cause of the bottlenecks. If the liveCache is the problem, you have to find out whether a general problem is involved or whether merely individual transactions are extremely CPU-intensive and have long runtimes.

Analysis Procedure Perform a general system health check using “Workload overview”. Use the guidelines listed in the above-mentioned references and try to locate the problem area, if any, in SAP Basis. In comparison to these guidelines, the APO system analysis also has it’s own specifics, for example, there is no common typical transaction runtime value such as 1 second for dialog transaction in R/3 system.


© 2005 SAP AG

13

Duration of APO transactions depends on application and amount of data to be processed and can be estimated by a benchmark. Relatively low database activity, and high background and RFC activity are typical for an APO system.

System Load Analysis (ST03N) Transaction ST03N provides initial information for determining where the time is spend. In particular, the following key figures provide useful information about the share of time spent in the liveCache:

• % DP Proc. Time; tab strip “Parts of response time”. % DP Proc. Time displays the liveCache share (in percent) of the total response time.

• Ø Response time compared to Ø DB Proc. (ms) in tab page “Times”. Shows the average absolute liveCache share of the average response time in milliseconds.


© 2005 SAP AG

14

If the key figures indicate a low (<30%) liveCache share, this means the liveCache is only partially responsible for the poor performance. In this case, the main cause for generally poor APO performance probably does not lie with the liveCache. On the other hand, while a high (>60%) share is not necessarily an indicator for general liveCache problems, APO performance is clearly dominated by liveCache, which means you will find the most likely optimization potential there. A general upper limit for dialog response times, like you know from R/3 (around 2 seconds maximum dialog response time and database share of under 50%) has not been defined yet for APO.

If you see high dbproc (liveCache) time, you should analyse further which com routine consumes the most time. Therefore you have to enable the collection of related statistical data

Parameter stat/dbprocrec Parameter stat/dbprocrec enables you to determine CPU-intensive COM routines. If you set the value of the parameter in transaction ST03N (collector and performance DB >> statistics records & file >> online parameters >> Dialog Step statistics) to “10”, for example, the 10 most CPU-intensive COM routines are determined. This parameter is an online parameter that takes effect immediately when set.

Caution: The COM routines with the highest accumulated runtimes are determined, not the COM routines with the highest individual times for a call.


© 2005 SAP AG

15

Determining CPU-Intensive COM Routines To display CPU-intensive COM routines, select the transaction to be examined in ST03N and then choose “Single records”.

Do not forget to select the instance whose parameters you want to set!


© 2005 SAP AG

16

Double-click on a line

The “Top 5”

The 5 most CPU-intensive COM routines


© 2005 SAP AG

17

Create a temporary variant for changing parameters such as the maximum runtime

Note: Single Records in ST03N are only evaluated of the last 30 minutes (Workload Total selected) or 2 hours (Workload for a specific instance selected). If the transaction was executed outside this time frame, no data will be displayed in ST03N>>Single Records. To expand the time window, choose Button New Selection in the Single Record Window and choose a matching time window.

Note: Up to SAP basis 6.20, COM routines of background jobs are not stored in the statistical record of the background job, but in the statistical record of report RSBTCRTE. Therefore to obtain the most expensive COM routines of background jobs, it may be necessary to identify the corresponding report RSBTCRTE that started at the same time as the batch job. Use transaction STAD (!) to identify and display the data.

Now you should see which com routine consumes the highest part of response time. If you think that the this com routine is the reason for the bad performance please open an OSS Message to get help from SAP Support for further analysis..

ABAP Trace (SE30) If you see high CPU time in ST03N or STAD files, the ABAP trace helps you to identify runtime-intensive transactions, ABAP programs and function modules, as well as the most CPU-intensive COM routine calls. You can even use the trace to analyze ongoing transactions. This feature is particularly useful when an ongoing transaction in a system cannot be stopped without further consequences. The information provided on individual transactions is more detailed than that in ST03N. You will see the programs which are consuming the highest part of the CPU related part of the overall run time

To date, the overview display does not differentiate between APO database and liveCache accesses. However, you can analyze liveCache accesses – that is, calls from COM routines – in the detailed display.

SE30 initial screen


© 2005 SAP AG

18

Overview of analysis results

Switch to a parallel session if you need to analyze an active work process.


© 2005 SAP AG

19

Display of analysis details

The display should be sorted by “NET” time, in order to display the calls with the highest accumulated runtime.

Calls to liveCache are displayed as Native SQL Procedure Calls.


© 2005 SAP AG

20

SAP Buffer Analysis with Transaction ST02 Another important step in the system performance analysis is SAP buffer analysis with transaction ST02. Important points to check are current usage of SAP memory areas and hit ratio of buffers.

If you found swaps from Table buffers, go to “Detailed analysis menu” and check buffered objects. Sort the table by “Database activity – rows affected” and check the size of the tables at the top of the list. Usually fully or generically buffered customer and SAP standard application tables should not be larger than several Mb. If they are it is recommended that they not be buffered. If you find large buffered tables, consult application specialists about the typical size of these tables. Very large tables can indicate application errors.

For detailed guidelines for analysis see SAP Training Course BC315 “Workload Analysis”, the SAP book: Thomas Schneider “SAP Performance Optimization” and SAP Notes 103747, 97497, 146289, 146528, 33576, 38052, 68544, and 88416.


© 2005 SAP AG

21

Operating System Monitor - Transactions OS07/OS06 Operating system monitors are intended for monitoring local (OS06) and remote (OS07) servers. The most important information which can be obtained from these transactions is CPU statistics, disks, and memory usage.

The data presented on the main screen, is a snapshot, which can be refreshed with 10 seconds interval. Short time workload peaks (for example, only 5% idle CPU) itself do not indicate bottlenecks. Average hourly values can be monitored in “Detailed analysis menu” -> “Previous hours”. Bottlenecks indicators are average idle CPU of less than 20%, average paging in for Windows and paging out for UNIX of more than 20% of main memory, or disk utilization of more than 80%.

In case of a hardware (CPU, memory) bottleneck, transactions with high resource consumption should be identified (use transaction ST03N) and analysed. There are many SAP services recommended for detailed transaction analysis such as EarlyWatch Session Package (remote), onsite SMO Packages “SQL Statement Optimisation” and “Customer Program Optimisation”. If optimisation potential is completed, consider a re-sizing procedure and a hardware upgrade.

Another important aspect is the dynamics of hourly average values. Permanently high values are more dangerous for system performance than natural daily dynamics with workload peak hours, pauses and hours of even workload (for example, a night background job run). You can use transaction OS07 to display operating system information. If the liveCache runs on a separate system, install the CCMS Agent as described in SAP note number 371023.

Double-click for detailed CPU info

Double-click for detailed memory info


© 2005 SAP AG

22

The following parameters are relevant:

Operating system paging drastically reduces overall system performance. Make sure a share of more than 10%/h “paged in” under NT and more than 10%/h “Paged out” physical main memory in UNIX is not exceeded. Remember that all the programs running in a given system are responsible for paging. Therefore, services such as terminal servers should never run on the liveCache server.

• CPU utilization

The idle time should be greater than 20% .


© 2005 SAP AG

23

LiveCache Monitoring liveCache is the most performance critical component in an APO system and should be monitored permanently. The most important steps in liveCache monitoring are monitoring of Data Cache, liveCache Heap, Data Devspaces and COM Routines. These procedures are different depending on the version of liveCache and SAP Basis Support Package.

Transaction LC10 provides important information about liveCache settings and activities, which can be partially interpreted by the customer’s system administrators during primary performance analysis.

Accesses to the Data Cache (Current Status >> Memory Areas >> Caches Table “Cache accesses”) A hit rate of at least 99.8% is required.

A lower hit rate could be due to the following causes:

• The data cache is too small

• Long-running transactional simulations / OMS versions

Exercise caution when interpreting the statistics from LC10.

Composition of the LC data.

Data cache fill level

Hit rate: minimum 99.8%

See Accesses to the


© 2005 SAP AG

24

Reliable data can only be expected with 50,000 or more called COM routines. To display the number of called COM routines, choose Problem Analysis >> Performance >> Monitor >> external DBPROC calls.

In an optimally configured liveCache the Data Cache hit rate should be at least 99, 8% calculated as the number of successful accesses divided by the total number of accesses and multiplied by 100%. Sometimes the reason for a poor Data Cache hit rate is a high number of long running versions or transactions. To keep the consistent state of the versions or transactions, liveCache is forced to store a large number of history pages, which fill the Cache and lead to a roll out of data and history pages to the data devspaces (on the Harddisks). To check this, go to “liveCache Monitoring” -> “Problem Analysis” -> “Performance” -> “OMS versions” in transaction LC10 and check for all the versions older than 4 hours. If there are any, find out the users, who have started those versions and contact the functional department in order to optimise business processes and avoid such long running versions if possible or increase the size of dataCache.

Generally Data Cache filling level should be below 100%. A value below 80% is not critical for system performance. If the data Cache usage reaches 100% while Hit Rate is below 100%, an extension of the Data Cache may be required. If the value is between 80% and 100%, the most important characteristic is the correspondence between the number of OMS data and history pages, which normally should be around 4:1. With lower values, check for long running versions as described above and optimise business processes to avoid long running versions.

If a named consistent view remains open for a long time, the data cache is filled with history pages (before images of objects). The history data cannot be released as long as the named consistent view is open, because the consistent view needs to see the liveCache as it was when the named consistent view started. When the database filling level exceeds 90%, the transactional simulation is deleted and history is released by liveCache garbage collectors. To avoid this situation, schedule the report /SAPAPO/OM_REORG_DAILY to run at least once a day. Among other things, this report deletes transactional simulations older than 8 hours (see SAP Note 208317). Also schedule the report /SAPAPO/OM_DELETE_OLD_SIMSESS to run every 30 minutes. This report checks for zombies of transactional simulations (see SAP Note 490992).

Size of Planning Versions

When Data Cache filling level reaches critical values, it is necessary to define which applications, or planning versions consume the system resources. Transaction /SAPAPO/OM16 provides information about sizes of planning versions in the tab strip “Plng Versions” -> execute.


© 2005 SAP AG

25

To calculate the sizes of the listed planning versions in kBytes, click the word “Calculate” in the last table column. The first column from the left “Size in kB” provides the information about the space in pages occupied by the planning version in liveCache. The second column from the left shows the real data size for the corresponding planning version. The difference between these two values gives an idea about the fragmentation of the pages used by the planning version.

The total size of the Data Cache should be not less than the sum of the sizes of all or the active and inactive (reference) versions.

liveCache Heap Monitoring

The OMS heap is a critical resource in liveCache. It contains copies of OMS objects from Data Cache and data needed for internal purposes in COM routines. Accesses to OMS objects are very fast, several times faster than accesses to data in the global data cache. The maximum size of the OMS heap is configured by the parameter OMS_HEAP_LIMIT

liveCache Heap usage can be monitored under LC10 -> “liveCache Monitoring” -> “Current Status” -> “Memory areas”.


© 2005 SAP AG

26

The configuration parameter OMS_HEAP_COUNT defines the number of heap segments (allocators). Value “Occupied” is the currently used heap memory in bytes. Value “Size” indicates the maximal heap memory usage in bytes per segment since the liveCache server was last restarted. The heap memory can be allocated by application until the value “Size” (sum for all the segments) reaches the value defined by parameter OMS_HEAP_LIMIT. Never set OMS_HEAP_LIMIT to 0, because in this case all of the available virtual memory could be occupied by heap. If a lot of disk space is used the performance will drop dramatically and in the worst case it could lead to a server standstill because of lack of free memory for the operating system.

Detailed information about Heap usage by separate versions can be obtained using SQL Studio command:

SELECT oms_version_id, heap_usage, unloaded FROM oms_versions

The number of COM Routine errors due to memory shortage in Heap can be obtained using another SQL Studio command:

SELECT SUM(OutOfMemoryExceptions) FROM monitor_oms

Out of Memory Exceptions should not occur at all because then the process will be terminated. If you encounter these kind of short dumps in ST22 or if you see it in liveCache the heap memory is too small.

In this case two things should be carried out:

1. check if the data selection can be reduced. This has to be checked by your application team

2. Check if it is possible to increase the OMS_HEAP_LIMIT parameter if you have available memory. If not consider increasing the physical memory of the server.


© 2005 SAP AG

27

Monitoring of Devspaces

Data devspaces can be monitored in transaction LC10: choose “liveCache Monitoring” -> “Current Status” -> “Memory areas” -> “Data area”.

The most important parameter to be monitored is “Used area”. If it exceeds 80%, an alert should be configured in CCMS and new devspace has to be added. At a value of 90% the situation gets critical because the garbage collectors will delete history data to free space. This could lead to short dumps.

“Used area” is the sum of “Permanent used area” and “Temporary used area”. Persistent OMS objects and history pages are stored as permanent pages. Permanent pages are available after a restart of liveCache. Only permanent pages are included in checkpoints and backups. Named consistent views, which were swapped from private OMS cache to global data cache, are stored in temporary pages. All temporary pages are released after a restart of liveCache.

Monitoring of COM Routines The screen LC10 -> “liveCache Monitoring” -> “Problem Analysis” -> “Performance” displays information about the COM routines running in the actual liveCache. Monitoring the COM routines may give indicators for the reason for performance problems caused by the APO application (high runtime, high memory consumption in liveCache heap etc.). Very detailed statistics about COM Routines calls are displayed. Therefore very detailed knowledge is required to understand the impact of LiveCache performance. This section should be carried out by SAP support only.


© 2005 SAP AG

28

Activity Monitoring (Tasks)

The current liveCache activity can be monitored under LC10 -> “liveCache Monitoring” -> “Current Status” -> “Kernel Threads” -> “Task Manager” -> “Active Tasks”.

You can also monitore the current liveCache activity via: LC10 -> “liveCache Console” -> “Active”.


© 2005 SAP AG

29

In a running system, possible states are:

• no display - the liveCache is idle and is waiting for an order from the APO server. There is no acute performance problem to be expected.

• Running - The respective session uses the CPU. 'Running' means that the application is in the liveCache code and not in the code of the COM routine. You may encounter a CPU bottleneck if the number of sessions with statuses 'Running' and 'DcomObjCalled' exceeds the CPU number over a longer period.

• Runnable - task is in kernel code of liveCache and is waiting for a free slot in its thread (UKT)

• DcomObjCalled - The session uses the CPU and is in the code of a COM routine. You may encounter a CPU bottleneck if the number of sessions with statuses 'Running' and 'DcomObjCalled' exceeds the CPU number over a long period.

• IOWait (R) or IOWait (W) - The liveCache session waits for the end of a physical I/O order. This status should not occur in a correctly configured and balanced system, because the liveCache should supply its data requests from the DATA_CACHE. Several sessions with status 'IO Wait' indicate a sequentialization of the whole liveCache on physical I/O and hold a considerable performance risk that should be examined.

• Vbegexcl or Vsuspend - The liveCache session waits for the release of a liveCache-internal lock (analog to Oracle term 'latch'). If several sessions are simultaneously in such a status over a longer period, there is a sequentialization of the whole liveCache on an internal resource.

• Vwait - The liveCache session waits for the release of a lock on a liveCache object. In rare cases, this can be a lock on an SQL table which was explicitly set by the APO application and can only be released by a COMMIT.

• LogIOwait - task is waiting for log I/O performed by the logwriter task

Application PID is the process ID of the connected APO work process, which can be identified in transaction SM50 or SM66 (column “PID”).


© 2005 SAP AG

30

Performance Check with Report /SAPAPO/OM_PERFORMANCE

Report /SAPAPO/OM_PERFORMANCE is a simple benchmark that gives you a rough overview of performance in the liveCache. You should conduct this test during every monitoring run. The “5-second test” contains several individual tests, whose runtimes should lie between 5 and 15 seconds. Longer runtimes are an indication of a general liveCache problem.

If only individual test benchmarks lie above the required maximum 15 seconds, this could be due to the concurrent execution of other transactions. In this case, run the benchmark again.

Report /SAPAPO/OM_PERFORMANCE provides short and simple liveCache runtime measurements for different kinds of APO applications. Report /SAPAPO/OM_PERFORMANCE can be used to compare the performance between different hardware configurations, to compare the performance between different parameter settings, to estimate the influence of network transfers on the performance, and to compare the performance between different APO service packs.

The actual runtime is highly dependent on the actual configuration, hardware and workload. Many different runtimes for different objects indicate performance problems in corresponding applications.

The detail results can automatically be displayed in an Excel sheet (checkbox “Create Excel Sheet”). This provides an easy way to store the results for comparisons with other measurements. Frequent storage of the results of /SAPAPO/OM_PERFORMANCE enables judgment of the influence of alternative configurations.


© 2005 SAP AG

31

System Messages

liveCache system messages are available in transaction LC10 -> “liveCache Monitoring” -> “Problem Analysis” -> “Messages”.

This path provides access to the file KNLDIAG which contains all the liveCache system messages since liveCache start and initialisation log. Error messages can be found under the tab strip “Error messages”. This information can be interpreted by experts and should be provided for SAP Support if required.

Integration of CCMS

The CCMS (Computer Center Management System) integrates most of the monitoring functionality in one transaction RZ20. The usage of RZ20 for liveCache monitoring is described in more detail below.

COM Trace COM tracing is a run time mode of the COM routines which can be switched on and off in transaction /SAPAPO/OM02. COM tracing is always active and by default all fatal errors are written in the COM trace file. A high COM trace level over a longer period can create very long COM trace files and thus potentially fill up the file system. This is avoided with the usage of cyclic COM trace, which is now used by default. COM tracing should only be activated and used by very experienced administrators if asked for by SAP support.

Generally there is no common guideline for estimation of COM Routines runtime. The typical runtimes depend on application and amount of data to be processed. COM Trace provides the information about runtime, memory consumption and number of object accesses for each COM routine. All these data give indications as to which COM routine can be problematic.

Writing COM Trace is performance critical. Follow the instructions given by SAP Support in configuration and running the COM Trace. It is configured in transaction /SAPAPO/OM02 and the results are presented in the transaction /SAPAPO/OM01.


© 2005 SAP AG

32

Database Analyzer The Database Analyzer is an analysis tool for SAP DB and liveCache. It collects performance relevant data with variable time intervals, classifies analysis messages as information and warnings and facilitates the search for possible error causes such as configuration (caches, parameters), synchronization (locks, semaphores, heap), processing of queries (indexes, optimiser statistics), hardware configuration (I/O system).

Database Analyzer functionality is integrated in the transaction LC10 under LC10 -> “liveCache Monitoring” -> “Problem Analysis” -> “Performance” -> “Bottlenecks”. It contains not only current performance values and actual status, but also historical values which can be exported in local files and analysed independently. Installation procedure and references to Database Analyzer descriptions are stated in the SAP Note 530394.


© 2005 SAP AG

33

“Expert Analysis” view provides access to the protocol files with detailed statistics for the selected day.

Performance relevant liveCache Parameters There are many SAP Notes with recommendations for liveCache parameters settings. For more details please refer to the list of recommended SAP Notes below. Normally these parameters are checked during the GoingLive Services. Initial recommendations are given which should be good for the start of production.

Due to increase of workload after some time these parameters should be adapted to the current load. SAP supports you with recommendations within available remote services e.g. earlywatch session.

Here is an overview of the most important parameters. For some of them rough initial values are given.

CACHE_SIZE – Size of the global cache (Data Cache) in 8 kB pages. Detailed recommendations about this parameter are given during the sizing of the system. Roughly this parameter can be set as 50% of the available physical memory. Refer to the SAP Note 487972.

CAT_CACHE_SUPPLY – Maximal size of the liveCache catalog cache in 8kB pages.

KERNELDIAGSIZE – Size of the file for liveCache messages in 8 kB pages. The file is cyclically overwritten.

MAXCPU – Defines the maximal number of processors that liveCache uses for parallel processing of application orders. For dedicated liveCache servers it is recommended that the value be set to the


© 2005 SAP AG

34

number of server CPUs. When other applications are running on the same server with liveCache, follow the recommendations from detailed sizing. Roughly optimization can be started with the value 2/3*(number of CPUs). See also SAP Notes 410002, 425051 and 487972.

MAXLOCKS – Maximal number of SQL locks. Refer to SAP Note 337773 for more details.

MAXUSERTASKS – Maximal number of liveCache connections. Each user task allocates approximately 1.5 MB memory. Minimum: (Number of APO work processes)*2 + 4.Refer to SAP Notes 205220 and 487972 for more details.

OMS_HEAP_COUNT (as of 7.4.2) – Segmentation of the liveCache heap to increase the parallel processing in multiprocessor environments. Unit: Number of segments (min. 1, max. 7 in 7.4.2. As of liveCache version 7.4.3 you can configure: min. 1, max. 128 Initial value: Depends on the operating system (see Note 496318. Also see Note 516661.)

OMS_HEAP_LIMIT - Size of the liveCache heap in kB. Detailed recommendations about this parameter are given during the sizing of the system. Roughly this parameter can be set as 75% (64-bit Operating system).

LOAD_BALANCING_CHK

As of LiveCache software versions 7.4.03 Build 36 or higher, 7.5.00 Build 18 or higher load balancing can be used. With the help of load balancing, user tasks can change the UKT (user kernel thread). This allows the database instance to optimize the use of the processors that are available to the database instance in accordance with the MAXCPU database parameter. See note 695721.

A description of all liveCache parameters can be found in SAP Note: 490958


© 2005 SAP AG

35

Core Interface Monitoring Core Interface uses the technology of queued RFC to provide reliable communication between APO and R/3 systems. General performance of an APO system can be affected sufficiently by problems occurring during data transmission via CIF. These problems can affect performance directly (slow transmission because of insufficient resources) or indirectly (by producing, for example, high database load due to bad ABAP or SQL performance). Sometimes termination of queue processing can cause cancellation of scheduled background jobs and therefore incomplete data development.

The most important monitoring points are qRFC versions installed, status of queue transmission (error messages), status of qRFC tables in database. Remember that the performance analysis of the Core Interface should be carried out in cooperation with the application team in order to interpret the messages correctly and to apply the actions avoiding data inconsistencies.

qRFC Versions Installed qRFC version and supplement can be found in transactions SMQ1 or SMQ2 -> execute -> “Information” -> “Version”.

It is recommended that the same qRFC versions and supplements be used for both communicating systems in order to provide better compatibility, performance and reliability. At the same time it is also recommended that the latest available qRFC Version and supplement be used due to performance enhancements and new functionality. The latest available qRFC Version and installation guides can be found in SAP Note 438015.

Monitoring of the Queue Status Queue transmission process can be monitored in transactions SMQ1 (outbound queues) and SMQ2 (inbound queues). All the error messages appearing in these monitors should be analysed and resolved as soon as possible in order to complete data transmission within the necessary time frame and to avoid delay or cancellation of succeeding jobs.

Monitoring of the Outbound Queues

Transaction SMQ1 offers options for monitoring of specific queues, specific destinations, hanging queues or any combination of these criteria.

The following operation and error statuses can be displayed in the column “Status” for each entry (see also SAP Note 378903): READY The queue is ready for transmission. This status should only be a temporary status. However, in the following case this status can also be a permanent status: a queue was locked manually via transaction SMQ1 or via a program and then unlocked without being activated at the same time. This queue must be activated explicitly.


© 2005 SAP AG

36

RUNNING The first LUW of this queue is currently being processed. If a queue in this STATUS hangs for more than 30 minutes, this may mean that the work process responsible for sending this LUW has terminated.In this case you can activate this queue again. Note that activating a queue in status RUNNING may cause a LUW to be executed several times if this LUW is processed in the target system at that time.We therefore recommend a waiting time of at least 30 minutes before you activate the queue again. EXECUTED The first LUW of this queue is processed. The system waits for a qRFC-internal confirmation from the target system before further LUWs are processed. If a queue in this STATUS hangs for more than 30 minutes, this might mean that the work process responsible for sending this LUW has terminated. In contrast to status RUNNING, this current LUW has definitely been executed successfully. You can Jump to running queues display and activate this queue again without problems. The qRFC Manager will automatically delete the LUW already executed and send the next LUW. SYSLOAD At the time of the qRFC call, no DIALOG work processes were free in the sending system for sending the LUW asynchronously. A batch job for subsequent sending has already been scheduled (see SAP Note 319860 for more details). SYSFAIL A serious error occurred in the target system while the first LUW of this queue was executed. The execution was interrupted. When you double-click on this status, the system displays an error text. You can find additional information on this error in the corresponding short dump in the target system (ST22).No batch job is scheduled for a repetition and the queue is no longer processed. To solve the problem,information from the affected application is required. Refer to SAP Note 335162 for the special error text "connection closed". CPICERR During transmission or processing of the first LUW in the target system, a network or communication error occurred. When you double-click on this status, the system displays an error text. You can find additional information on this error in the syslog (SM21), the trace files dev_rd or dev_rfc*. Depending on the definition in transaction SM59 for the destination used. A batch job is scheduled for subsequent sending. Status CPICERR may also exist in the following cases although no communication error occurred: a qRFC application finds out that a LUW cannot be processed any further due to a temporary error in the application and therefore calls the RESTART_OF_BACKGROUNDTASK function module to prompt the qRFC Manager to cancel the execution of this LUW and to repeat this LUW later in accordance with the specification in transaction SM59.In this case, qRFC simulates a communication error with the text "Command to tRFC/qRFC: Execute LUW once again." If this error occurs very often,you must contact the corresponding application team. STOP On this queue or a generic queue (for example BASIS_*) a lock was set explicitly (SMQ1 or programs).Note that the qRFC never locks a queue in its processing. After having informed the corresponding application team, you can unlock and activate this queue using transaction SMQ1. WAITSTOP The first LUW of this queue has dependencies to other queues, and at least one of these queues is currently still locked. WAITING The first LUW of this queue has dependencies to other queues, and at least one of these queues contains other LUWs with higher priorities. NOSEND LUWs of this queue are never sent but retrieved by a special application. These queues are only used internally at SAP (BW or CRM during communication with Mobile Clients). Even if a LUW was read by the corresponding application (BW, CRM), this status does not change. This LUW is only deleted from the queue if this application confirms collection (collective confirmation possible). Under no circumstances should this status be reset using transaction SMQ1 and the queue activated.


© 2005 SAP AG

37

NOSENDS During the qRFC call, the application determines at the same time that the current LUW is not sent immediately. This is used to debug the execution of an LUW via transaction SMQ1. Contact the corresponding qRFC application to clarify this status since this is either a programming or configuration problem. WAITUPDA This status is set if qRFC is called within a transaction that also contains one or more update functions.As a result of this status, the LUW and thus the queue is blocked until the update has been completed successfully. If this status takes longer than a few minutes, check the status of the update or the update requests using transaction SM13. After a successful retroactive update, the blocked LUW is sent automatically. You can also restart the LUWs manually in the WAITUPDA status without a successful retroactive update (via transaction SMQ1, Reset status, Activate queue). However, you may only perform this action after having informed the qRFC application (for example, APO, BW, CRM). You must do this to avoid possible inconsistencies. This WAITUPDA problem may be avoided as follows: If both qRFC calls and update calls occur within a transaction, qRFC must be executed exclusively within the update. In this case, the qRFC LUW is only created after the update has been completed successfully. Caution: If you are using SAP Basis Releases 4.0x, 4.5x, 4.6A or 4.6B, and an update function with the "collective run" type exists in a LUW, an error in the kernel may cause this status. The queue also hangs in this case. This error has already been corrected with a kernel patch (see SAP Note 333878). RETRY / ARETRY During LUW execution the application has diagnosed a temporary problem and has prompted the qRFC Manager in the sending system via a specific qRFC call to schedule a batch job for a repetition on the basis of the definition in transaction SM59 (Menu: Destination > TRFC Options) ANORETRY During the LUW execution, the application has found a serious error and prompted the qRFC Manager via a specific qRFC call to cancel processing of this LUW. Information from the affected application is required to solve the problem. MODIFY Processing this queue is locked temporarily because the LUW data is being modified.


© 2005 SAP AG

38

Monitoring of the qRFC Tables Where the communication errors described above occur, corresponding entries are written in the tRFC/qRFC tables in the database. These tables are ARFCSSTATE, ARFCSDATA, ARFCRSTATE, TRFCQDATA, TRFCQIN, TRFCQOUT and TRFCQSTATE. They are requested during qRFC communication. When the errors are not resolved in a reasonable time, or when errors appear much faster than they can be resolved, the size of these tables grows and therefore performance degrades dramatically. The number of entries can be checked in transaction SE16 -> “Number of entries”. See SAP Note 375566 for detailed guidelines on how to interpret the data.

Monitoring of Optimizers The main criterion for the optimiser program performance is the runtime of the appropriate background job. This can be monitored in transaction SM37 or with an external monitoring tool, if any. Optimizer run statistics are collected in transaction /SAPAPO/OPT11 “History of Optimization Runs”.

With suspected performance problems, the main object which should be monitored and analysed is the hardware (RAM, CPU) of the Optimizer server. Generally there is no way to tune the performance of an optimizer program itself. The most important task for providing high performance of optimizers is a proper sizing for the optimizer servers. Tuning potential can be found in tuning of the operation system or resizing.

SAP recommends establishing a pcANYWHERE connection to the optimizer servers for remote performance monitoring. It is also possible to configure appropriate nodes in the CCMS transaction RZ20 (see below).

Network Monitoring Once the liveCache and the application servers are installed on the different physical servers, the network between liveCache and application servers contributes to the general system performance. It is recommended that high-speed connections be configured: at least 100 Mbit/sec, or 1 Gbit/sec. The bandwidth of the real connection can be tested in the transaction /SAPAPO/OM13 -> tab strip “Network”.


© 2005 SAP AG

39

In reality, the measured bandwidth will never reach theoretically configured values, but the measured value should be of the same value order as the theory. In the example given above: 28225,81 kbytes/sec corresponds to 225806,48 kbit/sec, therefore 1 Gbit/sec network connection is configured.

Apart from this you could use the NiPing command to check the network between LiveCache and APO Server with help of SAPNote: 458221

Network connections between presentation, application and database servers can be tested in transaction OS06/OS07 -> “Detailed Analysis Menu” -> “LAN Check by Ping”. Select the desired servers for testing and options “1 x ping” or “10 x ping”.


© 2005 SAP AG

40

The test results include information about minimal, average and maximal runtime of pings as well as quantity of lost pings. Typical response times for 4 kByte packages are <20 ms for LAN, <50 ms for WAN and <250 ms for modem connection. Losses should not appear at all.

Alert Monitor RZ20 / CCMS The main advantage of the Computer Center Management System (CCMS) in comparison with other monitoring and management tools is the possibility of monitoring different hardware and software components belonging to different landscapes and produced by different manufacturers and of monitoring different performance indicators of these components in one transaction. This functionality is widely used by the SAP Solution Manager. Installation of the Solution Manager is strongly recommended for the customers with complicated multi-systems landscapes.

Collection of performance statistics for these different components is performed by agents. These agents are partially provided by SAP and pre-configured by default. Other agents can be requested and installed additionally. A customer can also develop and install his own agents for specific monitoring purposes.

CCMS monitoring is carried out in transaction RZ20. Default configuration of the transaction RZ20 provides monitors for operating system, database, R3 instance tasks, R3 Basis, RFC and ALE/EDI interfaces and others. Transaction RZ20 allows monitoring of the current values of system parameters with automatic configurable refresh. These values are rated depending on the configured thresholds as “white” (no threshold is configured), “green”, “yellow” and “red”. The values rated “yellow” and “red” produce alerts in the CCMS that can be sent to system administrators per e-mail or as mobile SMS.


© 2005 SAP AG

41

In the context of APO system monitoring, transaction RZ20 provides functionality for monitoring status and alerts for the whole system and all system components: instances, APO database, and operating systems on the servers, Core Interface and liveCache. Refer to SAP Note 415376 in order to configure liveCache monitoring in transaction RZ20 and to integrate RZ20 monitoring functionality into transaction LC10. CCMS liveCache monitor can be used in transaction LC10 -> “liveCache Alert Monitor”. It is possible to configure monitoring of several liveCaches within one RZ20 monitor.


© 2005 SAP AG

42

5. Performance Optimization When a performance problem is analyzed and located, the next step should be optimization of the parameters settings (system configuration). Some practical recommendations for parameters optimization are described below.

When the configuration changes do not lead to the desired performance improvement, the next steps can be verification of the system sizing and, if necessary, hardware upgrade.

At the same time, the functional department should revise and optimize the core business processes in order to reduce the system workload. Further information can be found on the SAP Service Marketplace, alias \solutionmanagerbp


© 2005 SAP AG

43

Workload Balancing Optimal workload distribution is achieved by changes in system configuration (parameters) in order to balance given workload and utilization of the available system resources. First, workload should be analyzed and structured in order to split it into several logical groups. Then, system resources should be configured in order to provide maximal availability for different specific workload groups (parameters settings).

System Configuration Initial requirements for system capacity are defined during system sizing. Sizing answers questions regarding how many CPUs with defined parameters and how much physical memory is needed for the given workload. How the resources are distributed over physical servers should be clarified with help of the hardware vendor and the system administration team. Either one big server or a landscape of several different servers are possible as long as the requirements of the sizing, and of high availability are met. Please keep in mind that in a distributed landscape you should have a good network connection. Absolute minimum would be 100 MBit/sec, 1 GBit recommended.

Logon Groups Logon groups are used to distribute system users over the instances. Users can be grouped into logon groups by applications, physical locations (countries), company units, etc.

Generally user groups (logon groups) occupy different system resources. For large multi instance systems it is recommended that the central instance be left free from interactive and interface workload in order to be able to access the system for maintenance and analysis in case of performance bottlenecks.

Logon groups can be configured using transaction SMLG.


© 2005 SAP AG

44

Server Groups A group of application servers can be defined using transaction RZ12. Server groups are used as a workload distribution tool in order to assign specific workload to specific hardware resources. This functionality is used in an APO system, for example, in CIF Outbound and Inbound Schedulers. See SAP Note 74141 for more details.

System Communication Outgoing and incoming Core Interface workload can be distributed using Outbound (QOUT, transaction SMQS) and Inbound (QIN, transaction SMQR) Schedulers.

The workload produced on the receiving system depends on the number of requests that have to be processed at the same time. If the number of requests sent simultaneously to the receiving system is high, the performance of the receiving system may be impaired. Therefore, it is desirable to control the resources consumed on the receiving system.

The Outbound Scheduler provides the functionality for distributing the outgoing workload over hardware resources of the sending system (pre-defined server groups) and to limit the outgoing workload in order to avoid a bottleneck on the receiving side.

You can configure the Outbound Scheduler in transaction SMQS.


© 2005 SAP AG

45

The Inbound Scheduler provides the functionality for limiting incoming workload in order to avoid bottlenecks due to insufficient resources. The Inbound Scheduler is configured on the basis of queue names. If new queue names are used, they must be registered, otherwise their entries will not be processed. The Inbound Scheduler does not process the LUWs but triggers their processing in asynchronously started work processes. An RFC server group can be configured to limit the resources available for processing. See SAP Note 369007 for configuration of the Inbound Scheduler.

Core Interface Parameters Detailed description of qRFC and gateway relevant parameters and recommended values is given in the SAP Notes 384971 and 384077. The most important of them are the following.

rdisp/rfc_min_wait_dia_wp – controls how many dialog work processes will not be used for internal RFC processing and for sending RFCs. This number of dialog processes is kept free for online dialog users or for RFCs coming from external system.


© 2005 SAP AG

46

rdisp/rfc_max_own_used_wp – defines allowed percentage of the total number of dialog work processes to be occupied by one RFC user.

rdisp/rfc_max_used_wp – defines allowed percentage of the total number of dialog work processes to be occupied by all RFC users.

rdisp/max_comm_entries – defines number of communication entries for an instance.

rdisp/rfc_max_comm_entries – defines allowed percentage of communication entries occupied by RFC.

rdisp/tm_max_no – defines number of connections to an instance.

gw/max_conn – defines number of logical connections to a gateway.

gw/max_overflow_size – defines maximal swap space for CPIC requests in a gateway.

Database Statistics for RFC Tables During qRFC communication, all remote function calls are documented in the qRFC tables ARFCSSTATE, ARFCSDATA, ARFCRSTATE, TRFCQDATA, TRFCQIN, TRFCQOUT and TRFCQSTATE. The entries exist in the tables only during runtime of an RFC and are deleted upon successful completion of a call. In systems with high CIF communication, the size of these tables can vary over a short time. In order to provide fast access to these tables, the collection of database statistics for Cost Based Optimizer should be deactivated for these tables. For more detailed information see the SAP Note 371068

Furthermore the Index of these communication tables should be frequently rebuilt.

These qRFC and tRFC tables grow and shrink very quickly, which causes their index structure to become highly fragmented even after a short period of time. For Oracle databases, this can lead to database performance problems if the indexes are not rebuilt regularly.

This will significantly reduce the database load and thus result in reduced database response times, especially for all index accesses against the qRFC tables. Refer to SAP Notes 407125 and 332677 for instructions.

The index rebuild might take up to several minutes for big tables and should be carried out during times with low RFC traffic.

Software Change Management SAP APO software is constantly updated with new functionality, improvements and corrections of errors. Monitoring of the latest available versions of the most critical components is necessary, for example, liveCache, COM Routines, qRFC and others, as described above. Certain component versions are issued specifically for performance improvement purposes.

At the same time, system operation and stability can be affected when software change is implemented without being recommended by SAP. In any case make sure that software changes are tested before appling them in the production system.

Sizing Verification System capacity should be verified if system tuning actions do not achieve the desired performance improvements. A customer using the Quicksizer program in the SAP Service Marketplace can carry out a simple sizing verification. It can also be carried out remotely within an EarlyWatch session performed by experienced SAP service engineers.

SAP Note 537210 describes how to obtain the real size of planning versions in liveCache using the reports /SAPAPO/OM_LVC_OBJECTS and /SAPAPO/OM_TS_FILLRATE.


© 2005 SAP AG

47

It is recommended that SAP standard remote services be ordered if you plan to increase your workload, or system performance has degraded after workload increase.

Resizing and Hardware Upgrade If sizing verification has indicated insufficient hardware resources, an upgrade project should be started and remote GoingLive services ordered. The system resources will be checked in a more detailed fashion and recommendations worked out within an upgrade project. In simple cases, a CPU upgrade (faster CPUs, more CPUs) and/or a main memory upgrade can solve most performance problems.

6. Monitoring Roadmaps The following monitoring roadmaps can help you to verify your monitoring procedure.

© SAP AG

Workload Monitor (Transaction ST03N)

High share of DB Procedure time (DB Proc > 25%) in average response time

Performance problem persists independently of peak hours

Analyse network connection to the liveCache

Performance problem occurs during peak hours only

Analyse liveCache configuration

?

?

?

Analyse liveCache hardware capacity

? There are only a few transactions with long running COM Routines

Analyse COM Routines


© 2005 SAP AG

48

© SAP AG

SAP Memory Monitor (Transaction ST02)

Table buffer swaps

Too many buffered objects

Analyse buffered objects

Large buffered objects

Analyse buffered objects

?

?

?

© SAP AG

Operating System Monitor (Transaction OS06/OS07)

Low average idle CPU rate (<20%)

Identify and analyse transactions with high CPU consumption

High paging rate (>20%)

Identify and analyse transactions with high memory consumption

??

?

? High disk utilization rate (>80%)

Analyse activities on that disk


© 2005 SAP AG

49

© SAP AG

liveCache Monitor (Transaction LC10)

Low Data Cache hit rate (<99.8%)

Analyse long running versions

Data Cache filling level is 100%

Check garbage collectors

?

?

Analyse long running versions

? Heap memory consumption is close to the limit (OMS_HEAP_LIMIT or RAM)

Check long running transactional simulations

Analyse size of planning versions

? High Devspace filling level (>90%)

Add Devspace

? Long running COM Routines

Analyse COM Routines

? High collosion rate (>30%)

Analyse critical regions

© SAP AG

qRFC Monitor (Transactions SMQ1, SMQ2)

Status STOP Lock set by application

Consult with application, unlock the queue with SMQ1, and restart the queue

Lock set explicitly by SMQ1, SMQ2

Contact Basis staff for settings for profile parameters rdisp/rfc*, consider increasing the number of dialog processes

Consult with the person who set the lock, unlock the queue with SMQ1, and restart the queue

??

?

?

Use ST22 to check the short dump or use SM21 to check the system log in the receiving system for possible errors

Status SYSLOAD

? Enough DIA processes allocated

Check gateway configuration for the number of connections allowed? Status SYSFAIL

Status CPICERR ?Use SM21 to check the system log in the receiving system, and identify the possible source of errors in the trace file dev_rd or dev_rfc*


© 2005 SAP AG

50

© SAP AG

qRFC Monitor (Transactions SMQ1, SMQ2) continued

Status WAITSTOP The first LUW of this queue has dependencies on other queues: resolve the causes of locking on one of these queues and restart the queue

The first LUW of this queue has dependencies on other queues: execution of this queue cannot start until other queues are completed up to the waiting LUW

?

? Status WAITING

One of the other queues with a higher priority LUW has an error status ?Follow this roadmap to correct errors on the queue with the higher priority LUW


© 2005 SAP AG

51

Further Information

Troubleshooting The performance monitoring and tuning procedure described in this Best Practice introduces a company’s system administrator to the first steps in problem analysis. Every company system has its own configuration and it is impossible to describe all possible problems and solutions. The main aim of this Best Practice is to provide guidelines for solving the simplest and most common problems immediately.

If you cannot solve a problem within a reasonable time frame, contact SAP Active Global Support by creating a Support Notification in SAP Service Marketplace. The results of the preliminary analysis according to the procedure described above contain important information for further problem analysis by SAP Active Global Support and should therefore be attached to the message. If the problem cannot be solved with the information provided, ordering a standard remote EarlyWatch service or an onsite Solution Management Assessment Service is recommended.

Background Information and References The following referencs can help you to build up the technical background necessary to perform workload analysis and performance tuning according to the procedures described above as well as to let your system be analyzed by SAP specialists.

Remote Services

• GoingLive

• EarlyWatch Alert

• EarlyWatch

• SAP Remote Performance Optimization

• Onsite Services

• Solution Management Assessment

• Solution Management Optimization

• Service Marketplace

• SAP Service Marketplace can be reached via transaction OSS1 or in internet www.service.sap.com.

• SAP Training Courses

• BC315 “Workload Analysis”

• BC555 “liveCache Administration”

• BC556 “APO System Administration”

• TEWA60 “liveCache Monitoring”

SAP Books

http://service.sap.com/books

• Thomas Schneider, “SAP R/3 Performance Optimization”

• Liane Will, “SAP R/3 System Administration”

• Liane Will, “SAP APO System Administration”


© 2005 SAP AG

52

Glossary

CCMS (Computer Center Management System) – monitoring, management and configuration tool for mySAP.com components.

Change List – specification of a COM Routine release.

COM Routines – APO application programs running in a liveCache

Core Interface – an interface between APO and SAP R/3 system based on qRFC technology.

Critical Regions – liveCache memory areas containing internal data structures.

Devspaces – hard disk regions used by a liveCache.

Garbage Collectors – liveCache memory management tool for deletion of unnecessary history pages.

Inbound Queue – a queue of Remote Function Calls in a receiving system

KNLDIAG – file containing system messages of a liveCache kernel.

liveCache – a program based on SAP DB technology for intelligent high performance management of objects used by COM routines.

liveCache Data Cache – memory region for SQL and OMS data in a liveCache.

liveCache Heap (Private OMS Cache) – memory region in a liveCache, allocated by demand.

Logon Group – logical group of users accessing to the same system resources.

Optimizer – an application program for optimization of planning results

Outbound Queue – a queue of Remote Function Calls in a sending system

Performance Analysis – detailed monitoring and tuning procedures performed in a system and directed to provide optimal and balanced system performance.

Planning Version – a set of logically consistent planning input data and results.

Server Group – group of physical servers dedicated to perform determined functions.

Workload Analysis – detailed monitoring and tuning procedures performed in a system and directed at balancing a given workload and available system resources.

Relevant SAP Notes 021960 Several instances/systems on one UNIX computer

023984 Workload analysis: duration of data storage

026317 Set up for LOGON group for autom. load balancing

035010 Service connections: Composite note (overview)

037001 Telnet link to customer systems (via OSS)

038052 System Panic, terminations due to low swapspace

039412 How many work processes to configure

051789 Poor user distribution in logon distribution

067739 Priorities of problem messages

074141 Resource Management for tRFC and aRFC

085524 Sizing mySAP.com (Quick Sizer)

088416 Zero administration memory management from 4.0A/NT

100740 Set up pcANYWHERE connection in OSS

103747 Performance in 4.0/4.5/4.6: Parameter

146289 Parameter Recommendations for 64-Bit SAP Kernel


© 2005 SAP AG

53

146528 Configuration of R/3 on hosts with much RAM

202344 Setting up DB connection in OSS

203924 Performance 4.6 - collective note

205220 Minimum size: MAXUSERTASKS in the liveCache

207223 Activating the SAP EarlyWatch Alert

208317 Performance problems in the liveCache

209834 CCMS agent technology (composite SAP note)

215361 FAQs on the Early Watch Alert topic

319860 tRFC/qRFC: Improved performance after SYSLOAD

326494 List of SAPAPO COM object builds for APO 3.0

335162 Error "connection closed" in tRFC/qRFC monitors

337445 liveCache and storage management

337773 Lock escalations in liveCache

369007 qRFC: Configuration for the QIN Scheduler

371068 tRFC/qRFC: Measures for better performance

375566 Many entries in tRFC and qRFC tables

378903 Queue status in SMQ1, SMQ2 and table ARFCRSTATE

384077 APO: Optimizing CIF Communication

384971 Gateway parameters for a high interface load

410002 Setting for MAXCPU as of liveCache 7.2.5 Build 4

415376 Using LC10 for liveCache monitoring

419634 Non-optimal default values for OMS_VERS_THRESHOLD

425051 liveCache performance problems with MAXCPU > 1

431299 Determining the size of plan versions

438015 Latest qRFC version and supplement for 3.x, 4.x,

457373 Termination of COM routine due to insufficient space

458369 Increase in liveCache due to increase in history

487972 Operating system parameterization of liveCache

490992 Earlier deletion of hanging transactional simulations

516661 liveCache performance problems with OMS_HEAP_COUNT=1

530394 Bottleneck analysis using the Database Analyzer

537210 Determination of the liveCache main memory

Relevant Transactions DB02 Database Monitor

LC10 liveCache Monitor

OS06 Local Operating System Monitor

OS07 Remote Operating System Monitor

OSS1 SAP Service Marketplace

RZ12 RFC Server Groups


© 2005 SAP AG

54

RZ20 CCMS Monitor

SE16 Database Table Contents

SE38 ABAP Editor

SE38 -> /SAPAPO/OM_PERFORMANCE Report “Performance Benchmark”

SE38 -> /SAPAPO/OM_LVC_OBJECTS liveCache Sizing Verification

SE38 -> /SAPAPO/OM_TS_FILLRATE liveCache Sizing Verification

SM12 Lock Monitor

SM13 Update Records

SM21 System Log

SM37 Batch Job Monitor

SM50 Work Process Overview

SM51 SAP Servers

SM59 RFC Destinations

SM66 Global Work Process Overview

SMLG Logon Groups

SMQ1 qRFC Monitor (Outbound Queue)

SMQ2 qRFC Monitor (Inbound Queue)

SMQR qRFC Monitor (QIN Scheduler)

SMQS qRFC Monitor (QOUT Scheduler)

ST02 SAP Memory Configuration

ST03N Workload Analysis

ST04 Database Performance Analysis

ST22 ABAP Dump Analysis

STAD Statistical Records

/SAPAPO/OM01 COM Trace Display

/SAPAPO/OM02 COM Trace

/SAPAPO/OM04 COM Module Versions

/SAPAPO/OM11 Logging Log

/SAPAPO/OM13 liveCache and COM Objects Analyser

/SAPAPO/OM14 Test Cockpit

/SAPAPO/OM16 liveCache Data Browser

/SAPAPO/OPT09 Version Overview Optimization Server

/SAPAPO/OPT11 History of Optimization Runs

Feedback and Questions Send any feedback by formulating an SAP customer message to component SV-SMG-SER. You can do this at http://service.sap.com/message.


© 2005 SAP AG

55

© Copyright 2005 SAP AG. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.

IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries, pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, and Informix are trademarks or registered trademarks of IBM Corporation.

Oracle is a registered trademark of Oracle Corporation.

UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.

Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc.

HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.

Java is a registered trademark of Sun Microsystems, Inc.

JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape.

MaxDB is a trademark of MySQL AB, Sweden.

SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.

best practice for sap performance monitoring for sap scm

Documents