the sap transaction model: know your applications...operational: run on multiple dbms products...

SYSTEMATIC THOUGHT LEADERSHIP FOR INNOVATIVE BUSINESS

The SAPTransaction Model:

Know YourApplications

Shel Finkelstein, Rainer Brendle, Dean Jacobs,Manfred Hirsch and Ulrich Marquard

SAP Labs{firstname.lastname}@sap.com

© SAP ACM SIGMOD Conference, June, 2008 2

Outline

SAP Applications and DataExamples, Requirements, Properties

SAP Data Management ArchitectureApplication ArchitectureDatabase Execution Model

SAP Scalability and PerformanceApplication ScalabilityBenchmarks


SAP Applications and Data


SAP Customer Application Requirements

SAP ERPOn a laptop

SAP for ConsumerProducts5000+ concurrentlyactive users

SAP SCM4.5 million characteristiccombinations & 256 GBmemory in live cache

SAP for Retail1.3+ million salesorder items per daySAP

NetWeaver BICurrently productive:10 TB database(planned 60 TB)

SAP ERP - HCMPayroll calculation for 500,000employees and retirees in 4 hours

SAP for Utilities25 million business partners85 million service and salesorders per year

SAP NetWeaverPortal100,000 concurrent users


Solution Areas and Data Access

Order ProcessingEnter order with line items; schedule delivery; handle changesRead mostly; most updates are inserts; most updates don’t conflict

Supply Chain Management (SCM)Planning for procurement, sales and demand, production, distribution; controland managementRead mostly, most updates are inserts; most updates don’t conflictBusiness processes handle impacts of updates

Integrated Product and Process Engineering (IPPE)Master data that provide high integration of engineering processes andmanufacturing processesRead mostly; most updates don’t conflictCollaboration, but changes usually involve independent subtrees


Data Access Patterns for Applications

Applications may be conversational, with multiple conversational userinteraction steps in a transactionMost business data is read-onlyUpdate operations are insert-mostlyMost database transactions are shortFor all updates, conflicts are rare in practicePotential hotspots (e.g., inventory stock; sequence numbers that must beconsecutive for legal reasons) must be addressed


Data Management Principles

Technical: Maintain integrity of Business Data across appsMultiple solution areas use the same Business Objects

Legal: Handle most updates as inserts of new dataSupport for Governance, Risk and Compliance (GRC) and auditing

Operational: Run on multiple DBMS productsDon’t depend on special properties of any DBMSDB must not be a bottleneck

Programming Model: Message-based consistencyLike normalization and serializability, transactions should only be used whereappropriateUse transactions within a deployment unit; avoid distributed commit acrossdeployment units


SAP Data Management Architecture


App

licat

ion

Serv

er DispatcherGateway

SharedMemory

andBuffers

Basic Application Server Architecture

DialogHandler

User User User User

DialogHandler

DialogHandler

DialogHandler

UserUser

Database


Screen

COMMIT WORK

Screen Screen

Two-Stage Execution Model

Read from databaseAcquire locksQueue descriptors of update operationsNo writesRollback possible

Perform update operations in deadlock-free orderSingle message database transactionNo rollbacksRelease locks upon completion

Dialog Stage Update Stage

Rediscovered in H-Store, where it is used to eliminate the undo log


Lock Manager

The ProblemMany applications have multiple screens with long user think timesMaintaining open database connections would hurt performanceImplicit Pessimistic and Optimistic Concurrency Control have disadvantages

E.g., scalability; rollback after user completes work

The SolutionA business-level logical lock manager (Enqueue Manager) outside the database

Business-level locking is very flexible approachNo danger of lock escalation or index locking in the database

Locks are explicitly requested; a “contract” between applications, using frameworkLock types: shared, intention, exclusiveRather than getting blocked, applications get errors that they can handleCan request that individual locks be held after commit


Database

App

licat

ion

Serv

er 2

App

licat

ion

Serv

er 1

DialogHandler

DialogHandler

DialogHandler

Dispatcher Dispatcher

LockManager

User User User

MessageServer

LockTable

SharedMemory

1

2

34

56

Steps in Two-Stage Execution

Gateway


Asynchronous Updates

The packaging of rollback-free queued updates into a bundle allows them to beexecuted after the transaction completes (write behind)Doing so allows the user to continue on to other work soonerThe locks are released after the updates have been performed so subsequent writeoperations never get stale dataAppropriate when subsequent operations go on to other work, so they never readstale dataThe updates are performed by an Update Handler, which reduces the number ofhandlers that do writesThe updates from several transactions may be delayed indefinitely until acoordinator picks them up and performs them all together

Example: A background process that updates statistical data


Screen

SAVE COMMIT WORK

Screen Screen Screen

Three-Stage Execution Model

Dialog Stage Update Stage Write StageDB COMMIT

Read from databaseAcquire logical locksNo writesRollback possible

Read update operations fromdatabase and perform them indeadlock-free orderSingle message databasetransactionNo rollbacksRelease locks uponcompletion

Write descriptors ofupdate operations tothe databaseNo rollbacks


Database Manager

Database

App

licat

ion

Serv

er 2

App

licat

ion

Serv

er 1

DialogHandler

DialogHandler

UpdateHandler

UpdateHandler

Dispatcher Dispatcher

LockManager

Application data Data to be updated

MessageServer

LockTable

1

2 3

4

56

Steps in Three-Stage Execution

7

Gateway

User User User


Comparison of the Models

Two-StageUpdates always visible immediatelyafter transaction completesLower database load for singletransactionSynchronous execution of updateshelps to build process chains

Three-StageUser can continue sooner, since returnto user as soon as update descriptor isin DBLower database load when multipletransactions are boxcarred togetherShorter locking times and/or log forsequence number assignment andother hotspot operations


What Does SAP Use Database For?

Store/Retrieve dataTransaction commit

Internal locks are held very brieflyCanonical ordering of locked resources avoids deadlocksRollbacks for internal DB reasons only (can retry)

Operational utilities, including disaster recoveryIndexing

Internal indexesBut external indexes are maintained in main memory, kept consistent by post-commit processing


Hot Spots: Sequence Numbers

Many ERP and Financial applications require that sequence numbers beassigned to transactions or documentsNumber allocation requirements depend on legal requirements

Weaker requirements permit implementations that scale betterStrictest guarantee: chronological assignment with no gaps

Number assignment must occur inside the transaction and be rolled back incase of abortForces serialized access to the next available number, which reducesconcurrencyEssential to hold the (database) lock for as short a time as possibleTwo-stage model: lock held during updates and writesThree-stage model: lock held during updates but not writes

Weaker guarantee: Gaps must be logged for auditingThree-stage model: pending updates provide a log


Hot Spots: Aggregates

Applications often must update shared aggregate data,as well as exclusive data

Order Processing focuses on a specific OrderProcessing an Order also affects aggregate data

Total expense for Contract associated with that OrderStock remaining for Products sold in that Order

Transaction execution strategy is designed for scalabilityAcquire exclusive logical lock on specific OrderAfter Commit, perform updates on OrderAfter Commit, perform (commutative) delta update on aggregatedataSAP also can perform merge update, e.g., for collaborativeapplications such as Integrated Product and Process Engineering


Reviewing SAP Transaction Execution Stages

During transactionLogically lock data that needs to be changedRead Business Data into an application contextBefore commit, perform updates only on the application context

After commitStage 1: Application does logical consistency checks, then queuesupdate descriptor

Due to logical locks, there can be no conflicts with concurrent transactionsStage 2: Transaction’s queued update descriptor is written to DB insingle message transaction (with boxcarring)

Return to user after its single message transaction completesStage 3: Updates to Business Data in DB are performed

Updates are executed in canonical order to avoid deadlocksDelta updates to aggregated executed last, with coarsest aggregationsexecuted last of all


Scalability and Performance


Performance typically defined by two metricsResponse time - speed of task completion (latency)

System throughput - amount of work done in a givenamount of time

Good performance if metrics match customerneeds or Service Level Agreements

Most important Key Performance Indicators (KPIs)CPU time consumedPeak memory usedDisk spaceNetwork load

Proven and predictable scalabilityPrecondition for translating business requirements intohardware configurations (sizing) with linear growthMinimize load on central resources (e.g., DB)No system wait times, no bottlenecks

What Exactly Is Good Performance?

Application ServerApplication Server(s)

Database

Web Server(s)

ResponseTime = t[i]

ResponseTime

t[1]

t[2]t[3]

t[4]

t[5]

t[6]t[i]


Vertical & Horizontal Scalability(Cross-Tier Scaling and ScaleOut)

From one tier (laptop demo) tomulti-tier systems

PresentationMore than 150K very activeusers (fat transactions)

InternetMore than tens of thousands ofhits /sec10 servers at one of our largestcustomers

Application and IntegrationOver 150 app serversconnected to DB

DatabaseMore than 100 CPUs; morethan 10 TB DB sizeScalability through SMPdatabase serverScalability through parallelpartitioned databases

©SAP 24

Architecture of SAP Solutions

CentralServicesMessage Services

Application Server

ABAP

UserUser

HTTP Dispatcher HTTP Dispatcher

Workprocess

SAP Web Dispatcher

DBMS

I/O sub system

HTTP, SOAPHTTP

RFC

DB WPDB WP

RFC

Application Server

ABAPABAP

Dispatcher

Workprocess

Enqueue Services (Logical Locks)

ABAP Dispatcher

Sharedmemory &buffers

Sharedmemory &buffers


Simulation of User Interaction Load

The Sales and Distribution(SD)benchmark is the mostcommon SAP benchmark

Core business process (selling, shipping, delivery)High scalability demands from customersFocuses on CPU utilizationEasy to installEasy to use"Well understood and accepted"Highly credible and influential high-end OLTPbusiness application benchmarkCertified benchmarks run since 1995


0. Logon 10. Choose customer order1. Main screen 11. Change delivery info for order

2. Create customer order 12. Posts goods issue (schedule delivery)3. Enter order info 13. List orders4. Enter details for 5 items 14. Choose set of orders

5. Save 15. Create invoice6. Create a delivery 16. Save

7. Enter delivery info 17. End8. Save 18. Confirm logoff

9. Display customer orderSteps 2 to 16 are repeated n times; each 15 step run takes at least 150 seconds due to 10 secondthink time between steps.

Business aspect:One run (steps 2 to 16) corresponds to the selling of 5 line items.

User Interaction Steps –SAP SD Standard Application Benchmark


2005

Proven Scalability Using SD Benchmarks

SD Benchmark, Two-Tier Internet Architecture: Scale with Hardware1996 1997 1998 1999 2000 2001 2002 2003 2004

http://www.sap.com/benchmark

21,000

6001,410 1,708

3,0004,100

7,8008,000

13,000

20,000

2006

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

23,456

30,000

35,4002007 2008

Number of SDBenchmark users

http://www.sap.com/benchmark


Key Figures for the 168,300 UserThree-Tier SD Benchmark (2005)

8,392 DB transactions per second (commits)

323 MB written to disk per second on average

611 GB data written in one hour (~ 170 MB per second on average)

1,157,000 network packets per second on average, with packet size of 512 bytes

469,091 SQL statements per second

2,240 GB database disk space

5,632 SAP (as opposed to DB) transactions / second

14,081 Screen changes / second

Database used a 32-processors SMP server

R/3 was running on 712 processors

16,896,670 fully business processed order line items per hour ( 4,694 per second )


Conclusion

Data and transaction management should be based on applications andtheir data utilization

Traditional DBMS are not ideally suited for thisSAP has had its own external data and transaction layer since 1985– This works; DB hasn’t been a bottleneck for us

Get the programming model right for your appsExplicit locking for applications, based on frameworkUse distributed commit only within a deployment unit– Across deployment units, use message/process-based (loose) consistency

Know your applicationsOne size hasn’t fit all, for SAP, for other companies, and for our customers


Further Information

Public ReferencesSAP Standard Application BenchmarksSAP Standard Benchmark Certification 2005021Rudiger Buck-Emden, “The SAP R/3 System: A

Client/Server Technology”, Addison-Wesley, 1996Horst Keller, “The Official ABAP Reference”, SAP Press,

2005Gerhard Knolmayer, “Supply Chain Management Based on

SAP Systems: Processes and Architecture”, Springer, 2002Burkhard Neidecker-Lutz, Frying your infrastructure: Are

transactions really useful in a distributed realtimeenterprise system?, HPTS 2005[H-Store] Mike Stonebraker, The End of an Architectural

Era (It's Time for a Complete Rewrite), VLDB 2007


Further Information

SAP Service Marketplace (SAP customers)http://service.sap.com/benchmarkhttp://service.sap.com/performancehttp://service.sap.com/sizinghttp://service.sap.com/quicksizing

http://service.sap.com/benchmark

http://service.sap.com/performance

http://service.sap.com/sizing

http://service.sap.com/quicksizing


Copyright 2008 SAP AGAll rights reservedNo part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changedwithout prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

SAP, R/3, xApps, xApp, SAP NetWeaver, Duet, SAP Business ByDesign, ByDesign, PartnerEdge and other SAP products and services mentioned herein as well as their respective logos aretrademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned and associated logos displayedare the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This documentcontains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy,and/or development. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, orother items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties ofmerchantability, fitness for a particular purpose, or non-infringement.

SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitationshall not apply in cases of intent or gross negligence.

The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in thesematerials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages

Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durchSAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden.

Einige von der SAP AG und deren Vertriebspartnern vertriebene Softwareprodukte können Softwarekomponenten umfassen, die Eigentum anderer Softwarehersteller sind.

SAP, R/3, xApps, xApp, SAP NetWeaver, Duet, SAP Business ByDesign, ByDesign, PartnerEdge und andere in diesem Dokument erwähnte SAP-Produkte und Services sowie diedazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und in mehreren anderen Ländern weltweit. Alle anderen in diesem Dokument erwähnten Namenvon Produkten und Services sowie die damit verbundenen Firmenlogos sind Marken der jeweiligen Unternehmen. Die Angaben im Text sind unverbindlich und dienen lediglich zuInformationszwecken. Produkte können länderspezifische Unterschiede aufweisen.

Die in diesem Dokument enthaltenen Informationen sind Eigentum von SAP. Dieses Dokument ist eine Vorabversion und unterliegt nicht Ihrer Lizenzvereinbarung oder einer anderenVereinbarung mit SAP. Dieses Dokument enthält nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP®-Produkts und ist für SAP nicht bindend, einen bestimmtenGeschäftsweg, eine Produktstrategie bzw. -entwicklung einzuschlagen. SAP übernimmt keine Verantwortung für Fehler oder Auslassungen in diesen Materialien. SAP garantiert nicht dieRichtigkeit oder Vollständigkeit der Informationen, Texte, Grafiken, Links oder anderer in diesen Materialien enthaltenen Elemente. Diese Publikation wird ohne jegliche Gewähr, wederausdrücklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschließlich, hinsichtlich der Gewährleistung der Marktgängigkeit und der Eignung für einen bestimmten Zwecksowie für die Gewährleistung der Nichtverletzung geltenden Rechts.

SAP übernimmt keine Haftung für Schäden jeglicher Art, einschließlich und ohne Einschränkung für direkte, spezielle, indirekte oder Folgeschäden im Zusammenhang mit der Verwendungdieser Unterlagen. Diese Einschränkung gilt nicht bei Vorsatz oder grober Fahrlässigkeit.

Die gesetzliche Haftung bei Personenschäden oder die Produkthaftung bleibt unberührt. Die Informationen, auf die Sie möglicherweise über die in diesem Material enthaltenen Hotlinkszugreifen, unterliegen nicht dem Einfluss von SAP, und SAP unterstützt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewährleistungen oder Zusagen überInternetseiten Dritter ab.

Alle Rechte vorbehalten.


The SAP Transaction Model:Know Your Applications

Shel Finkelstein, Rainer Brendle, Dean Jacobs,Manfred Hirsch and Ulrich Marquard

SAP Labs{firstname.lastname}@sap.com

the sap transaction model: know your applications...operational: run on multiple dbms products...

Documents