Are RDBMS really useful in a distributed realtime enterprise system ?

Frying your infrastructure

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 2

Overview

Quick SAP application infrastructure runthrough

SAP application evolution: Enterprise Services & Smart Items

Oops: frying the old infrastructure

Lessons from the Web Services rewrite

Conclusions and SAP Research ideas

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 3

SAP: Lots and lots of business applications...

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 4

...on a common infrastructure

Portal (People, Roles, KM, Collaboration)

Exchange Infrastructure (Process Integration, Messaging)

Business Warehouse (Analytics)

APO (Planning, Optimization)

Search (TREX)

Web Application Server (Everything...)

(i.e. SAP takes a pretty broad definition of „application server“)

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 5

Anatomy of a workhorse: SAP Web Application Server

Web BrowserWeb Client

Web Application ServerABAP/J2EE

Database Server

Internet Communication Manager (HTTP/SOAP/etc.)

Connectivity Layer

Database Interface (JDBC/OpenSQL)

Persistence Layer

Dispatcher

Presentation Layer (JSP/etc.)

• Presentation Logic• Navigation• Interaction with Client

Dispatcher

Business Layer (J2EE/ABAP)

• Business Logic• Status Administration• Locking/Enqueue

Dispatcher

Integration Layer

Integration Tools:

• Java Connector• .NET Connector• XML/XSLT• SOAP• WSDL Proxies• BAPI/RFC• IDoc• etc.

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 6

Remarks on SAP Applications

SAP Web Application Server

Very robust, complete resource isolation (even for J2EE)

RDBMS (ab-) used as a persistence layer for almost everything

Not particularly lightfooted Heavyweight sessions

Accrued portability layering over the years

SAP Application Design

Database transactions only used for error rollbacks

Locking mostly handled at AppServer level (Enqueues)

Isolation mostly handled at application level

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 7

Global Industry Survey: The world in 2010

The ability to adapt strategy and business models quickly will be a critical source of competitive advantage

Firms will focus increasingly on speed of innovation and customer retention to create long-term value

Market consolidation will continue, lots of mergers and splits

People are the biggest asset and the biggest cost factor

Interviewing more than 4000 executives and managers

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 8

Flexibility: From monolithic applications to Web Services

Information TechnologySolutions

ProcessInnovationSolutions

SAP NetWeaver

UI UI UI

Functionalcomponents

DB DB

Composite Application

3 tier applications Coded transactions Best practices

Service-oriented applications Model-driven applications Best practices and platform

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 9

Attaching the real world: Eliminating media breaks

Source: M-Lab, 2001 (http://www.m-lab.ch/)

Degree of automation

RF

ID

Digital world („Bits“):• Inter- and cross-company

information systems (e.g.: ERP systems)

• Local, regional and global communication networks (e.g.: Internet)

Physical world („Atoms“):• Human beings • Products • Production means

State-of-the-art Next Generation Smart Items

Ma

nu

al

da

ta e

ntr

y

Vo

ice

in

pu

t

Ba

rco

de

sc

an

nin

g

Sm

art

Ite

ms

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 10

Performance matters: Realtime Application Scenarios

Asset Tracking

Hospital inventory

Machines and parts

Supply Chains

Emergency Response

Vehicle dispatch & tracking

Workplace safety

Security

Behavior mining

Intrusion detection

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 11

Putting it all together: Retail example

Enterprise & Merchandise Operations

Distribution Center Operations

Store Operations Multi-Channel Sales Operations

POS

CustomerLoyalty Mgmt

e-Commerce,Catalog &

Call CenterStore

Inventory Mgmt

In-StoreCustomer Service

PersonalizedPromotions

WarehouseManagement

FinanceHuman

Resource

RetailMaster Data Operational

Purchasing

StoreWorkforce/Task

Mgmt

InventoryMgmt

Bu

siness D

ata W

areho

use

…

Multi-ChannelOrder

Fulfillment Glo

bal

Mas

ter

Dat

a M

ana

gem

ent

(Ex

chan

ge

& C

on

soli

dat

ion

)

TransportationManagement

CustomerOrder Mgmt

Supply ChainEvent Mgmt

Sourcing

Product Design & Introduction

Foreign TradeManagement

EnterpriseOperations

Demand Intelligence & Analytics

Merchandise & Assortment Planning &

Optimization

RevenuePlanning &

Optimization

Multi Echelon Inventory Planning

& Optimization

Workforce Planning &

Optimization

Collaborative

Collaborative Collaborative

SupplyPlanning

Demand

Forecast

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 12

Infrastructure challenges

Event Rates Hundreds of readers in a warehouse

Bursts of hundreds of events per reader per second

Reaction times on the order of seconds in some scenarios

Data Volumes Billions of objects in a large organization (retail forecast & replenishment)

Some organizations are required to maintain history over 3 or more years

Analytics need to deal

Cost efficiency Sometimes do this on a shoestring budget

Deal with unpredictable load patterns (the curse of business flexibility)

Keep it all manageable

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 13

Net result: Frying your infrastructure

General purpose RDBMS no longer a good fit for these tasks

Good at

Transactions on a few records at a time

Search with well (pre-) designed queries

Not so good at:

Very long running transactions

Mass operations

Ad hoc analytics

Streams

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 14

Business Process Platform: The big rewrite

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 15

Observations from the Enterprise Services rewrite

You can do without distributed transactions for even very complex business systems

Database-based consistency can be replaced with message-based consistency

Extremely stylized use of DB model generated

transaction design patterns

A lot of DB functionality factored out (by design): No cross-component transactions (design rule)

Search (capturing updates allows for separate engine)

Analytics designed for separate engine

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 16

SAP Enterprise Services

SAP comitted to Enterprise Services across the board

New design opens up lots of flexibility in the lower stack Sufficiently coherent data replication in multiple special purpose

engines possible

Web services semantics require application level recovery already, negating a lot of RDBMS „comforts“

Special engines already (partly) there: Search (TREX)

Analytics (EUCLID)

Planning/Optimization (APO)

SAP AG 2005, Frying your infrastructure, Burkhard Neidecker-Lutz / 17

Research Idea: Special purpose engines for everything

Mass operations (prediction) Computation -> Data (integrated processing, LiveCache)

Data -> Computation (mass data services, SAP Research)

Streams Luckily, not yet a big problem for us (other than prediction)

SAP Research looking for partners

Very fast „simple“ RDBMS for the rest Fast keyed search, update

Speaking of device evolution: Give me a few Mbyte of MRAM per machine