five years of framework building lessons learned acm oopslakurt madsen october 26 – 30, 2003

Five Years of Framework Building

Lessons Learned

ACM OOPSLA Kurt MadsenOctober 26 – 30, 2003 www.MetaTech.us

© 2003 MetaTech

Copyright Notice

• Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.

• To copy otherwise or republish, to post on servers, or to redistribute to lists, requires prior specific permission.

© 2003 MetaTech

Software Engineering on Long-Term Projects

• Given scarce resources, the goal of software engineering is to optimize value by…– Analyzing trade-offs

– Making decisions

– Measuring quality parameters

• On long-term software projects, trade-offs and quality parameters may not be clear at project inception

Application frameworks balance up-front structure with long-term software evolution

© 2003 MetaTech

What is an Application Framework?

A reusable, semi-complete application that can be specialized to produce custom applications [Johnson, Foote]

• Benefits– Benefits of object-oriented application frameworks stem from

modularity, reusability, extensibility, and inversion of control. [Fayad, Schmidt]

• Today’s Talk– EXPRESS: Application integration for financial services

– GSCE: Model execution environment for telecom provisioning

© 2003 MetaTech

Funding the Bank: Wholesale Bond Trading

InvestorInvestment Bank

Money

Bonds

Problem Domain

© 2003 MetaTech

• Trade Details:– Trade Type = Repurchase Agreement

– Quantity = 2,000

– Price = $10,000 USD

– Trade Amount = $20,000,000 USD

$

Bonds

Bonds

$ + interest

Investment Bank Investor(e.g., California)

Problem Domain

Funding the Bank: Wholesale Bond Trading

© 2003 MetaTech

The Bank Funding Pyramid Scheme

Assets = LiabilitiesC = Capital

0

…

…etc…

…

4,050,000,000

4,500,000,000

5,000,000,000 = Co

165,000,000,000

…

…etc…

…

13.55B

9.5B

5B

11 TTT CiCC

T = Trade

∞

…

…etc…

3

2

1

n/a i = ReserveRequirements

Problem Domain

© 2003 MetaTech

Trade Flow Through Back Office Systems

Front Office• Trade Capture

Middle Office• Trade Confirm

Back Office• Settlement

Intranet Extranet

Settlement

Networks

Other Banks

Depositories

Problem Domain

© 2003 MetaTech

The Challenge of B2B Integration




Settlement

Networks

Other Banks

Depositories

Number of interfaces: n(n-1) / 2

Problem Domain

© 2003 MetaTech

Use Two-Steps to Integrate for B2B Applications




Express

Settlement

Networks

Other Banks

Depositories

Number of interfaces: n

Lesson #1

© 2003 MetaTech

Protocol Conversion (e.g. Control Msgs)Protocol Conversion (e.g. Control Msgs)

struct {char id[9];float price;double qty;

} Security;

struct {char bank_name[15];char acct_num[12];

} Counter_party;

struct {charmsg_type_code[4];Security security;Counter_party cpty;

} Deliver_vs_payment;

struct {char id[9];float price;double qty;

} Security;

struct {char bank_name[15];char acct_num[12];

} Counter_party;

struct {charmsg_type_code[4];Security security;Counter_party cpty;

} Deliver_vs_payment;

<Trade><Type>DeliverVsPayment</Type><Security>

<ID>ISIN123</ID><Price>87 3/8</Price><Qty>100,000</Qty>

</Security><Counter-Party>

<Name>Other Bank</Name><Account>ABA123</Account>

</Counter-Party></Trade>

<Trade><Type>DeliverVsPayment</Type><Security>

<ID>ISIN123</ID><Price>87 3/8</Price><Qty>100,000</Qty>

</Security><Counter-Party>

<Name>Other Bank</Name><Account>ABA123</Account>

</Counter-Party></Trade>

Source Message Format Destination Message Format

Legacy Message XML Message

Data Format Conversion

– Syntactic

– Semantic

Data Format Conversion

– Syntactic

– Semantic

Tokens

TranslatorParser <pymt> <amt> 32.50 </amt></bill>

Pymt Data

An Example of Two-Step Integration

© 2003 MetaTech

Security

TypeCounter Party

Trade

Deliver vs. Payment

ID

ISIN123

Price

87 3/8

Quantity

100,000

Name Account

Other Bank

ABA123

Security

TypeCounter Party

Trade

Deliver vs. Payment

ID

ISIN123

Price

87 3/8

Quantity

100,000

Name Account

Other Bank

ABA123

Name Account

Other Bank

ABA123

An Example of a Parsed, Intermediate Message

© 2003 MetaTech

Distinguish Between Framework and ApplicationLesson #2

Application Layer

App

Format Layer

Communications Layer

+processMsg()+initialize()+shutdown()

Manager

1 1+processMsg()

Handler

Message

Comm_1 Comm_2

Controller

* *

App_1 App_2

Plug-IN Plug-IN

Fra

mew

ork

Ap

plic

atio

n

© 2003 MetaTech

Avoid Unnecessary Layers of Abstraction

MQ Series Database Proprietary

Lesson #3

Decision: should we wrap middleware APIs?

• Advantages– Improved modularity– Middleware independence

• Disadvantages– Increased complexity– Loss of middleware features

© 2003 MetaTech Bank 1

Application

Formatting

National

Regional

Local

MsgQs

SWIFT

Fed Wire

Bank 2

Avoid Unnecessary Layers of AbstractionLesson #3

“Things should be made as simple as possible, but no simpler.” - A. Einstein

Our decision that the wrapper was necessary led to a flexible switching network

And, we never needed those low-level middleware features

© 2003 MetaTech

Improve Reliability with Different Component Types

Normal processing through the New York data center

New York

New Jersey

Back Office• Equity Trades

Federal Reserve BankBroker Dealer System

Federal Reserve BankManual Processing

InputMessage Log

OutputMessage Log

1 2

3Q ID Alias Type

1 Equity-In Database Q

2 Equity-Out MQ Series

3 Equity-Audit Socket

Lesson #4

© 2003 MetaTech


Contingency processing when the primary data link fails

New York

New Jersey


Federal Reserve BankManual Processing

Q ID Alias Type

1 Equity-In Database Q

3 Equity-Audit Socket

4 Equity-Out PrintQ or FaxQ

1

3 4

Back Office• Equity Trades

Lesson #4

InputMessage Log

OutputMessage Log

© 2003 MetaTech


New Jersey


Audit TrailInput

Q ID Alias Type

5 Equity-In File Q

6 MQ Series5

6

Test processing emulates applications by replaying audit trails

• permits testing with different teams in stages

• It also facilitates regression testing

Lesson #4

© 2003 MetaTech

Leverage Type-Safety to Reduce Software DefectsLesson #5

FmtA1_Msg.o FmtA2_Msg.omain.o

aFmtA1_Msg aFmtA2_MsgMsgFactory

Source Message

© 2003 MetaTech

Leverage Type-Safety to Reduce Software Defects

Compiler and link editor precluded mapping defects

// Plug-IN Bstatic int map( *FmtA_1, *FmtB_1);static int map( *FmtA_2, *FmtB_2 );

// Plug-IN Bstatic int map( *FmtA_1, *FmtB_1);static int map( *FmtA_2, *FmtB_2 );

// Plug-IN Cstatic int map( *FmtA_1, *FmtC_1);static int map( *FmtA_2, *FmtC_2 );

// Plug-IN Cstatic int map( *FmtA_1, *FmtC_1);static int map( *FmtA_2, *FmtC_2 );

Lesson #5

© 2003 MetaTech

Avoid Data-Driven Control of Program Execution

+processMsg(in ExMsg)

FmtMgr

1 1

Translator

Parser

+prcoessMsg(in ExMsg)

Determiner

FmtAFmtA

FmtAFmtB

msgType : StringsrcMsg : StringdestMsg : StringtokenList : HashMapsrcChannel : intdestChannel : int

ExMsg

aa

SRC MSG

DEST MSG

Lesson #6

© 2003 MetaTech

FromMany Applications

ToMany Banks

Mechanism, Not Policy

IIOP

Application

Data Formatting

Communications

Application

Data Formatting

Communications

Enforce architectural constraints in the framework

Give application builders freedom of usage

They will discover unforeseen ways to apply the framework

Horizontal Scaling

Lesson #7

© 2003 MetaTech

FromMany Applications

ToMany Banks

Mechanism, Not Policy

IIOP

Application

Data Formatting

Communications

Application

Data Formatting

Communications

Vertical Scaling

Decisions such as process configuration are deferred until application build- or even run-time

Our initial bias towards horizontal scaling was wrong.

Vertical scaling turned out to bemore flexible

Lesson #7

© 2003 MetaTech

Hong Kong

Frequent Releases Reduce Deployment Failure Risks

London

New York Bonds @ FED

Equities @ Oasys

Forex @ SWIFT

Mortgages @ PTC

Trade Confirmation

Settlement

Collateral Allocation

...etc…

Lesson #8

© 2003 MetaTech

Staging Environment Production and ContingencyEnvironments

InternalAudit

DevelopUser AcceptanceTest Plan

SystemTest

UATest

CustomerReview

Fail

System Quality Assurance

Reject

VerifyBuild

Compare

ArchiveSource

source code

executables

Roll Out

Success

reproducible

different

Roll Back

Software Deployment

Document Support Procedures- Administration, Escalation, etc.

End-user Product Training

Training

Mig

rati

on

Str

ateg

y

Hardware Deployment

Document DeploymentProcedures- Cut-Over Procedures- Contingency Plans- Risk Mitigation

Procurement

Deploy StagingHardware

Deploy Production andContingency Hardware

Reviews & Approvals- Project Management- Corporate Security- Network / Telecom- Internal Audit

DevelopSystemTest Plan

Specify HardwareConfiguration Capacity Planning R.A.S. Features

YesNo

executables

Fail Fail Fail

StandardsCompliance

Schedule

DevelopmentEnvironment

So

ftw

are

Rel

ease

Use

Cas

esTe

chn

ical

Arc

hit

ectu

re

> 1 week

Frequent Releases Require Well-Defined SDLC

© 2003 MetaTech

Most Stable Softw are Most Recent Changes

Version: 2.6.1State: ReleasedUser ID: ~PROD

Version: 2.6.3State: IntegrateUser ID: ~any

Version: 2.6.2State: SQAUser ID: ~CM_m gr

Version: 2.6.3State: W orkingUser ID: ~Colin

Version: 2.6.4State: W orkingUser ID: ~Troy

Dev

elop

er's

Wor

king

Cop

ies

Version: 2.5.3State: ReleasedUser ID: ~rel_m gr

P roductionE nvironm ent

Q AE nvironm ent

D eve lopm entE nvironm ent

Version: 2.6_baseState: W orkingUser ID: ~CM_m gr

In tegra tionE nvironm ent

Frequent Releases Require Strong Config. Mgt.

© 2003 MetaTech

Build a Business Case for Your Framework

Change Requests Implemented

Res

ourc

e E

ffor

t (F

TE

s)

Pre

dict

able

Fir

e F

ight

ing

Few Many

Budget Constraints

Available Resources

Migrate Legacy to Framework

Time

IT D

epar

tmen

t T

eam

Cu

stom

er

Sel

f-S

ervi

ce

Rel 3

Change is Costly 1

RapidTurn-around

1

= effort / deliverables

Release 1

Transition Period

Release 2

Rel 4

Lesson #9

Exp

and

ing

Sco

pe

© 2003 MetaTech

Rapid Delivery Leads to Requirements Discovery

New

Fea

ture

s O

nly

IT CustomerService

Network

Tools Com ponents

B iz AnalystTech.

Param eters UniversalApplication

PartitionedProduction

Environm ent

Generated

Application

New

Fea

ture

s O

nly

P rototype Application(in contro lled production

environment)

Service CreationEnvironm ent

EVALUATE4

CustomerDomain

IT AssetProcurement(reuse, buy, build)

End Users

Analysis

Design

Build

Integrate

Test

Analysis

Design

Build

Integrate

Test

Fee

dbac

k

Fea

ture

s

DEPLOY

3 etcCO NFIGURE

2 6

CO NTINUO USREQUIREM ENTSCAPTURE

1 5Turn around in m inutes

Lesson #10

© 2003 MetaTech

Summary of Lessons Learned

1. Use Two-Steps to Integrate B2B Applications

2. Distinguish between framework and application

3. Avoid unnecessary layers of abstraction: unless necessary

4. Improve reliability with different component types

5. Leverage type-safety to reduce software defects

6. Avoid data-driven control of program execution

7. Mechanism, not policy

8. Frequent releases reduce deployment failure risks

9. Build a business case for your framework

10. Rapid delivery leads to requirements discovery

Five Years of Framework Building

Lessons Learned

ACM OOPSLA Kurt MadsenOctober 26 – 30, 2003 www.MetaTech.us

five years of framework building lessons learned acm oopslakurt madsen october 26 – 30, 2003

Documents

trade type

californiaproblem domain

intermediate message

application integration

complete application

application frameworks

metatechtrade flow

software projects