dependable workflow technology

1© Gerhard Weikum

DependableWorkflow Technology

Gerhard WeikumUniversity of the Saarland, Germany

[email protected]

http://www-dbs.cs.uni-sb.de/

2© Gerhard Weikum

Guiding Mottos- 20 Years Ago and Now -

„We don‘t know where we are heading,but we want to be there first!“

1983:

„Time to market is everything!“2002:

3© Gerhard Weikum

Conclusion

Time to market, featurism, and $$$•Dependability and service guarantees ???•Shift gears to build highly dependable systemswith predictable, guaranteed behavior !!!

•

Provably correct behavior•World-wide failure masking•Guaranteed QoS with„autonomic“ systems

•

Dependable workflow technology:

http://www-dbs.cs.uni-sb.de/~mlite/

4© Gerhard Weikum

What I Can Offer

• Overview of the area

• Relevant foundations

• Interesting research problems

• Logic, formal spec, verification• Fault-tolerant computing• Stochastic performance modeling

What Do You Want?

5© Gerhard Weikum

Outline

Part A: WF Specification and Verification

Part B: WF System Architectureand Configuration

• What Is It All About?

• WF Specification Techniques

• Statecharts

• CTL and Model Checking

• Summary and Open Research Issues

• WF Execution Infrastructure

• Failure Handling

• Stochastic Modeling

• WF System Configuration


6© Gerhard Weikum

Outline



What Is It All About?

• WF Specification Techniques

• Statecharts








7© Gerhard Weikum

Workflow Application Example 1:Credit Request Processing

EnterCreditRequest

CheckCreditWorthiness

CheckRisk

MakeDecision

8© Gerhard Weikum

Receivesubmittedpaper

Choosereferees

Makeeditorialdecision

Notifyauthor

Contactreferee 1

Sendpaper

Remindreferee 1

Receivereview 1

Workflow Application Example 2:Journal Refereeing Process

Contactreferee 2 ...

Contactreferee 3 ...

9© Gerhard Weikum

What is Workflow Management?

Credit requests, insurance claims, etc.

Tax declaration, real estate purchase, etc.

Student exams, journal refereeing, etc.

Electronic commerce, virtual enterprises, etc.

Application examples:

Computer-supported business processes:coordination of control and data flow between distributed - automated or intellectual - activities

10© Gerhard Weikum

Workflow Management System Architecture

Ms2.lnk Ms1.lnkMs3.lnk...

Workflow server

Applications

Workflowspecification

11© Gerhard Weikum

Workflowspecification

Workflow ManagementSystem Architecture

Ms2.lnk Ms1.lnkMs3.lnk...

Workflow server

Baroque specification

Non-scalableperformance

Failure-proneexecution

Applications

12© Gerhard Weikum

The Great Vision

“And, as amoebas, you’ll have no problems recruiting other sales reps ... just keep dividing and selling, dividing and selling.”

Make e-Businessas simple as amoeba business !

13© Gerhard Weikum

Business Benefits ofWorkflow Technology

Business process automation(to the extent possible and reasonable)

Fast & easy adaptationBusiness Process Reengineering (BPR)

shorter turnaround time, less errors,higher customer satisfaction

better use of intellectual resourcesfor exceptional cases

Transparencyunderstanding & analyzing the enterprise

14© Gerhard Weikum

Technical Benefits ofWorkflow Technology

Application Integration(by loose coupling of activities)

Scalability, Reliability, Availability, Manageability

without having to tackleenterprise-wide data integration problems

supports incremental long-term migration fromstand-alone applications to electronic processes

Support for Legacy Applicationsby wrapping them into business activities

Extends Transactions to Long-lived Processes

15© Gerhard Weikum

Workflow Management Systems (WFMS):Products and Research Prototypes

Opera (ETH Zurich)

MQSeries Workflow /WebSphere (IBM)

•

•

Wide and CrossFlow(EU projects)

•

Mentor-lite (U Saarland)•CMI (MCC)•

+ workflow technology embedded in E-Commerce products and ERP systems

Staffware•

Changeengine / E-Speak (HP)•

InConcert (Tibco)••

Meteor (U Georgia)•

Adept (U Ulm)•

jFlow / WebLogic (BEA)•

.. .

BizTalk (MS)

.. .

SAP Workflow•

Wasa (U Muenster))•

16© Gerhard Weikum

Workflow ManagementSystem (WFMS)

WfMC Reference Architecture

ProcessDefinitionTools

WorkflowClientApplications

InvokedApplications

Administration& MonitoringTools

Other WFEnactmentServicesWorkflow Enactment Service

Workflow Engine

17© Gerhard Weikum

Integration with Internet Technologies

Ms3.lnk

XML (WSFL, XLANG, ...)

HTTP,DHTML

WSDL, SOAP,EJB, CORBA

XML (ebXML, ...)

UDDI

18© Gerhard Weikum

Hard Issues and Research Directions

business (bureaucratic) complexity Rap problems (e-complete)

system complexity Techno problems (DB-complete)

computational complexity Blues problems (NP-complete)

semantic complexity Psychedelic problems (AI-complete)

19© Gerhard Weikum

Outline




WF Specification Techniques

• Statecharts








20© Gerhard Weikum

Specification in WFMS Products

imprecise or ad hoc semantics

<flowModel name="totalSupplyFlow" <serviceProviderType="totalSupply"> <serviceProvider name="buyer" type="buyer" /> ...<activity name="submitPO"> ...</activity><controlLink source="submitPO" target="processPO"/> <controlLink source="processPO" target="processPayment"/> ...<dataLink source="submitPO" target="processPO"> <map sourceMessage="purchaseOrder" targetMessage="purchaseOrder"/> </dataLink> ...

graphs ... ...and scripts

21© Gerhard Weikum

Specification Methods

Requirements: Solutions:

Visualization •Refinement &Composability

•

Rigorous Semantics•Interoperability withother methods & tools

•

Wide acceptance &standard compliance

• Statecharts included inUML industry standard(Unified Modeling Language,OMG 1997))

Import / exportBPR tools WFMS WFMS

Statecharts (Harel et al. 1987)

(alt.: Petri Net variants,temporal logic, process algebra,script language)

22© Gerhard Weikum

describes process structure•nodes: activities•edges: data flow

Example of Harel-style Activitychart

23© Gerhard Weikum

describes process behavior•nodes: execution states•edges: control flow•transition labels:•event [condition] / action rules

Example of Harel-style Statechart

24© Gerhard Weikum

Refinement of Harel-style Statechart

25© Gerhard Weikum

Example of Workflow-style Activitychart

DE CCW RSK

DEC ERROR

CREDIT_REQUEST_AC

CustomerData

CustomerData

CustomerData

CustomerData

DE: Data EntryCCW: Check Credit WorthinessRSK: Risk AssessmentDEC: Decision

26© Gerhard Weikum

Example of Workflow-style Statechart

DE_S

RSK_SCCW_S

DEC_S

END_S

CR_SINIT_S

ERROR_S

CREDIT_REQUEST_SC

[DE_NOK orCCW_NOK orRSK_NOK orDEC_NOK or]

[DE_OK and not (Amount < 1000]

[DE_OK and Amount < 1000]

[DEC_OK]

[CCW_OK andRSK_OK]]

27© Gerhard Weikum

More Sophisticated Statechart Example

SelectConf

CheckFlight

CheckHotel

CheckCost

Go

No

/ Budget:=1000; Trials:=1;

[Fok & Eok]/ Cost := ConfFee + TravelExpenses

[Cost Budget]

[Cost > Budget & Trials < 3] / Trials++

[Cost > Budget & Trials 3]

[!Found]

[Found] / Cost:=0

SelectTutorials Compute

Fee

CheckAirfare

CheckHotel

CheckTravelExpenses

CheckConfFee

28© Gerhard Weikum

E-Commerce Workflow: Activitychart

ECommerce_AC

NewOrderCreditCardCharge

CheckStore

FindStore

@ECommerce_SC

Notify

CreditCardCheck

Payment

OrderNumber, EmailAddress., ...

OrderNumber, Address, ...

OrderNumber,ItemList, ... StoreID,

ItemList, ...Acknowledgement

Name, Address, OrderNumber, ...

Name, Date, CreditCardNumber, ...

CreditCardNumber, Amount, ...

29© Gerhard Weikum

E-Commerce Workflow: Statechart

Shipment_S

CreditCardCheck_S NewOrder_S [PayByCreditCard and

NewOrder_DONE]

[PayByBill and NewOrder_DONE]

[CreditCardOK and CreditCardCheck_DONE]

[CreditCardNotOK and CreditCardCheck_DONE]

[in(Notify_EXIT_S) and in(Delivery_EXIT_S) and

PayByCreditCard] CreditCardCharge_S

EC_EXIT_S [CreditCardCharge_DONE] Payment_S

[Payment_DONE]

[in(Notify_EXIT_S) and in(Delivery_EXIT_S) and

PayByBill]

ECommerce_SC INIT_S

Notify_S Notify_EXIT_S

[Notify_DONE]

Notify_INIT_S

FindStore_S CheckStore_S [ItemsLeft and

FindStore_DONE]

/fs!(ItemAvailable)

[ItemAvailable and CheckStore_DONE]

[AllItemsProcessed]

Delivery_EXIT_S

Delivery_INIT_S

30© Gerhard Weikum

Workflow AdministrationFrom Organizational Viewpoint

• Worklist Management:Who is assigned which pieces of work?

• Work History Management and Evaluation:Which processes are late?Which process types have inherent bottlenecks?How can we improve work effectivity?

31© Gerhard Weikum

Worklist Management• Assignment: Work Items Actors

(where a work item is a non-automated activity that is ready to be started)

• Static Mapping onto Roles

• Dynamic Resolution of Roles into Actors(based on competence, availability, experince, etc.)

+ additional functions:- enforcing constraints (e.g., dual control)- monitoring of deadlines and alerting- priority control- load balancing

32© Gerhard Weikum

Worklist Management Implementation

Typical solution:worklist manager and worklist DB on server ,worklist GUI for clients

33© Gerhard Weikum

Worklist Management Example

Find all actors who are capable of performing the role, have the necessary permissions, and are currently available. Among those, assign the work item to the actor with the lowest current workload.

34© Gerhard Weikum

Worklist Management Strategies

• organizational structure of the enterprise• actors´ expertise and experience• actors´ availability and load• workflow-instance-specific restrictions

Parameters to be considered:

Implementation of a worklist strategy:

• specifying the strategy as a workflow• implement the activities (queries against organizational databases)• integrate the strategy into the workflow

35© Gerhard Weikum

Integration of Worklist Strategies

Original specification

Work assignment strategy included as nested statechart

S2

. . .E1[C1]

E2[C2]/st!(activity2)

S2.1 S2.n

.../st!(insertWL)

. . .

AcceptWI/st!(activity1)

. . .S1

. . .



S1 S2

Rationale: Worklist strategies are workflows themselves!

36© Gerhard Weikum

Event Process Chains (EPCs) for Business Process Modeling

event

function

condition

actor(role)

inputdata

outputdata

action

popular in BPR toolsused in SAP Workflow

37© Gerhard Weikum

Event Process Chains: Control Flow Constructs

function

condition 1 condition 2

event 1 event 2

... ...

branching

function

event 1 event 2... ...

(fork-join)split

38© Gerhard Weikum

Event Process Chains: Simple Example

DEStart

DE_OK

Amount< 1000

Amount 1000

CCW

CCW_OK

RSK

RSK_OK

DEC

DE: Data EntryCCW: Check Credit WorthinessRSK: Risk AssessmentDEC: Decision

39© Gerhard Weikum

Import from BPR Tools

Event process chains (EPCs à la Aris Toolset):

- process decomposed into functions

- completed functions raise events that trigger further functions- control-flow connectors

40© Gerhard Weikum

Import from BPR Tools (continued)

41© Gerhard Weikum

Automatic Conversion EPC SC

Event process chainscan (often) be automatically converted into statecharts

42© Gerhard Weikum

Automatic Conversion EPC SC

43© Gerhard Weikum

Outline





Statecharts








44© Gerhard Weikum

Abstract Syntax of Statecharts (1)

State set S

State tree (with node types AND or XOR)

Transition t: (source, target, [c]/a)

Transition set T

Variable set V

A

B

C

F

D

ML

K

HG

E

J

45© Gerhard Weikum

Abstract Syntax of Statecharts (2)

A

B J

C F K ML

D E G H

AND XOR

XOR XOR XOR XOR XOR

XOR XOR XOR XOR

XOR

46© Gerhard Weikum

Operational Semantics of Statecharts (1)

Execution state of statechart (S,T,V): subset states S of currently active states s.t.• root of S is in states• if s in states and type of s is AND then all children of s are in states• if s in states and type of s is XOR then exactly one child of s is in states

Execution context of statechart (S,T,V): current values of variables defined by val: V Dom

Configuration of statechart (S,T,V): (states, val)Initial configuration

47© Gerhard Weikum


Evaluation of expression in configuration:eval (expr, conf) defined inductively

Effect of action on context: modification of variable values in val

fire(conf) = set of transitions t = (source, target, [cond]/action) with source(t) in states for which eval(cond, conf) = true

48© Gerhard Weikum


for transition t:• a = lca (source(t), target(t))• src(t) = child of a in subtree of source(t)• tgt(t) = child of a in subtree of target(t)

when t fires:• set of left states source*(t):

• src(t) is in source*(t)• if s in source*(t) then all children of s are in source*(t)

• set of entered states target*(t):• tgt(t) and target(t) are in target*(t)• if s in target*(t) and type of s is AND then all children of s are in target*(t)• if s in target*(t) and type of s is XOR then exactly one child of s with initial transition is in target*(t)

49© Gerhard Weikum


For a given configuration conf = (states, val) a successor configuration conf‘ = (states‘, val‘) is derivedby selecting one transition t from fire(conf) with the effect:• states‘ = states – source*(t) target*(t)• val‘ captures the effect of action(t) and equals val otherwise

The operational semantics of a statechart (S,V,T) is theset of all possible executions along configurationsconf0, conf1, conf2, ... with• initial configuration conf0 and• confi+1 being a successor configuration of confi

51© Gerhard Weikum

Guaranteed Behavior and Outcomeof Mission-critical Workflows

Crucial for workflows inbanking, medical applications, electronic commerce, etc.

Formalization of properties

Verification method

Mathematical model

Model checking

Temporal logic

Finite-state automaton

• Safety properties (invariants): nothing bad ever happens• Liveness properties (termination, fairness, etc.): something good eventually happens

52© Gerhard Weikum

Mapping Statecharts into FSAs

Represent SC configurations as states of a finite state automaton:

Step 1:abstract conditions on infinite-domain variables into Boolean variablesformal mapping: 1: val B1 B2 ... Bm

Step 2:capture set of active SC states (along SC hierarchy and in components)by powerset automaton 2: states 2S =: Z

Step 3:encode SC context into extended state space of FSAby an injective mapping 3: Z B1 B2 ... Bm Z’ such that there is a transition from z1 to z2 in the FSAiff 3-1(z2) is a possible successor configuration of 3-1(z1) in the SC

53© Gerhard Weikum

Example: From SC To FSA (1)

SelectConf

CheckFlight

CheckHotel

CheckCost

Go

No

/ Budget:=1000; Trials:=1;

[Fok & Eok]/ Cost := ConfFee + TravelExpenses

[Cost Budget]

[Cost > Budget & Trials < 3] / Trials++

[Cost > Budget & Trials 3]

[!Found]

[Found] / Cost:=0

SelectTutorials Compute

Fee

CheckAirfare

CheckHotel

CheckTravelExpenses

CheckConfFee

54© Gerhard Weikum

Example: From SC To FSA (2)

SelectConf,!F,!Fok,!Eok,

!Bok,TokCheckCost,F,Fok,Eok,Bok,!Tok

No,!F,!Fok,!Eok,

!Bok,Tok

CheckCost,F,Fok,Eok,

Bok,Tok

CheckCost,F,Fok,Eok,!Bok,Tok

No,F,Fok,Eok,!Bok,!Tok

Go,F,Fok,Eok,

Bok,Tok

...13

2

4

5

7

8

9CheckCost,F,Fok,Eok,!Bok,!Tok

6

CheckConfFee,CheckTravelExpenses,

F,!Fok,!Eok,Bok,Tok

55© Gerhard Weikum

CTL: Computation Tree Logic

propositional logic formulasquantifiers ranging over execution pathsmodal operators referring to future states

EF AG p

combination:

all globally:

AG p

all finally(inevitably):

AF p

exists globally:

EG p

exists finally(possibly):

EF p

allnext:

AX p

existsnext:

EX p

56© Gerhard Weikum

Critical Properties of the Example Workflow

Do we always eventually reach a decision ?

AF ( in(Go) or in(No) )

Can we ever exceed the budget ?

not EF ( in(Go) and !Bok )

AG ( not in(Go) or Bok )

formalized in CTL (Computation Tree Logic)

Can the trip still be approved after a proposalthat would have exceeded the budget ?

EF ( (in(CheckCost) and !Bok) => ( EF (in(Go)) ) )

57© Gerhard Weikum

CTL SyntaxDefinition:An atomic CTL formula is a propositional logic formula over elementary propositions (i.e., Boolean variables).The set of CTL formulas is defined inductively:• Every atomic CTL formula is a formula..• If P and Q are formulas then

EX (P), AX (P), EG (P), AG (P), EF (P), AF (P), (P), P, PQ, PQ, PQ and PQ are formulas..

58© Gerhard Weikum

CTL Semantics (1)Definition:Consider a set P of elementary propositions. A Kripke structure M over P is a 4-tuple (S, s0, R, L) with• a finite state set S,• an initial state s0 S,• a transition relation R S S,• a function L: S 2P that assigns true propositions to each state.

59© Gerhard Weikum

CTL Semantics (2)Definition:The interpretation of formula F over elementary propositions P is a mapping onto a Kripke structure M=(S, s0, R, L) over propositions P such that the truth value of subformulas p, p1, p2 of F in state s, denoted M,s |= p, is defined as follows:

(i) M,s |= p with propositional formula p holds iff p L(s); (ii) M,s |= p holds iff M,s |= p does not hold;(iii) M,s |= p1 p2 iff M,s |= p1 and M,s |= p2;(iv) M,s |= p1 p2 iff M,s |= p1 or M,s |= p2;(v) M,s |= EX p iff there exists tS with (s,t)R and M,t |= p;(vi) M,s |= AX p iff for all tS with (s,t)R M,t |= p holds;(vii) M,s |= EG p if there exists t1, ..., tk S with t1=s, (ti, ti+1)R for all i and tk=tj for some j:1j<k or tk has no successors, such that M,ti |= p for all i;(viii) M,s |= AG p iff for all tS with (s,t)R+ M,t |= p holds;(ix) M,s |= EF p iff there exists tS with (s,t)R+ and M,t |= p;(x) M,s |= AF p iff for all tS with (s,t)R+ there exists t’S with a) (t,t’)R+ or b) (s,t’)R+ and (t’,t)R+, such that M,t’ |= p holds.

60© Gerhard Weikum

CTL Semantics (3)

Definition:A Kripke structure M = (S, s0, R, L) is a model of formula F if F is true in s0, denoted M,s0 |= F.A formula is satisfiable if it has at least one model, otherwise it is unsatisfiable. A formula is valid (or called a tautology) if everyKripke structure over the elementary propositions of F is a model of F.

61© Gerhard Weikum

Model CheckingFor CTL formula F and transition system (Kripke structure) Mcheck if M is a model of F by inductively marking all states of M in which subformula q of F holds with the label q.

Let q be a subformula of F, let p, p1, p2 direct subformulas of q,and let P, P1, P2 be the sets of states of M with labels p, p1, p2, resp.(i) q is an elementary proposition (Boolean variable): label all states s with qL(s) with label q(ii) q is of the form p: label S – P with label q(iii) q is of the form p1 p2: label P1 P2 with label q(iv) q is of the form p1 p2: label P1 P2 with label q(v) q is of the form EX p: label all predecessors of P with label q (i.e., all sS for which there exists xP with R(s,x) )(vi) q is of the form AX p: label s with q if all successors of s are labeled with p

62© Gerhard Weikum

Model Checking: EF Case

(vii) q has the form EF p: solve recursion EF p p EX (EF p). (fixpoint computation Q = P pred(Q) )

Q := P; Qnew := Q pred(Q);while not (Q = Qnew) do Q := Qnew; Qnew := Q pred(Q); od;

63© Gerhard Weikum

Model Checking: EG Case

(viii) q has the form EG p: solve recursion EG p p EX (EG p) : Q := P; Qnew := Q ;repeatfor each s in Q do

if s has successors and no successor of s is in Q then Qnew := Q - {s}; fi; od;until (Q = Qnew);

64© Gerhard Weikum

Model Checking: AG Case (ix) q has the form AG p: solve recursion AG p p AX (AG p) Q := P; repeat Qnew := Q; for each s in Q do if s has successors and one successor of s is not in Q then Q := Q - {s} fi; od;until (Q = Qnew); Alternatively, because of AG p EF (p): compute state set Q’ labeled EF (p) and label S – Q’ with label q.

65© Gerhard Weikum

Model Checking: AF Case

(x) q has the form AF p: solve recursion AF p p AX (AF p)

Q := P;repeat Qnew := Q;

for each s in pred(Q) do if all successors of s are in Q

then Q := Q {s}; fi; od;

until (Q = Qnew); Alternatively, because of AF p EG (p): compute state set Q’ labeled EG (p) and label S – Q’ with label q.

66© Gerhard Weikum

Model Checking: Example 1



No,!F,!Fok,!Eok,

!Bok,Tok


Bok,Tok



Go,F,Fok,Eok,

Bok,Tok

...1

3

2

4

5

7

8


6


F,!Fok,!Eok,Bok,Tok

AG ( not in(Go) or Bok )

Labelwith Bok :with in(Go) :with in(Go) :with (Bok in(Go)) :with AG (Bok in(Go)) :

3, 4, 5, 881, 2, 3, 4, 5, 6, 7, 91, 2, 3, 4, 5, 6, 7, 8, 91, 2, 3, 4, 5, 6, 7, 8, 9

67© Gerhard Weikum




No,!F,!Fok,!Eok,

!Bok,Tok


Bok,Tok



Go,F,Fok,Eok,

Bok,Tok

...1

3

2

4

5

7

8


6


F,!Fok,!Eok,Bok,Tok

Labelwith in(Go) :with in(No) :with in(Go) in(No) :with AF (in(Go) in(No)) :

AF (in(Go) in(No))

82, 92, 8, 92, 4, 5, 6, 8, 9

68© Gerhard Weikum




No,!F,!Fok,!Eok,

!Bok,Tok


Bok,Tok



Go,F,Fok,Eok,

Bok,Tok

...1

3

2

4

5

7

8


6


F,!Fok,!Eok,Bok,Tok

Labelwith in(Go) :with EF (in(Go)) :with not in(CheckCost) or Bok :with (in(CheckCost) and !Bok) => ( EF (in(Go)) :with EF ( (in(CheckCost) and !Bok) => ( EF (in(Go)) ) ) :

EF ( (in(CheckCost) and !Bok) => ( EF (in(Go)) ) )

81, 3, 4, 5, 7, 81, 2, 3, 4, 5, 81, 2, 3, 4, 5, 7, 81, 2, 3, 4, 5, 7, 8

69© Gerhard Weikum

Guaranteed Behavior of Workflows

Efficiency gain with encoding of FSM as OBDD

Leverage computer-aided verification techniques for finite-state concurrent systems

Further requirements:

- More expressive logic

- Adding real-time (clock variables)

- User-friendly macros for CTL

Preserving guaranteed behaviorin distributed, failure-prone system environment System guarantees

- Adding assertions on behavior of invoked apps

71© Gerhard Weikum

Summary and Open Research IssuesFormal specification and verification methods are crucialif we want to have high confidence in the correctness of workflow models

Statecharts and model checking are a good example

Interesting research topics for graduate students:Formal semantics of XML-based workflow spec languagesand automatic translation between languages

Comprehensive, user-friendly workflow verification workbench

Extended model checking orcombinations with theorem proving & constraint solvingfor enhanced verification

Comprehensive framework for correctness-preserving run-time modifications of workflow specifications

•

•

•

•

dependable workflow technology

Documents