soft state in publish/subscribe

Overview System Model Soft State Evaluation Summary 1 of 23 slides

Soft State in Publish/SubscribeDEBS 2009

Zbigniew Jerzak and Christof Fetzer

Systems Engineering Group, Dresden University of Technology

Thursday 9th July, 2009

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 2 of 23 slides

What is Soft State?

System state which can be lost without permanent disruption ofsystem features [Cla88]

I Soft state system =⇒ a survivable system:I can cope with transient failuresI state is held at the edge of the systemI node crash =⇒ loss of state owned by it

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 2 of 23 slides

What is Soft State?

System state which can be lost without permanent disruption ofsystem features [Cla88]

I Soft state system =⇒ a survivable system:I can cope with transient failuresI state is held at the edge of the systemI node crash =⇒ loss of state owned by it

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 3 of 23 slides

In This Talk

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 3 of 23 slides

In This Talk

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 3 of 23 slides

In This Talk

I How to build a soft pub/sub system using a practical systemmodel, i.e., one which:

I makes realistic assumptions about the type and frequency offailures

I can be implemented in a real, distributed environment, i.e.,PlanetLab

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 4 of 23 slides

Approach Overview

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 4 of 23 slides

Approach Overview

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 4 of 23 slides

Approach Overview

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 4 of 23 slides

Approach Overview

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 4 of 23 slides

Approach Overview

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 4 of 23 slides

Approach Overview

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 4 of 23 slides

Approach Overview

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 4 of 23 slides

Approach Overview

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 4 of 23 slides

Approach Overview

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 4 of 23 slides

Approach Overview

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 4 of 23 slides

Approach Overview

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 4 of 23 slides

Approach Overview

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 5 of 23 slides

Challenges

0.25

0.5

0.75

1

10-1

100

101

102

103

104

CD

F

rtt [ms]

LAN: inf.tu-dresden.deMAN: tu-dresden.de

WAN: tu-dresden->tu-berlin

Figure 1: Network latencies cannot be predicted, nor bounded!

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 5 of 23 slides

Challenges

0.25

0.5

0.75

1

100

101

102

103

104

105

CD

F

request processing time [µs]

Lab PCPlanetLab node

Figure 1: Request processing time cannot be predicted, nor bounded!

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 6 of 23 slides

Challenges – contd.

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 6 of 23 slides

Challenges – contd.

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 6 of 23 slides

Challenges – contd.

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 6 of 23 slides

Challenges – contd.

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 6 of 23 slides

Challenges – contd.

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 6 of 23 slides

Challenges – contd.

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 6 of 23 slides

Challenges – contd.

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 6 of 23 slides

Challenges – contd.

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 6 of 23 slides

Challenges – contd.

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 6 of 23 slides

Challenges – contd.

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 7 of 23 slides

Timed Asynchronous Distributed System Model [CF99]

I All processes are timed (σmax)I . . . however, no guarantee that a request is indeed answered

within σmax

I Processes expose crash-stop failure semanticsI . . . and we know [WF07] how to enforce it

I Messages might be arbitrarily delayed or droppedI . . . specifically, no upper bound on message transmission delay

I Only local hardware clocks, with bounded drift rate (ρmax)I . . . and crash-stop failure semantics

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 7 of 23 slides

Timed Asynchronous Distributed System Model [CF99]

I All processes are timed (σmax)I . . . however, no guarantee that a request is indeed answered

within σmax

I Processes expose crash-stop failure semanticsI . . . and we know [WF07] how to enforce it

I Messages might be arbitrarily delayed or droppedI . . . specifically, no upper bound on message transmission delay

I Only local hardware clocks, with bounded drift rate (ρmax)I . . . and crash-stop failure semantics

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 7 of 23 slides

Timed Asynchronous Distributed System Model [CF99]

I All processes are timed (σmax)I . . . however, no guarantee that a request is indeed answered

within σmax

I Processes expose crash-stop failure semanticsI . . . and we know [WF07] how to enforce it

I Messages might be arbitrarily delayed or droppedI . . . specifically, no upper bound on message transmission delay

I Only local hardware clocks, with bounded drift rate (ρmax)I . . . and crash-stop failure semantics

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 7 of 23 slides

Timed Asynchronous Distributed System Model [CF99]

I All processes are timed (σmax)I . . . however, no guarantee that a request is indeed answered

within σmax

I Processes expose crash-stop failure semanticsI . . . and we know [WF07] how to enforce it

I Messages might be arbitrarily delayed or droppedI . . . specifically, no upper bound on message transmission delay

I Only local hardware clocks, with bounded drift rate (ρmax)I . . . and crash-stop failure semantics

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 8 of 23 slides

Publish/Subscribe Model

I Our approach works with any pub/sub modelI topic-basedI type-basedI content-basedI rule-based

I It does not require bounds on message propagation delays

I It does not require clock synchronization

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 8 of 23 slides

Publish/Subscribe Model

I Our approach works with any pub/sub model

I It does not require bounds on message propagation delays

I propagation delay = transmission delay + processing delay

I It does not require clock synchronization

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 8 of 23 slides

Publish/Subscribe Model

I Our approach works with any pub/sub model

I It does not require bounds on message propagation delays

I It does not require clock synchronization

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 9 of 23 slides

Approach

Use upper bound on message propagation delayto estimate the link uncertainty

I Link uncertainty:I a measure of the link latency variationI calculated using upper bound on propagation delay [JFF07]

Extend the advertisement/subscription validityusing calculated link uncertainty

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 9 of 23 slides

Approach

Use upper bound on message propagation delayto estimate the link uncertainty

I Link uncertainty:I a measure of the link latency variationI calculated using upper bound on propagation delay [JFF07]

Extend the advertisement/subscription validityusing calculated link uncertainty

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 9 of 23 slides

Approach

Use upper bound on message propagation delayto estimate the link uncertainty

I Link uncertainty:I a measure of the link latency variationI calculated using upper bound on propagation delay [JFF07]

Extend the advertisement/subscription validityusing calculated link uncertainty

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 10 of 23 slides

Extending the Lease Time

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 10 of 23 slides

Extending the Lease Time

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 10 of 23 slides

Extending the Lease Time

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 10 of 23 slides

Extending the Lease Time

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 10 of 23 slides

Extending the Lease Time

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 11 of 23 slides

Estimating (One-Way) Link Latencies

latency(sub) 6= (D − C )

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 11 of 23 slides

Estimating (One-Way) Link Latencies

latency(sub) 6= (D − C )

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 11 of 23 slides

Estimating (One-Way) Link Latencies

latency(sub) ≥ (D − A)

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 11 of 23 slides

Estimating (One-Way) Link Latencies

latency(sub) ≥ (D − A)

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 11 of 23 slides

Estimating (One-Way) Link Latencies

latency(sub) ≥ (D − A)− (C − B)

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 11 of 23 slides

Estimating (One-Way) Link Latencies

latency(sub) ≥ (D − A)− (C − B)− δmin

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 11 of 23 slides

Estimating (One-Way) Link Latencies

latency(sub) ≥ (D − A)(1 + ρmax)− (C − B)(1− ρmax)− δmin

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 12 of 23 slides

Evaluation Environment

I F-A P/SI Zbigniew Jerzak, Robert Fach, and Christof Fetzer. Fail-aware

publish/subscribe. In NCA 07: Sixth IEEE InternationalSymposium on Network Computing and Applications, pages113125, Cambridge, MA, USA, July 2007.

I XSiena

I Apache Mina

I PlanetLab

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 12 of 23 slides

Evaluation Environment

I F-A P/SI XSiena

I http://wwwse.inf.tu-dresden.de/xsiena/

I Apache Mina

I PlanetLab

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 12 of 23 slides

Evaluation Environment

I F-A P/S

I XSienaI Apache Mina

I http://mina.apache.org/

I PlanetLab

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 12 of 23 slides

Evaluation Environment

I F-A P/S

I XSiena

I Apache MinaI PlanetLab

I http://www.planet-lab.eu/

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 13 of 23 slides

Local Setup

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 14 of 23 slides

Local Setup – Upper Bound on Processing Delay

0

2

4

6

8

10

12

14

16

18

0 10 20 30 40 50

tim

e [m

s]

event number

u-b pd (p1->s3)r-t pd (p1->s3)

u-b pd (p1->b2)r-t pd (p1->b2)

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 15 of 23 slides

European PlanetLab Setup

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 16 of 23 slides

PlanetLab Europe – First Broker

10

100

1000

300 320 340 360 380 400

tim

e/u

nce

rtai

nty

[m

s]

refresh message number

last 200 msgs (di.unito.it)last 20s (di.unito.it)u-b pd (di.unito.it)

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 17 of 23 slides

PlanetLab Europe – Last Broker

10

100

1000

300 320 340 360 380 400

tim

e/u

nce

rtai

nty

[m

s]

refresh message number

last 200 msgs (cs.uit.no)last 20s (cs.uit.no)u-b pd (cs.uit.no)

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 18 of 23 slides

Intercontinental PlanetLab Setup

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 19 of 23 slides

PlanetLab World – Advertisement Presence

no

yes

no

yes

no

yes

no

yes

26 28 30 32 34

adv

erti

sem

ent

pre

sen

t

time [s]

uoit.ca - extuoit.ca - noext

hiit.fi - exthiit.fi - noext

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 20 of 23 slides

PlanetLab World – Total Time without Advertisement

0

20

40

60

80

100

1 10 100

tota

l ti

me w

ith

ou

t ad

vert

isem

en

t [%

]

time [s]

uoit.ca - extuoit.ca - noext

hiit.fi - exthiit.fi - noext

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 21 of 23 slides

Summary

I Keep the state with the state owner

I Limit the impact of failures

I Use upper bound on propagation delay to ensure eventuallconsistency

I For more information:I Jerzak, Z.; Fach, R. & Fetzer, C. Fail-Aware Publish/Subscribe

NCA ’07: Proceedings of the Sixth IEEE InternationalSymposium on Network Computing and Applications, IEEEComputer Society, 2007, 113-125

I Jerzak, Z.; Fach, R. & Fetzer, C. Adaptive Internal ClockSynchronization SRDS ’08: 27th International Symposium onReliable Distributed Systems, IEEE Computer Society, 2008,217-226

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 22 of 23 slides

Thank You!http://wwwse.inf.tu-dresden.de/xsiena/softstate

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer

Overview System Model Soft State Evaluation Summary 23 of 23 slides

References

Flaviu Cristian and Christof Fetzer.The timed asynchronous distributed system model.IEEE Transactions on Parallel and Distributed Systems, 10(6):642–657, June1999.

D. Clark.The design philosophy of the DARPA internet protocols.In SIGCOMM ’88: Symposium proceedings on Communications architecturesand protocols, pages 106–114, New York, NY, USA, 1988. ACM.

Zbigniew Jerzak, Robert Fach, and Christof Fetzer.Fail-aware publish/subscribe.In NCA ’07: Proceedings of the Sixth IEEE International Symposium on NetworkComputing and Applications, pages 113–125, Cambridge, MA, USA, July 2007.IEEE Computer Society.

Ute Wappler and Christof Fetzer.Software encoded processing: Building dependable systems with commodityhardware.In Francesca Saglietti and Norbert Oster, editors, SAFECOMP ’07: 26thInternational Conference on Computer Safety, Reliability, and Security, volume4680 of Lecture Notes in Computer Science, pages 356–369, Nuremberg,Germany, September 2007. Springer.

Soft State in Publish/Subscribe Zbigniew Jerzak and Christof Fetzer