Sapna E. George, Ing-Ray Chen

Presented ByYinan Li, Shuo Miao

April 14, 2009

AgendaAgendaIntroductionIntroductionThe Proposed ApproachThe Proposed Approach

System ModelSystem ModelCheckpointing, Logging, and RecoveryCheckpointing, Logging, and Recovery

Performance AnalysisPerformance AnalysisThe SPN ModelThe SPN ModelResults and AnalysisResults and Analysis

Summary and ApplicabilitySummary and Applicability

Failure-Prone Mobile ComputingFailure-Prone Mobile ComputingMobile ComputingMobile Computing

A type of distributed computing involving hosts that are mobile while retailing network connection through unreliable wireless communication

Prone to failuresWhy mobile computing is prone to failures?Why mobile computing is prone to failures?

Host mobility Link breakage Limiting the use of backend power source

Limited battery lifeWireless communication

Unreliable wireless links Intermittent connectivity Bandwidth limitations

Failure Recovery in Traditional Failure Recovery in Traditional Distributed SystemsDistributed SystemsCheckpointingCheckpointing

Process state is periodically saved to stable storage such as hard disks, namely checkpoint Checkpoints may be large, e.g., the size of a checkpoint may be several KB

Sometimes called a snapshotIndependent VS. coordinated

Independent checkpointing Each process independently and asynchronously takes local

checkpoints Coordinated checkpointing

All processes synchronize to jointly build global checkpointsAn expensive operation

During the process of taking checkpoints, the process has to be suspended

The size of checkpoints may be large How can we reduce the number of checkpointing operations, while still

enable rollback recovery?

Failure Recovery in Traditional Failure Recovery in Traditional Distributed SystemsDistributed SystemsLoggingLogging

Proposed as a complimentary to checkpointing, to reduce the number of expensive checkpointing operations, but still enable effective recovery

Recording all the non-deterministic events happening in a process between two checkpoints that potentially change the state of the process, e.g., receipt of a message, user input, etc., and the information necessary to reply these events

Asynchronous recovery is achieved by combining checkpointing with logging

Failure Recovery in Traditional Failure Recovery in Traditional Distributed SystemsDistributed SystemsRollback recoveryRollback recovery

Upon failure, the state of the failed process is rolled back to the most recent checkpoint

Events found in the logs that happened after the most recent checkpoint are replayed in their original order

Independent recovery Each process recreates its pre-failure state independently

(asynchronously)Computation is restarted after rollback

New Challenges Brought by Mobile New Challenges Brought by Mobile ComputingComputingAre traditional methods still directly applicable?Are traditional methods still directly applicable?

Rethinking …Properties of mobile computing that complicate the situationProperties of mobile computing that complicate the situation

Mobility of hostsUnreliable wireless network connectivityLow wireless bandwidthLimited battery life of mobile devicesLack of stable storageDifferent types of failures

Voluntary disconnection (to save battery life) Hardware failure Transmission errors due to noises Software failure

The Proposed Approach (1/2)The Proposed Approach (1/2)Why it is proposed?Why it is proposed?

Traditional recovery schemes may not work Due to the characteristics of mobile computing And the challenges imposed by mobile applications

The limitation of existing methods Checkpoints are taken periodically

If the frequency of checkpointing is high, the additional overhead is large If the frequency is low, the recovery cost may be very large (the distance

between the current MSS of the MH and the MSS that stores the checkpoint may be large, and the logs may be widely dispersed)

Rigid mechanism for the whole system, lacking the flexibility of per-user methods

The Proposed Approach (2/2)The Proposed Approach (2/2)Proposed approachProposed approach

Movement-based, instead of periodic Checkpoints are taken only when a mobile host has made M handoffs Goal: looking for the optimal M that minimizes the total recovery cost

AdvantagesAdvantagesMinimization of the total recovery cost

The optimal threshold M ensures that Checkpoints stay close to the recovery MSS Logs are not too widely dispersed Overhead of unnecessary checkpoints & logs is avoided

Per-user based M is a function of the failure rate, log arrive rate, and mobility rate of

individual MHs, which is adaptive to specific user behaviorWide-range applicability

System Model (1/2)System Model (1/2)

System Model (2/2)System Model (2/2)MHs are not assumed to have stable storageMHs are not assumed to have stable storageMSSs are equipped with enough stage storageMSSs are equipped with enough stage storage

However, efficient storage management is still necessaryInteractions between MH and the network infrastructure Interactions between MH and the network infrastructure

most relevant to failure recoverymost relevant to failure recoveryHandoff, disconnect, and reconnect

Process stateProcess stateNormal: computing, receiving user inputs, sending/receiving

messages Save

Checkpointing: checkpoints are sent to MSS for stable storage Logging: events happening between two checkpoints that cause process

state changes, e.g., receipt of messages or user inputs, are logged by MSS to stable storage

Recovery: rollback

Checkpointing & Logging (1/3)Checkpointing & Logging (1/3)

Checkpointing & Logging (2/3)Checkpointing & Logging (2/3)Each MH maintains several variables locallyEach MH maintains several variables locally

handoff_counter: stores the #handoffscp_seq: store the sequence number of the most recent

checkpoint cp_loc: stores the ID of the MSS that stores the most

recent checkpointMSScp: the MSS that stores the most recent checkpointlog_set: contains a list of IDs of MSSs that stores the logsMSSlogs: the set of MSSs that stores the logs These variables must survive the host failure

Checkpointing & Logging (3/3)Checkpointing & Logging (3/3)ProcedureProcedure

When handoff_counter = M A new checkpoint is taken

The checkpoint is sent to the current MSS for stable storage cp_seq is set to the sequence number of this new checkpoint cp_loc is updated with the current MSS handoff_counter is reset to zero log_set is cleared

When a new log is recorded The log is sent the to current MSS for stable storage The ID of the MSS is inserted into log_set if it is not present

When a handoff occurs handoff_counter is increased by 1 and checked against M

Independent RecoveryIndependent RecoveryEach MH recovers independently without global coordinationEach MH recovers independently without global coordination

Asynchronous recoveryRecovery procedureRecovery procedure

When a MH reconnects to a MSS after a failure, it sends cp_seq and cp_loc to its current MSS

The MSS initiates the process of collecting the checkpoint and the logs on behalf of the MH The current MSS sends a request to MSScp MSScp responds with the most recent checkpoint The current MSS sends requests to all MSSs in MSSlogs Each MSS in MSSlogs responds with the log entry

The current MSS compiles the checkpoint and all the logs, sends it back to the MH

The MH rolls back to the checkpoint and replays all the logs

Performance Analysis: SPN ModelPerformance Analysis: SPN Model

SPN Model ParametersSPN Model Parameters

The SPN Model (1/3)The SPN Model (1/3)PlacesPlaces

Handoff: represents the state in which handoff_counter since the last checkpoint does not exceed M

Failure: stands for the state in which a MH failure has occurred.

Initially, mark(“handoff”)=0 and mark(“failure”) = 0

TransitionsTransitionsMove (inter-cell): with a mobility rate of σCheckpointing: with a checkpoint rate of θk

Fail: with a failure rate of λf

Recovery: with a recovery rate of θi, which depends on i=handoff_counter

The SPN Model (2/3)The SPN Model (2/3)Parameters elaborationParameters elaboration

θθkk – checkpointing rate – checkpointing rate The MH takes a snapshot of its current state and sends it to the MSS for

stable storage The dominating factor is the transmission of the checkpoint to the MSS Thus, θθk = k = 11/T/Tckp_wckp_w

θθi i - recovery rate - recovery rate This represents the recovery rate, which is the reverse of the average

recovery time The average recovery time is the sum of

1. the time needed to send recovery request information to all the MSSs that stores the most recent checkpoint and all the logs

2. the time needed to transmit the most recent checkpoint from MSScp to MSSrec , and from MSSrec to the MH

3. The time needed to transfer all the logs from respective MSSs where they are located to MSSrec, and from MSSrec to the MH

4. The time required to rollback and replay all the logs

The SPN Model (3/3)The SPN Model (3/3)Transition rulesTransition rules

Whenever the MH encounters a handoff, the number of tokens in the place “Handoff” is increased by 1

The MH stays in the state represented by the place “Handoff” if the number of cumulative handoffs is less than M before a failure

When the number of tokens in the place “Handoff” is equal to M, the transition “Checkpointing” is fired, and this consumes all the tokens in the place “Handoff”, i.e., handoff_counter is reset

When a MH failure occurs (the transition “Fail” is fired), all the tokens in the place “Handoff” are moved to the place “Failure”; the recovery rate θθi i depends on the number of handoffs since the last checkpoint, which is denoted as i = mark(“Fail”) = #(Handoffs)

The only inhibitor arc ensures that the number of handoffs between two checkpoints does not exceed the threshold M

Calculations (1/5)Calculations (1/5)

3/)2(16

30

6

21

6

1)1(1,

jjD jmss

iN smss log_

Variables required to compute :

The number of MSSs storing logs, this is the size of the list log_set

The average hop count between two MSSs separated by j handoffs

irT

1/6 probability to move backward

2/6 probability to have sideway

moves

3/6 probability to move forward

The first move

For simplicity, it is assumed that this equals to the number of handoffs, i.e., i

Calculations (2/5)Calculations (2/5)Total time to recover after i handoffs is the sum of the following components

i

jwjmssreqrec TrDT

1log_,_ )(

wckpwckpjmsstxckp TTrDT __,_

Time for recovery info requests, assuming that the size of a request packet is no more than the size of a log

Time for transferring the most recent checkpoint to the MH

The time required for MSSrec to transfer one request packet, assuming that the time to send a request packet from MH to MSSrec through wireless network is wTlog_

The time required to transfer the checkpoint from MSScp to MSSrec through wired link

The time required to transfer the checkpoint from MSSrec to the MH through wireless link

Calculations (3/5)Calculations (3/5)

ww

i

jmssjmsstx TnTrnDT log_log_

1,log_ )(

Time for transferring the logs to the MH; n is the number of logs since the most recent checkpoint, and nmss is the number of log entries per MSS

)]1(/[)( ;/)( iinin mss

logeeckprec TnTT

Time for rolling back and replaying all the n logs

The time required to transfer the logs from MSSlogs to MSSrec through wired link, assuming that each MSS in MSSlogs stores nmss logs

The time required to transfer the logs from MSSrec to the MH through wireless link

rectxtxckpreqrecir TTTTT ),max( log___

12

0

M

j

irjr TPT

Thus we have:

The total time to recovery after i handoffs; 1/ θi

The average recovery time per failureThe underlying Markov model has 2M+1 states, with the probability of state j as Pj

Calculations (4/5)Calculations (4/5)

The max operator is used to take the maximum of checkpoint transfer time and log transfer time since these messages can be transmitted simultaneously

wfwckpfc TTMT log__ )/())/((

rct TwTwT 21cos

12

1

)1(}{PrM

S

TSrr

SePTTobF

The recovery probability, defined as the probability that the recovery time is less than or equal to the recovery time deadline T; θS corresponds to θi if the MH has made i handoffs in state S

The total time spent on checkpointing and logging before a failure

The total cost of recovery per failure is the weighted sum of the average recovery time per failure and the total time spent on checkpointing and logging per failurew1 = w2 = 0.5 in the result analysis

Calculations (5/5)Calculations (5/5)

Total number of checkpoints taken

before a failure

Total number of log entries taken

before a failure

Results and Analysis (1/4)Results and Analysis (1/4)

Log arrival rate↑ ecover Probability ↓Mobility Rate ↑ Recover Probability ↑

Mobility Rate ↑ Checkpoint interval ↓ Number of logs ↓ Recovery time ↓

Flat segment:Recall

Deadline ↑ Recover Probability ↑Mobility Rate ↑ Recover Probability ↑

Mobility Rate ↑ Checkpoint interval ↓ Number of logs ↓ Recovery time ↓

rectxtxckpreqrecir TTTTT ),max( log___

Results and Analysis (2/4)Results and Analysis (2/4)

Failure Rate ↑ Number of logs ↓ Recovery time ↓ Recover Probability ↑

Checkpoint Size↑ ↑ Recovery Probability↓ When size of checkpoint is sufficiently large (32K), dominates Mobility Rate becomes insensitive.

wckpT _

wTlog_

Results and Analysis (3/4)Results and Analysis (3/4)

M↑ Time interval↑ Number of logs ↑ Recovery time ↑ Recover Probability ↓

M is NOT the smaller the better A smaller M brings more overhead during failure free operation. So there is a tradeoff between recovery time and total time.

Results and Analysis (4/4)Results and Analysis (4/4)

The curves indicate there is a optimal M.Small M Tc dominates TrM ↑ Tc ↓ , Tr ↑ and Total cost ↓Large M Tr dominates Tc Total cost ↑

Crossover point at M=8Large M Tr dominates TcMobility Rate ↓ Checkpoint interval ↑# of logs ↑ Recovery time ↑

Tr: Recovery TimeTc: Total time spent on checkpointing and logging per failure

SummarySummaryAn efficient failure recovery scheme for mobile An efficient failure recovery scheme for mobile

computing systems based on movement-based computing systems based on movement-based checkpointing and logging.checkpointing and logging.

A Performance model based on stochastic Petri NetsA Performance model based on stochastic Petri NetsIdentify optimal movement threshold MMinimize cost of recovery per failureCalculate failure recoverability

ApplicabilityApplicabilityBuild a table covering possible parameter valuesBuild a table covering possible parameter values

Mobility rate, failure rate, log arrival rate and etc.Optimal M may be selected dynamically to minimize costOptimal M may be selected dynamically to minimize costOptimal M must satisfy specified recovery probabilityOptimal M must satisfy specified recovery probability

Questions?Questions?