Update Propagation with Variable Connectivity

Prasun Dewan

Department of Computer Science University of North Carolina

dewan@unc.edu

2

Evolution of wireless networks Early days: disconnected computing (Coda’91)

Laptops plugged in at home or office No wireless network

Later: weakly connected computing (Coda ‘95) Assume a wireless network available, but Performance may be poor Cost may be high Energy consumption may be high Intermittent disconnectivity causes involuntary breaks

3

Issues raised by Coda’91 Considers strong connection and

disconnection weak connection? hoarding, emulation, or

something else?

4

Design Principles for Weak Connectivity

Don’t make strongly-connected clients suffer E.g. wait for weakly connected client to give token

Weak-connection should be better than disconnection Do communication tasks in background if possible

user patience threshold In weakly connected mode, communicate

the more important info what is important?

When in doubt ask, user Should be able to work without user advice

5

Disadvantages of Disconnection Updates not visible to others Updates at risk due to theft, loss, damage Conflicts more likely Cache misses may impede progress Exhaustion of cache space

6

State Transition Diagram

Hoarding

MergingEmulation

• Provide access to cached, possibly stale data• Track changes to cached data

• Detect conflicts• Resolve conflicts

Disconnection

Physical Reconnection

Logical Reconnection

• Serve cache miss• Hoard walk• Replace stale data• Write through

Write disconnected

Weak connection

• Serve cache miss• Mark stale data• Background adapted hoard walk• Track changes & trickle merging

Strong Connection,

User command

Weak connection

7

Immediate vs. Delayed Service of Cache Miss

On cache miss

if fetch time < user-patience-threshold

get remote object (in foreground)

else

add to items fetched during next hoard walk (in background)

(2 + e0.01p )

8

Replacing vs. Marking Stale Data Replacing stale data would make strongly

connected clients wait for weakly connected ones. Stale data replaced by invoking client callbacks Stale data marked by breaking a client callback

Requires sending of message to client Only on first remote update to cached object Client probably sends message to break callback

so that server does not have know who is weakly connected

Could give feedback to user, disallow changes or viewing of stale data, but not mentioned in paper

9

Volume Callbacks Volume callbacks another efficiency mechanism

Broken when any object in a volume changes Also make merging more efficient

If entire volume is unchanged then individual objects not examined

Can still invoke (any) file callbacks if volume callback broken

Adaptation to fine-grained (Sync-like) merging Hierarchical callbacks

10

Hoard Walk

Done every h time units (10 minutes) or user request

Not the same as connected hoard walk Stale small files automatically included User has option to marks large files to be

fetched

11

Trickle Merging

Do not propagate local changes immediately reduces # of messages sent reduces size of data sent

buffering changes allows log compression important because whole file sent what if small (Sync-like) changes sent?

12

Trickle Merging vs. Write-back Caching

Do not propagate local changes immediately reduce network traffic

Weak consistency Conflicts

Changes buffered in persistent storage

Buffering time large Many Minutes, days

Do not propagate local changes reduce latency

Strong consistency No conflicts

Changes buffered in volatile memory

Buffering time small Seconds, few minutes

13

Trickle Merging Implementation

write f1 mkdir d1

TMt1 < A TM

mkdir d2 mkdir d3

t1 > A

t1 < A

14

Trickle Merging Implementation

write f1 mkdir d1

TM

mkdir d2 mkdir d3

Do high priority task (e.g. servicing cache miss)

15

Trickle Merging Implementation

mkdir d1

TM

mkdir d2 mkdir d3 rm d1

Committed items not candidate for compression

16

Trickle Merging Implementation

mkdir d3 rm d1

17

Trickle Merging Implementation Consider those log changes created > A time units (600 s)

ago Allows change to age 50% compression effectiveness on all traces

Break these into chunks (C time units) Amount of data transmitted in time C based on available

bandwidth Urgent messages sent between chunk transmission

Cache miss services Trickle merging not done atomically

New log changes not combined with changes chosen at merge start time not sent for merging

Not specified when initiated Periodic, during inactivity, when earliest item ages

18

Merging Times chosen to get desired level of compression Desired level of consistency?

19

Trading Consistency for Performance

Consistency (Divergence)

Per

form

ance

20

Two methods to control divergence

Rover: Mix pessimistic and optimistic Worry about W-W conflicts but not R-W conflicts

Some objects may be cached in read-only mode. Verify object not changed before allowing change

• Pull model: client can check

• Push model: server can inform (Coda)

Worry about R-W conflicts. Expire object after timeout.

Lotus Notes, Suite, Coda:Keep optimistic model Reduce transaction length

based on real-time push model see other ways in TACT

21

TACT: Combining and extending aspects of the two approaches

Optimistic and pessimistic combined Can choose which conflicts are delayed

Optimistic transaction length bounded Real-time used as in Rover and Coda Logical properties of operations also used

Three tunable parameters per replicating host Can simulate pessimistic, optimistic and points in between. Control staleness of data read and tentativeness of local

writes Model called continuous consistency

22

Time-based StalenessStaleness of data controlled based on real-time

Pull or push? Push requires a host’s synchronization times to be

known by all remote hosts How to guarantee staleness level given message

delay TACT local host pulls data from remote host

after it has not heard form the latter t seconds guaranteed to not see data more stale than t

seconds• blocks during pull, so message delays do not count• performance probably worse

Called staleness error “Physical Staleness”

23

Content-based StalenessStaleness of data controlled based on how much remote

values diverge from local value. Measuring value divergence?

Could be number of remote writes, but• some writes more major than others

– allocate five seats vs. 1 seat

• some writes cancel others

– allocate vs. deallocate seat Sum of (+/-) weights of remote writes

• alloc n seats given weight n

• decalloc n seats given weight -n

Local host pull vs. remote hosts push push because remote hosts knows what changes made

24

Content-based Staleness (contd) For each host i, one value, e i, defined for all other hosts

conceptually want to think of divergence of value v I at host i from final value with all distributed changes taken into account

Value divided evenly among all hosts each remote host checks abs(divergence) i < e i /n -1 if not sends its local changes to host i

thus each remote host processes independently decentralized process (local decision on each write) change of parameter requires broadcast and reallocation

e i is called numeric error since it is a number in case of numeric objects, represents error of local version but also applicable to non-numeric results

• bulletin board represents a numeric value for staleness depending on

content of object logical staleness

25

Tentativeness of Local Writes Local writes tentative until ordered wrt to remote writes Parameter determines how many local writes buffered

until resolution process occurs a subset of remote peers may be sent the writes

primary server quorum

Called order error assumes merge process simply orders the writes

other approaches possible• taking avg, min• discarding• order not important for commuting operations

In case list insertions, causes insertions to be out of order bulletin board example

26

TACT ExampleHost A

X = 3

Y = 6

X += 1

Y += 2

X += 2

Y += 4

Host B

X = 1

Y = 3

X += 1

Y += 3

Staleness = 3 Tentativeness = 3

Staleness = 8 Tentativeness = 1

27

Simulating Optimistic (Weak Consistency)

No special process used to reduce conflicts Content-based staleness = infinity Time-based staleness = infinity Write tentativeness = infinity A replica uses regular mechanisms to merge

export in Rover anti-entropy in Bayou strong connection in Coda’91

28

Simulating Pessimistic Each operation checked for global conflict

Content-based staleness = 0 or Time-based staleness = 0 or Tentativeness = 0 of all replicas

A replica syncs with all other replicas on each operation

29

Staleness vs. Tentativeness Not orthogonal

Reducing staleness reduces tentativeness implies more frequent synchronization

Not equivalent Pull vs push Reducing tentativeness may not reduce staleness

all hosts may not be contacted Tentativeness = 0 vs Staleness = 0

In both cases all local allowed changes will be committed in the correct order

But some remote changes may be reordered Chances of reordering smaller if staleness = 0 Serialiazability vs Linearizability (Strong consistency)

30

Possible Extensions Use type-based serializability

for numeric values such as int, real automatically determine weights

table automatically determine

• put(k, e) cancelled by remove(e)• put(k1, e1) commutes with put (k2, e2) and remove(k2)

– do not have to worry about merging them or order errors among them

sequence automatically determine

• insert (i, e) cancelled by del(e)

simulate Coda and Rover weak merging

31

Simulating Trickle Merging

Merge with some period, p Content-based staleness = infinity Time-based staleness = p Write tentativeness = infinity Not the same, because pull in TACT instead of

push in CODA’95