Flexible Network Striping

Asfandyar Qureshi

RQE: 21st August 2006

Medical Emergencies

Due to the EMTs’ lack of medical training, doctors do not believe they can depend on EMTs for much beyond speedy transport.

Emergency» 911 call» Ambulance

arrives» EMTs’ initial

examination» Transport to

hospital» Doctors begin

treatment

› Remote diagnosis and treatment» Provide advanced remote diagnostics capabilities» Allow doctors to examine patients in-transit

› What we want to send» Unidirectional VIDEO (~600 kbit/sec)» Bidirectional AUDIO (~30 kbit/sec)» Low-rate Physiological data (EKG, Heart-rate, etc)

Mobile Telemedicine

System Deployment

› Plan to deploy the system in order to conduct multiple medical studies» Initial deployment: Spring ’07

› Project partners» Boston: Massachusetts General Hospital» Orlando: Orange County Florida’s Emergency

Medical Services

› Economic constraints» Limited/progressive deployment must be easy» Initial medical study: two ambulances» Cannot amortize the cost over many

ambulances» Use existing wireless infrastructure

Talk Structure› Motivation

› Wireless Wide Area Networks (WWANs)» Performance overview

› Horde: network striping middleware» Network striping» Horde API» Horde internals

Channel managers and transmission slots

Packet scheduling

WWAN Performance Overview› CDMA EV-DO interfaces

» US providers: Verizon and Sprint › Low throughput

» Download < 400 kbits/sec» Upload < 150 kbits/sec

› High and variable packet RTTs» 500 ms ± 200 ms

› Realistic WWAN performance data is hard to come by» Lots of hype/disinformation

WWAN Experiments› We spent some time running WWAN

experiments in Orlando and Boston» Drove around in a car» Simultaneously used multiple interfaces» Measured UDP throughput and RTTs» Used GPS to derive coverage maps» Also logged 802.11 APs

› Not a rigorous analysis» Nonetheless, provided a great deal of

insight into WWAN behaviour

Throughput

VERIZON-1 mean = 109 kbps, stdev = 21 kbps

VERIZON-2 mean = 91 kbps, stdev = 32 kbps

SPRINT mean = 111 kbps, stdev = 36 kbps

AGGREGATE mean = 306 kbps, stdev = 64 kbps

High and variable RTTsS

tatio

nary

pin

gs (B

osto

n)

› ‘Stripe’ Application Data across Multiple Network Channels» Simultaneously use many networks» Wireless networks can be dissimilar and

unstable

Network Striping

A

N network channels

M data streams

M data streams

B

Striping: Related Work› Wired networks

» e.g., n-ISDN lines as a virtual T1 line» SIGCOMM ‘96: “A Reliable and Scalable Striping

Protocol”, ADISESHU, PARULKAR, AND VARGHESE.» Mostly concerned with getting TCP to run well.

› Wireless networks» Globecom ’99: “Adaptive Inverse Multiplexing for Wide-

Area Wireless Networks”, SNOEREN.» Mobisys 2004: “MAR: A Commuter Router Infrastructure

for the Mobile Internet”, RODRIGUEZ, CHAKRAVORTY, CHESTERFIELD, PRATT, AND BANERJEE.

› Multi-path video streaming» 2001: “Unbalanced Multiple Description Video

Communication using Path Diversity”, APOSTOLOPOULOS AND WEE.

Challenges› Network

» Network channels can be quite dissimilar» Channel QoS varies significantly in time» Number of channels varies

› Application» Bandwidth limited» Different types of streams with dissimilar

network QoS sensitivities» Want applications to be independent of

the number and types of channels

Our approach: Horde› Network striping middleware

» Exposes striping operation to applications» Apps can abstractly modulate striping policy» Flexible striping mechanism» Per-channel congestion control

› Does not support unmodified legacy applications» Horde is targeted at developers of new mobile

applications» Legacy support is relatively easy to provide…

Virtual TUN interfaces TESLA (Salz et al, Usenix ‘03)

Forward Compatibility› Applications do not have a short shelf-life

» Depend on Horde’s abstract API

› The modular design of the middleware, allows our applications to be forward compatible» Networks are likely to evolve.

Sprint moving to WiMax Verizon moving to EV-DO rev A

» Our middleware is designed to simultaneously handle many different types of networks and its functionality can be extended easily.

› Each host runs a local Horde daemon» Exposes a RPC-like interface to applications

› Before any data is sent, peering applications must negotiate a named session.» Multiple sessions (unique names) are allowed

Horde API: Sessions

Host A

HordeBHordeA

ApplicationA ApplicationB

“Tavarua” “Tavarua”

Host B

N network channels

› Within a session, one or more streams can be negotiated.» Streams are (relaxed) bi-directional FIFOs of

Application Data Units (ADUs). Maximum reordering (maximum number of channels)

» ADUs can be fragmented, partially lost, etc.

Horde API: Streams & ADUs

Host A

HordeBHordeA

ApplicationA ApplicationB

“Tavarua” “Tavarua”“video”

“audio”

“data”

Host B

API: Data Send Timeline› Host A: send data

» App requests that Horde send an ADU» Data is scheduled (ADU fragments → packets)

One or more ADU fragments per packet

› Host B: receive data» Data is unpacked (packets → ADU fragments)» App notified about received fragments» ACKs sent for received packets

› Host A: receive ACKs» ACKs mapped to ADU fragment info» Losses detected from ACK stream» App notified about ACK’d or Lost ADU fragments

Horde API: QoS Objectives› ADUs can be ‘tagged’ with Quality-of-service

objectives» Tags are hints to packet scheduler» An abstract way to modulate the packet

scheduling policy

› Example tags:» ADU priority» Correlation group

Minimize the joint loss probability P(X lost ∧ Y lost), if the two ADUs X and Y are in the same correlation group.

Elements in the same correlation group are less likely to experience correlated failures.

E.g., President and Vice president would have the same group.

Horde Internals

APPLICATION

API

Network Channel Management

Packet Scheduler (iMux)

IPC

Applications

Horde Daemon

ADUs

Packets

O/S NETWORK SERVICES

Session

MUXctrl

Scheduler and network layer can have per-session configurations. Multiple implementations of these.

Channel Managers

O/S NETWORK SERVICES

Packet Scheduler

› A single channel manager instance for each active network interface» Pool manager creates and destroys the Mi

» The Mi implement congestion control» Multiple implementations of the channel manager

interface Pool manager chooses one based on per-interface settings

M1 M2 MN

Network Management

Layer

Channel Managers› Primary services provided by MX

» Throttle packet sends on interface X.» Generate ACK and LOSS notifications for

packets sent on X.

› Each MX runs a congestion control session» Congestion control logic belongs below the

striping layer Multiple independent congestion domains, need

multiple independent sessions.

» Channel-specific optimizations possible Pick congestion control algorithm based on the

channel type (e.g., 802.11, EV-DO, WiMax) Implementations: AIMD, CBR, AIMD_EVDO, DCCP*, …

Transmission Slots› TxSlot objects are the interface between

the scheduler and the network layer» For the scheduler, each channel manager is a

source of TxSlots.» Each TxSlot provides a packet TX capability.» When a slot becomes available, the scheduler

maps data to that slot.

› A TxSlot provides expected QoS for that TX» E.g., expected RTT and expected loss probability.» The scheduler can use the expected QoS fields to

determine the best data for that slot.

TxSlot Life Cycle

Managerk

Packet Scheduler

Ti

DiTi

generate_tx_slot(…)TxSlot allocation

consume_tx_slot(…)TxSlot deallocation

schedule_tx_slot(…)TxSlot→Data mapping

Packet

ADUADU

ADU

One or more ADU fragments packed together

Hoard Scheduler

» schedule_tx_slot(slot): streams = set of streams with unsent data while (slot not full) and (streams not empty): stream = highest priority from streams adu = ADU at the head of stream f = largest fragment that will fit in slot if test_constraints(slot, f) is okay:

read f from adu into slot if (no more unsent data in adu): pop adu from stream if (stream empty): remove from streams if (slot has data): consume_tx_slot(slot)

Currently Supported Objectives› Optimization

» Static ADU priority› Constraints

» Stream FIFO ordering» ADU loss probability threshold» ADU latency threshold» Stream latency variance threshold» ADU correlation groups

› Other» Resilient ADU sends

Conclusion› Mobile telemedicine deployment

» Horde is a big part of a system that is being deployed soon.

» Will allow doctors to try things they wouldn’t have been able to try for another 5-10 years.

› Support the development of demanding mobile applications» Transparently exploit all available wireless

resources.› Powerful abstractions (internal and API)

» Objectives, transmission slots, etc.» Allowed us to modularly experiment with

different congestion control schemes and scheduling strategies.