scalable multi-module switches with quality of service thesis defense santosh krishnan...

Scalable Multi-module Switches with Quality of Service

Thesis Defense

Santosh [email protected]

May 1, 2006

Advisor: Prof. Henning G. Schulzrinne

Co-advisor: Dr. Fabio M. Chiussi

mailto:[email protected]

May 1, 2006 2

Thesis: Santosh Krishnan

Outline

• Problem Definition– Motivations, list of contributions

• Switching Model: Components• Related work: Formal methods in switching

• Buffered Clos Switches– Concept of functional equivalence

• BCS: Throughput and Quality of Service– Single-path BCS: CIOQ, aggregation, pipelining– Multi-path BCS: Parallelization

• Conclusions

May 1, 2006 3


Problem Definition

• How to methodically construct a high-capacity switch?• How to design high-performance algorithms for such switches?

Goals:

• Physical layer improvements: 10-G Ethernet, OC-768• Converged network requiring QoS: IPTV, MPLS VPN• Case for modular design: component reuse

• Ad-hoc approach to switch design• No benchmarks, varying performance satisfaction

– Non-blocking, 100% throughput, nominal capacity

Importance:

What exists:

May 1, 2006 4


Contributions

• Taxonomy of multi-module switches: Buffered Clos Switches• Performance framework: Functional equivalence with ideal switch

• QoS: Online maximal matching

• Throughput: Critical matching

• Strict stability: Maximal matching, SOQF

• Switched Fair Airport matching

• Flow-based PPS: Clos fitting

• Cell-based PPS: Striping, Equal Dispatch

• Shadow CIOQ and Decompose

• Virtual Element Queueing

• Striping and Equal Dispatch

• Concurrent Dispatch: 3D matching

• Combination methods

• Recursive BCS

Applications

Combined I/O Queueing

Parallelization

Aggregation

Pipelining

Memory Space Memory

Mimics circuit-switching rigor

May 1, 2006 5


Switching Model

• Basic property: Contention• Flows: Guaranteed QoS, Best-effort• Ideal Switch: Provide bandwidth trunks, sustain link capacity

– Black box for network engineering purposes

PPU

PPU

PPU

SwitchFabric

CPU

PPU

PPU

PPU

Inpu

ts

Out

puts

Fast Path

Slow Path

May 1, 2006 6


Switching Model: Components

Architecture: Interconnect memory and space elements Algorithms: Meaningfully emulate the ideal switch for throughput and QoS

Mes

h

Link SchedulingBuffers Matching: 2D

Memory Element Space Element

OQ Switch: Ideal IQ Switch

Memory bandwidthFull-mesh circuitry

Conflict-free propertyMatching complexity

Monolithic

Constraints:

May 1, 2006 7


Background: Clos Networks

• Strictly non-blocking: K ≥ 2M – 1 (Clos theorem)• Re-arrangeable: K ≥ M (Slepian-Duguid)

Inpu

ts-

One

circ

uit

Out

puts

M

K

Fitting Algorithms

• Space-time duality• Fitting: matrix decomposition

Recognize:

Inspiration: Replace selected elements with memory

May 1, 2006 8


Background: CIOQ Switches

• Low memory bandwidth

Complexity of matching:• Switch size• Frequency• Reconfiguration rate

Pro:

Con:

5 3 0

0 1 7

0 5 0

1 0 0

0 0 1

0 1 0

• Offline: Templates• Maximum, Maximal, Critical• Heuristics

Queue State Configuration

What performance results when applied to a changing queue state?

May 1, 2006 9


Background: CIOQ Switch Results

Bandwidth Trunks • Birkhoff-Von Neumann decomposition (Chang ‘99)• Min/Double decomposition (Towles-Dally ‘02)• Low-jitter decomposition (Keslassy ‘03)

Exact 100% Throughput • Batch maximal matching• Batch critical-maximum matching

100% Throughput • iSLIP for Bernoulli uniform (McKeown ‘95) • Maximum weight matching (McKeown ‘97)• Maximal matching (Dai-Prabhakar ‘00)• Critical-maximum for Bernoulli (Iyer-McKeown ‘02)

OQ Emulation • LOOFA for work conservation (Krishna ‘98)• Exact emulation: stable marriages (Chuang ‘99)

(Weller-Hajek ‘97)

QoS

Thr

ough

put

Auxiliary Results: Envelope matching (Kar ‘00), Packet-mode matching (Marsan ‘02)

Based on combinatorics and stability theory

May 1, 2006 10


Framework: Buffered Clos Switches

Isomorphism: Non-blocking Clos network Properties: Multi-stage, fully connected, symmetric, uniform

• Switch size

• Type of elements

• Number in first stage

• Number in second

• Speedup

Definition:

Aggregate: Smaller elements

Pipeline: Lower speed, complexity

Parallelize: Pool memory resources

PPS

CIOQ-A, G-MSM

CIOQ-P, G-MSM

May 1, 2006 11


Framework: Functional Equivalence

Emulate an ideal switch: exact, asymptotic Bandwidth trunks, independent throughput optimization

Characterize relative performance: Functional equivalence

f1: Allocate known rates

f2: Relative stability for admissible traffic

f3: Per-output relative stability

f4: Strict relative stability: all pairs

f5: Exact emulation

Shape: Bandwidth trunks

Literature: 100% throughput

Work conserving

May 1, 2006 12


CIOQ: Bandwidth Trunks

Shaping plus online matching is sufficient for bandwidth guarantees

Arb

itra

ry A

rriv

als

Weight Scheduler

Rate Matrix BVN TemplatesOffline

Shape/Batch VOQ

Onl

ine

Online: Maximal (s=2)Online: Critical (s=1)

Cons:Template StorageCentralized rate processing

Split time into intervals: T = GCD (R) Batch traffic in each interval: Simple counters

Extension of Weller-Hajek maximal matching theorem Clos analogy: Maximal matching as a strategy for orderly assignments

May 1, 2006 13


CIOQ: Admissible Traffic

• No speedup: MWM (McKeown et al.), Speedup 2: Maximal (Dai-Prabhakar)• Can a simple maximum size matching suffice for admissible traffic?

Best Throughput Results:

Red Herring!

Critical matching suffices for asymptotic 100% throughput (f2)

6 3 0

0 1 7

0 5 0

Que

ue S

tate 6 3 0

1 1 7

2 5 2

Augment MSMCritical Matching

Intuition: 2x2 Line buckets

R1 R2 C1 C2 Max

May 1, 2006 14


CIOQ: Strict Relative Stability

Maximal matching: Keeps under-subscribed outputs stable (f3) (s=2)

Shortest Output-Queue First: (f4) (s=3) Output element scheduler: Identical to the one in emulated switch Intuition: Give preference to less congested pairs at the output Asymptotic emulation of an ideal switch: long-term fairness

May 1, 2006 15


Switched Fair Airport

Integrate two policies M1 and M2:

• M1: Provides bandwidth trunks given rate reservations

• M2: Optimize throughput independent of above rates

Multi-phase Combination Exclusive Combination

BVNS = 1

MaximalS = 2

CriticalS = 1

MWM/CriticalS = 1

2 3 2

MaximalS = 2

2 2 2

Maximal matching is additive to any other policy, hence needs the least speedup

Speedup Required:

M1M2

May 1, 2006 16


CIOQ-A: Aggregation

Shadow-Decompose: CIOQ emulation (f5)

VEQ Matching: Less complex, only for admissible traffic (f2)

Smaller space elementLower arbitration complexityHeterogeneous subports

Advantages:

May 1, 2006 17


CIOQ-P: Pipelining

Sequential Dispatch: CIOQ emulation (f5)

Concurrent Dispatch: Limited candidates: stale-state issues 3D Maximal Matching for relative stability

Striping: Shadow on envelope basis Equal Dispatch:

Explicitly equalize load Separate occupancy counters for each SE

Slower space elementLower arbitration complexity

Advantages:

Implement arbitrarily complex policies!

May 1, 2006 18


G-MSM: Combination

Combination methods: CIOQ-A/PNo need for independent analysisRecursion possible

May 1, 2006 19


PPS: Architecture

Pool the resources on several switching paths Dual of a CIOQ-P switch

Matching algorithm replaced by load balancing Sequence control might be necessary

Demux Mux

Core

Reuse low-capacity core switch

Implement arbitrarily slow memories!

Advantages:

Memoryless first and third stages

Performance: Emulates OQ switch

provided

May 1, 2006 20


PPS: Flow-based

Model for clustered routers: Per-flow path assignment: explicit or hashed No need for sequence control

Memory in first stage High speedup (Clos fitting)

Unbalanced load assignment

Requires knowledge of loads

Split flows

May 1, 2006 21


PPS: Cell-based

Uniformly distribute the load of each flow Premise: Each core element receives 1/K cells of each flow

Striping Equal Dispatch

First-stage buffers NK cells NK cells

Third-stage buffers 0 NK cells (w/ backpressure)

Issues Infinite latency Sequence control

Equal dispatch and striping suffice for asymptotic OQ emulation Bandwidth trunks: Large buffers required

May 1, 2006 22


Summary: A Recipe Book

• Taxonomy of multi-module switches: Buffered Clos Switches• Performance framework: Functional equivalence with ideal switch

• QoS: Online maximal matching

• Throughput: Critical matching

• Strict stability: Maximal matching, SOQF

• Switched Fair Airport matching

• Flow-based PPS: Clos fitting

• Cell-based PPS: Striping, Equal Dispatch

• Shadow and Decompose

• Virtual Element Queueing

• Striping and Equal Dispatch

• Concurrent Dispatch: 3D matching

• Combination methods

• Recursive BCS

Applications


Parallelization

Aggregation

Pipelining

Memory Space Memory

May 1, 2006 23


Avenues for Follow-on Research

• Efficient policies for multicast• Similar treatment on other interconnection networks• Theory of backpressure:

– Recent interest in buffered crossbars

• Quality of stability: Average delay analysis• Short-timescale equivalence• Emulation of a finite-memory ideal switch

– Interplay of buffer management with matching algorithms

Supporting Slides

May 1, 2006 25


Relevant Publications

• Dynamic Partitioning: Switch Memory Management, Infocom ’99

• Packet Switches with QoS Support, Hot Interconnects ’00

• Feedback Control for Distributed Scheduling, Globecomm ’00

• Buffered Clos Switches, Columbia TR ’02

• Inverse Multiplexing for Switches, Globecom ’98

• Switched Connections Inverse Multiplexing, Intl. Conf. ATM ’99

• Recognition of Parallel Packet Switches, GBN, Infocom ’01

• Stability Analysis of Parallel Packet Switches, ICC ’01

• Open-loop Schemes for Multi-path Switches, ICC ‘03

SwitchingAlgorithms

ParallelSwitches

May 1, 2006 26


Proposal Conjectures

• Maximal matching is sufficient to isolate oversubscribed outputs: DONE

• SOQF is sufficient for strict relative stability: DONE

• Equal dispatch for strict stability in CIOQ-P: DONE

• Equal dispatch plus decomposition for strict stability in G-MSM: DONE

• Rate shaping plus maximal matching suffices for QoS in CIOQ: DONE

• SOQF suffices for long-term fairness in CIOQ: DONE

Plus many more to round out the work

Proposal: six conjectures

May 1, 2006 27


Additional Contributions

• Maximal Matching: Delay analysis

• Perfect Sequences: Uniform Traffic

• Multicast support using Recycling

• SMM Switches: PPS without backpressure

• Fractional Dispatch for memoryless inputs

• Batch Decomposition (Optical)

• Support for Heterogeneous Subports

• Concurrent Dispatch: BVN and SPS


Parallelization

Aggregation

Pipelining

Background: Survey of formal methods in switching– a new perspective

Applications

May 1, 2006 28


Matching Flavors

• Maximal matching: Non-idling, greedy

• Maximum-size matching: Maximum flow in a bipartite graph– Ford-Fulkerson, Hopcroft-Karp

6 3 0

0 1 7

0 5 0

Que

ue S

tate

At least one connectionin the marked lines

Non-empty

Invariant:

May 1, 2006 29


Matching Flavors (continued)

• Critical Matching: Covers all critical rows and columns– Critical line: A line with the maximum sum

• Perfect Matching: Each configuration is a permutation• Maximum Weight Matching: Use queue length as weights

– Optimization problem: simplex method

• Template Matchings:– BVN: Decompose rate matrix as convex combination of permutations

– Double: Lower number of permutations, wasted slots

– Min: N permutations will cover all entries, large number of wasted slots

• Stable Matching: Gale-Shapely algorithm

May 1, 2006 30


• Lyapunov functions: Kumar-Meyn ‘95– Mechanism to extend Foster’s criterion to a system of queues– Weighted cartesian product of queue lengths– Symmetric and co-positive

• Fluid limits: Dai-Prabhakar ‘00– Function of discrete time: Interpolate– Limit: Scale time to infinity

– The scaling parameter may be drawn from an increasing sequence rn

Stability Theory

F(t) = lim 1/r f(rt)r ∞

May 1, 2006 31


CIOQ: Bandwidth Trunks

Arrivals into GQ:Bounded admissible

Bandwidth Trunk:Timescale = 1/GCD(R)

Covers all entries inGQ before next batch

Delay comparable to BVN rate decomposition

May 1, 2006 32


CIOQ: Perfect Sequences

• Sub-maximal Perfect Sequence:– A sequence of N permutations that covers the unit matrix– A repeating sequence guarantees 1/N to each pair– Suffices for 100% throughput to uniform traffic

• Simple implementation: Staggered round-robin– Not even maximal!

Concurrent SPS for CIOQ-P:K turns in KN slots

Basis for iSLIPBasis for Atlanta arbitration

May 1, 2006 33


Hierarchical Scheduling

May 1, 2006 34


CIOQ-P: Equal Dispatch

Explicitly equalize the load for each input-output pair

Implemented as countersNo mis-sequencing issues

May 1, 2006 35


CIOQ-P: 3D Maximal Matching

Concurrent traversal of queue state matrix

Pointers do not coincide with each other

May 1, 2006 36


Recursive G-MSM

Memory element of a G-MSM:Replace with a CIOQ switch

Virtual Element QueuesOrganized per space element

SPS Any matching SPS

May 1, 2006 37


PPS: Data Path

scalable multi-module switches with quality of service thesis defense santosh krishnan...

Documents