programmable packet scheduling at line rate - sigcomm · 2016. 8. 26. · the pifo abstraction in...
TRANSCRIPT
![Page 1: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/1.jpg)
Programmable Packet Scheduling at Line Rate
Anirudh Sivaraman, Suvinay Subramanian, Mohammad Alizadeh, Sharad Chole, Shang-Tse Chuang, Anurag Agrawal, Hari Balakrishnan, Tom Edsall, Sachin Katti, Nick McKeown
![Page 2: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/2.jpg)
Programmable scheduling at line rate• Motivation: Can’t deploy new schedulers in production networks• The status quo in line-rate switches
InOut
Parser DeparserIngress pipeline Egress pipeline
2
Queues/Scheduler
RMT RMT, Domino RMTRMT, Domino???
The scheduler is still fixed
![Page 3: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/3.jpg)
Why is programmable scheduling hard?
• Many algorithms, yet no consensus on abstractions, cf.• Parse graphs for parsing• Match-action tables for forwarding• Packet transactions for data-plane algorithms
• Scheduler has tight timing requirements• Can’t simply use an FPGA/CPU
Need expressive abstraction that can run at line rate
![Page 4: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/4.jpg)
What does the scheduler do?
It decides• In what order are packets sent• e.g., FCFS, priorities, weighted fair queueing
• At what time are packets sent• e.g., Token bucket shaping
![Page 5: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/5.jpg)
A strawman programmable scheduler
• Very little time on the dequeue side => limited programmability• Can we move programmability to the enqueue side instead?
Classification Programmable logic to decide order or time
Packets
![Page 6: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/6.jpg)
The Push-In First-Out QueueKey observation• In many cases, relative order of buffered packets does not change• i.e., a packet’s place in the scheduling order is known at enqueue
The Push-In First-Out Queue (PIFO): Packets are pushed into an arbitrary location based on a rank, and dequeued from the head
259 791013
8
![Page 7: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/7.jpg)
A programmable scheduler
To program the scheduler, program the rank computation
Rank Computation
(programmable) (fixed logic)
29 8 5
PIFO Scheduler
f = flow(pkt) …...p.rank= T[f] + p.len
![Page 8: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/8.jpg)
In Out
Parser DeparserIngress pipeline Egress pipeline
Rank Computation
Queues/Scheduler
PIFO Scheduler
A programmable scheduler
Rank computation is a packet transaction (Domino, SIGCOMM’ 16)
![Page 9: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/9.jpg)
In Out
Parser DeparserIngress pipeline Egress pipelineQueues/
SchedulerPIFO
Scheduler
Rank Computation 1. f = flow(p)2. p.start = max(T[f].finish,
virtual_time)3. T[f].finish = p.start + p.len4. p.rank = p.start
Fair queuing
![Page 10: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/10.jpg)
In Out
Parser DeparserIngress pipeline Egress pipelineQueues/
SchedulerPIFO
Scheduler
1. tokens = min(tokens + rate * (now – last), burst)
2. p.send = now + max( (p.len – tokens) / rate, 0)
3. tokens = tokens - p.len4. last = now5. p.rank = p.send
Rank Computation
Token bucket shaping
![Page 11: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/11.jpg)
29 8 5
PIFO Scheduler
Shortest remaining flow size
11
In Out
Parser DeparserIngress pipeline Egress pipelineQueues/
SchedulerPIFO
Scheduler
![Page 12: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/12.jpg)
Shortest remaining flow size
Rank Computation 1. f = flow(p)2. p.rank = f.rem_size
29 8 5
PIFO Scheduler
12
![Page 13: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/13.jpg)
Beyond a single PIFO
x1
y1
x2
b1b2b3y2
a1
Hierarchical scheduling algorithms need hierarchy of PIFOs
Red (0.5) Blue (0.5)
a(0.99)
b(0.01)
x(0.5)
y(0.5)
root
HierarchicalPacket Fair Queuing
![Page 14: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/14.jpg)
b1
b3
b2
a1
Tree of PIFOs
Red (0.5) Blue (0.5)
a(0.99)
b(0.01)
x(0.5)
y(0.5)
root
HierarchicalPacket Fair Queuing
PIFO-Red(WFQ on a & b)
PIFO-root (WFQ on Red & Blue)
x1
x2
y1
y2
PIFO-Blue(WFQ on x & y)
a1
a1
BRBB RRBR
![Page 15: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/15.jpg)
Expressiveness of PIFOs
• Fine-grained priorities: shortest-flow first, earliest deadline first, service-curve EDF• Hierarchical scheduling: HPFQ, Class-Based Queuing• Non-work-conserving algorithms: Token buckets, Stop-And-Go, Rate
Controlled Service Disciplines• Least Slack Time First• Service Curve Earliest Deadline First• Minimum and maximum rate limits on a flow• Cannot express some scheduling algorithms, e.g., output shaping.
15
![Page 16: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/16.jpg)
PIFO in hardware
• Performance targets for a shared-memory switch• 1 GHz pipeline (64 ports * 10 Gbit/s)• 1K flows/physical queues• 60K packets (12 MB packet buffer, 200 byte cell)• Scheduler is shared across ports
• Naive solution: flat, sorted array is infeasible
• Exploit observation that ranks increase within a flow
16
![Page 17: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/17.jpg)
A single PIFO block
2
Rank Store(SRAM)
Flow Scheduler(flip-flops)
AB
Dequeue Enqueue
A 0 B 1 C 3 C24
345 C 6D 4
A 2
D
• 1 enqueue + 1 dequeue per clock cycle• Can be shared among multiple logical PIFOs
17
![Page 18: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/18.jpg)
Hardware feasibility
• The rank store is just a bank of FIFOs (well-understood design)
• Flow scheduler for 1K flows meets timing at 1GHz on 16-nm transistor library• Continues to meet timing until 2048 flows, fails timing at 4096
• 7 mm2 area for 5-level programmable hierarchical scheduler• < 4% for a typical chip.
18
![Page 19: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/19.jpg)
Related work
• PIFO: Used in theoretical work by Chuang et. al. in the 90s
• Universal Packet Scheduling (UPS): Uses LSTF to replay all schedules, end point sets slack• Assumes fixed switches => cannot express fair queueing, shaping• Assumes single priority queue => cannot express hierarchies
![Page 20: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/20.jpg)
Conclusion
• Programmable scheduling at line rate is within reach
• Two benefits:• Express new schedulers for different performance objectives• Express existing schedulers as software, not hardware
• Code: http://web.mit.edu/pifo
![Page 21: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/21.jpg)
Backup slides
![Page 22: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/22.jpg)
Limitations of PIFOs
• Output shaping: PIFOs rate limit input to a queue, not output
• Shaping and scheduling are coupled.
![Page 23: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/23.jpg)
PIFO mesh
![Page 24: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/24.jpg)
Proposal: scheduling in P4
• Currently not modeled at all, blackbox left to vendor
• Only part of the switch that isn’t programmable
• PIFOs present a candidate
• Concurrent work on Universal Packet Scheduling also requires a priority queue that is identical to a PIFO
![Page 25: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/25.jpg)
Hardware implementation
25
Rank
== comparators> comparators
Priority encoderPriority encoder
Shift elements based on push, pop indices
Pop(DEQ)
Push 1(ENQ)
Push 2(reinsert)
LogicalPIFO ID
Rank
Rank
LogicalPIFO ID Rank Logical
PIFO IDRank Logical
PIFO ID
§ Meets timing (1 GHz) for up to 2048 flows at 16 nm§ Less than 4% area overhead (~7 mm2) for 5-level scheduler
![Page 26: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/26.jpg)
A PIFO block
Enqueue:(logical PIFO,
rank, flow)
Dequeue:(logical PIFO)
ALU
26
![Page 27: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/27.jpg)
A PIFO mesh
Enq Deq
Enq
ALU
DeqEnq
ALU
Deq
Next-hop lookup
ALU
Next-hop lookup
Next-hop lookup
27
![Page 28: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/28.jpg)
Proposal: scheduling in P4
• Need to model a PIFO (or priority queue) in P4
• Requires an extern instance to model a PIFO• Can start by including it in a target-specific library• Later migrate to standard library if there’s sufficient interest• Section 16 of P4v1.1
• Transactions themselves can be compiled down to P4 code using the Domino DSL for stateful algorithms.
![Page 29: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/29.jpg)
Hardware feasibility of PIFOs
• Number of flows handled by a PIFO affects timing.
• Number of logical PIFOs within a PIFO, priority and metadata width, and number of PIFO blocks only increases area.
![Page 30: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/30.jpg)
Composing PIFOs: min. rate guarantees
Minimum rate guarantees:
Provide each flow a guaranteedrate provided the sum of theseguarantees is below capacity.
PIFO-RootPrioritize flows undermin. rate
PIFO-A(FIFO for flow A)
PIFO-B(FIFO for flow B)
1 32 42
ABABA
Composing PIFOs
![Page 31: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/31.jpg)
Traffic Shaping
1. update tokens2. p.send = now +
(p.len - tokens) / rate;3. p.prio =p.send Push-In-First-Out
(PIFO) Queue
SchedulerIngress Pipeline
![Page 32: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/32.jpg)
LSTF
Add transmission delay to slack Push-In-First-Out
(PIFO) Queue
SchedulerIngress Pipeline
Decrement wait time in queue
from slack
Initialize slackvalues
![Page 33: Programmable Packet Scheduling at Line Rate - SIGCOMM · 2016. 8. 26. · The PIFO abstraction in one slide • PIFO: A sorted array that let us insert an entry (packet or PIFO pointer)](https://reader036.vdocuments.site/reader036/viewer/2022071419/611886ed88a01943f10303fb/html5/thumbnails/33.jpg)
The PIFO abstraction in one slide
• PIFO: A sorted array that let us insert an entry (packet or PIFO pointer) into a PIFO based on a programmable priority• Entries are always dequeued from the head• If an entry is a packet, dequeue and transmit it• If an entry is a PIFO, dequeue it, and continue recursively