scalable rule management for data centers masoud moshref, minlan yu, abhishek sharma, ramesh...
TRANSCRIPT
Scalable Rule Management for Data Centers
Masoud Moshref, Minlan Yu, Abhishek Sharma, Ramesh Govindan
4/3/2013
NSDI’13
2Motivation Motivation Design EvaluationIntroduction
Introduction: Definitions
Datacenters use rules to implement management policiesDatacenters use rules to implement management policies
• Access control• Rate limiting• Traffic measurement • Traffic engineering
Datacenters use rules to implement management policies
An action on a set of ranges on flow fieldsAn action on a set of ranges on flow fields
Examples:• Deny• Accept• Enqueue
Flow fields examples:• Src IP / Dst IP• Protocol• Src Port / Dst Port
3Motivation Motivation Design EvaluationIntroduction
Introduction: Definitions
Datacenters use rules to implement management policies
R2
R1
Src IP R1: Accept SrcIP: 12.0.0.0/7 DstIP: 10.0.0.0/8
An action on a set of ranges on flow fields
Dst
IP
R2: Deny SrcIP: 12.0.0.0/8 DstIP: 8.0.0.0/6
4Motivation Motivation Design EvaluationIntroduction
Agg1 Agg2
ToR1 ToR2 ToR3
Agg1 Agg2
ToR1 ToR2 ToR3
Current practice
R0R4
R3
R0R4
R3
R0R4
R3
R0R4
R3
R0R4
R3
R0R4
R3
R0R4
R3
R0R4
R3
R0R4
R3
R0R4
R3
R0R4
R3
Rules are saved on predefined fixed machines
On hypervisors On switches
5Motivation Motivation Design EvaluationIntroduction
Machines have limited resources
TCAM
Top-of-Rack switch
Network Interface Card
Software switches on servers
6Motivation Motivation Design EvaluationIntroduction
Future datacenters will have many fine-grained rules
Regulating VM pair communication• Access control (CloudPolice)• Bandwidth allocation (Seawall)
Per flow decision• Flow measurement for traffic engineering (MicroTE, Hedera)
VLAN per server• Traffic management (NetLord, Spain)
1B – 20B rules
10M – 100M rules
1M rules
7Introduction ArchitectureMotivation Design Evaluation
Rule location trade-off (resource vs. bandwidth usage)
Storing rules at hypervisor incurs CPU overhead
TCAM
R0
8Introduction ArchitectureMotivation Design Evaluation
Rule location trade-off (resource vs. bandwidth usage)
Storing rules at hypervisor incurs CPU overhead
TCAM
Move the rule to ToR switch and forward traffic
R0
9Introduction ArchitectureMotivation Design Evaluation
Rule location trade-off: Offload to servers
TCAM R1
10Introduction ArchitectureMotivation Design Evaluation
Challenges: Concrete example
Src IP
Dst
IP
Agg1 Agg2
ToR1 ToR2
S1 S2 S3 S4 S5 S6
ToR3
R0
R1 R5
R6
R2
R3
R4
0 1 2 3 4 5 6 7
1234567
0
VM0
VM1
VM2
VM3
VM4
VM5
VM6
VM7
11Introduction ArchitectureMotivation Design Evaluation
Challenges: Overlapping rules
R0
R1 R5
R6
R2
R3
R4
0 1 2 3 4 5 6 7
1234567
0
Agg1 Agg2
ToR1 ToR2
S1 S2 S3 S4 S5 S6
ToR3
VM2
VM6
R0
R1R2
R3
R4
0 1 2 3 4 5 6 7
1234567
0
Source Placement: Saving rules on the source machine means minimum overhead
Src IP
Dst
IP
12Introduction ArchitectureMotivation Design Evaluation
Agg1 Agg2
ToR1 ToR2
S1 S2 S3 S4 S5 S6
ToR3
Challenges: Overlapping rules
R0
R1R2
R3
R4
0 1 2 3 4 5 6 7
1234567
0
VM2
VM6
R4
R0
R1R2
R3
0 1 2 3 4 5 6 7
1234567
0
If Source Placement is not feasible
Src IP
Dst
IP
13Introduction ArchitectureMotivation Design Evaluation
Heterogeneous devices
Challenges
Respect resource constraints
Minimize traffic overhead
Traffic changesRule changesVM Migration
Preserve the semantics of overlapping rules
Handle Dynamics
14Introduction DesignMotivation Evaluation
Contribution: vCRIB, a Virtual Cloud Rule Information Base
vCRIB
Network State
Proactive rule placement abstraction layer
Optimize traffic given resource constraints & changes
Agg1 Agg2
ToR1 ToR2 ToR3
Rules
R1 R2R3R4 R3
15Introduction DesignMotivation Evaluation
vCRIB design
Source Partitioning with
Replication
Topology &Routing
Rules
Partitions
Overlapping Rules
Minimum Traffic Feasible
Placement
16Introduction DesignMotivation Evaluation
R0
R 1
R5
R7R8
R2
R3 R6
R4
Partitioning with replication
P1 P3P2
P1 P3
0 1 2 3 4 5 6 7
1234567
0
R0
R 1
R8
R3
R4
Smaller partitions have more flexibility Cutting causes rule inflation
0 1 2 3 4 5 6 7
1234567
0
R0
R3
R1
R7 R6
0 1 2 3 4 5 6 7
1234567
0
R0
R1
R5R2
R3
0 1 2 3 4 5 6 7
1234567
0
P2
R0R 1
R8
R3
R4 R0
R5
R 1
R7
R2
R3 R6
17Introduction DesignMotivation Evaluation
Partitioning with replication
R0
R3
R1 A7
R6
R0
R 1
R8
R3
R4 R0
R1
R5
R2
R3R0
R1 R5R2
R3
R7 R6Introduce the concept of
similarity to mitigate inflation
P1 (5 rules)
P3 (5 rules)P2(5 rules)
0 1 2 3 4 5 6 7
1
2
3
4
5
6
7
00 1 2 3 4 5 6 7
1
2
3
4
5
6
7
0
0 1 2 3 4 5 6 7
1
2
3
4
5
6
7
0
P1 P2(7 rules)
18Introduction DesignMotivation Evaluation
Per-source partitions
R0
R1 R5
R6
R2
R3
R4
0 1 2 3 4 5 6 7
1234567
0
• Limited resource for forwarding• No need for replication to
approximate source-placement• Closer partitions are more similar
Src IP
Dst
IP
19Introduction DesignMotivation Evaluation
vCRIB design: Placement
Source Partitioning with
Replication
Topology &Routing
Rules
Partitions
PlacementT11
T21 T22
T23
T32
T33
ToR1
Traffic Overhead
Resource Constraints
Minimum Traffic Feasible
Placement
20Introduction DesignMotivation Evaluation
vCRIB design: Placement
Source Partitioning with
Replication
Topology &Routing
Rules
Partitions
Minimum Traffic Feasible
Placement
Feasible Placemen
tTraffic Overhead
Resource Constraints
Traffic Overhead
Resource-Aware Placement
Traffic-Aware Refinement
Traffic-Aware Refinement
21Introduction DesignMotivation Evaluation
FFDS (First Fit Decreasing Similarity)
1. Put a random partition on an empty device2. Add the most similar partitions to the initial partition
until the device is full
Find the lower bound for optimal solution for rulesProve the algorithm is a 2-approximation of the lower bound
22Introduction DesignMotivation Evaluation
vCRIB design: Heterogeneous resources
Source Partitioning with
Replication
Topology &Routing
Rules
Partitions
Minimum Traffic Feasible
Placement
Resource-Aware Placement
Traffic-Aware Refinement
Feasible Placemen
t
Resource Heterogeneity
Resource Usage
Function
23Introduction DesignMotivation Evaluation
vCRIB design: Traffic-Aware Refinement
Source Partitioning with
Replication
Resource-Aware Placement
Traffic-Aware Refinement
Partitions
Feasible Placemen
t
Minimum Traffic Feasible
Placement
Resource Usage
FunctionTraffic Overhead
Topology &Routing
Rules
24Introduction DesignMotivation Evaluation
Traffic-aware refinement Overhead greedy approach
1. Pick maximum overhead partition2. Put it where minimizes the overhead and maintains feasibility
Agg1 Agg2
ToR1 ToR2
S1 S2 S3 S4 S5 S6
ToR3
VM2 VM4
P2
P4
25Introduction DesignMotivation Evaluation
Traffic-aware refinement Overhead greedy approach
1. Pick maximum overhead partition2. Put it where minimizes the overhead and maintains feasibility
Problem: Local minima Our approach: Benefit greedy
Agg1 Agg2
ToR1 ToR2
S1 S2 S3 S4 S5 S6
ToR3
VM2 VM4
P2
P4
26Introduction DesignMotivation Evaluation
vCRIB design: Dynamics
Source Partitioning with
Replication
Resource-Aware Placement
Traffic-Aware Refinement
Rule/VM Dynamics
Partitions
Feasible Placemen
t
Minimum Traffic Feasible
Placement
Resource Usage
Function
Major Traffic
Changes
Dynamics
Topology &Routing
Rules
27Introduction DesignMotivation Evaluation
vCRIB design
Source Partitioning with
Replication
Resource-Aware Placement
Traffic-Aware Refinement
Rule/VM Dynamics
Partitions
Feasible Placemen
t
Minimum Traffic Feasible
Placement
Resource Usage
Function
Traffic OverheadMajor Traffic
ChangesDynamics
Topology &Routing
Rules
Resource Constraints
Resource Heterogeneity
Overlapping Rules
28EvaluationIntroduction Motivation Design
Evaluation Comparing vCRIB vs. Source-Placement Parameter sensitivity analysis
Rules in partitionsTraffic localityVMs per serverDifferent memory sizes
Where is the traffic overhead added? Traffic-aware refinement for online scenarios Heterogeneous resource constraints Switch-only scenarios
29EvaluationIntroduction Motivation Design
Simulation setup 1k servers with 20 VMs per server in a Fat-tree network 200k rules generated by ClassBench and random action IPs are assigned in two ways:
RandomRange
FlowsSize follows long-tail distributionLocal traffic matrix (0.5 same rack, 0.3 same pod, 0.2 interpod)
0 1 2 3 4 5 6 7
30EvaluationIntroduction Motivation Design
Comparing vCRIB vs. Source-Placement
4k_0 4k_4k 4k_6k0
0.05
0.1
0.15
0.2
0.25
0.3
Server memory_Switch memory
Tra
ffic
ove
rhe
ad
ra
tio
RangeRandom
Maximum Load is 5K Capacity is 4K
Range is better as similar partitions are from the same source
Random: Average load is 4.2K
Adding more resources helps vCRIB reduce traffic overhead
vCRIB finds low traffic feasible solution
31EvaluationIntroduction Motivation Design
0 1000 20000
1000
2000
Average Partition SizeA
vera
ge
Sim
ilarit
y
Parameter sensitivity analysis: Rules in partitions
Total space
vCRIB
Defined by maximum load on a server
Feas
ible
Source placeme
nt
32EvaluationIntroduction Motivation Design
0 1000 20000
1000
2000
Average Partition SizeA
vera
ge
Sim
ilarit
y
Total space
vCRIB
Parameter sensitivity analysis: Rules in partitions
Lower traffic overhead for smaller partitions andmore similar ones
<10% Tr
affic
vCRIB<10% Traffic
Source placeme
nt
A
A
33EvaluationIntroduction Motivation Design
Future work
Conclusion
Conclusion and future work
vCRIB allows operators and users to specify rules, and manages their placement in a way that respects
resource constraints and minimizes traffic overhead.
• Support reactive placement by adding the controller in the loop
• Break a partition for large number of rules per VM• Test for other rulesets
Scalable Rule Management for Data Centers
Masoud Moshref, Minlan Yu, Abhishek Sharma, Ramesh Govindan
4/3/2013
NSDI’13