2DFQ: Two-Dimensional Fair Queuing for Multi-Tenant Cloud Services

Jonathan MaceBrown University

Peter BodikMicrosoft

Madanlal MusuvathiMicrosoft

Rodrigo FonsecaBrown University

Krishnan VaradarajanMicrosoft

2

Containers / VMs

2

Containers / VMs

Shared Systems:Storage, Database,

Queueing, etc.

2

Containers / VMs

Shared Systems:Storage, Database,

Queueing, etc.

Shared Process

3

Shared ProcessTenants

Request Queues

…

3

Shared ProcessTenants

Request Queues

time

Threads

…

3

Shared ProcessTenants

Request Queues

time

Threads

…

3

Shared ProcessTenants

Request Queues

time

Threads

…

FIFO or ad-hoc

3

Shared ProcessTenants

Request Queues

time

Threads

…

4

Shared ProcessTenants

Request Queues

time

Threads

…

Flows Packet

4

Shared ProcessTenants

Request Queues

time

Threads

…

Flows Packet Switch

Link

4

Shared ProcessTenants

Request Queues

time

Threads

…

Flows Packet Switch

Link

Weighted Fair Queueing (WFQ)

4

Shared ProcessTenants

Request Queues

time

Threads

…

Weighted Fair Queueing (WFQ)

Multiple threadsMultiple concurrent requests

4

…

5

…

5

…

Two-Dimensional Fair Queueing (2DFQ)

5

6

Asize =1

(fast database read)

6

Dsize =10

Asize =1

(fast database read) (slow database scan)

6

Dsize =10

Asize =1

One thread:

time

(fast database read) (slow database scan)

6

Dsize =10

Asize =1

One thread:

time

(fast database read) (slow database scan)

6

Dsize =10

Asize =1

One thread:

time

(fast database read) (slow database scan)

6

Dsize =10

Asize =1

…

One thread:

time

(fast database read) (slow database scan)

6

Dsize =10

Asize =1

…

One thread:

time

high-throughput burst

zero-throughput lull

thro

ugh

pu

t

(fast database read) (slow database scan)

6

Dsize =10

Asize =1

…

One thread:

time

high-throughput burst

zero-throughput lull

Ideal:

thro

ugh

pu

t

(fast database read) (slow database scan)

6

Dsize =10

Asize =1

…

One thread:

time

high-throughput burst

zero-throughput lull

Ideal:

thro

ugh

pu

t

(fast database read) (slow database scan)

6

Dsize =10

Asize =1

…

One thread:

time

high-throughput burst

zero-throughput lull

Ideal:

thro

ugh

pu

t

(fast database read) (slow database scan)

6

Dsize =10

Asize =1

7

Dsize =10

Asize =1

CB

7

Dsize =10

Asize =1

CB

time

7

Dsize =10

Asize =1

CB

time

7

Dsize =10

Asize =1

CB

time

7

Dsize =10

Asize =1

CB

time

time

7

Dsize =10

Asize =1

CB

time

time

7

Dsize =10

Asize =1

CB

time

time

7

Dsize =10

Asize =1

CB

time

time

7

Dsize =10

Asize =1

CB

time

time

thro

ugh

pu

tth

rou

ghp

ut

7

Dsize =10

Asize =1

CB

time

time

thro

ugh

pu

tth

rou

ghp

ut

WFQ, SFQ, …Weighted Fair Queuing

Start-time Fair Queueing

7

Dsize =10

Asize =1

CB

time

time

thro

ugh

pu

tth

rou

ghp

ut

WF²QWorst-Case Fair

Weighted Fair Queuing

WFQ, SFQ, …Weighted Fair Queuing

Start-time Fair Queueing

7

Dsize =10

Asize =1

CB

8

Dsize =10

Asize =1

CB

time

8

Dsize =10

Asize =1

CB

time

8

Dsize =10

Asize =1

CB

time

8

Dsize =10

Asize =1

CB

time

8

Dsize =10

Asize =1

CB

time

time

8

Dsize =10

Asize =1

CB

time

time

8

Dsize =10

Asize =1

CB

time

time

8

Dsize =10

Asize =1

CB

time

time

8

Dsize =10

Asize =1

CB

time

time

Ideal:

8

Dsize =10

Asize =1

CB

time

time

More threads → Opportunity to reduce burstiness

8

Dsize =10

Asize =1

CB

time

time

More threads → Opportunity to reduce burstiness

WFQ, SFQ, WF²Q, …Weighted Fair QueueingStart-time Fair Queueing

Worst-case Fair Weighted Fair Queueing

8

Dsize =10

Asize =1

CB

time

time

More threads → Opportunity to reduce burstiness

WFQ, SFQ, WF²Q, …Weighted Fair QueueingStart-time Fair Queueing

Worst-case Fair Weighted Fair Queueing

2DFQTwo-dimensional Fair

Queueing

8

Challenges

9

A size =1

Tenants with small requests are affected

10

C size =10A size =1

Tenants with small requests are affected

Burstiness is proportional to size of large requests

10

C size =10A size =1

Cloud services:

4+ orders of magnitude variation in cost

Tenants with small requests are affected

Burstiness is proportional to size of large requests

10

C size =10A size =1

Cloud services:

4+ orders of magnitude variation in cost

Tenants with small requests are affected

Burstiness is proportional to size of large requests

size =10A size =1

Size is used by scheduler to make scheduling decisions

C

10

C size =10A size =1

Cloud services:

4+ orders of magnitude variation in cost

Tenants with small requests are affected

Burstiness is proportional to size of large requests

A size =?size =?

Size is used by scheduler to make scheduling decisions

C

10

C size =10A size =1

Cloud services:

4+ orders of magnitude variation in cost

Tenants with small requests are affected

Burstiness is proportional to size of large requests

A

Cloud services:

Estimation using model or moving averages

size =?size =?

Size is used by scheduler to make scheduling decisions

C

10

Key

11

1st quartile 3rd quartile

distribution shape

1st percentilemedian 99th percentile

outliers

Key

11

1st quartile 3rd quartile

distribution shape

1st percentilemedian 99th percentile

outliers

12

UnpredictabilityCost Variation

Quality OfService

12

UnpredictabilityCost Variation

Quality OfService

12

UnpredictabilityCost Variation

Quality OfService

12

Two-DimensionalFair Queueing

13

Dsize =10

Asize =1

CB

time

WFQ

14

Dsize =10

Asize =1

CB

time

WFQ

14

Dsize =10

Asize =1

CB

0 0 0 0

time

WFQ

14

Dsize =10

Asize =1

CB

0 0 0 0

time

WFQ

15

Dsize =10

Asize =1

CB

0 0 0 0

time

WFQ

15

Dsize =10

Asize =1

CB

0 0 0 0

time

WFQ

15

Dsize =10

Asize =1

CB

0 0 0 0

time

WFQ

15

Dsize =10

Asize =1

CB

1 0 0 0

time

WFQ

16

Dsize =10

Asize =1

CB

1 1 0 0

time

WFQ

17

Dsize =10

Asize =1

CB

1 1 0 0

time

WFQ

18

Dsize =10

Asize =1

CB

1 1 0 0

time

v(t)=½

WFQ

18

Dsize =10

Asize =1

CB

2 2 0 0

time

v(t)=1

WFQ

19

Dsize =10

Asize =1

CB

3 3 0 0

time

v(t)=1½

WFQ

20

Dsize =10

Asize =1

CB

4 4 0 0

time

v(t)=2

WFQ

21

Dsize =10

Asize =1

CB

5 5 0 0

time

v(t)=2½

WFQ

22

Dsize =10

Asize =1

CB

6 6 0 0

time

v(t)=3

WFQ

23

Dsize =10

Asize =1

CB

7 7 0 0

time

v(t)=3½

WFQ

24

Dsize =10

Asize =1

CB

8 8 0 0

time

v(t)=4

WFQ

25

Dsize =10

Asize =1

CB

9 9 0 0

time

v(t)=4½

WFQ

26

Dsize =10

Asize =1

CB

10 10 0 0

time

v(t)=5

WFQ

27

Dsize =10

Asize =1

CB

10 10 0 0

time

v(t)=5

WFQ

27

Dsize =10

Asize =1

CB

10 10 10 10

time

v(t)=10

WFQ

28

Dsize =10

Asize =1

CB

0 0 0 0

time

v(t)=0

WF²Q

29

Dsize =10

Asize =1

CB

0 0 0 0

time

v(t)=0

WF²Q

29

Dsize =10

Asize =1

CB

1 1 0 0

time

v(t)=½

WF²Q

30

Dsize =10

Asize =1

CB

1 1 0 0

time

v(t)=½

WF²Q

30

Dsize =10

Asize =1

CB

1 1 0 0

time

v(t)=½

WF²Q

30

Dsize =10

Asize =1

CB

1 1 10 10

time

v(t)=5½

WF²Q

31

Dsize =10

Asize =1

CB

2 2 10 10

time

v(t)=6

WF²Q

32

Dsize =10

Asize =1

CB

3 3 10 10

time

v(t)=6 ½

WF²Q

33

Dsize =10

Asize =1

CB

4 4 10 10

time

v(t)=7

WF²Q

34

Dsize =10

Asize =1

CB

5 5 10 10

time

v(t)=7 ½

WF²Q

35

Dsize =10

Asize =1

CB

6 6 10 10

time

v(t)=8

WF²Q

36

Dsize =10

Asize =1

CB

7 7 10 10

time

v(t)=8½

WF²Q

37

Dsize =10

Asize =1

CB

8 8 10 10

time

v(t)=9

WF²Q

38

Dsize =10

Asize =1

CB

9 9 10 10

time

v(t)=9½

WF²Q

39

Dsize =10

Asize =1

CB

10 10 10 10

time

v(t)=10

WF²Q

40

Dsize =10

Asize =1

CB

11 11 10 10

time

v(t)=10½

WF²Q

41

Dsize =10

Asize =1

CB

11 11 10 10

time

v(t)=10½

WF²Q

41

Dsize =10

Asize =1

CB

11 11 10 10

time

v(t)=10½

WF²Q

41

Dsize =10

Asize =1

CB

0 0 0 0

time

v(t)=0

2DFQ

42

Dsize =10

Asize =1

CB

0 0 0 0

time

v(t)=0

2DFQ

42

Dsize =10

Asize =1

CB

1 1 0 0

time

v(t)=½

2DFQ

43

Dsize =10

Asize =1

CB

1 1 0 0

time

v(t)=½

2DFQ

43

Dsize =10

Asize =1

CB

1 1 10 0

time

v(t)=½

2DFQ

44

Dsize =10

Asize =1

CB

1 1 10 0

time

v(t)=½

2DFQ

44

Dsize =10

Asize =1

CB

1 1 10 0

time

v(t)=½ 1 1

2DFQ

44

Dsize =10

Asize =1

CB

2 1 10 0

time

v(t)=1

2DFQ

45

Dsize =10

Asize =1

CB

2 1 10 0

time

v(t)=1

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

45

Dsize =10

Asize =1

CB

2 2 10 0

time

v(t)=1½

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

46

Dsize =10

Asize =1

CB

3 2 10 0

time

v(t)=2

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

47

Dsize =10

Asize =1

CB

3 3 10 0

time

v(t)=2½

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

48

Dsize =10

Asize =1

CB

4 3 10 0

time

v(t)=3

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

49

Dsize =10

Asize =1

CB

4 4 10 0

time

v(t)=3½

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

50

Dsize =10

Asize =1

CB

5 4 10 0

time

v(t)=4

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

51

Dsize =10

Asize =1

CB

5 5 10 0

time

v(t)=4½

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

52

Dsize =10

Asize =1

CB

6 5 10 0

time

v(t)=5

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

53

Dsize =10

Asize =1

CB

6 6 10 0

time

v(t)=5½

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

54

Dsize =10

Asize =1

CB

6 6 10 0

time

v(t)=5½

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

54

Dsize =10

Asize =1

CB

6 6 10 0

time

v(t)=5½

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

54

Dsize =10

Asize =1

CB

6 6 10 0

time

v(t)=5½

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

6 654

Dsize =10

Asize =1

CB

7 6 10 10

time

v(t)=6

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

55

Dsize =10

Asize =1

CB

7 7 10 10

time

v(t)=6½

Eligible 𝑟𝑗 , 𝑖 = Start 𝑟𝑗 − 𝑖𝑛× size 𝑟𝑗

2DFQ

56

Unknown Costs

57

Unknown Costs

57

Unknown Costs

57

Unknown Costs

57

Unknown Costs

57

Unknown Costs

58

Unknown Costs

58

Unknown Costs

Pessimistic cost estimation

58

Evaluation

Compare 2DFQ to WFQ and WF²Q

Discrete event simulator with Azure Storage workloads

More experiment results in the paper, evaluating:• Burstiness

• Fairness

• Tail latency

59

60

A

50 tenants with size ≈ 1

60

A

50 tenants with size ≈ 1

C

50 tenants with size ≈ 1000

60

A

50 tenants with size ≈ 1

C

50 tenants with size ≈ 1000

16 threads1000 units/second

Costs known by scheduler

60

CA

50 tenants with size ≈ 1 50 tenants with size ≈ 1000

16 threads1000 units/second

Costs known by scheduler

60

A’s

Ser

vice

Rat

e

CA

50 tenants with size ≈ 1 50 tenants with size ≈ 1000

16 threads1000 units/second

Costs known by scheduler

60

A’s

Ser

vice

Rat

e

CA

50 tenants with size ≈ 1 50 tenants with size ≈ 1000

16 threads1000 units/second

Costs known by scheduler

60

A’s

Ser

vice

Rat

e

CA

50 tenants with size ≈ 1 50 tenants with size ≈ 1000

16 threads1000 units/second

Costs known by scheduler

60

A’s

Ser

vice

Rat

e

CA

50 tenants with size ≈ 1 50 tenants with size ≈ 1000

16 threads1000 units/second

Costs known by scheduler

60

A’s

Ser

vice

Rat

e

CA

50 tenants with size ≈ 1 50 tenants with size ≈ 1000

16 threads1000 units/second

Costs known by scheduler

60

A’s

Ser

vice

Rat

e

250 Azure Storage tenants32 threads

1 million units/secondCosts known by scheduler

61

250 Azure Storage tenants32 threads

1 million units/secondCosts known by scheduler

61

250 Azure Storage tenants32 threads

1 million units/secondCosts known by scheduler

T1

61

250 Azure Storage workloads32 threads

1 million units/secondCosts known by scheduler

T1

62

250 Azure Storage workloads32 threads

1 million units/secondCosts known by scheduler

T1

62

250 Azure Storage workloads32 threads

1 million units/secondCosts known by scheduler

T1

62

250 Azure Storage workloads32 threads

1 million units/secondCosts known by scheduler

T1

62

250 Azure Storage workloads32 threads

1 million units/secondCosts known by scheduler

T1

62

250 Azure Storage workloads32 threads

1 million units/secondCosts known by scheduler

T1

62

250 Azure Storage workloads32 threads

1 million units/secondCosts known by scheduler

T1

62

250 Azure Storage workloads32 threads

1 million units/secondCosts known by scheduler

T1

62

Unpredictable WorkloadsWidely Varying Costs

Quality OfService

63

Unpredictable WorkloadsWidely Varying Costs

Quality OfService

63

Unpredictable WorkloadsWidely Varying Costs

Quality OfService

63

Unpredictable WorkloadsWidely Varying Costs

Quality OfService

63

Predictable

2/3 predictable1/3 unpredictable

1/3 predictable2/3 unpredictable

T1

T1

T1

64

Predictable

2/3 predictable1/3 unpredictable

1/3 predictable2/3 unpredictable

T1

T1

T1

64

Predictable

2/3 predictable1/3 unpredictable

1/3 predictable2/3 unpredictable

T1

T1

T1

64

Predictable

2/3 predictable1/3 unpredictable

1/3 predictable2/3 unpredictable

T1

T1

T1

64

Predictable

2/3 predictable1/3 unpredictable

1/3 predictable2/3 unpredictable

T1

T1

T1

64

Predictable

2/3 predictable1/3 unpredictable

1/3 predictable2/3 unpredictable

65

Predictable

2/3 predictable1/3 unpredictable

1/3 predictable2/3 unpredictable

65

Predictable

2/3 predictable1/3 unpredictable

1/3 predictable2/3 unpredictable

65

Predictable

2/3 predictable1/3 unpredictable

1/3 predictable2/3 unpredictable

65

Two-Dimensional Fair Queueing

66

More threads → Opportunity to reduce burstiness

Two-Dimensional Fair Queueing

66

More threads → Opportunity to reduce burstiness

Two-Dimensional Fair Queueing

Partitions requests across threads by size

66

More threads → Opportunity to reduce burstiness

Two-Dimensional Fair Queueing

Partitions requests across threads by size

Co-locates unpredictable and expensive workloads

66

More threads → Opportunity to reduce burstiness

Two-Dimensional Fair Queueing

Partitions requests across threads by size

Co-locates unpredictable and expensive workloads

Unpredictable WorkloadsWidely Varying Costs

Quality OfService

66

More threads → Opportunity to reduce burstiness

Two-Dimensional Fair Queueing

Partitions requests across threads by size

Co-locates unpredictable and expensive workloads

Less burstiness

Reduced tail latency

Unpredictable WorkloadsWidely Varying Costs

Quality OfService

66