tail latency in node.js · instrument the node.js runtime so that at runtime we could easily...

Wenzhi Cui Daniel Richins Yuhao Zhu Vijay Janapa Reddi

Tail Latency in Node.js: Energy Efficient Turbo Boosting for Long Tail

Requests in JavaScript

�3

Connecting People (2010s)

�4

Connecting People (2010s)

�5

Connecting Things (2020s)

�5

Connecting Things (2020s)

50 Billion Devices

“The Internet of Things” — Cisco www.cisco.com/web/about/ac79/docs/innov/IoT_IBSG_0411FINAL.pdf

�6

Thread-based Programming: Traditional Approach

Client Requests

�6


Client Requests

Blocking I/O

Client Response

�6


Client Requests

Limited resources & thrashing

Blocking I/O

Client Response

�7

Thread-based Programming

[Welsh et al. ’00]

�8

Event-driven Programming: Emerging Approach

�8

Even

t Que

ueHead

Tail


�8

ApplicationTasks

Even

t Que

ueHead

Tail


�8

ApplicationTasks

Even

t Que

ueHead

Tail

fs.readFile(‘input.txt’, function (err, data) { if (err) return console.error(err); console.log(data.toString()); } );

console.log("Program continues…”);


�8

Single-threaded event loop

ApplicationTasks

Even

t Que

ueHead

Tail




�8

Single-threaded event loop

DB AccessFile I/O

Network

ApplicationTasks

Even

t Que

ueHead

Tail

Asynchronous I/O




�9

Thread-based Programming

Event-driven Programming

[Welsh et al. ’00]

The Changing Programming Language Landscape

�10

�11

ManagedLanguage

Event-drivenExecution Model

Taming Tail Latencies in Event-Driven Web Services

�13

Frac

tion

of R

eque

sts

Request Latency

Tail

Experimental Setup

�14

(1 Gbps Network)

Intel i7-4790K, 4 physical cores with hyper-threading, 32 GB DRAM, 240GB SSD

Wrk

2: A

cus

tom

ized

Load

Tes

ting

Tool

, sim

ulat

e re

al-w

orld

wor

kload

s

Applications

�15

Benchmarks I/O Type #Requests Description

Etherpad lite N/A 20K Real time word processor.

Todo Redis 40K Online Task Manager.

Lighter Disk 40K Blogging Engine.

Let’s Chat MongoDB 10K Web-based Chat Application.

Client Manager MongoDB 40K Online Address book.

Github Repo: https://github.com/nodebenchmark/benchmarks

Etherpad: An Example

�16

1.0

0.8

0.6

0.4

0.2

0.0

CDF

605040302010Latency (ms)


�17

1.0

0.8

0.6

0.4

0.2

0.0

CDF

605040302010Latency (ms)

(1.85, 50%)


�18

1.0

0.8

0.6

0.4

0.2

0.0

CDF

605040302010Latency (ms)

(1.85, 50%)

(7.30, 90%)


�19

1.0

0.8

0.6

0.4

0.2

0.0

CDF

605040302010Latency (ms)

(1.85, 50%)

(7.30, 90%)(36.91, 99.9%)


�19

Tail Region

1.0

0.8

0.6

0.4

0.2

0.0

CDF

605040302010Latency (ms)

(1.85, 50%)

(7.30, 90%)(36.91, 99.9%)

Tail RegionTail Region


�20

1.0

0.8

0.6

0.4

0.2

0.0

CDF

3.02.01.0Latency (ms)

(0.47, 50%)

(1.12, 90%)(1.80, 99.9%)

1.0

0.8

0.6

0.4

0.2

0.0

CDF

161284Latency (ms)

(0.81, 50%)

(1.40, 90%)(8.75, 99.9%)

1.0

0.8

0.6

0.4

0.2

0.0

CDF

40302010Latency (ms)

(12.52, 50%)

(20.30, 90%)(39.54, 99.9%)

1.0

0.8

0.6

0.4

0.2

0.0

CDF

2015105Latency (ms)

(1.65, 50%)

(2.66, 90%)(12.99, 99.9%)

Todo Lighter

Client ManagerLet’s Chat



�20

1.0

0.8

0.6

0.4

0.2

0.0

CDF

3.02.01.0Latency (ms)

(0.47, 50%)

(1.12, 90%)(1.80, 99.9%)

1.0

0.8

0.6

0.4

0.2

0.0

CDF

161284Latency (ms)

(0.81, 50%)

(1.40, 90%)(8.75, 99.9%)

1.0

0.8

0.6

0.4

0.2

0.0

CDF

40302010Latency (ms)

(12.52, 50%)

(20.30, 90%)(39.54, 99.9%)

1.0

0.8

0.6

0.4

0.2

0.0

CDF

2015105Latency (ms)

(1.65, 50%)

(2.66, 90%)(12.99, 99.9%)

Todo Lighter


Tail latency (99.9%) is 9.1x longerthan median request latency

System Overview

�21

Step 1 Step 2 Step 3

System Overview

�21


Tools to Root-cause Tail in

Node.js

System Overview

�21

Tail LatencyReconstruction

Event-driven

runtime(libuv)

JS to C++ bindings

Req1 Req2 …

EventData

e1

e5

e3e4

e2

EventCritical Path

JavaScript runtime

(V8)

File/Net JS libraries

Event-DependencyGraph (EDG)


System Overview

�21


Event-driven

runtime(libuv)

JS to C++ bindings

Req1 Req2 …

EventData

e1

e5

e3e4

e2

EventCritical Path

JavaScript runtime

(V8)




Root-causing Tail in Node.js

System Overview

�21


Event-driven

runtime(libuv)

JS to C++ bindings

Req1 Req2 …

EventData

e1

e5

e3e4

e2

EventCritical Path

JavaScript runtime

(V8)



Exec

Queue

I/O

Req1Exec

Queue

I/OReq2

Requ

est L

aten

cy

Tail LatencyBottleneck Anaysis

IO (%)Queue (%)Exec (%) GC (%) JIT (%) …


System Overview

�21


Event-driven

runtime(libuv)

JS to C++ bindings

Req1 Req2 …

EventData

e1

e5

e3e4

e2

EventCritical Path

JavaScript runtime

(V8)



Exec

Queue

I/O

Req1Exec

Queue

I/OReq2

Requ

est L

aten

cy




Mitigating Tail in Node.js

System Overview

�21


Event-driven

runtime(libuv)

JS to C++ bindings

Req1 Req2 …

EventData

e1

e5

e3e4

e2

EventCritical Path

JavaScript runtime

(V8)



Exec

Queue

I/O

Req1Exec

Queue

I/OReq2

Requ

est L

aten

cy



Tail LatencyOptimization

Queue Boosting

VM Boosting

VM Optimization VM Tuning

Event Queue


System Overview

�21


Event-driven

runtime(libuv)

JS to C++ bindings

Req1 Req2 …

EventData

e1

e5

e3e4

e2

EventCritical Path

JavaScript runtime

(V8)



Exec

Queue

I/O

Req1Exec

Queue

I/OReq2

Requ

est L

aten

cy




Queue Boosting

VM Boosting


Event Queue


Static Instrumentation

System Overview

�21


Event-driven

runtime(libuv)

JS to C++ bindings

Req1 Req2 …

EventData

e1

e5

e3e4

e2

EventCritical Path

JavaScript runtime

(V8)



Exec

Queue

I/O

Req1Exec

Queue

I/OReq2

Requ

est L

aten

cy




Queue Boosting

VM Boosting


Event Queue


Static Instrumentation

Dynamic Analysis & Optimization

Step 1: Latency Reconstruction

�22

var count = N;fs.readdir(“/data”, function dir(err, files) { files.foreach(function file(f, index) { var fname = …; fs.readFile(fname, function read(err, data) { count -= 1; if (count == 0) sendResponse(); }); });});

Step 1: Latency Reconstruction

�22

var count = N;fs.readdir(“/data”, function dir(err, files) { files.foreach(function file(f, index) { var fname = …; fs.readFile(fname, function read(err, data) { count -= 1; if (count == 0) sendResponse(); }); });});

Source

readdir

readFile1 readFile2Sink readFile3 readFileN

Event Dependency Graph (EDG)

�23


�23

e1

e5

e3e4

e2

EventCritical Path



�23

e1

e5

e3e4

e2

EventCritical Path


How do we obtain the latency of each event?

Deconstructing Event Latency

Sever-side Latency


Sever-side Latency

Queue. Exec.I/O


Sever-side Latency

Queue. Exec.I/O Node.js RuntimeCompute Time


Sever-side Latency

Queue. Exec.I/O Node.js Runtime

User CodeGC Interrupt IC Miss JIT Engine

Compute Time

Offline Instrumentation + Online Analysis

�25

Event-driven

runtime(libuv)

JS to C++ bindings

Req1 Req2 …

JavaScript runtime

(V8)


Instrument the Node.js runtime so that at runtime

we could easily obtain: EDG & event latency info.

Offline Instrumentation + Online Analysis

�25

Exec

Queue

I/O

Req1

Exec

Queue

I/O

Req2

Requ

est L

aten

cy



Event-driven

runtime(libuv)

JS to C++ bindings

Req1 Req2 …

JavaScript runtime

(V8)


Instrument the Node.js runtime so that at runtime

we could easily obtain: EDG & event latency info.

Identify root-causes of long tails at runtime.

Step 2: EDG-based Bottleneck Analysis

�26

5040302010

0

Late

ncy

(ms)

A n B n C n D nAvg n A t B t C t D t

Avg t

Request Type

IO Queue ExecTailNon-tail

Step 2: EDG-based Bottleneck Analysis

�26

5040302010

0

Late

ncy

(ms)

A n B n C n D nAvg n A t B t C t D t

Avg t

Request Type


Etherpad: Queue and Exec. latency increases in tails, and I/O latency is not dominant for this particular application.

EDG-based Bottleneck Analysis

�27

16

12

8

4

0

Late

ncy

(ms)

V n W n X nAvg n V t W t X t

Avg t

Request Type



�27

16

12

8

4

0

Late

ncy

(ms)


Avg t

Request Type


Client Manager: Queue and Exec. latency dominate in tails, but unlike Etherpad I/O plays a notable role in the requests.


�27

16

12

8

4

0

Late

ncy

(ms)


Avg t

Request Type


Client Manager: Queue and Exec. latency dominate in tails, but unlike Etherpad I/O plays a notable role in the requests.

On average, queuing and native code execution time contribute to ~80% of the tail latencies.

Breakdown Within Compute

�28

Compute Time

Breakdown Within Compute

�28

Compute Time

We should focus optimization efforts on Garbage Collection and Generated Native Code

Step 3: Tail Latency Optimization

▸ Leveraging the turbo boosting capability of modern CPUs

▸ Key: wisely choose what to boost

▸ GC Boosting

▹ Boost GC

▸ Queue Boosting

▹ Boost when the system is “busy”

▹ Use event queue stats as “hints”

�29


Queue Boosting

VM Boosting


Event Queue

(Intel Turbo Boosting)

Optimization 1: VM Optimization (GC Boost)


▸ Observations:

▹ GCs are infrequent, little overall energy overhead

▹ IPC during GC is relatively high: ~1.3


▸ Observations:

▹ GCs are infrequent, little overall energy overhead

▹ IPC during GC is relatively high: ~1.3

▸ Implementation

▹ User-space DVFS embedded in Google V8: enter boosting at GC prologues and exit boosting at GC epilogues

▹ More benefits if we have access to fine-grained per-core DVFS mechanism

Optimization 2: Queue Boost

�31

…

Queue Monitor

DVFS


�31

▸ More general compute acceleration: Boost when the system is “busy”

…

Queue Monitor

DVFS


�31

▸ More general compute acceleration: Boost when the system is “busy”▸ How do you detect that?

…

Queue Monitor

DVFS


�31

▸ More general compute acceleration: Boost when the system is “busy”▸ How do you detect that?▸ Rely on two queue-related heuristics: ▹ # of events in the queue

▹ Processing time of the head-of-line event

…

Queue Monitor

DVFS


�31

▸ More general compute acceleration: Boost when the system is “busy”▸ How do you detect that?▸ Rely on two queue-related heuristics: ▹ # of events in the queue

▹ Processing time of the head-of-line event

…

Queue Monitor

DVFS▸ Implementation: ▸ Periodic Sampling: Every 1 ms

▸ Dynamic Thresholding

▹ Sample the average value of event number and per event processing time

▹ Amplify the average value to decide a dynamic threshold by 2-3x

Evaluation

▸ Our system: normal operating frequency at 2.6 GHz, boosts to max 4.0 GHz during GC boost and Queue boost

�32

Evaluation


▸ Different Variants:

�32

Evaluation



▹ GC Boost

�32

Evaluation



▹ GC Boost

▹ GC Boost with GC Parameter Tuning

�32

Evaluation



▹ GC Boost


▹ Queue Boost

�32

Evaluation



▹ GC Boost


▹ Queue Boost

▹ GC Tuning + GC Boost + Queue Boost

�32

Evaluation



▹ GC Boost


▹ Queue Boost

▹ GC Tuning + GC Boost + Queue Boost▸ Baseline

�32

Evaluation



▹ GC Boost


▹ Queue Boost

▹ GC Tuning + GC Boost + Queue Boost▸ Baseline▸ Static Frequency: 3.3GHz and 4.0GHz

�32

Evaluation



▹ GC Boost


▹ Queue Boost


▸ Adrenaline [HPCA 2015]

�32

Evaluation



▹ GC Boost


▹ Queue Boost



▸ Rubik [MICRO 2015]

�32

Evaluation



▹ GC Boost


▹ Queue Boost



▸ Rubik [MICRO 2015]

�32

2.4

2.1

1.8

1.5

1.2

0.9

Nor

m. E

nerg

y

3228242016128

Tail Reduction (%)

Combined GC Boost GC Boost+Tuning QBoost

Etherpad Lite

2.4

2.1

1.8

1.5

1.2

0.9

Nor

m. E

nerg

y

3228242016128

Tail Reduction (%)


2.8

2.3

1.8

1.3

0.8

Nor

m. E

nerg

y

1086420Tail Reduction (%)

Adrenaline Rubik 3.3 GHz 4.0 GHz

3.0

2.3

1.6

0.9

Nor

m. E

nerg

y

4032241680

Tail Reduction (%)2.8

2.4

2.0

1.6

1.2

0.8

Nor

m. E

nerg

y

262116116

Tail Reduction (%)

3.2

2.6

2.0

1.4

0.8N

orm

. Ene

rgy

3227221712

Tail Reduction (%)

Todo Lighter


2.4

2.1

1.8

1.5

1.2

0.9

Nor

m. E

nerg

y

3228242016128

Tail Reduction (%)


2.8

2.3

1.8

1.3

0.8

Nor

m. E

nerg

y

1086420Tail Reduction (%)

Adrenaline Rubik 3.3 GHz 4.0 GHz

3.0

2.3

1.6

0.9

Nor

m. E

nerg

y

4032241680

Tail Reduction (%)2.8

2.4

2.0

1.6

1.2

0.8

Nor

m. E

nerg

y

262116116

Tail Reduction (%)

3.2

2.6

2.0

1.4

0.8N

orm

. Ene

rgy

3227221712

Tail Reduction (%)

Todo Lighter


Pareto-dominate existing solutions; 14-21% tail reduction with only 3-14% energy overhead over baseline.

Conclusions

Node.js uniquely combines event-driven programming model and managed language runtime, presenting new landscape and challenges to tail latency optimizations.

�34

Conclusions


�34

e1

e5

e3e4

e2

EventCritical Path


Event-dependency graph (EDG) and event-critical path (ECP) critical to deconstruct tail latency in Node.js.

Conclusions


�34

e1

e5

e3e4

e2

EventCritical Path


Event-dependency graph (EDG) and event-critical path (ECP) critical to deconstruct tail latency in Node.js.


Queue Boosting

VM Boosting


Event Queue


Queue Boosting

VM Boosting


Event Queue

Intelligently leverage existing hardware features, turbo boosting in particular, to reduce latency with little to none energy overhead.

tail latency in node.js · instrument the node.js runtime so that at runtime we could easily...

Documents