Real-Life Node.js TroubleshootingDamian Schenkelman, Auth0

4 years

Usual suspects

Memory Leaks

–Sebastian Peyrott (a co-worker)

“The main cause for leaks in garbage collected languages are unwanted references”

Memory Modelhttps://developers.google.com/web/tools/chrome-

devtools/memory-problems/memory-101

A

B

C

F

E

D

GGC

H I

b.d=null

A

B

C

F

E

D

GGC

H I

D not referenced

A

B

C

F

E

D

GGC

H I

D is GCed

A

B

C

F

E GGC

H I

Sawtooth

Memory Leak

A

B

C

F

E

D

G

Things we don’t need and are taking up a LOT of space

GC

Memory Leak

A

B

C

F

E

D

G

Things we don’t need and are taking up a LOT of space

GC

More things we don’t need

Crashes

Memory leak busting

Take Control

--max-old-space-sizeis your friend

Drain connections

Restarts

Heap Snapshot

v8-profilerhttps://github.com/node-inspector/v8-profiler

varprofiler=require('v8-profiler');

varsnapshot=profiler.takeSnapshot();

snapshot.export(function(error,result){fs.writeFileSync('snapshot.json',result);snapshot.delete();});

Demo

Mental Picture

{'kinesis.us-west-1.amazonaws.co:443':}

{sockets:}

[…,…,…,…,…,…,…,…,…,TLSSocket

]

Something related to logging…

Not Easyhttps://github.com/auth0/kinesis-writable/pull/6/files

forever-agenthttps://github.com/request/forever-agent/blob/

ece900a6e8dfac734186db3080c00875c0300450/index.js#L70

if(this.freeSockets[name]&&this.freeSockets[name].length>0&&!req.useChunkedEncodingByDefault){varidleSocket=this.freeSockets[name].pop()idleSocket.removeListener('error',idleSocket._onIdleError)deleteidleSocket._onIdleErrorreq._reusedSocket=truereq.onSocket(idleSocket)}else{this.addRequestNoreuse(req,host,port)}

Fixed

CPU bottlenecks

Auth Service

DB

Scenario

Client

Expected response times

Actual response times

Flame graphsa()

b()

c()

d()

h()

i()

e() f()

g()

Flame graphs

Demo

The problem

Scale

Faster Hash

Caching

Scale up

Multiple Auth Services

LB

Auth Service

Auth Service

Auth Service

Auth Service

BaaShttps://github.com/auth0/node-baas

//comparebcryptforreq.passwordtodbhashbaas.compare(req.body.password,user.passwordHash,(err,success)=>{res.send(err||!success?401:200);});

Auto Scaling

Auth Service LB

BaaS

BaaS

BaaS

BaaS

Cost comparison

Price / (1M req) #req per sec / vCPU

t2-micro $0.36 10.00

t2-medium $0.76 9.50

c4-large $1.53 10.00

c3-8xlarge $1.64 8.88

Fail gracefully

Cat picture

Memory Leaks• Memory terminology: https://developers.google.com/web/tools/

chrome-devtools/memory-problems/memory-101

• Memory leaks in JS: https://auth0.com/blog/four-types-of-leaks-in-your-javascript-code-and-how-to-get-rid-of-them/

• Roots: https://stackoverflow.com/questions/9748358/when-does-the-js-engine-create-a-garbage-collection-root

• Increase heap size: https://twitter.com/tjholowaychuk/status/480753206301966336

• v8 GC logs speaking to you: https://github.com/joyeecheung/v8-gc-talk

Memory Leaks (2)• Connection draining: https://cloud.google.com/

compute/docs/load-balancing/enabling-connection-draining

• master-process: https://github.com/jfromaniello/master-process

• Express connection draining: https://github.com/expressjs/express/issues/1366

• v8-profiler: https://github.com/node-inspector/v8-profiler

Memory Leaks (3)• AWS SDK issues: https://github.com/aws/aws-sdk-js/issues/329,

https://github.com/aws/aws-sdk-js/issues/855

• Kinesis event fix: https://github.com/auth0/kinesis-writable/pull/6/files

• Forever agent: https://github.com/request/forever-agent/blob/ece900a6e8dfac734186db3080c00875c0300450/index.js#L70

• Memory profiling for mere mortals: http://thlorenz.com/talks/memory-profiling/book/memory_profiling_for_mere_mortals_/memory_profiling_for_mere_mortals__0.html

• @thlorenz notes on mem profiling: https://github.com/thlorenz/v8-perf/blob/master/memory-profiling.md

Performance• http://security.stackexchange.com/a/83382

• http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html

• https://github.com/davidmarkclements/0x

• https://github.com/thlorenz/v8-perf/issues/4

• https://github.com/auth0/node-baas

• http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/US_SetUpASLBApp.html

• https://gist.github.com/trevnorris/9616784

• https://github.com/dschenkelman/bcrypt-sample

Thankshttps://github.com/dschenkelman/node-troubleshooting

@dschenkelman npmidschenkelman