defensive programming in javascript and node.js
DESCRIPTION
Various factors to consider when trying to adopt a defensive programming mindset, methodology and process. This is especially useful for teams working with node.js.TRANSCRIPT
Defensive Programming
in Javascript & node.js
Wednesday, May 29, 13
INTRODUCTION
Ruben Tan Long Zheng 陈龙正
VP of Engineering, OnApp CDN KL
Lead Engineer, 40 Square Sdn Bhd
Javascript > 5 years
@roguejs
Organizer of Nodehack KL
Wednesday, May 29, 13
OVERVIEW
Dependency Awareness
Javascript Mastery
Methodology Improvements
Wednesday, May 29, 13
SEASON 1
DEPENDENCY AWARENESS
Wednesday, May 29, 13
Internal dependencies
Libraries
require(), include()
External dependencies
Services, files, databases, etc
socket.connect(), db.open()
DEPENDENCY TYPES
Wednesday, May 29, 13
NEVER ASSUME!
Never assume a dependency is reliable!
var db = require(‘database’);
db.open();
db.write(‘foo bar’, function (err, data) { // ... do something ... });
Wednesday, May 29, 13
NEVER ASSUME!
var db = require(‘database’);
db.open();
db.write(‘foo bar’, function (err, data) { // ... do something ... });
What if this failed?will write() throw an error? will open() throwan exception?
Wednesday, May 29, 13
NEVER ASSUME!
var db = require(‘database’);
db.open(function (err) { db.write(‘mr-big’, bigData, function (err, data) { // ... unrelated logic db.close(); });
db.read(‘foo2’, function (err, data) { // ... some work done });});
Accidents happen...
Wednesday, May 29, 13
NEVER ASSUME!
var db = require(‘database’);
db.open(function (err) { db.write(‘mr-big’, bigData, function (err, data) { // ... unrelated logic db.close(); });
db.read(‘foo2’, function (err, data) { // ... some work done });});
close() might affect read()
Wednesday, May 29, 13
A MORE COMPLEX EXAMPLE...
Wednesday, May 29, 13
VIDEO STREAMING SERVICE
VideoStreamer
Origin
StatsLogger
VODClient
User
Accounting
UploadLiveClient
UserStream
Stream
LogReport
Render
Render
Wednesday, May 29, 13
VIDEO STREAMING SERVICE
VideoStreamer
Origin
StatsLogger
VODClient
User
Accounting
UploadLiveClient
UserStream
Stream
LogReport
Render
Render1
2
3
4
5
6
7
89
1011
Wednesday, May 29, 13
DEPENDENCY AWARENESS
What can fail, WILL FAIL!
Never assume a dependency is reliable!
Contingency plans - failover, redundancy, fail-fast, etc
Pro-active monitoring
Load test, stress test, chaos monkey, etc
Remember, what can fail, WILL FAIL!
Wednesday, May 29, 13
SEASON 2
JAVASCRIPT MASTERY
Wednesday, May 29, 13
JAVASCRIPT MASTERY
Code Execution Order
Sanitization & Validation
Scope
Control Flow
Wednesday, May 29, 13
I KNOW CODE-FU!
Wednesday, May 29, 13
EXECUTION ORDER
var mq = require(‘mq’);
mq.conn(...);
mq.on(‘ready’, function () { mq.send(‘batman’);
mq.on(‘message’, function (msg) { console.log(msg); mq.close(); });});
mq is never closed!
send() executes before on()
Wednesday, May 29, 13
DOIN’ IT RIGHT!
var mq = require(‘mq’);
mq.conn(...);
mq.on(‘ready’, function () { mq.on(‘message’, function (msg) { console.log(msg); mq.close(); });
mq.send(‘batman’);});
Swap places
Wednesday, May 29, 13
SANITIZATION & VALIDATION
function foodForKittens(num) { return num * 10;}
foodForKittens();
num is not validated, is undefined
this will fail!
Wednesday, May 29, 13
TOO SIMPLE?
Wednesday, May 29, 13
SANITIZATION & VALIDATION
var db = require(‘database’);var conn = db.open(...);
function writeToDb(conn, cb) { conn.write(bigData, function (err, res) { if (err) { cb(err); return; }
cb(null, res); });});
writeToDb(conn, ghostCallback);
Wednesday, May 29, 13
Wednesday, May 29, 13
var db = require(‘database’);var conn = db.open(...);
function writeToDb(conn, cb) { conn.write(bigData, function (err, res) { if (err) { cb(err); return; }
cb(null, res); });});
writeToDb(conn, ghostCallback);
what if open() returned undefined?
this will throw an exception!
Wednesday, May 29, 13
var db = require(‘database’);var conn = db.open(...);
function writeToDb(conn, cb) { conn.write(bigData, function (err, res) { if (err) { cb(err); return; }
cb(null, res); });});
writeToDb(conn, ghostCallback);
What if ghostCallback is undefined?
These will fail too!
Wednesday, May 29, 13
DOIN’ IT RIGHT!var db = require(‘database’);var conn = db.open(...);
function writeToDb(conn, cb) { if (typeof conn !== ‘object’) { // ... handle error ... }
if (typeof cb !== ‘function’) { // ... handle error ... }
conn.write(bigData, function (err, res) { if (err) { cb(err); return; }
cb(null, res); });});
writeToDb(conn, ghostCallback);
Validate your input,especially when theyinvolve functions ormethods that you need toinvoke in your code.
These are not the time tofail-fast!
Wednesday, May 29, 13
DON’T GO OVERBOARD...
Validate only necessary parameters
Method invocations (anObject.method())
Function invocations (aFunction())
Have a proper error/exception handling policy
Validate for correctness, not existence
Correctness: typeof a === ‘object’
Existence: a !== undefined
Wednesday, May 29, 13
SCOPE AWARENESS
Plagues most callback-based code
Bad practice leads to costly debugging waste
New JS programmers not aware of scoping
JS scoping is a simple but weird thing (to non-JS programmers)
Wednesday, May 29, 13
SCOPE!!!
var a = ‘outside’;
if (true) { var a = ‘inside’; console.log(a);}
console.log(a);
What is the output?
> node test.jsinsideinside
Wednesday, May 29, 13
SCOPE!!!
Non-JS programmers:
a inside the if block is “inside”
a outside the if block is “outside”
JS programmers:
they are both “inside”
JS scope by function
Wednesday, May 29, 13
SCOPE CHAINS!!!var avar = 1;
(function outer1() { var avar = 2; (function inner1() { var avar = 3; console.log(avar); // outputs 3 })();
(function inner2() { console.log(avar); // outputs 2 })();})();
(function outer2() { (function inner3() { console.log(avar); // outputs 1 })();})();
inner1()
local - found!
inner2()
local - nope
outer1() - found!
inner3()
local - nope
outer2() - nope
global - found!
Wednesday, May 29, 13
HOISTING VARIABLESfunction () { for (var i = 0; i < 10; i++) { for (var j = 0; j < 10; j++) { // ... do something } }}
function () { var i, j; // now the scope is clear for i & j
for (i = 0; i < 10; i++) { for (j = 0; j < 10; j++) { // ... do something } }}
Below is far clearer what individual variable scopes are:
Wednesday, May 29, 13
CONTROL FLOW
Node.js’ async nature makes it unintuitive to predict control flow
I <3 async (github.com/caolan/async)
Control flow is ugly. Welcome to Javascript.
Async will save your life. Use it.
Wednesday, May 29, 13
CONTROL FLOWvar fs;
fs = require(‘fs’);
fs.readFile(‘./myfile.txt’, function (err, data) { if (err) { console.log(err); return; }
fs.writeFile(‘./myfile2.txt’, data, function (err) { if (err) { console.log(err); return; }
// ... do stuff ... });})
Wednesday, May 29, 13
CONTROL FLOW
Callback hell!
Step 1
Step 2
Step 3
Step 4
Step 5
Wednesday, May 29, 13
mod.step1(function () { mod.step2(function () { mod.step3(function () { mod.step4(function () { mod.step5(function () { // ... too many levels ... }); }); } });});
Wednesday, May 29, 13
CONTROL FLOW
var async, fs;
async = require(‘async’);fs = require(‘fs’);
async.waterfall([ function step1(callback) { fs.readFile(‘./myfile.txt’, callback); },
function step2(data, callback) { fs.writeFile(‘./myfile2.txt’, data, callback); }], function (err) { // ... execute something in the end ...});
Wednesday, May 29, 13
SEASON 3
METHODOLOGY IMPROVEMENTS
Wednesday, May 29, 13
GOLDEN RULES
Golden Rules of Defensive Programming
Proper error handling policy
Intelligent logging
Design for failure
Wednesday, May 29, 13
ERROR HANDLING
Never, ever HIDE errors
> node app.js 2>&1 /dev/null
ob.callback(function (err, data) { if (err) {} console.log(data);});
socket.on(‘error’, function () {});
Wednesday, May 29, 13
ERROR HANDLING
I WILL FIND YOU
AND I WILL CRASH YOU
Wednesday, May 29, 13
ERROR HANDLING
Standardize error handling in the app
Log to error DB
Output to error file
Output error to a stream
Use a logging library
Ask a leprechaun to manage it
etc
Wednesday, May 29, 13
LOGGING
How do you feel if your “log” looks like this?
> tail -f error.log[12:01:55] ERROR - General error detected[12:01:56] ERROR - General error detected[12:01:57] ERROR - General error detected[12:01:58] ERROR - General error detected[12:01:59] ERROR - General error detected[12:02:00] ERROR - General error detected[12:02:01] ERROR - General error detected
Wednesday, May 29, 13
LOGGING
Wednesday, May 29, 13
LOGGING
Logs are the first place you go to find out what happened
Standardize a log location for each app
Make logs easy to access for developers
Wednesday, May 29, 13
DESIGN FOR FAILURE
Common steps to designing software:
1 - what should it do?
2 - how do I do it?
3 - how do I deploy?
4 - done
Wednesday, May 29, 13
DESIGN FOR FAILURE
Proper steps in defensive programming:
1 - what should it do?
2 - how many ways can it fail?
3 - how do I know when it fails?
4 - how do I prevent it from failing?
5 - write code accordingly
Wednesday, May 29, 13
DESIGN FOR FAILURE
Nothing is reliable
TCP can fail
Network can go down
Servers can run out of memory
Cows might fly through the sky crashing into your datacenter and flooding the server rooms with milk and destroying everything
Wednesday, May 29, 13
DESIGN FOR FAILURE
Designing for failure mindset & methodologies:
Identify SPOF (single point of failures)
Redundancy, failover, monitoring
Fail-fast, start-fast
Persist important data
Reliability & Consistency > Speed
Code is liability
Wednesday, May 29, 13
~ The End ~
Wednesday, May 29, 13