node in production at aviary
DESCRIPTION
Aviary's customizable SDK powers cross-platform photo editing for over 6,500 partners and over 70 million monthly active users across the globe. Some of our notable partners include Walgreens, Squarespace, Yahoo Mail, Flickr, Photobucket, and Wix. At Aviary, we use node.js for several mission-critical projects in production and have seen extremely positive results. In this talk, we will discuss how we approach some common situations that developers deploying node.js projects will likely need to tackle. We will walk you through our routing mechanism, our automated deployment system, some of our custom middleware, and our testing philosophy.TRANSCRIPT
Node.js in Production at AviaryNYC Node.js MeetupMarch 5, 2014
Aviary
Fully-Baked UI
Configurable, High-Quality Tools
Over 6,500 Partners
Over 70 Million Monthly Users
Over 6 Billion Photos Edited
iOS, Android, Web, Server
Photo-Editing SDK & Apps
J
Who Are We?
JackDirector of Engineering
Nir Lead Serverside Engineer
● Automated deployment● Big-O notation● Brainteasers
Likes:
● Cilantro
Hates:
● Parallelizing processes● DRY code● Seltzer
Likes:
● Food after the sell-by date
Hates:
Who Are We?
AriDeveloperEvangelist
Jeff Serverside Engineer
● Performance Profiling● Spaces, not tabs● Bikes
Likes:
● His Photo
Hates:
● Empowering Developers● Refactoring/Patterns● Dancing
Likes:
● Forrest Gump
Hates:
How Do We Use Node?
● In Production:○ Analytics Dashboard○ Content Delivery (CDS)○ Public Website○ Receipts Collection○ Monitoring Tools
● Future:○ Server-side Rendering API○ Automated billing
Why Do We Use Node?● Extremely fast and lightweight● Easy to iterate on● Common language for client and server● JSON● Cross Platform● npm● express module● Active Community
Setting Up Your Server
Request RoutingOur API servers all require(routes.json)
{“home”: {
“getVersion”: {“verb”: “get”,“path”: “/version”
}},“queue”: {
“updateContent”: {“verb”: ”put”,“path”: “/content/:id”,“permissions”: [“content”]
}}
}
for (var controllerName in routes) {var controller = require(ctrlrDir + controllerName);for (var endpointName in routes[controllerName]) {
var endpoint = routes[controllerName][endpointName];
var callback = controller[endpointName];app[endpoint.verb](
endpoint.path, ensurePermissions(endpoint.permissions), callback
);}
}
Authentication: Overview
Request Server listens
Middleware Logged in?
Redirected to login page
Authenticated user saved in cookie
NoYes
Does user have permission?
Request handler takes overResponse
Authentication: Loginpassport.use(new GoogleStrategy({ returnURL: DOMAIN + '/auth/return' },
function (identifier, profile, done) {
var userInfo = {
name: profile.email.value,
fullName: profile.name
};
userRepository.findUserByName(userInfo.name, function (findErr, foundUser) {
// ...
if (foundUser.length === 0) {
done('Invalid user', null);
return;
}
userInfo.userId = foundUser.user_id;
userInfo.permissions = foundUser.permissions;
done(null, userInfo);
});
}
));
Working with JSON
Validation - JSON Schema
● JSON-based
● Like XML Schema
● Validation modules
● Used throughout Aviary’s systems
{ “type”: “object” “additionalProperties”: false “properties”: { “status”: { “type”: “string”, “enum”: [“ok”, “error”], “required”: true }, “data”: { “type”: “object”, “required”: false } }}
{ “status”: “ok”}
{ “status”: “error”, “data”: { “reason”: “hoisting” }}
{ “status”: “gladys”, “node”: “meetup”}
SCHEMA JSON
Advanced JSON - ContentEffects Frames Stickers Messages
The One-To-Many Problem
Android expects responses to look like this:
iOS 1.0 expects responses to look like this:
iOS 2.0 expects responses to look like this:
Response Formatting - The ModelContent Entry Response Formats Responses
JSON document describing content item
JSON documents defining mappings from entry to responses
Actual JSON responses delivered to devices
Response Formatting - Details
"id": {
"type": "string",
"dataKey": "identifier"
},
"isPaid": {
"type": "boolean",
"dataKey": "isFree",
"transformations": ["negateBool"]
},
"iconImagePath": {
"type": "string",
"dataKey": "icon.path-100"
},
"stickers": {
"type": "array",
"dataKey": "items"
"identifier": "com.aviary.stickers.234fe"
"isFree": false,
"icon": {
"path": "cds/hats/icon.png"
"path-100": "cds/hats/icon100.png"
},
"items": [
{
"identifier": "1"
"imageUrl": "cds/hats/1.png"
}
]
"id": "com.aviary.stickers.234fe",
"isPaid": true,
"iconImagePath": "cds/hats/icon100.png"
"stickers": [
{
"identifier": "1"
"imageUrl": "cds/hats/1.png"
}
]
The Single Content Entry The Response Format The Response
Code Sample (Dumbed Down)
var formattedResponse = {};
for (var propName in responseFormat) {
var val = contentEntry[responseFormat[propName].dataKey];
for (var transformation in responseFormat[propName].transformations) {
val = transformationModule[transformation](val);
}
formattedResponse[propName] = val;
}
return formattedResponse;
Interacting with External Processes
Image RenderingChallenge: Use our existing image rendering .NET/C++ process from node server
Solution:require(‘child_process’).spawn(‘renderer.exe’)
Benefits: Easy IPC, asynchronous workflow
Code Samplevar spawn = require(‘child_process’).spawn;
var renderer = spawn(‘renderer.exe’, [‘-i’, ‘inputImage.jpg’, … ]);
// read text
renderer.stderr.setEncoding(‘utf8’);
renderer.stderr.on(‘data’, function (data) { json += data; });
// or binary data
renderer.stdout.on(‘data’, function (data) { buffers.push(data); });
renderer.on(‘close’, function (code, signal) {
// respond to exit code, signal (e.g. ‘SIGTERM’), process output
var diagnostics = JSON.parse(json);
var img = Buffer.concat(buffers);
});
Going Live
Testing Philosophy
● Unit tests (sparingly)
● End-to-end integration tests
● Mocha
● Enforced before push ○ (master / development)
Example Integration Test#!/bin/bash
mocha scopecreation &&
mocha cmsformatcreation &&
mocha crfcreation &&
mocha mrfcreation &&
mocha rflcreation &&
mocha appcreation &&
mocha contentcreation &&
mocha manifestcreation &&
mocha push &&
mocha cmsformatupdate &&
mocha crfaddition &&
mocha rfladdition &&
mocha contentupdate &&
mocha manifestupdate
● Bash script
● Independent files
● Shared configuration
● Single failure stops process
Automated Deployment: Overview
Git
1) Code ispushed tomaster
Jenkins
2) Jenkins polls for repo changes
S3
3) Code is zipped and uploaded to S3
AWS API
4) Get a list of live servers in this group
5) SSH into each server and run
the bootstrap script
Automated Deployment: Bootstrap5) SSH into each
server and run the bootstrap script
#!/bin/bash
ZIP_LOCATION="s3://aviary/projectX/deployment.zip";
cd ~/projectX;
sudo apt-get -y -q install [email protected];
sudo apt-get -y -q install s3cmd;
sudo npm install -g [email protected];
# Missing step: create s3 configuration file
s3cmd -c /usr/local/s3cfg.config get "$ZIP_LOCATION" theCds.zip;
unzip -q -o deployment.zip
iptables -t nat -A PREROUTING -p tcp --dport 80 -j
REDIRECT --to-ports 8142;
forever stopall;
forever start server.js;
Goals of the bootstrap.sh:1. Ensure all dependencies are
installed2. Download and extract project3. Ensure HTTP traffic is routed to the
proper port4. Keep the old version of the project
live until the moment the new one is ready to go live
Summary
Lessons Learned (1)
● Integration tests!
● Watch out for node and npm updates○ Hardcode the node version you’re using○ If you’re using package.json, version everything
● Node.js + MongoDb are a great couple
● Make sure you understand hoisting
Lessons Learned (2)● Always callback in async functions
● Always return after a callback
● Node doesn’t always run the same on all platforms
● Use middleware only when necessary
● Always store dates as Unix Timestamps○ Timezones are a pain in your future
● Throwing unhandled errors will crash your process
ConclusionToday, our production node servers:
● serve dynamic content to 20MM people (soon 70MM)
● power our website: aviary.com
● log real-time receipt data for every in-app purchase
● allow us to analyze hundreds of millions of events daily
● power quick scripts and one-off internal tools
Questions?Comments also welcome
[email protected] - [email protected] - [email protected] - [email protected]
…and by the way, WE’RE HIRING!