web mashups with nodejs
TRANSCRIPT
Richard Thomson
Principal Architect for Modeling
DAZ 3D
@LegalizeAdulthood
http://LegalizeAdulthood.wordpress.com
http://github.com/LegalizeAdulthood
Wikipedia
"A mashup is a web page that uses content
from more than one source to create a single
new service displayed in a single graphical
interface.
[...]
The term implies easy, fast integration,
using APIs and data sources to produce
enriched results [...]."
What is NodeJS?
"Node.js is a platform built on Chrome's
JavaScript runtime for easily building fast,
scalable network applications. Node.js uses an
event-driven, non-blocking I/O model that
makes it lightweight and efficient, perfect for
data-intensive real-time applications that run
across distributed devices."
http://nodejs.org
What is a Mashup?
Web server application
Aggregates data from multiple sources
Renders your interpretation of the data
Data gathering is inherently asynchronous
Good time to use NodeJS!
A Software Team's Mashup
Sprint dates
Bug tracker queries
Builds
TODO Comments
Cross-repository synchronization
Copyright notices
Sources of Internal Data
Source code (local / version control)
Bug tracker
Wiki
QA/build servers
build logs
static analysis reports
code reviews (review board, gerrit, etc.)
Sources of External Data
REST apis
RSS feeds
HTML pages
Serial port
Simple Mashup Server
Mashup Server with
Persistence
Mashup Agent
Exploring Available Data
Use a REST api exploration tool (not curl)
Advanced REST Client for Chrome is great
Explore other data sources from Node.js
command line
Handy npm Modules
request
Simplifies making HTTP requests
moment
Date/Time manipulation
async
Orchestrate data sources and updates
underscore / lodash
General utility
cheerio
Lax HTML parsing for screen scraping
http://npmjs.com
https://github.com/caolan/async
http://momentjs.com/
http://underscorejs.org/
https://github.com/cheeriojs/cheerio
Web Server
http.createServer
express + jade
MEAN stack
(MongoDB, Express, AngularJS, NodeJS)
Others
https://npmjs.com/package/express
http://jade-lang.com/
http://en.wikipedia.org/wiki/MEAN
Data Gathering
Use NodeJS style callbacks
Periodic updates
Use an interval timer and a cache
Use an interval timer and events
Use async to get parallelism
One NodeJS module per data source
Screen Scraping with Cheerio
Cheerio is a lax HTML parser
Forgiving of crappy HTML all over the web
Provides a jQuery-like API for accessing the
parsed document
Data Rendering
Server side:
Templates with data model substitution
Client side:
Server static client app (AngularJS, etc.)
Client app pulls model data from REST api
Advice
Start simple and evolve
Separate responsibilities into Node modules
Inject dependent modules for unit testing
Look for npm modules for your data source
For large data models, consider rendering a REST api and serve client-side app