divide and conquer – microservices with node.js

Divide and conquer

Christian Steiner / pixelio.de

Basti• Sebastian Springer

• from Munich

• work @ MaibornWolff GmbH

• https://github.com/sspringer82

• @basti_springer

• JavaScript Developer

AgendaHTTP Microservice with Express

• Routing • Structure • Database • Logging • Tests • Container

Commandbased Microservices with Seneca

• Patterns • Plugins • Queueing

Rainer Sturm / pixelio.de

MicroservicesSmall

Scaleable

Resilient

Independent Testable

Focused

JavaScript is nearly everywhere. There are a lot of OSS Modules. Node.js was built for the web

Node.js supports most of the protocols, systems and databases.

The platform is lightweight.

Example

A microservice that delivers articles for a web shop.

M. Hermsdorf / pixelio.de

Webshop

Client API

Articles

Payment

Users

DB

DB

DB

Auth

Logger

List of articlesRouter

Controller

Model

Datenbank

Logger

Packages

S. Hofschlaeger / pixelio.de

Express.js

Express is a lightweight (< 2MB) web application framework for Node.js.

Express supports HTTP and HTTPS in version 1 and 2. Express handles routing within a web application for you.

Additionally it provides you with a plugin interface (Middleware).

Installation

yarn init

yarn add express

Structure of a Microservice. ├── db │   ├── config.js │   └── db.sqlite3 ├── package.json ├── spec └── src    ├── config.js    ├── controller.js    ├── index.js    ├── logger.js    ├── model.js    └── router.js

Entry pointindex.js

Entry point

const express = require('express'); const router = require('./router');

const app = express();

router(app);

app.listen(8080, () => { console.log('Service is listening to http://localhost:8080'); });

Entry point

In this file all global packages are included and the app is configured.

Additional features should be put in separate files.

Routerrouter.js

Router

const controller = require('./controller');

module.exports = (app) => { app.get('/article', controller.getAll); app.get('/article/:id', controller.getOne); app.post('/article', controller.create); app.put('/article', controller.update); app.delete('/article', controller.remove); }

Router

The router defines all available routes of a microservice.

A route consists of a HTTP method and a path. By adding variables to routes they become more flexible.

To keep the code of your router readable, you should put the routing callbacks in a separate file.

Controllercontroller.js

Controllerconst model = require('./model');

module.exports = { getAll(req, res) { res.json( model.getAll() ); }, getOne(req, res) {}, create(req, res) {}, update(req, res) {}, remove(req, res) {} }

ControllerThe controller holds your routing callback functions. It extracts information from the requests and hands the

information over to the model.

To access variables of your routes, you can use the req.params object.

To deal with information within the request body, you should install the body-parser middleware. It introduces the req.body

object which represents the message body.

Modelmodel.js

Model

The model contains the business logic of your application. It also controls access to the database.

Usually every microservice has its own database, which increases independence between services.

ModelNode.js supports all common databases such as OracleDB,

MySQL, Redis, MongoDB.

To access your database, you have to install a database driver first.

yarn add sqlite3

To further simplify the access, you can use various ORMs or ODMs

yarn add orm

ORM

An ORM handles CRUD operations for you. It already implements all common use cases. You just have to

configure it with the structure of your database.

The ORM also takes care of some security features such as escaping.

ORMconst express = require('express'); const orm = require('orm'); const router = require('./router'); const dbConfig = require('./db/config.js'); const {dbPath} = require('./config.js');

const app = express();

app.use(orm.express(dbPath, dbConfig));

router(app);

app.listen(8080, () => { console.log('Service is listening to http://localhost:8080'); });

ORM

module.exports = { define(db, models, next) { models.articles = db.define('articles', { id: Number, title: String, price: Number }); next(); } }

Model

Besides encapsulating database communication your model takes care of additional tasks such as validation of user input

and various calculations.

Most of these operations are asynchronous. To provide a clean API you should think about using promises, async

functions or streams instead of plain callbacks.

async getAll(req, res) { try { const articles = await model.getAll(req); res.json(articles); } catch (e) { res.status(500).send(e); } }

getAll(req) { return new Promise((resolve, reject) => { req.models.articles.all((err, results) => { if (err) { reject(err); } else { resolve(results); } }); }); }

controller

model

Logging

Tim Reckmann / pixelio.de

LoggingWithin your microservice errors might occur at any time. Of course you should be prepared to catch and handle them.

But you should also keep a log of errors somewhere.

A logger should not be a fixed part of a certain microservice. logging is a shared service which is available for all services

in your application.

Centralised logging provides some advantages over local logging such as scalability and an improved maintainability.

Logging

You can use log4js for remote logging in your application. This library provides a plugin interface for external appenders which support various log targets such as files or logstash.

For remote logging you could use the logstash appender and a centralised logstash server.

Loggingconst log4js = require('log4js'); log4js.configure({ appenders: [{ "host": "127.0.0.1", "port": 10001, "type": "logstashUDP", "logType": "database", "layout": { "type": "pattern", "pattern": "%m" }, "category": "database" }], categories: { default: { appenders: ['database'], level: 'error' } } });

module.exports = log4js;

Loggingconst log4js = require('./logger');

module.exports = { getAll(req) { return new Promise((resolve, reject) => { req.models.articles.all((err, results) => { if (err) { reject(err); log4js.getLogger('database').error(err); } else { resolve(results); } }); }); } }

Tests

Dieter Schütz / pixelio.de

TestsQuality and stability are two very important aspects of

microservices. To ensure this, you need the ability to automatically test your services.

There are two levels of tests for your microservice application: unittests for smaller units of code and integration tests for

external interfaces.

For unittests you can use Jasmine as a framework. With Frisby.js you can test your interfaces.

Unittests

yarn add jasmine

node_modules/.bin/jasmine init

Unittestsconst model = require('../model');

describe('model', () => { it('should handle a database error correctly', () => { const req = { models: { articles: { all: (cb) => {cb('error', null);} } } }

model.getAll(req).catch((e) => { expect(e).toBe('error'); }); }) });

Unittests

To execute your tests, just issue the command npx jasmine.

You have two variants for organising your tests. You can either store them in a separate directory, usually named “spec” or

you put your tests where your source code is located.

Mockery

In your microservice application you have to deal with a lot of dependencies. They are resolved via the node module

system. Mockery is a tool that helps you with dealing with dependencies. Mockery replaces libraries with test doubles.

yarn add -D mockery

Mockeryconst mockery = require('mockery');

beforeEach(() => { mockery.enable();

const fsMock = { stat: function (path, cb) {...} }; mockery.registerMock('fs', fsMock); });

afterEach(() => { mockery.disable(); });

Integration tests

Frisby.js is a library for testing REST interfaces. Frisby is an extension for Jasmine. So you don’t have to learn an additonal

technology. In order to run Frisby.js, you have to install jasmine-node.

yarn add -D frisby jasmine-node

Integration testsrequire('jasmine-core'); var frisby = require('frisby'); frisby.create('Get all the articles') .get('http://localhost:8080/article') .expectStatus(200) .expectHeaderContains('content-type', 'application/json') .expectJSON('0', { id: function (val) { expect(val).toBe(1);}, title: 'Mannesmann Schlosserhammer', price: 7 }) .toss();

PM2

Node.js is single threaded. Basically that’s not a problem because of its nonblocking I/O nature. To optimally use all

available resources, you can use the child_process module.

A more convenient way of locally scaling your application is using PM2.

yarn add pm2

PM2pm2 start app.js

pm2 start app.js -i 4

pm2 reload all

pm2 scale <app-name> 10

pm2 list

pm2 stop

pm2 delete

Docker

All your services run in independent, self contained containers.

You can start as many instances of a certain container as you need.

To extend the features of your application, you simply add additional services by starting containers.

DockerfileFROM node:7.10

# Create app directory RUN mkdir -p /usr/src/app WORKDIR /usr/src/app

# Install app dependencies COPY package.json /usr/src/app/ RUN yarn install

# Bundle app source COPY . /usr/src/app

EXPOSE 8080 CMD [ "yarn", "start" ]

Docker

docker build -t basti/microservice .

docker run -p 8080:8080 -d basti/microservice

API Gateway

Jürgen Reitböck / pixelio.de

API Gateway

Each service of your application has to focus on its purpose. In order to accomplish this, you have to centralise certain

services. A typical example for a central service is authentication.

An API Gateway forwards authorised requests to the services of your application, receives the answers and forwards them to

the client.

Seneca

Seneca follows a completely different approach. All services of your application communicate via messages and become

independent of the transport layer.

Installation

yarn add seneca

Service Definition

const seneca = require('seneca')(); seneca.add({role: 'math', cmd: 'sum'}, controller.getAll);

The first argument of the add method is the pattern, which describes the service. You are free to define the pattern as you want. A common best practice is to define a role and a

command. The second argument is an action, a function that handles incoming requests.

Service Handlerasync getAll(msg, reply) { try { const articles = await model.getAll(req); reply(null, JSON.stringify(articles)); } catch (e) { reply(e); } }

Similar to express a service handler receives the representation of a request. It also gets a reply function. To

create a response, you call the reply function with an error object and the response body.

Service Callseneca.act({role: 'article', cmd: 'get'}, (err, result) => { if (err) { return console.error(err); } console.log(result); });

With seneca.act you consume a microservice. You supply the object representation of a message and a callback function. With the object, you trigger the service. As soon as the answer

of the service is available, the callback is executed.

Patterns

Helmut / pixelio.deHelmut / pixelio.de

Patterns

You can create multiple patterns of the same type. If a service call matches multiple patterns, the most specific is

used.

You can use this behaviour to implement versioning of interfaces.

Plugins

Klicker / pixelio.de

Plugins

A plugin is a collection of patterns. There are multiple sources of plugins: built-in, your own plugins and 3rd party

plugins.

Plugins are used for logging or debugging.

Plugins

You organise your patterns with the use method. The name of the function is used for logging purposes.

You can pass options to your plugin to further configure it. The init pattern is used instead of a constructor.

const seneca = require('seneca')();

function articles(options) { this.add({role:'article',cmd:'get'}, controller.getAll); }

seneca.use(articles);

Pluginsconst seneca = require('seneca')();

function articles(options) { this.add({role:'article',cmd:'get'}, controller.getAll); this.wrap({role:'article'}, controller.verify); }

seneca.use(articles);

The wrap method defines features that are used by multiple patterns. With this.prior(msg, respond) the original service

can be called.

Client/Server

cre8tive / pixelio.de

Serverfunction articles(options) { this.add({role:'article',cmd:'get'}, controller.getAll); this.wrap({role:'article'}, controller.verify); }

require('seneca')() .use(articles) .listen(8080)

Just like in the ordinary web server the listen method binds the server to the port 8080. Your service can then be called by

the browser or another server:

http://localhost:8080/act?role=article&cmd=get

Client

require('seneca')() .client(8080) .act({role: 'article', cmd: 'get'}, console.log);

The client method lets you connect to a seneca microservice.

Change the transport// client seneca.client({ type: 'tcp', port: 8080 });

// server seneca.listen({ type: 'tcp', port: 8080 });

By providing the type ‘tcp’ the TCP protocol is used instead of HTTP as a communication protocol.

Integration in Express

Integration in Expressconst SenecaWeb = require('seneca-web'); const Express = require('express'); const router = new Express.Router(); const seneca = require('seneca')();

const app = Express();

const senecaWebConfig = { context: router, adapter: require('seneca-web-adapter-express'), options: { parseBody: false } }

app.use( require('body-parser').json() ) .use( router ) .listen(8080);

seneca.use(SenecaWeb, senecaWebConfig ) .use('api') .client( { type:'tcp', pin:'role:article' } );

Integration in Express

With this configuration seneca adds routes to your Express application.

You have to adopt your seneca patterns a little bit. All routes defined with seneca.act('role:web', {routes: routes}) are added

as Express routes.

Via the path-pattern a corresponding matching is done.

Communication via a queue

S. Hofschlaeger / pixelio.de

Queue

You can use various 3rd party plugins to communicate via a message queue instead of sending messages over network.

The main advantage of using a Queue is decoupling client and server. Your system becomes more robust.

Queue

yarn add seneca-servicebus-transport

Queuerequire('seneca')() .use('seneca-servicebus-transport') .use(articles) .listen({ type: 'servicebus', connection_string: '...' });

require('seneca')() .use('seneca-servicebus-transport') .client({ type: 'servicebus', connection_string: '...' }) .act({role: 'article', cmd: 'get'}, console.log);

Server

Client

Microservices, the silver bullet?

Microservices might be a good solution for certain problems but definitely not for all problems in web development.

Microservices introduce an increased complexity to your application. There is a lot of Node.js packages dealing with

this problem. Some of these packages are outdated or of bad quality.

Take a look at GitHub statistics and npmjs.com if it fits your needs.

Questions

Rainer Sturm / pixelio.de

CONTACT

Sebastian Springer sebastian.springer@maibornwolff.de

MaibornWolff GmbH Theresienhöhe 13 80339 München

@basti_springer

https://github.com/sspringer82