nixcon berlin 2015 - nix at logicblox

Smart database for next-generation applications

Nix and LogicBlox

INTRODUCTION

©2014. LogicBlox. All Rights Reserved.

▪ Rob Vermaas

▪ IRC: ikwildrpepper▪ Email: [email protected]

mailto:[email protected]

MY NIX HISTORY


▪ Contributed to:▪ Nixpkgs, Hydra, NixOps, Nix

▪ First Nix related commit▪ August 2004, adding Octave

▪ Infrastructure support

▪ NixOS Foundation

LOGICBLOX

▪ Develops a state-of-the-art database system that is used to develop applications for many top retailers and banks

▪ 150 people, all over the world▪ USA, Canada, Tunisia, The Netherlands, Greece, United Kingdom, Belarus, Sweden

▪ LogicBlox Nix-ers:▪ Eelco Dolstra

▪ Rob Vermaas

▪ Martin Bravenboer

▪ Shea Levy (former employee)

▪ Evgeny Egorochkin (contractor)


WHY DO I WANT TO USE NIX?


▪ Nix protects me against me

▪ Nix let’s me do things multiple times consistently, even on different machines

▪ Nix exposes the things I forget

▪ Nix, the one language to rule them all

HOW TO NIXIFY A COMPANY


Step 1:Find a nice company

Step 2:Apply Nix everywhere

Step 3:Profit

▪ 3 step plan to Nixify a software company

HOW TO NIXIFY A COMPANY


Step 1:Find a nice company

Step 2:Apply Nix everywhere

Step 3:Profit

▪ 3 step plan to Nixify a software company

BuildTest

Deploy

Nix

NIX - THE BIG FOUR


Build Test Deploy

Nix

NIX - THE BIG FOUR


Hydra

Build Test Deploy

NixNixOps / NixOS

NIX - THE BIG THREE


Hydra

Build Test Deploy

Nix

NIX - THE BIG THREE


Hydra

Build Test Deploy

▪ How to start? Too much to do! Can’t change everything immediately!

NixOps / NixOS

NIXIFYING EXISTING BUILDS

▪ Closed source software

▪ Buildscripts everywhere!

▪ Hardcoded locations

▪ Binary files/libraries


▪ Long builds

▪ Huge builds

▪ Network access

▪ Not just 1 language

HYDRA - CONTINUOUS BUILDS


HYDRA - CONTINUOUS BUILDS

▪ Why Hydra? ▪ All builds consolidated in one system, with one language to describe all▪ Less maintenance▪ Better traceability▪ Reproducibility of build▪ Caching of builds, very useful when composing builds▪ Integrates with our deployment tools

▪ Terminology▪ Projects▪ Jobsets / Jobs▪ Builds

▪ Features▪ Easy to clone jobsets and build branches▪ Ability to compose multiple builds as one logical build


HYDRA - WHAT DO WE BUILD

▪ Hydra structure:▪ 19 active projects

▪ 205 active jobsets


HYDRA - BUILDING PLATFORM


▪ What is built?▪ Builds, debug builds, regressions tests,

benchmarks, applications, binary releases

▪ Continuous builds▪ Each commit -> ~ 90 builds, ~ 3 buildhours

▪ Nightly integration builds▪ Each commit -> ~ 300 builds, ~ 120

buildhours

HYDRA - BUILDING CLIENT APPLICATIONS

{ platform_release ? "3.9.5", increment ? ./.}:with (import <config> {});let platform = getPlatform platform_release;in genericAppJobset { build = buildLBConfig { name = "increment-${version increment}"; src = increment; buildInputs = [ logicblox bloxweb ]; }; extraNixPath = { inherit increment; }; }

HYDRA - BUILDING CLIENT APPLICATIONS

Everything needed for deployment▪ LB application (installer)▪ Software dependencies (logicblox, bloxweb, nixpkgs)▪ Operating system (nixpkgs)▪ System configuration libraries for Logicblox (lbdevops)▪ Machine definitions (increment)

HYDRA - CHARON_CLOSURE (EHHHM, NIXOPS_CLOSURE)

HYDRA - BUILDING MACHINE IMAGES


▪ Also used for NixOS EC2/GCE/VB images:

▪ <nixpkgs/nixos/lib/make-disk-image.nix>

with (import <nixpkgs> {});

vmTools.runInLinuxVM (

stdenv.mkDerivation {

name = "need-root";

buildCommand = ''

echo Do stuff only root can do

'';

})

HYDRA - TESTING ON OTHER DISTRIBUTIONS


fedora19x86_64 =

testDownloadPackage

(diskImageFuns.fedora19x86_64 {

size = 2048;

extraPackages = [

"bash"

"java-1.7.0-openjdk"

"python"

"procps-ng"

];

});

HYDRA - BUILDING DOCUMENTATION


HYDRA - BENCHMARKING


▪ Dedicated benchmarking teamusing nix as benchmark composer

▪ Tracking performance over time

▪ Evaluate performance of PR’s

▪ Improvements▪ Scheduler

▪ Hydra EC2 provisioner

▪ Prioritization of builds

Softw

are

Hardware

Data

CPU x Memory x Storage

Systemx

Versionx

Benchmark

Scale x Data Set

BOB / HYDRA

▪ 2011 - 2013: Started out with 3 linux machines, up to 8 machines▪ On-premise hardware in the main office in Atlanta

▪ High initial investments

▪ Need IT people to handle hardware issues

▪ Need fast internet connection to scale out

▪ 2013-2015: 21 x Linux, 2 x Mac OS X▪ 8 general purpose

▪ 12 dedicated benchmark

▪ Can scale up on Amazon EC2, when needed▪ https://github.com/NixOS/hydra-provisioner


BOB / NUMBER OF BUILDS


BOB / NUMBER OF BUILD HOURS


NIXOPS

Tool to deploy networks of NixOS machines

▪ Single formalism for packaging, provisioning and configuration management▪ Easy to add software to deployment▪ Based on Nix package manager

▪ Expressive configuration language

▪ Supports separation of logical and physical aspects of a deployment▪ Reuse spec for multiple cloud backends

▪ Combined configuration and provisioning▪ Interaction between config and infrastructure


NIXOPS


+ Mixed spec

Logical spec

+ VirtualBox physical spec

+ EC2 physical spec

+ GCE physical spec

EC2

Google

Physical hardware

EC2

NIXOPS

▪ What to deploy?

▪ Where to deploy?


{ machine =

{ config, pkgs, ... }:

{ imports = [ ./nix-homepage.nix ]; };

}

{ machine =

{ deployment.targetEnv = "ec2";

deployment.ec2.accessKeyId = "ec2";

...

};

}

NIXOPS - ARGUMENTS

{ account ? "logicblox-dev"

, region ? "eu-west-1"

, instanceType ? "r3.xlarge"

, name

, enableBatch ? false

, production ? false

, ...

}:

{

…}


NIXOPS

▪ Do we use NixOps?▪ Hell yeah!

▪ Networks/clusters with up to 54machines per cluster


NIXOPS



▪ Last 30 days▪ 100 ‘nixops create’

▪ 88 ‘nixops delete’

▪ 1700 ‘nixops deploy’

▪ 300 ‘nixops destroy’

▪ 40 different people running deployments


NIXOPS



▪ Last 30 days▪ 100 ‘nixops create’

▪ 88 ‘nixops delete’

▪ 1700 ‘nixops deploy’

▪ 300 ‘nixops destroy’

▪ 40 different people running deployments


Type # Where?

Internal servers ~ 50 EC2, GCE, Hetzner, on-premise hardware

Client applications > 500 EC2, GCE

Dynamically scaled 0-1500 EC2

NIXOPS - OUR CURRENT SETUP

▪ Deployment servers▪ Log in via ssh, switch to specific deploy user via sudo, containing a group’s deployments

▪ Different deployment server for development / production


NIXOPS - OUR CURRENT SETUP



▪ Issues▪ Hard to use for non-technical users (project-managers, sales)

▪ Easy access to provider (AWS/GCE) credentials

▪ Inconsistent deployments (e.g. managing code clones)

▪ Hard to track what is going on


DEPLOYMENT - NIXOPS - OUR CURRENT SETUP



▪ Issues▪ Hard to use for non-technical users (project-managers, sales)

▪ Easy access to provider (AWS/GCE) credentials

▪ Inconsistent deployments (e.g. managing code clones)

▪ Hard to track what is going on

▪ New project: NixOps Dashboard▪ Web UI for NixOps , started November 1st

▪ Will be open-sourced once we have a usable version

▪ Created by Oussama Elkaceh and me.©2014. LogicBlox. All Rights Reserved.

DEPLOYMENT - NIXOPS DASHBOARD

▪ Improved security▪ By limiting the direct access to environments that have keys to access Cloud resources. Users

will use the Dashboard (RESTful API) to perform any deployment actions.

▪ Deeper operational visibility▪ Centralized, consistent and detailed view of the real-time status of all machines and

deployments.

▪ Audit trail▪ Easy access and insight into what is happening and what has happened with regards to

deployments. Also, all performed operations are tracked and visible to administrators.

▪ Usability, Consistency, Predefined Actions and Operations▪ Anyone can access and view the state of the existent deployments / historical deployments.


SUMMARY - LOGICBLOX AND NIX

▪ Nix / Hydra▪ Reproducible, composable builds, traceability

▪ NixOS▪ Reproducible system configuration

▪ NixOps▪ Automation of provisioning, creating reproducible networks

of machines, one click deployment!

▪ Nix helps us, so we help Nix!

▪ Giving back by contributing to:▪ Nix, Nixpkgs, Hydra, NixOps▪ Supporting the infrastructure running Nix*

▪ Binary caches, EC2 capacity


THANK YOU.


nixcon berlin 2015 - nix at logicblox

Technology