martin höfling tng technology consulting …...© tng technology consulting gmbh 3 / 54...

Martin HöflingJohannes Ebke

From Zero to Webscale:Evolving a Continuous Delivery Pipeline

TNG Technology Consulting

2 / 54© TNG Technology Consulting GmbH

What this talk is about

Introduction

►Context

►Challenges

►Technology

Evolution towards fully automatic deployment

Remarks & Conclusions

Overview

Introduction: Context

TNG Technology Consulting GmbH

~230 regular employees,

►> 50% with a PhD

►Physics, Mathematics, Computer Science

Founded in 2001, grown ever since, CAGR ~ 25%

Place of Business: Munich – Heart of Bavaria

Value-based consulting partnership

Focus on high end IT

Startup Characteristics

Few people

Moving targets

No dedicated operations

Motivated tech addicts

Fast decisions - flat hierarchy

Everybody feels responsible(ideally…)

The Product

Scaling

Introduction: Challenges

…sometimes many, sometimes fewer users

Complexity

Introduction: Challenges

Multiple backend types

Several Databases and Queues

Despite of Complexity: Daily Releases

What Technology are we using?

ReactWebpack

TornadoPython3.5 Elasticsearch

MongoDB RedisJenkins

Packer

Docker

SaltStackTerraform

Boto3 AWS

Vagrant

Icinga2 ELK

Backbone

Introduction: Technology

Introduction

►Phase 1: Starting up

►Phase 2: Automating deployment to the cloud

►Phase 3: Fully automated infrastructure setup

Overview

Starting Up

Set up CI pipeline

►Automatic testing

Create deployable artifact

►Deploy „somewhere“:

►Reproducible

Starting Up

Infrastructure at the Beginning

Starting Up

Local Cloud

developers

CI / Jenkinstest instance

production single EC2

What is SaltStack?

Starting Up

Quoting saltstack.com:

… „scalable and flexible configuration management“

… „event-driven automation of CloudOps, ITOps and DevOps“

Why SaltStack?

Key features:

Declarative configuration state management

Scalability to thousands of managed machines

Event driven reaction and reconfiguration:

►Predictive Orchestration

Nice to have:

Provider agnostic (Cloud-) Ops

Implemented / extensible in Python

Large, friendly community

Starting Up

Declarative Configuration: Salt States

Starting Up

/etc/secrets: file.managed: - mode: 600 - contents: | root:secret

nginx: service.running: - reload: True - watch: - file: /etc/secrets

/etc/secrets: file.managed: - mode: 600 - contents: | root:secret

nginx: service.running: - reload: True - watch: - file: /etc/secrets

ResourceState Function

Parameter

Dependency

Event Driven Orchestration with Salt

Starting Up

Salt Master Salt Minion loadbalancer-12

Salt Minion loadbalancer-12

subscribe

Salt Minion app-42

subscribe

Salt Minion monitoring

subscribe

Master Event Bus

Master - Minion Event Bus

Salt Minion elasticsearch-11

Starting Up

subscribe

Salt Minion app-42

subscribe

Master Event Bus

From: AWS type: new machine launched machine: elasticsearch-12

Starting Up

subscribe

Salt Minion app-42

subscribe

Master Event Bus

From: AWS type: new machine launched machine: elasticsearch-12

To: elasticsearch-12 Function: install_es Arguments: -version: 2.1.3

publish job

Starting Up

subscribe

Salt Minion app-42

subscribe

Master Event Bus

From: elasticsearch-12 JobId: 3032 Result: OK, installed

To: elasticsearch-12 Function: install_es Arguments: -version: 2.1.3

Starting Up

subscribe

Salt Minion app-42

subscribe

Master Event Bus

From: elasticsearch-12 JobId: 3032 Result: OK, installed

To: app-* Function: add_ES_to_config Arguments: -server: elasticsearch-36

publish job

Starting Up

subscribe

Salt Minion app-42

subscribe

Master Event Bus

From: app-[1..42] JobId: 3034 Result: OK, config

To: app-* Function: add_ES_to_config Arguments: -server: elasticsearch-36

Automating deployment with Salt

Local Clouddevelopers

test instances

production single EC2CI / Jenkins

Salt master

Salt / ØMQ Artifact

Starting Up

Starting Up - Lessons Learned

Keep testing and production in sync

Automate deployment early

►Select and get familiar with your deployment tool

Make the deployment accessible for the team

► Infrastructure as Code from the beginning

Starting Up

Introduction

►Phase 2: Automating deployment to the cloud

Overview

Automating Deployment into the Cloud

Rationale:

Cloud Machines are Cattle, not Pets ...

► ... they must be replaceable quickly

(Bill Baker, Microsoft)

Approach:

Automate dynamic resource configuration (e.g. deployment):

► resource allocation, software installation, …

► distributed systems with many moving parts

Manual configuration of static resources:

►VPC, VPN, Firewall, Routing and DNS

Modularize CI/CD

Automating Deployment

Dockerized Jenkins Pipeline

Reasons:

Flexibility

►Versioned and adapting CD pipeline

►Deployability in case of hardware failure

Transparency

►Docker test cluster also runs locally

Rapid feedback

►Scale beyond single machine

►Parallel integration tests

Dockerized Jenkins Pipeline – Build

test instances

production instance

Test / Build

Artifact

Dockerized Jenkins Pipeline – Deploy

test instances

production instance

Salt / ØMQ CI

Salt master

Salt Cloud – Machine Lifecyle made easy

Cloud Profile

►Different providers possible

Cloud Map

►List instances for each profile

Salt-cloud

►Creates / destroys machines

► Installs Salt

►Attaches machine to salt-master

appserver: provider: aws image: ami-bdc9dad1 size: t2.large

loadbalancer: provider: gce image: centos-6 size: n1-standard-1

appserver: provider: aws image: ami-bdc9dad1 size: t2.large

loadbalancer: provider: gce image: centos-6 size: n1-standard-1

loadbalancer: - lb1 - lb2

appserver: - app1 - app2 - app3

loadbalancer: - lb1 - lb2

appserver: - app1 - app2 - app3

Salt Cloud from Dockerized Salt Container

test instances

production instance

Salt / ØMQ CI

Salt master

Salt Cloud new instance

Distribute your Application

Reasons:

Improved availability

Horizontal scaling

Problems:

Adds complexity to your setup

►Deployment often requires complex orchestration

How to Create a Distributed Application?

Create Distributed Applications with Salt Cloud

all-in-one instances

distributed instance

Salt / ØMQ

Salt master

new instance master

new instance

Create Distributed Applications with Salt Cloud

Salt / ØMQ

Salt master

instance master

trigger salt-cloud

salt cloud

Deployment of a Distributed Application

Separate non critical and critical steps

Non-critical:

►Build and Push Frontend / Backend Package

►Update Instance Master

Critical step

►Apply configuration to (critical) systems

► Independent of local CI

Salt / ØMQ

Salt master

instance master

Salt / ØMQ

Salt master

instance master ELBs

trigger activation

Autoscaling

Goals:

Only use computing resources required at the moment

Automatically replace dead or disconnected instances

Approaches:

Autoscaling solely with Salt

AWS Autoscaling technology

Deployment with Autoscaled Application Tier

Salt master

instance master

Salt / ØMQ app template

trigger activation

Deployment with Autoscaled Application Tier

Salt master

instance master

app AMI

trigger activation

Python Salt / Boto3

Deployment – Third Party Services

Salt master

instance master

app AMI

ASGsElasticache

Python Salt / Boto3trigger activation

Automating Deployment – Lessons Learned

Know the capabilities and limits of the different services

►what scales and what does not scale

Integration of Ops tools in CI not trivial

►Salt Return Codes are “surprising”

Frequent deployments

►… are key to Reliability

Reliability

►… is key to frequent deployments

Introduction

►Phase 2: Automating deployment to the Cloud

Overview

Fully automated Infrastructure Setup

„Click here for New Datacenter“

Trigger:

Separate testing and production into two accounts

►Restricting access to production to a team subset

Define static infrastructure as code

Salt for Infrastructure – and its limits

Use and extend Salt fordynamic configuration ofinfrastructure:

DNS, ELB, ASG

Orchestration Scripts:

Deployment, Backup &Restore

Problem:

►Static Infrastructure

#!python

import boto3#!python import boto3

#!python

import boto3

Static Infrastructure (~200 Resources per App Instance)

Local CloudDevOps team

CDN distributions

Route53 DNS Zones

S3 Storage

multiple Acounts

multiple VPCsinternet / NAT / VPN

gateways

VPN connections

routing / subnets

SEIPs, Security Groups

Terraforming the Static Infrastructure

Local CloudDevOps team

CDN distributions

Route53 DNS Zones

S3 Storage

multiple Acounts

multiple VPCs

AWS API

internet / NAT / VPN gateways

VPN connections

routing / subnets

SEIPs, Security Groups

Terraform in Action

$ terraform plan

$ terraform apply

Differences to SaltStack:

Keeps track of state

Implicit dependencies by referencing resources

Plan changes beforehand to avoid disruption

resource "aws_vpc" "main" { cidr_block = "10.1.0.0/16" tags { Name = "Testing" }}

resource "aws_internet_gateway" "gw" { vpc_id = "${aws_vpc.main.id}"}

resource "aws_vpc" "main" { cidr_block = "10.1.0.0/16" tags { Name = "Testing" }}

resource "aws_internet_gateway" "gw" { vpc_id = "${aws_vpc.main.id}"}

Terraform – Lessons Learned

Modularize: Keep testing andproduction as similar as possible

Make sure you check the documentationthat the features you require are there.

Terraform enabled us to easily manage~200 resources and ~500 dependencies.

Introduction

Overview

Distribute Knowledge in the Team

Not all need in depth knowledge

►Two of us focus on CI/CD and infrastructure

Make deployment visible and accessible

► Jenkins pipelines everybody can use

►Check in deployment code into git

Simplify deployment wherever possible

Why Automating and DefiningInfrastructure as Code?

In software development...

… you test your code – right?

►Automatic & reproducible

… you review your code?

►Quality and Maintainability

So why is infrastructure often…

… not reviewed?

… not automatically tested?

… so broken?

We have the tools!

Let‘s go and fix this!

Automate!

Thank you for listening!

Questions?

Contact

Martin Höfling

martin.hoefling@tngtech.com

@martoss13

martinhoefling

Johannes Ebke

johannes.ebke@tngtech.com

JohannesEbke

martin höfling tng technology consulting …...© tng technology consulting gmbh 3 / 54...

Documents

consulting • technology • managed services ·...

microsoft project server enterprise project solution shawn...

health information technology consulting

robot operating system version 2 - tng | tng technology...

hardware hacking team - tng technology consulting ·...

effektiver tooleinsatz für scrum-projekte im...

caster technology consulting en

athos consulting technology

technology consulting in the community consulting … ·...

tai hou-tng - co-creating with community in technology...

lernen sie von ihren kunden, sonst tut es ihre konkurrenz...

ntepa - appendix e...tng limited (tng) is committed to...

emstell technology consulting

dynamic soaring - tng | tng technology consulting gmbh

hjk consulting engineers - project management – technology...

cover consulting services technology solutions … document...

enterprise technology consulting

technology needs for lynx: mirrors, coatings and...

job title: technology consulting analyst, accenture...

aus aus der lokalen ptimierungsfalle - tngtech.com · tng...