Upload File

into the cloud

45
Into the Cloud Making our Continuous Delivery fly.

Upload: tomas-riha

Post on 18-Jul-2015

63 views

Category:

Technology


0 download

Report

Tags:

TRANSCRIPT

Page 1: Into the cloud

Into the Cloud

Making our Continuous Delivery fly.

Page 2: Into the cloud

Tomas Riha

Architect @ VGT/WirelessCar

Passionate about creativity, change and improvement

Horrible at following instructions and performing repetitive tasks

MAJOR Project Liability

mail: [email protected]

twitter: @TomasRihaSE

blog: continuous-delivery-and-more.blogspot.com

Page 3: Into the cloud

Four Years ago

New Telematics Delivery Platform

Sent us on road to

Micro Services

Continuous Delivery

Page 4: Into the cloud

Infrastructure at that time

Low level of virtualization

“Two servers and a Oracle DB” per delivery

Split Operations and Development

Infrastructure and Application Architecture separated

Long process to get a server

Page 5: Into the cloud

Micro Service Architecture

Page 6: Into the cloud

Reality

Page 7: Into the cloud

Continuous Delivery

Build Release Deploy Test

Build Release Deploy Test

Build Release Deploy Test

Build Release Deploy Test

Build Release Deploy Test

Assemble Release Deploy Test

Service Pipes

100s of tests run in parallel testing

Solution Pipes

10s Use Case Requirements Verified

Page 8: Into the cloud

Here is your test server

Build Release Deploy Test

DB

Test

Server

Page 9: Into the cloud

Static virtual instances

Master Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Custom DeliveryEngine GUI

Master

Mind

Gerrit

Page 10: Into the cloud

Capacity Planning & Peaks

Master Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

?

Page 11: Into the cloud

Pampering the Snowflakes

Each static server needs maintenance.

Server Maintenance one of the top task the team does

Each server provisioned by different version of our chef scripts, where baseline

is a moving target

Page 12: Into the cloud

Continuous Delivery as a Service

Ok so this Continuous Delivery thing is nice....

....what does it take to deliver it to the entire organization as a Service?

400+ developers

Page 13: Into the cloud

Continuous Delivery as a Service

Blue skies, Sunshine

and ohh yes a CLOUD PROVIDER!!

Amazon Web Services

Page 14: Into the cloud

Load Tests

We have used AWS a bit for Load Tests

Page 15: Into the cloud

Go!

Continuous Delivery in the AWS Cloud

Demo Environments in the AWS Cloud

Page 16: Into the cloud

Build in Cloud host on premises and at partner

AWS

Volvo

Hosting

Partner

Hosting

Partner

Compatibility in

Environments!

Page 17: Into the cloud

What we got

Automated provisioning on our Continuous Delivery infrastructure using chef

Automated provisioning of our Jenkins jobs into build pipes

Rudimentary Image Baking Capabilities from our AWS Load Tests

Basic understanding of how our services behave in a cloud environment

Page 18: Into the cloud

What we need to do

Build a Virtual Private Cloud with networks

Connect VPC to Volvo Network

Migrate about

~800 Jenkins jobs

60 Jenkins master/slaves

1 Nexus server, 2 test databases, 1 Sonar, 1 Elk

2 custom applications, 1 MongoDB

Page 19: Into the cloud

Our approach

Change as little as possible during migration

Automate everything

Establish responsibility between Operations and Development

Page 20: Into the cloud

Responsibilities

Volvo

Network Team

Delivery Engine Team

Page 21: Into the cloud

Defining Application Topology

Load Balancer

Server Instance Server Instance

Data Storage

In Amazon Web Services

● Elastic Load Balancer

○ Security Groups

● Auto Scaling Group

○ Launch Configuration

○ Amazon Machine Image

○ Security Groups

● Data Storages

○ Simple Storage Service

○ Relational Data Service

■ Oracle

■ Postgres

○ Custom Databases

■ MongoDB

○ Security Groups

Page 22: Into the cloud

Defining Application Environment

Load Balancer

Server Instance Server Instance

Data Storage

In Amazon Web Services

● Virtual Private Cloud

○ Subnets

● Auto Scaling Group

○ Scaling Rules

■ Up/Down

■ Max/min

○ Instance Size

■ Mem/Cpu

● Data Storages

○ Size

■ Mem/Cpu/Storage

Page 23: Into the cloud

Infrastructure as Code

Dev TestLoad

TestUAT

INT

Test

Provisioning Interface

Vagrant Impl Cloud Provider Impl PDF Document Impl

Topology

Spec Env Spec

Legacy

PROD

Environment Definition

Networking

Loadbalancing

Servers

Databases

Location

Integration

Scaling

Page 24: Into the cloud

Infrastructure as Code

TestLoad

Test

Provisioning Interface

AWS Cloudformation Impl

Topology

Spec Env Spec

Demo

Delivery

Engine

Tool

Page 25: Into the cloud

Baking Machine Images

Git Repo Base

Packer

Scripts

Bakery

Instance

Git Repo Delta

Packer

Scripts

AMI Alpha

RHEL

AMI

Bakery

Instance

AMI Delta

Page 26: Into the cloud

Baking Machine Images

RHEL

AMI

VGT Base

Java Base Jenkins Base Nexus Graphite ELK

Jenkins Master Jenkins SlaveApplications

Applications

Applications

Page 27: Into the cloud

Chef Issues

Never share same scripts for baking image that go from known

state to a delta with scripts that manage snowflakes!

Page 28: Into the cloud

Phase 1

Politics, Cross Organizational handovers & PoC

Way too much time

Phase 2

Build & Automate Setup

Build & Tear Up the VPC a few 100 times

1.5 months ~ 2 engineers

Phase 3

Migration

Start moving over the repositories.

Add .delivery-engine.yml files to repositories

Disable build pipes in old environment

1 month ~ 3 engineers

The Migration

Page 29: Into the cloud

Daily Cost tracking during migration!!

Did we do something really stupid yesterday?

Bosses love you when you report cost and think about optimizing cost

Cost Tracking

Page 30: Into the cloud

Prepare

Automate

Verify

Rip The Bandaid!!!

Lessons Learnt

Page 31: Into the cloud

Since the migration started we have not launched

one single instance through the AWS Console.

Console only used to terminate instances, never to

create instances.

Done well

Page 32: Into the cloud

Delivery Engine after Migration

More or less the same setup

Different distribution of jobs

Auto Scaling Workers

Orchestration Worker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker WorkerWorker Worker Worker

Orchestration Worker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

WorkerWorker

Custom DeliveryEngine GUI

Master

Mind

Gerrit

Page 33: Into the cloud

Immediate Benefits

Maintenance down at least 80% at least due to worker recycling

No more capacity planning

Orchestration Worker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker WorkerWorker Worker Worker

Orchestration Worker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

WorkerWorker

Custom DeliveryEngine GUI

Master

Mind

Gerrit

Page 34: Into the cloud

Not High Availability

Setup is still stateful

Orchestration Worker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker WorkerWorker Worker Worker

Orchestration Worker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

Worker

Worker

WorkerWorker

Worker Worker

WorkerWorker

Custom DeliveryEngine GUI

Master

Mind

Gerrit

Page 35: Into the cloud

250 Git repositories building, testing, releasing

Components, Applications, Solutions

For closer to 150 developers

Usage

Page 36: Into the cloud

Growth during migration wasn't expected but was about 50 repos

Over 100 Git repos added after migration

New teams constantly onboarding to Delivery Engine

Expecting 200 plus developers by end of year

Growth

Page 37: Into the cloud

Grow number of users and content not the size of our Team Size!

We do it through a

Event Driven, Stateless, Application developed using ALL the

Continuous Delivery principles.

Using Continuous Delivery to Deliver Continuous Delivery.

We need to ensure scalability and availability

Page 38: Into the cloud

Stateless Build Environment

Pipes as Code

Unit test pipe task

Integration test pipes

JobPipe as

CodeClient

Page 39: Into the cloud

Stateless Build Environment

Build Data and reports away from Build Environment

JobPipe as

CodeClient

Page 40: Into the cloud

Stateless Build Environment

No Dependency between GIT Repository and Build Job

Only one job exists

JobPipe as

CodeClient

Queue

Gerrit Listener

Page 41: Into the cloud

Job

Stateless Build Environment

Aws Tools

Pipe as

CodeClient

Test Environment

Queue

Gerrit Listener

Split Test Environment from Build Environment

Page 42: Into the cloud

HA Build environment

Test Environment

Rabbit MQ Cluster

Coordinator Jenkins

Factory

ForemanJenkins

WorkersJenkins SlavesWorkers

Jenkins SlavesWorkersJenkins Slaves

Gerrit Factory

ForemanJenkins

WorkersJenkins SlavesWorkers

Jenkins SlavesWorkersJenkins Slaves

Test Environment

Test Environment

Test Environment

ELK Graphite

S3Master

Mind

Custom DeliveryEngine GUI Slack

Page 43: Into the cloud

From idea to rolled out in about 2 months, 1 man month of work

Without AWS not possible

HA Rewrite

Page 44: Into the cloud

+

Lead time for Infrastructural change drastically reduced

Ability to build HA and scalability on a level previously not possible

No more capacity planning

Demo Environment using same tooling and fully up and running

QA and Integration testing environments coming up as well

DevOps

Amazon Web Services works GREAT!

-

Different tooling for creating aws infrastructure to creating in house infrastructure

Retrospective

Page 45: Into the cloud

Questions?

here: @TheBar

mail: [email protected]

twitter: @TomasRihaSE

blog: continuous-delivery-and-more.blogspot.com

Thats it!


Java EE 7 - Into the Cloud

Cloud-enabled Development: Putting the Agile into the Infrastructure

Taking Education Into the Cloud

Bringing the cloud into the classroom

your Services into the Cloud - AT&T · your into the Cloud Tap in and transform ... your mobile reach. ... Scale your services into the cloud to meet dynamic

Virtualization Into Cloud

Venturing into the cloud

Into The Cloud

In-Vehicle Infotainment Journeys Into the Cloud

‘Insights’ into Cloud Security · If you are deploying virtual servers (IaaS) into the cloud, you as the cloud customer will be responsible for most of the security controls for

Elastic HPC Extending the Cluster into the Cloud

SharePoint Extending into the Cloud

MOVING SOCIAL NETWORKING APPLICATIONS INTO THE CLOUD

CloudX – Expand Your Cloud into the Future

Getting into the cloud is like getting into a swimming pool

Visual Frog - Leap into the Cloud

INTO THE CLOUD: NETWORKING MUSEUM INFORMATION 04.06.09

Plug into the Cloud with Oracle Database 12ci.dell.com/.../en/Documents/plug-into-cloud-12c.pdf · Plug into the Cloud with Oracle Database 12 c Introducing Oracle Database 12c Oracle

ENTERPRISES ADVANCE INTO THE CLOUD - Oracle · Unisphere Research is the market research unit ... 3 Enterprises Advance into the Cloud: ... private cloud computing services in place—defined

Plug into the Cloud with Oracle Database 12c Plug Into The ... · PDF filecloning of pluggable databases can occur almost instantaneously. ... Plug into the Cloud with Oracle Database

Unifying Events and Logs into the Cloud

Out of the Basement and into the Cloud

Spring Into the Cloud

Plug into the Cloud with Oracle Database 12c Plug Into The ... · Plug into the Cloud with Oracle Database 12c Introducing Oracle Database 12c Designed for the cloud, Oracle Database

5 workloads to migrate into the cloud

Extend Your Enterprise Into The Cloud Presentation

Into the Cloud - An introduction to cloud computing

Migrating Into the Cloud: The Brownfield vs. Greenfield Opportunity

INTO THE CLOUD - IAPP · Cloud Adoption and Enhancing Compliance Posture in the Cloud ... •Application usage and user ... –We must find a way to transfer data into the cloud in

Journey into the Cloud

Into the cloud white paper

GETTING YOUR HEAD INTO THE CLOUD: SOLUTIONS FOR … · PYTHIAN CLOUD SERVICES GETTING YOUR HEAD INTO THE CLOUD: SOLUTIONS FOR OPTIMIZING GROWTH AND PERFORMANCE HIGHLIGHTS • Develop

Stepping into the Cloud - University of Washington

Get Your Head Into the Cloud

The Cloud: Releasing Software into Controlled Chaos