vmworld 2015 san francisco - inf5432 - infrastructure as code - ban snowflake deployments

1© Copyright 2015 EMC Corporation. All rights reserved.

INFRASTRUCTURE AS CODEBAN SNOWFLAKE SERVERS

1© Copyright 2015 EMC Corporation. All rights reserved.

2© Copyright 2015 EMC Corporation. All rights reserved. 2© Copyright 2015 EMC Corporation. All rights reserved.

Jonas RoslandDeveloper AdvocateEMC {code}

@jonasrosland

Loves retro gamingTalks a lot

3© Copyright 2015 EMC Corporation. All rights reserved. 3© Copyright 2015 EMC Corporation. All rights reserved.

Reid VandewieleSolutions EngineerPuppet Labs®

@reidmv

Biking, HikingCode

4© Copyright 2015 EMC Corporation. All rights reserved.

5© Copyright 2015 EMC Corporation. All rights reserved.

WHAT IS DEVOPS?

6© Copyright 2015 EMC Corporation. All rights reserved.

AN IDEA OR COLLABORATIVE CULTURE/PHILOSOPHY BETWEEN

TECHNICAL TEAMS

7© Copyright 2015 EMC Corporation. All rights reserved.

UNIFIED PROCESSES AND TOOLS FOR FASTER END-TO-END DELIVERY

OF QUALITY SOFTWARE

8© Copyright 2015 EMC Corporation. All rights reserved.

NOT A JOB DESCRIPTION!

9© Copyright 2015 EMC Corporation. All rights reserved.

SO WHY SHOULD YOU CARE?

10© Copyright 2015 EMC Corporation. All rights reserved.

YOU CAN'T WAIT 6 MONTHS FOR A FEATURE UPGRADE/BUG FIX/ETC

11© Copyright 2015 EMC Corporation. All rights reserved.

SOLVE CONFIGURATION DRIFT ISSUES QUICKLY

12© Copyright 2015 EMC Corporation. All rights reserved.

KNOW EXACTLY WHAT'S WRONG, WHERE AND WHY

13© Copyright 2015 EMC Corporation. All rights reserved.http://46zwyrvli634e39iq2l9mv8g.wpengine.netdna-cdn.com/wp-content/uploads/2013/06/DevOps-infinity-loop2.png

14© Copyright 2015 EMC Corporation. All rights reserved.

LAST BUT NOT LEAST, IMPROVE TEAM COMMUNICATION

15© Copyright 2015 EMC Corporation. All rights reserved.

SO WHAT'S THE RESULT OF DEVOPS?

16© Copyright 2015 EMC Corporation. All rights reserved.

IT'S REALLY BIG ACTUALLY...

17© Copyright 2015 EMC Corporation. All rights reserved.

DEVOPS PRACTICES AND IT PERFORMANCE IMPACT

ORGANIZATIONAL PERFORMANCE

http://www.slideshare.net/realgenekim/2014-state-of-devops-findings-velocity-conference

18© Copyright 2015 EMC Corporation. All rights reserved.

LET'S START WITH

INFRASTRUCTURE AS CODE

19© Copyright 2015 EMC Corporation. All rights reserved.

WHAT DO WE DEFINE AS INFRASTRUCTURE?

20© Copyright 2015 EMC Corporation. All rights reserved.

21© Copyright 2015 EMC Corporation. All rights reserved.

22© Copyright 2015 EMC Corporation. All rights reserved.

23© Copyright 2015 EMC Corporation. All rights reserved.

• Deploy, monitor and connect together all the pieces needed to run services for the organization• Processes and tools for faster end-to-end delivery of

quality services• Automation comes built-in

WHY DEFINE INFRASTRUCTURE AS CODE?

24© Copyright 2015 EMC Corporation. All rights reserved.

• Desired state specified in text files• Autonomic (self-corrects to desired state)• State should be known through monitoring• Remove snowflake servers

HOW DO WE DO THIS?

25© Copyright 2015 EMC Corporation. All rights reserved.

• Easy to read and edit• Shareable• Can use standard version control like Git or SVN• Becomes executable documentation

WHY STORE THEM IN TEXT FILES?

26© Copyright 2015 EMC Corporation. All rights reserved.

node 'www2' {

class { 'apache': } # use apache module

apache::vhost { 'awesomewebsite.com': # define vhost

port => '80',

docroot => '/var/www/html'

}

}

SIMPLE EXAMPLE

27© Copyright 2015 EMC Corporation. All rights reserved.

user { root: ensure => present, password => '$ecretP@ssw0rd',}

CHANGE ROOT PASSWORD

28© Copyright 2015 EMC Corporation. All rights reserved.

MANUALLY CONFIGURED ENVIRONMENTS ARE LIKE A HOUSE

OF CARDS IN A CHINA SHOP

- NEAL FORD

29© Copyright 2015 EMC Corporation. All rights reserved.

• Deploying, provisioning and scaling automatically is virtually impossible if every server is unique• Adds friction between the requestor and the deployer• Mistakes happen• We're all human

SNOWFLAKE SERVERS

30© Copyright 2015 EMC Corporation. All rights reserved.

WHAT CAN HAPPEN IF YOU DON'T TREAT

INFRASTRUCTURE AS CODE?

31© Copyright 2015 EMC Corporation. All rights reserved.

UP TO 60% OF FAILURES ARE CAUSED BY HUMAN ERROR, NOT

HARDWARE FAILURE

32© Copyright 2015 EMC Corporation. All rights reserved.

EXAMPLE - KNIGHT CAPITAL GROUP

https://infocus.emc.com/wp-content/uploads/2012/08/KCG.jpg

33© Copyright 2015 EMC Corporation. All rights reserved.

• Manual deployment of new trading software• 7 of 8 servers correctly updated• Old function still alive on the 8th server led to…

KNIGHT CAPITAL GROUP - WHAT HAPPENED?

34© Copyright 2015 EMC Corporation. All rights reserved.

$440 MILLION LOSSIN 45 MINUTES

35© Copyright 2015 EMC Corporation. All rights reserved.

SO

36© Copyright 2015 EMC Corporation. All rights reserved.

TREAT YOUR INFRASTRUCTURE AS CODE

37© Copyright 2015 EMC Corporation. All rights reserved.

BARE METAL APP STACK

INFRASTRUCTURE AS CODE

38© Copyright 2015 EMC Corporation. All rights reserved.

• Razor– Text / JSON configuration– Bare-metal provisioning– Policy based

• Puppet– Text configuration manifests– Software configuration management

INFRASTRUCTURE AS CODE

39© Copyright 2015 EMC Corporation. All rights reserved.

OS – DNS – DHCP - DB

Storage – Network - Compute

INFRASTRUCTURE AS CODERA

ZOR

40© Copyright 2015 EMC Corporation. All rights reserved.

OS – DNS – DHCP - DB

Storage – Network - Compute

App Queue Business Logic

PUPP

ETINFRASTRUCTURE AS CODE

41© Copyright 2015 EMC Corporation. All rights reserved.

STARTING ENVIRONMENT:INFRASTRUCTURE AS CODE

One Razor ServerOne Puppet Server

42© Copyright 2015 EMC Corporation. All rights reserved.

LIVE DEMO

INFRASTRUCTURE AS CODE

43© Copyright 2015 EMC Corporation. All rights reserved.

INFRASTRUCTURE AS CODE

Boot PXE To Razor

OS Installed Puppetize! Decommissio

n

One-time Provisioning

Lifetime Management

44© Copyright 2015 EMC Corporation. All rights reserved.

Boot PXE To Razor

OS Installed Puppetize! Decommissio

n

One-time Provisioning

Lifetime Management

INFRASTRUCTURE AS CODE

45© Copyright 2015 EMC Corporation. All rights reserved.

• Self documenting infrastructure• You now have source code for how anything in your

datacenter is setup• Executable documentation

RECAP OF THE BENEFITS

46© Copyright 2015 EMC Corporation. All rights reserved.

ALRIGHT

47© Copyright 2015 EMC Corporation. All rights reserved.

THEN WHAT?

48© Copyright 2015 EMC Corporation. All rights reserved.

HOW ABOUTAUTOMATING SOME MORE?

49© Copyright 2015 EMC Corporation. All rights reserved.

• Jenkins CI• Travis CI• Codeship• CloudFoundry• OpenShift

DEPLOYMENT AUTOMATION

50© Copyright 2015 EMC Corporation. All rights reserved.

• Docker• Framework for application container management• HUGE community

CONTAINERS

51© Copyright 2015 EMC Corporation. All rights reserved.

• Splunk• Loggly• LogInsight• ELK Stack

– ElasticSearch– LogStash– Kibana

LOG MANAGEMENT

52© Copyright 2015 EMC Corporation. All rights reserved.

• vCenter Operations• New Relic• AppDynamics• DataDog

PERFORMANCE MANAGEMENT

53© Copyright 2015 EMC Corporation. All rights reserved.

• Nagios• Sensu• VictorOps• PagerDuty

MONITORING

54© Copyright 2015 EMC Corporation. All rights reserved.

• Slack• HipChat• Trello• Asana

COMMUNICATION

55© Copyright 2015 EMC Corporation. All rights reserved.

• Github• Quip

COLLABORATION

56© Copyright 2015 EMC Corporation. All rights reserved.

• And by smartly we mean cooperatively between Devs and Ops• Will help operators understand what's running and

happening in the infrastructure• Will give developers insight into the underlying

infrastructure and the way it behaves under load

USE THESE TOOLS SMARTLY

57© Copyright 2015 EMC Corporation. All rights reserved.

• Start talking about it• Choose low-hanging fruit• Learn, make wrong decisions, learn again• Involve the broader team

– Give books, presentations, blog posts etc.

SO HOW DO WE GET STARTED?

58© Copyright 2015 EMC Corporation. All rights reserved.

QUESTIONS?Jonas Rosland

Developer Advocate @ EMC {code}@jonasrosland