devops irl (in real life) by sarah zelechoski and jonathan frappier - boston vmware user conference...

DevOps IRL

Sarah Zelechoski & Jonathan Frappier

Sarah ZelechoskiLead Site Reliability Engineer

@ReactiveOps

@szelechoski

/in/szelechoski

reactiveops.com

github.com/sairezgithub.com/reactiveops

Jonathan Frappier

Senior Technical Consultant, Education Services @EMCCorp

@jfrappier

/in/jonathanfrappier

#vBrownBag professionalvmware.combit.ly/vbrownbagitunes

#Commitmasgithub.com/jfrappiergithub.com/commitmasbit.ly/Commitmas2

Agenda

● Trust Exercise● This is a story all about how

we...● Discovered DevOps● Enabled business through

automation● Supported new business

requirements in weeks not months (or years)

*Disclaimer

As always…

● Assume when we say how awesome something is, we really mean:

“In most cases…” everything is awesome “if it meets your business requirements”

● If you think we are repeating things, it’s because we are, but we did that on purpose

Trust Exercise...

via 9gag.com

It’s coming...

Actually it’s here!

Actually it’s here!

Actually it’s here!

Actually it’s here!

What is DevOps?

Cultural● what your company look like today?

○ interactions between groups○ silos of expertise○ responsibility/blame○ bottlenecks/waiting○ methods of communications○ different toolsets#notDevOps

developer <> build <> QA <> ops <> support <> marketing <> financeWho are your customers?

developer <> build <> QA <> ops <> support <> marketing <> finance

“DevOps is a cultural and operational model that fosters collaboration to enable high-performance IT to achieve business goals”

Brian Gracely, Wikibon

Cultural● management supported

○ no more silos○ cross-pollination between groups○ understanding motivations○ increase communication○ shared responsibility○ learn to trust#DevOps

Operational

● from feature code to deploy

○ all groups have to work in concert

to get product out the door● pipeline has to function

○ consistent effort■ no bottlenecks■ waiting derails productivity

○ be continuous■ fits and starts■ feedback cycle

#DevOps

Collaboration

● everyone on the same page ○ what you’re working on

■ new feature■ new tests■ new requirements■ new tools■ new campaign

○ how it affects me● focus effort

○ not on task○ on supporting the project

#DevOps

Enable#DevOps

● to achieve business goals○ make your customers happy○ everyone is your customer

■ marketing■ support■ sales

○ help me help you

Big DevOps

ZOMG!

DevOps IRL

“There’s no new problem that someone hasn’t already had and written about it in a book”

Will Smith

● you are not a beautiful and unique snowflake ○ business needs in common with others○ working with the same groups

● you don’t have the resources or time to develop new tools new and techniques○ focus on your own product dev!

● techniques and tools already exist to help○ open source software○ pipeline/workflow examples○ collaboration/communication

methodologies● community

○ real stories of success○ help when you need it

● if it’s sink or swim○ don’t jump into the deep end first○ nothing wrong with swimming

lessons○ use existing tools to help buoy you○ good technique will move you faster

and more efficiently○ use the buddy system○ practice makes perfect

Storytime

Cultural

StorySJ -2

● Devs dev’d● QA QA’d● IT IT’d

StorySJ

● Management support● Team collaboration● Empathy● Blameless post-mortems● Continuous improvement● Go play darts...or pool...or

cards...or make a run

Techniques● Join team meetings/standups

(hopefully standups)● Retrospectives● Shared task boards● Go sit somewhere else● Go play darts...or pool...or

cards...or make a run

Tools● Slack, HipChat, giant group

Lync chat● Social media● Conferences, meetups● Human-ie stuff like talking to

other “humans”

Community

● The Phoenix Project● Mentorship Programs

○ bit.ly/mentorex1○ bit.ly/mentorex2○ vsensei.expert *

● Etsy Blameless Post-Mortems○ bit.ly/ETSYbp-m

● 8 DevOps Secrets○ bit.ly/8devopsec

● Breaking Down Silos Doesn’t Happen Overnight○ bit.ly/breakingsilos

● Unicorns and the language of otherness○ bit.ly/igniteunicorns

Operational

StorySJ -2

● hand-crafted artisan infrastructure○ installs done by who? when? ○ vmware fusion, samba, postgres

● devs/qa○ had local VMs or

workstation/system setup

● “staging” ○ rolled by hand / modified by all

parties

● production was under CM○ layered bash scripts

○ long running instances / never

updated

● deployment

StorySJ

● codify infrastructure○ make repeatable from dev to qa

to staging to production

○ try for no hand-rolled or modified

anything

● config management ○ idempotent ansible

● strict procedure ○ hand rolled gold images○ eventually moved to packer

● build products for everyone○ vagrant boxes○ vm templates

● convincing everyone to participate

Techniques

● source control○ quality control○ infrastructure reviewed/audited○ input from all groups

● leverage automation ○ avoid repeating effort○ eliminate hand rolled resources

● configuration management● immutable infrastructure

○ for everything (even vcenter)○ dev has same box as production

Tools

● source control○ github

● automation○ PowerShell/PowerCLI (vcenter)○ ansible○ packer

● configuration management○ puppet/ansible/chef

● immutable infrastructure○ vagrant, VMware, terraform

vSphere provider○ packer

Community

● About Immutable Infrastructure via Codeship○ https://blog.codeship.

com/immutable-deployments/

● devopsdays○ http://www.devopsdays.org/

● The Ship Show○ http://theshipshow.com

● Hacking Ansible○ http://t.co/IFYdUBImqn

Collaboration

StorySJ -2

● Switch replacement take 1…○ Teams were siloed○ Dev was... something… ¯\_(ツ)_/¯

○ No knowledge of work process

between groups

StorySJ

● Switch replacement take 2 except…○ Management support on culture

○ Dev was Agile(ish) and Ops was

Kanban(ish) (different but

compatible)

○ More knowledge of work process

between groups

● Inserted into the Dev process

“The kind of trust that makes a team great is vulnerability trust...the ability to say I don’t know the answer” - Patrick Lencioni

Techniques● Lots of methodologies

○ Agile, Kanban○ Okay to morph what fits your org

● Human communication…○ Yup...I’m gonna talk about pool

and again

Tools● Trello, Asana● Jira (all things for small

teams), Zendesk● Git, GitHub, any SCM● Slack, HipChat● Whiteboards & Post-Its

Community● bit.ly/usingtrello● bit.ly/brenderasana● https://geekspeak.slack.com● http://community.emccode.

com/ ● #vBrownBag Slack coming

soon

Enable

StorySJ

● everyone a customer● enabled dev with repeatable

dev boxes ○ moved to vagrant○ provided upgrade paths

● enabled build & qa with standardized resources○ eliminated IT waste

○ long standing boxes and archived

versions were unnecessary○ immutable vm guests○ shorter lead time

HELP ME HELP YOU

HELP ME HELP YOU

StorySJ

● enabled external customers○ coherent monitoring and alerting○ less downtime○ quicker releases (hotfixes, etc)○ heartbleed, critical security fixes

● enabled ourselves○ made everything repeatable and

reliable○ free DR○ self documenting

Techniques

● Shared responsibility ○ dev helping with alerts/support

○ ops helping with feature

requirements

● Feedback loop ○ including sales/marketing

● Immutable infrastructure● Continuous Build/Deployment● Performance for Business

Tools

● Ticketing & Tasking○ JIRA, Zendesk, Asana, Trello

● Logging○ loggly, logentries, ELK, papertrail

● Metrics○ statsd, Graphite, Jenkins

● Infrastructure Monitoring○ Splunk, Nagios, Veeam One,

Sensu

● Application Monitoring○ New Relic, App Dynamics

● Alerting & SLA Tracking○ Pager Duty, Pingdom

Community● Foundations of a Successful

DevOps Team○ https://www.pagerduty.

com/blog/devops-foundations/

● The Fallacy of Fast○ https://www.youtube.com/watch?

v=iIT18m6bHmw

Where do you Start?

Culture● Own it● Make a new friend● Lunch ‘n Learn / Beer ‘n Learn● Desk swap / desk move● Get in on meetings● Goal board

Operational

● Infrastructure as code○ CM and SCM of all things○ collaborate in Github (or other)

■ everyone has access + input■ @ people for conversations■ enforce code review

● Stop using the GUI○ start writing your VM infra as

code○ PowerCLI or ansible 2○ orchestrate stand-up/tear down

○ try orchestrating the GUI with

Onyx

● Start small○ pick a task you have to do

manually and automate it

○ if you find yourself hardcoding

something, extract it to a variable

Collaboration

● Pick a starting point○ Agile? Kanban? Whatever…

● Pick a tool, try it for a small project○ Asana, Trello, Jira…○ If it doesn’t work, punt

● Find the process and tools that work for you○ Talk to the community

○ Remember someone has tackled

this problem○ Different != wrong, ask why

● Talk more○ Slack, HipChat, actual talking

Enable

● Pick a project with organization wide visibility

● Give everyone a part to play○ how do they contribute?○ how do they give feedback?○ how do they get help?○ how do they get work done?

● Everyone knows the goal○ agree on well defined success

criteria

○ turns in to SLA or KPI. business

enabled.

● Over-communicate ● Trust, empower, encourage,

reward○

Learning

● #vBrownBag DevOps Series○ http://professionalvmware.

com/vbrownbag-devops-series/

● #Commitmas○ http://bit.ly/Commitmas2

○ https://github.

com/commitmas/30-days-of-

commitmas-2015

● Pluralsight.com● CodeSchool.com● FreeCodeSchool.com● LinuxAcademy.com/devops

Thank You