velocity conference nyc 2014 - real world devops
DESCRIPTION
In a world where agility has become a requirement, business and engineering demands have decreed the death of the “Department of No”. This talk will cover the journey of an IT Operations department from a single DevOps team to a business-wide cultural shift that has affected the way people interact and work with each other. In order to make sure that our DevOps initiative would be successful, we needed to make changes to the corporate organization, rearrange teams and roles in several areas, and make sure that everyone fully understand where we were being headed to. All these steps will be covered in this talk that will demonstrate some common pitfalls and misconceptions that jeopardize the DevOps adoption, particularly in large enterprises with several compliancy requirements and some outdated bureaucracy.TRANSCRIPT
Real World DevOpsRodrigo Campos (@xinu)
Velocity NY 2014
Hi I’m Rodrigo!
• Proudly crimping cables since 1992
• System Administrator
• Operations Director at walmart.com in Latin America
I’m not going to talk about
• Silver Bullets
• Unicorns
• Tools and Utilities
Instead I’ll focus on
• The challenges we faced
• How DevOps has helped us
• Bumps and bruises along the way
Our challenges
Our challenges
• Deploy a whole new platform before Black Friday
Our challenges
• Deploy a whole new platform before Black Friday
• Expand our private infrastructure to withstand web traffic spikes
Our challenges
• Deploy a whole new platform before Black Friday
• Expand our private infrastructure to withstand web traffic spikes
• Develop a company-wide agile mindset
Our challenges
• Deploy a whole new platform before Black Friday
• Expand our private infrastructure to withstand web traffic spikes
• Develop a company-wide agile mindset
• Rebuild trust between tech and business
Different companies, same problems
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
• BOFH attitude started to kick-in
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
• BOFH attitude started to kick-in
• Over the wall production deployments
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
• BOFH attitude started to kick-in
• Over the wall production deployments
• Tug of war between Dev and Ops
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
• BOFH attitude started to kick-in
• Over the wall production deployments
• Tug of war between Dev and Ops
• Lots of bureaucracy and red-tape
Some phrases (roughly translated)
Some phrases (roughly translated)
• “I'll go to the 9th floor to help you kick everyone’s a**”
Some phrases (roughly translated)
• “I'll go to the 9th floor to help you kick everyone’s a**”
• “I don’t care if it’s stable as long as it passed QA testing”
Some phrases (roughly translated)
• “I'll go to the 9th floor to help you kick everyone’s a**”
• “I don’t care if it’s stable as long as it passed QA testing”
• “This is operations’ job”
Some phrases (roughly translated)
• “I'll go to the 9th floor to help you kick everyone’s a**”
• “I don’t care if it’s stable as long as it passed QA testing”
• “This is operations’ job”
• “Consider this a favor”
How DevOps was seen
So what is DevOps anyway?
So what is DevOps anyway?
• Being loosely defined can actually be a good thing!
So what is DevOps anyway?
• Being loosely defined can actually be a good thing!
• For us: “DevOps is a culture that fosters communication, transparency and collaboration"
So what is DevOps anyway?
So what is DevOps anyway?
• Communication: I’ll stand up and go to your desk if I need to talk to you, you’re more than welcome to do the same!
So what is DevOps anyway?
• Communication: I’ll stand up and go to your desk if I need to talk to you, you’re more than welcome to do the same!
• Transparency: I want to know your problems, I’ll make sure you know mine!
So what is DevOps anyway?
• Communication: I’ll stand up and go to your desk if I need to talk to you, you’re more than welcome to do the same!
• Transparency: I want to know your problems, I’ll make sure you know mine!
• Collaboration: We’re in the same boat and I really mean it!
Operations Developers
UsThem InfoSec
Works on my Machine
We needed to change people's mindset
Source: http://youtu.be/Fx8OBeNmaWw
We needed to change people's mindset
We needed to change people's mindset
• Presentations to technology, business and human capital teams
We needed to change people's mindset
• Presentations to technology, business and human capital teams
• Reinforcing an inclusive and welcoming culture was essential
We needed to change people's mindset
• Presentations to technology, business and human capital teams
• Reinforcing an inclusive and welcoming culture was essential
• Everyone felt prepared and willing to change
We needed to change people's mindset
Typical Scenario
Infrastructure
Operations
Developers
Business
{ }
• Things go over the wall
• No alignment between tech and business
• One-way communications
• IT as a necessary evil (aka cost center)
We were already a little bit different…
Infrastructure
Operations
Developers
Business
{ }
DevOps Team
1st step
Infrastructure
Operations
Developers
Business
{ }Production Engineering
DevOps
2nd step
Infrastructure
Production ManagementDevelopers
Business
{ }
Production Engineering + { }
DevOps
InfoSec
• First and foremost: end the dossier culture
• Communication + collaboration + transparency = safer systems
InfoSec
• First and foremost: end the dossier culture
• Communication + collaboration + transparency = safer systems
• Integrate security tests into continuous integration
InfoSec
• First and foremost: end the dossier culture
• Communication + collaboration + transparency = safer systems
• Integrate security tests into continuous integration
• Rugged DevOps: http://slidesha.re/1r87Vs7
InfoSec
• No more FUD
• What is riskier to the business?
• Lack of speed is a risk too!
InfoSec
• ITIL: “You keep using that word, I don’t think it means what you think it means”
• People often blame ITIL as an excuse for badly designed processes
• Surprisingly ITIL compliance was not our biggest problem
Compliance
• Audits are still our biggest challenge
• In our case, we have a sponsor (the CTO)
• Someone (who happens to be me) will be held accountable if something goes wrong
Compliance
The bad apples
• We’ve adopted the “No asshole rule"
• There will be obstructive people that won’t embrace change
• If you can't get rid of them, simply let them fade into oblivion
KPIs
KPIs
• Deployment Frequency
• Speed of Deployment
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)
KPIs
• Deployment Frequency
• Speed of Deployment
• Deployment Success Rate
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)
KPIs
• Deployment Frequency
• Speed of Deployment
• Deployment Success Rate
• Service Recovery
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)
KPIs
• Deployment Frequency
• Speed of Deployment
• Deployment Success Rate
• Service Recovery
• CultureSource: Puppet Labs Blog (http://bit.ly/Z3gRpu)
Bumps & Bruises
Bumps & Bruises• DO:
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
• Use measurable KPIs
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
• Use measurable KPIs
• Explain to business areas what you’re doing
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
• Use measurable KPIs
• Explain to business areas what you’re doing
• Align with InfoSec and Compliance ASAP
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
• Use measurable KPIs
• Explain to business areas what you’re doing
• Align with InfoSec and Compliance ASAP
• Work hard on a sponsor
Bumps & Bruises
Bumps & Bruises• DON'T:
Bumps & Bruises• DON'T:
• Create a “Ninja Superstar team”
Bumps & Bruises• DON'T:
• Create a “Ninja Superstar team”
• Postpone communication
Bumps & Bruises• DON'T:
• Create a “Ninja Superstar team”
• Postpone communication
• Delay role changes
Bumps & Bruises• DON'T:
• Create a “Ninja Superstar team”
• Postpone communication
• Delay role changes
• Underestimate the managers' ability to influence behavior
• In the beginning it will be a lonely fight
• Some people will label you as the “hippie-anarchist”
Keep in mind
Results
Results• New eCommerce platform successfully deployed
Results• New eCommerce platform successfully deployed
• Only major eCommerce website in Brazil to have 100% uptime during Black Friday
Results• New eCommerce platform successfully deployed
• Only major eCommerce website in Brazil to have 100% uptime during Black Friday
• Increase in deployment frequency and success rate
Results• New eCommerce platform successfully deployed
• Only major eCommerce website in Brazil to have 100% uptime during Black Friday
• Increase in deployment frequency and success rate
• Business is incorporating agile methodologies
Any questions?
• Special thanks to: https://www.flickr.com/photos/51321237@N07/ https://www.flickr.com/photos/kh-67/ https://www.flickr.com/photos/jrparis/ https://www.flickr.com/photos/andrewdubock/ https://www.flickr.com/photos/rachelyoung/ !https://bitly.com/bundles/o_4p62vc3lid/5