does14 - aimee bechtle and bill donaldson - the mitre corp
DESCRIPTION
Bill and Aimee's Excellent DevOps Journey Bill Donaldson, Senior Principal Engineer, and Aimee Bechtle, Principal Software Engineer, The MITRE Corp In their presentation Bill Donaldson and Aimee Bechtle will share a summary of their tumultuous journey that has resulted in 75% of their corporate applications utilizing Continuous Integration with automated deployments, a 70% reduction in labor, and 288% reduction of cycle time. They will support these numbers with charts depicting the deployment volumes over the course of a 1.5 year adoption. They will share how by selecting the right applications, approach, and people, and using creative ways to advertise success, the new capabilities were embraced and adopted in their organization. The presentation will conclude with how Aimee has transformed her team to support Continuous Integration with auto deploy. Additionally, she will share why the 25% remain in the manual system and how she’s pursuing Continuous Delivery. In 2011 Bill Donaldson and Aimee Bechtle were leading development and deployment operations teams in The MITRE Corporation’s corporate IT department. MITRE is a not-for-profit corporation whose mission is to provide the US Government world class Systems Engineering. MITRE’s IT was running well with a mature and stable release process. Bill’s team had successfully adopted agile development practices and was adept at producing high quality software code quickly. However, the successful adoption of agile introduced a problem between the developers and the deployment operations team lead by Aimee. Developers had to “hurry up and wait” on operation’s 24-hour SLA to build and deploy their apps. The SLA frequently expired due to multiple handoffs and human errors. Tired of the bottlenecks and lengthy cycle times one day Bill said to Aimee “24 hours is 23.5 hours too long”. This simple requirement sparked the vision for a transformation of MITRE’s software delivery process. The deployment operations team was employing a manual, mature and repeatable process that had been in place for over 10 years. In fact, in 2012 the Release & Deployment Management process scored the highest among all the processes in an ITIL process evaluation. So why would Aimee be motivated to change? Because MITRE’s unique mission and culture values innovation, active cost management, and establishing MITRE as a showcase to their Government customers. Bill from Dev, and Aimee from Ops, partnered to successfully deliver an enterprise continuous integration and automated deployment capability. Aimee took Bill’s requirements and declared them as acceptance criteria. Together they formed an influential, cross-functional team that was critical to building momentum and adoption within the organization. And together they experienced the pitfalls and challenges of implementing change, working against the resistors and laggards. At times project goals were in question but through determinationTRANSCRIPT
MITREBill & Aimee’s
Excellent DevOps Journey
How a Not-for-profit Corporation
Adopted DevOps
Aimee [email protected]
Bill [email protected]
Approved for Public Release; Distribution Unlimited. Case Number: 14-3400 ©2014 The MITRE Corporation. ALL RIGHTS RESERVED.
MITREWhat is MITRE?
MITRE is
a not-for-profit
organization that operates
Federally Funded Research
and Development Centers
MITREWho Are We?
I’m BillI’m
Aimee
DevOps
Road
Trip!
MITRE
Production
1. Code, Build,
Unit Test
1...n
2. Commit
3. Run Build,
Development Test
1...n
3a. Export Code4. Writes/Maintains
CM Form, Creates Change
Ticket
6. Rejects
CM Form
17. Install
Failed
7a. Export Code
14. Email Instructions
9. Execute
Task
18. Export Code
19. Install Failed
Rollback & Restore
Integration
Development
DBA
5. Receives & Reviews
CM Form
Instructions
7. Executes Instructions
(logs on, Exports code,
Runs Build, runs scripts
Copies files, cleans up)
8. Email
Instructions
10. Notify
Complete
16. Notify Complete
Deploy
Mgr
Sys
Admin
13. Notify
Complete
11. Email
Instructions
12. Execute
Task
15. Execute
Task
Other
DEV
20. Use
Application
User
Sandbox
SCM
OPERATIONS
(Deploys)
The Starting Line
Paperwork
People
Processing• JAVA, .Net
•ANT, Maven
• Subversion (SVN)
•Oracle, SqlServer,
MySQL
•RHEL, Windows 2008
•Avg Cycle Time = 15
days (30 mins – 486
days)
• # Deploys/yr = 2579
MITRE
Ops
In The Beginning – Aug 2009
The service
we provide
is great!
They are
slowing
me down!
Dev
MITREConfronting the Chasm – May 2010
24 hour SLA
is 23.5 hours
too long!
OpsDev
Huh?
MITREBridging the Chasm – Aug 2010
Let me
show you
how
this could
work.
OK!
OpsaDev
MITRE
Let’s define
Acceptance
Criteria !
Defining Success – Fall 2010
DevOps
MITREAcceptance Criteria
I can live
with that.
Single command to build app given an SVN revision #
Single command to deploy to environment given an SVN revision #
Build once. No development tools utilized on target environment.
Deploy to test environment is self-service and can be done by any user without handoffs (no
SA, DBA) or additional passwords
Deploy to PROD is done by release team without handoffs (no SA, DBA)
Prior to deploying validates compliance with SONAR quality reports and code coverage
Can determine what SVN revision is deployed in each environment, by who and when
In PROD, on error or abort application is restored to prior state
No additional instruction documentation is needed.
In phase 1, utilize existing request system for approvals
System validation is manual and outside of scope
MITRE
MITRE
Build and ReleaseEngineer
Sys Admin
DBAJava, .NET Developers
The Launch – May 2011
DevOps
DevOps
Everyone
on Board!
MITREImplementation – (5/11 to 6/12)
Are we there
yet? Are we
there yet?
DevOpsDevOps
MITRECourse Correction – April 2012
Got Training Brought in Expertise
Freed Up Resources A New Plan!
MITRE
1 Developer develops commits code to repository
2 Commits automatically trigger builds
3 Depending on environment developer or
Release Engineer deploys build
Our DevOps Solution– June 2012
Source Code Repository
Build WorkflowCheckout
Compile
Run Unit Tests
Run Code Analysis
Store Artifacts
Deploy WorkflowCleanup Environment
Backup Environment
Deploy Binaries
Execute Rollback & Restore
Artifact Repository
Commit
Code &
Build
Files
Application &
Build Files
Release
ArtifactsRelease
Artifacts
Run
Build
(revision #)
Run
Deploy
(build #)
Repeated for each target
Environment using the
same Build
Build/CI Server
2
1
3
* Developers can deploy to Non-Production Environments
CI
DEV
Test
Prod
DEV OPS
MITREScaling Up – June 2012
Its working
Woo Hoooo!
MITREVisible Tracking
MITRERecipe For Onboarding
MITREGuides and Samples
MITRE
Water cooler Talk
Snackfest
Formal briefings to
CIO and Senior Mgt
Visible Metrics
Marketing
MITREThe Results
• 68 applications in system
• Multiple Platforms
o ANT, Maven, Gradle
o SVN, GIT
o JAVA, .Net, NodeJS
o Oracle, MySQL, SQL Server,
MongoDB
• Avg Cycle Time = 9 days (-40%)
(range 2 mins – 483 days)
• ~75% of deploys are automated
o # Deploys/year = 3555
o Auto = 2665
o Manual = 890 (-65%)
o 38% Throughput Increase
• Freed up 2.5 deployment staff
MITRE
Build & ReleaseEngineer
Sys Admin
DBAJava, .NET Developers
The Future: Continuous Delivery
aDevOps
aDevOps
OpsAutomationEngineer
TestAutomationEngineer
Let’s do it
again
Pleeease!
MITREBill & Aimee’s Excellent
DevOps Journey
Aug ’09:Old
Process
July ’10:Confront
May ’11:Success Defined, Launch
Jun ’12:Success!
• Anyone can do it
• Know where you want to go and why
• Patience - it takes time for change
• Education, education, education
• Use Agile (minimal product, iterations, product demonstrations)
Apr ’12:Got Help
Today:Results
MITRE
• Changing skillsets and mindsets to:
– Support infrastructure and test automation
– Break up silos and organize into cross-functional delivery teams
• Incorporating ERP and large COTS (or non-typical, non-web) applications into a
DevOps model
What We Need Help With
