on the importance of infrastructure as code
TRANSCRIPT
On the importance of Infrastructure as Code
Kris Buytaert
@krisbuytaert
Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,● Then Became an OpThen Became an Op● Chief Trolling Officer and Open Source Chief Trolling Officer and Open Source
Consultant @Consultant @inuits.euinuits.eu● Everything is an effing DNS ProblemEverything is an effing DNS Problem● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore● Some books, some papers, some blogsSome books, some papers, some blogs● Evangelizing devopsEvangelizing devops● Organiser of #devopsdays, #cfgmgmtcamp, Organiser of #devopsdays, #cfgmgmtcamp,
#loadays, ….#loadays, ….● Part of the travelling geek circusPart of the travelling geek circus
What's this devops What's this devops thing anyhow ? thing anyhow ?
C(L)AMSC(L)AMS● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● MeasurementMeasurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim
Why automate ?Why automate ?
Common ProblemsCommon Problems● Many manual changes to systems● Many undocumented changes ● Emergency Administration only● Disaster Recovery site is a Disaster● Time to deliver a box is to slow● All boxen are different● Computers don’t work hard enough for us
More ProblemsMore Problems● How long does it take to reinstall a machine from 0● To the exact same point as before ?● With different Hardware ? In a different cloud ?● What about your (customer/personal data )
Security ?Security ?● Monitoring that your platform hasn't changed.Monitoring that your platform hasn't changed.
• Why is selinux disabled ?Why is selinux disabled ?
• Who added / dropped that firewall ?Who added / dropped that firewall ?
• What did this originally look like ?What did this originally look like ?
• Is this file really what Bernd meant it to be ?Is this file really what Bernd meant it to be ?
#monitoringsucks#monitoringsucks● Monitoring is out of sync with realityMonitoring is out of sync with reality
● Managed manuallyManaged manually
● Can't keep up.. Can't keep up..
Do you want to ?Do you want to ?● Install these racks manuallyInstall these racks manually
● Over and over again ? Over and over again ?
● And can you guarantee that installs are And can you guarantee that installs are identical ? identical ?
● ““No simple admin taks is fun more than No simple admin taks is fun more than twice” twice”
● s/twice/once/g;s/twice/once/g;
● Repeating installs are boring and prone to Repeating installs are boring and prone to errorserrors
● Each installation is unintentionally UniqueEach installation is unintentionally Unique
● Manual installs DO NOT scaleManual installs DO NOT scale
ChallengesChallenges● ReproducabilityReproducability
● SpeedSpeed
● AuditingAuditing
● Keeping stuff in sync Keeping stuff in sync
• MonitoringMonitoring
• SecuritySecurity
• BackupBackup
The 10The 10thth floor test floor test● Grab a random machine (don’t take a backup before)
● Throw it out a 10th floor window
● Can you recover it in 10 minutes ?
Facts!Facts!● Data Backup is only a part● Sysadmin backup needs to be done
also● Manual Installations = bad● Bad installations = unusable
infrastructure● Bad installations = unproductive users● Bad installations = manual efforts● Manual efforts = no time● No time = no updates no patches no
security● Manual work = high costs● No security + high costs = Bancrupcy
Deploying an InfrastructureDeploying an Infrastructure● 1996 : Manual Installations1996 : Manual Installations
● 2001 : Mondo rescue 2001 : Mondo rescue (reproducable single instances)(reproducable single instances)
● 2003 : SystemImager2003 : SystemImager
• Reproducable Infrastructure , with Reproducable Infrastructure , with “OVERRIDES”“OVERRIDES”
• Fast Multicast Image deploymentsFast Multicast Image deployments
• Image Sprawl (thank you VMware)Image Sprawl (thank you VMware)
Deploying an InfrastructureDeploying an Infrastructure● 1996 : Manual Installations1996 : Manual Installations
● 2001 : Mondo rescue2001 : Mondo rescue
● 2003 : SystemImager2003 : SystemImager
● 2005 : 2005 : Kickstart / FAI Kickstart / FAI
• Dreaming of Jeos + IAC (Cfengine)Dreaming of Jeos + IAC (Cfengine)
Deploying an InfrastructureDeploying an Infrastructure● 1996 : Manual Installations1996 : Manual Installations
● 2001 : Mondo rescue2001 : Mondo rescue
● 2003 : SystemImager2003 : SystemImager
● 2005 : Dreaming of Jeos + IAC2005 : Dreaming of Jeos + IAC
● 2008 : Actual JeOS + IAC2008 : Actual JeOS + IAC
● 2010 : Vagrant for development 2010 : Vagrant for development
Imagesprawl AND Imagesprawl AND SnowflakesSnowflakes
● Image Sparwl :Image Sparwl :
• Copy vm 3xCopy vm 3x
• Modify 2x Modify 2x
• Copy 21x Copy 21x
• How the Heck did we get here ?How the Heck did we get here ?
● SnowFlakes :SnowFlakes :
• Don't touch this box it might breakDon't touch this box it might break
• Look how nice it is !Look how nice it is !
You never deploy You never deploy something “just” oncesomething “just” once
● Local test … experiment, Local test … experiment,
• Vagrant box / local containersVagrant box / local containers
● Integration PlatformIntegration Platform
• Same codebase,, different environmentSame codebase,, different environment
● Dev/ UAT/ Prod / DR … Dev/ UAT/ Prod / DR …
● Or your customer just forgot to renew the lease Or your customer just forgot to renew the lease on his VPS. #toldyousoon his VPS. #toldyouso
What's different in the cloud ?What's different in the cloud ?
● ScaleScale
● VelocityVelocity
● ChangeChange
Your machines as CattleYour machines as Cattle
Treat your people as petsTreat your people as pets
Configuration MgmtConfiguration Mgmt● Configure 1000 nodes,Configure 1000 nodes,
● Modify 15000 files, Modify 15000 files,
● Think : Think :
•Cfengine,Puppet, Chef, SaltCfengine,Puppet, Chef, Salt
● Put configs under version controlPut configs under version control
● Please don't roll your own ... Please don't roll your own ...
Infrastructure as CodeInfrastructure as Code● Treat configuration automation as code Treat configuration automation as code
● Development best practicesDevelopment best practices
• Model your infrastructureModel your infrastructure
• Version your cookbooks / manifestsVersion your cookbooks / manifests
• Test your cookbooks/ manifestsTest your cookbooks/ manifests
• Dev/ test /uat / prod for your infraDev/ test /uat / prod for your infra
● Model your infrastructureModel your infrastructure
● A working service = automated ( Application Code + Infrastructure A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring )Code + Security + Monitoring )
● IAC -ne scripting (or translating bash to yaml)IAC -ne scripting (or translating bash to yaml)
IAC Is a Testing IAC Is a Testing RequirementRequirement
● Stable reproducable starting pointStable reproducable starting point
AuditabilityAuditability● git loggit log
● git blamegit blame
● Review, Review,
● authorizationauthorization
File monitoringFile monitoring
Fixing Monitoring FatigueFixing Monitoring Fatigue
Stored ConfigsStored Configs
Collection and ExportCollection and Export
Export :Export :
@@resource { @@resource {
... }... }
Collect:Collect:
Resource <<| query |Resource <<| query |>>>>
Clean out nodes that dissapearClean out nodes that dissapear
puppet node clean puppet node clean
Use Cases:Use Cases:● Ssh keysSsh keys
● Reverse proxy configsReverse proxy configs
● Monitoring resourcesMonitoring resources
● Measuring resourcesMeasuring resources
Puppetmaster Example:Puppetmaster Example:
Defining a ServiceDefining a Service● profile that :profile that :
• Configures service using a standard Configures service using a standard module call with hiera based parametersmodule call with hiera based parameters
• Configures BackupConfigures Backup
• Configures logrotation Configures logrotation
• Configures logshipping Configures logshipping
• Exports Monitoring NeedsExports Monitoring Needs
Chronicle of a failed Chronicle of a failed private cloudprivate cloud
● Tool X provisions a VMTool X provisions a VM
• 3 weeks from the request / can only be done by 1 team3 weeks from the request / can only be done by 1 team
● Tool Y installs patches Tool Y installs patches
• 2 weeks2 weeks
● Team Z installs backupTeam Z installs backup
• 1 day1 day
● Team A installs monitoringTeam A installs monitoring
• 3 weeks3 weeks
● AppApp
• Manual deploy on wrong JVM, return to senderManual deploy on wrong JVM, return to sender
Application IncludedApplication Included● Application = Application =
• PackagePackage
• ConfigConfig
• ServiceService
● No manual scriptingNo manual scripting
● Think about your bootstrapping / scaleoutThink about your bootstrapping / scaleout
Automation of Automation of #monitoring #monitoring
brought back brought back thethe #love#love
ConclusionConclusion● IAC solves a lot of problems IAC solves a lot of problems
• Improves SecurityImproves Security
• Creates Monitoring LoveCreates Monitoring Love
• Creates SpeedCreates Speed
● But it still is code, and needs to be treated like But it still is code, and needs to be treated like code !code !
ContactContactKris Buytaert Kris Buytaert [email protected]@inuits.be
Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.be/http://www.inuits.be/
InuitsInuits
Essensteenweg 31Essensteenweg 31BrasschaatBrasschaatBelgiumBelgium891.514.231891.514.231
+32 475 961221+32 475 961221