Outline Goal A Problem to Solve Tools Practical Examples
Building a loosely coupled toolchain withRundeck and Puppet
Steven Meunier
21 June 2014
Outline Goal A Problem to Solve Tools Practical Examples
1 GoalWhat are we trying to achieve?InfluencesInfrastructure
2 A Problem to Solve
3 ToolsWhat do we need
Something to trigger a changeSomewhere to store the data for the changeSomething to perform the change
Helping the tools help us
4 Practical ExamplesGeneric Deployment FlowUpdating Hiera DataRundeck
Outline Goal A Problem to Solve Tools Practical Examples
GoalTo have a loosely coupled deployment toolchain that leveragesour desire to manage the servers based on state
Implies decisions have already been made regarding kind oftooling and server management:
Tools that perform a particular function and do it wellTools can be swapped out if neededConcept of state can be extended to deployments if viewedas a process of changing the server state
Outline Goal A Problem to Solve Tools Practical Examples
Influences
Deployment models from gluInfrastructure as data from ansiblePuppet server statehttp://dev2ops.org/2012/05/using-rundeck-and-chef-to-build-devops-toolchains-at-chefcon/
Outline Goal A Problem to Solve Tools Practical Examples
Normal Puppet Infrastructure
Puppet Infrastructure
Outline Goal A Problem to Solve Tools Practical Examples
Deployment Infrastructure Built On Top Of Puppet
Deployment Infrastructure
Outline Goal A Problem to Solve Tools Practical Examples
Ready . . . Set . . .
By tableatny (BXP135671) [CC-BY-2.0], Source:http://upload.wikimedia.org/wikipedia/commons/1/13/Athlete_at_starting_block.jpg
Outline Goal A Problem to Solve Tools Practical Examples
Go!
By David A. Brandenburg, U.S. Navy (http://www.navy.mil/; VIRIN: 100517-N-7643B-048) [Public domain], Source:http://upload.wikimedia.org/wikipedia/commons/2/2a/Sack_race.jpg
Outline Goal A Problem to Solve Tools Practical Examples
Know your Problem Domain
DeadlinesRestrictions imposed by Security TeamTeam requirements
Release management (easy interface, keep track ofversions, support all our applications and release process)Development (APIs for jenkins, low (preferably no) adoptionbarrier for development)Operations (operations maintain control, integrate withpuppet, mcollective)
Outline Goal A Problem to Solve Tools Practical Examples
What do we need?
Something to trigger a changeSomewhere to store the information for the changeSomething to perform the change
Outline Goal A Problem to Solve Tools Practical Examples
Something to trigger a changeRundeck
Rundeck is an automation consoleMakes use of existing puppet infrastructureCan automate whatever we want, not just deploymentsFine grained access controlsEasily extensible through plugins — even if you don’t knowJavaDeployments triggered manually or via Jenkins
Outline Goal A Problem to Solve Tools Practical Examples
Somewhere to store the data for the changeHiera data
“Infrastructure as code” → “Infrastructure as data”Hiera data for configuration using Puppet 3’s data bindingChange configuration without editing puppet manifests ortemplates
property_files::parent_prefix: /srv/tomcat6/foo/confproperty_files::property_files:- foo.properties- bar.properties
properties_foo.properties:database.name: fooDBtimeout: 100
Outline Goal A Problem to Solve Tools Practical Examples
Something to perform the changePuppet & MCollective
Puppet manages all aspects of the server — nothing ismanualMcollective is an orchestration tool using activemq totrigger agents across serversMCollective manage non-state managed aspects ortemporary state changes
starting/stopping queuesdatabase deployments
Outline Goal A Problem to Solve Tools Practical Examples
No matter what you think you’ve got...
Spaceballs, c©1987 MGM Studios, Source:http://s3.amazonaws.com/mgm-assets/assets/Image/1266267325/41135/SPACEBAL-still-003-x.jpg
Outline Goal A Problem to Solve Tools Practical Examples
Sometimes your princess is a hairy guy in a dress
Spaceballs, c©1987 MGM Studios, Source:http://img.pandawhale.com/65424-Spaceballs-Celebrity-stunt-dou-DZzt.jpeg
Outline Goal A Problem to Solve Tools Practical Examples
Tools need help to make them fit
Git: steep learning curveRundeck: needs extra plugins to fit your workflowPuppet: duplicate declarations; ordering; limitations ofhieraMCollective: can be unreliable if used as a source of nodetruth; time needed to build the agentsRuby: it’s critical if you’re using puppet + mcollective
Outline Goal A Problem to Solve Tools Practical Examples
Generic Deployment Flow
Create deployment window with no puppet interferenceBuild rpms from deployable unit (if applicable)Disable monitoringCompare and update configuration (if applicable)Update version parameter in hiera dataKick off puppet runRestart serviceRe-enable monitoring
Outline Goal A Problem to Solve Tools Practical Examples
Deployment Configuration Committer
Ruby script with the goal of reducing manual actionsrelated to configurationWorks with java properties files and context xml filesBuilds a hash of properties in the release and compares itagainst the current propertiesUploads any changes to Gerrit for review
Outline Goal A Problem to Solve Tools Practical Examples
An example tomcat application in Hiera
tomcat_context:ROOT:
version: xx.yy.zz
tomcat_instance:foo: {}
tomcat_context:ROOT:
artifact: fooartifact_provider: fileartifact_target: ROOT.warartifact_url: http//repo.example.com/warinstance: foo
rundeck_job:foo: {}
Outline Goal A Problem to Solve Tools Practical Examples
Puppet Rundeck Job Type
@@rundeck_job { "${project}/${group}/${name}_${hostname}":ensure => $ensure,jobname => $name,project => $project,group => $group,definition => template($template),
}
Outline Goal A Problem to Solve Tools Practical Examples
Rundeck Conditional Local Command
#!/bin/shCOMMAND_TRUE=$1COMMAND_FALSE=$2CONDITION="[ $3 ]"
EXIT_CODE=0
if [ "x$COMMAND_FALSE" == "x" ]then
if eval $CONDITIONthen
$COMMAND_TRUEEXIT_CODE=$?
fielse
if eval $CONDITIONthen
$COMMAND_TRUEEXIT_CODE=$?
else$COMMAND_FALSEEXIT_CODE=$?
fifi
exit $EXIT_CODE
Outline Goal A Problem to Solve Tools Practical Examples
Rundeck Conditional Local Command
Outline Goal A Problem to Solve Tools Practical Examples
Thank You