at your service: using jenkins in operations

At Your ServiceUsing Jenkins in Operations

mandi wallsVelocity NYC14 OCT 13

Monday, October 14, 13

whoami

• Mandi Walls

• Technical Practice Manager at Opscode

• @lnxchk


What is Jenkins?

• Open source continuous integration and build server

• A bit of complex history we don’t have to get into

• One of a family of similar tools commonly found on the “dev” side of the team

• http://jenkins-ci.org/


Other CI tools include Hudson and Travis. Many, if not all, commercial source code management systems have some kind of job builder included in their software package.

Common Uses of Jenkins

• Continuous build and integration of a project

• Hooks up directly to source control

• Go from code checkin to completed build hands-free


What is This Workshop?

• Not the be-all end-all definitive guide to Jenkins.

• Not a guide to installing Jenkins. It’s a package, yo.


This workshop will give you some idea of possible tasks you can farm out to jenkins. It’s not meant to be an exhaustive course on jenkins.

Jenkins is an open source project, and, like so many others, is a moving target. There are a lot of plugins focused on different aspects of job builds, build management, build visualization, all kinds of things. We’re going to work through a handful of sample jobs to give you an idea of what Jenkins can be doing for you and your team.

We’re also going to skip getting jenkins onto the box. it’s just not all that interesting. There’s plenty of other things to muck with in jenkins, like authentication, that we’re just going to handwave. You’re ops folks, you know how to install software to servers!

Why Would Ops Want It?

• Run commands the same way every time

• Guard against those fat fingers or forgotten steps

• Trigger processes off a main action, like checkin or other status

• Automatically perform tests and checks that get lost or skipped


Jenkins has a number of different job types. We’re going to talk about only a few of the features that you might find useful in a jenkins server, and there are lots more where they came from.

Choose Your Yaks WiselyMonday, October 14, 13

Could you slap something together to do what Jenkins does? sure, some kind of cron-launched polling thing, with bits of whatever else you deemed cool or necessary. And then you have to support it. Why? Why not get right to the building of stuff and running of tests and doing of awesome?

• http://$YOUR_IP_HERE:8080/

• The first page is the Jenkins dashboard.

• All of your jobs will show up here.

• You have no jobs, so there’s nothing listed yet.

First Look at Jenkins


Primitives

• Your build projects in Jenkins are called “jobs”

• Each job can have multiple steps

• Any time a job is run on the jenkins server, it creates a “build”

• For a build to be successful, all the steps in the project must succeed


What’s on the Dashboard


Other things on the dashboard, the like managing users on the server, seeing the whole build history, working with credentials. We’re not going to worry too much about those today.


add new jobs




add new jobsmanage the jenkins server





jobs in the queue





jobs in the queue

jobs executing



Managing Jenkins


Click on “manage jenkins” on the left. this screen should show up

Managing JenkinsFreaking open source. I just installed this...


Click on “manage jenkins” on the left. this screen should show up

Jenkins Plugins

• Click on “Manage Plugins” and then select the “Available” tab

• Plugins add features and functionality to Jenkins

• They help Jenkins integrate with other software, provide more sophisticated job management, add authentication to the server, etc

• There’s a whole lot of them


You can see from the list the types of software, services, and components that are available to be added to your Jenkins server. We’re certainly not going to look at most of these. We’re going to keep this simple.

When you add plugins to the Jenkins server, you can request that it restart on its own as long as no jobs are currently running. When you add new plugins, your existing jobs can be modified to take advantage of their features, you don’t have to recreate your jobs when you add new plugins.

Configure Jenkins Job• Return to the dashboard by clicking “Jenkins” in the upper left

• Click on “create new jobs” or “New Job” from your Jenkins dashboard


Types of Jobs

• Free-style: build projects, integrate with source code managers, do stuff with things

• Maven 2-3 project: build projects using Maven. Jenkins knows stuff about Maven and helps you with the configuration

• Monitor an External Job: run jobs on other systems with a Jenkins watcher, which reports in when the job finishes

• Other types: added by plugins, extensions, etc


Our first project will be free-style


Example Job 1: bind

1. Zonefiles are in a git repo on the host called “bindfiles”

2. Jenkins will check them out, build them into an rpm

3. The rpm will be added to a repository and the metadata updated

4. We’ll also load the files into the bind server

This builds on part of my tutorial from Velocity Santa Clara in June Monday, October 14, 13

I know people love to hate on my bind example. but it meets a couple of criteria: the server component is lightweight, so it’s fine to run on a loaded lab box; the syntax of the db files is well known, so i’m not teaching someone crazy syntax; and finally, there are already good-enough tools to borrow from the rest of the world for dealing with it.Would you put your zonefiles in an rpm? maybe not. is the world going to fall apart because you *can*? no.

Configure Bind Job in Jenkins

• Enter “Bind” in the “Job name” field

• Select “Build a free-style software project”

• Click “OK”

• Jenkins sets up the build template and you can now create the job


Job Set Up• Add a meaningful description.

“This project creates an rpm of our zonefiles from a git checkout of the “bindfiles” repo”


SCM Settings

• Select “Git” under “Source Code Management” for the expanded view

git@localhost:bindfiles.git


add the location of the repository to the project

Schedule• Click on the blue “?” on the right for details on scheduling jobs

Looks a lot like cron, also has splay features



Build on a schedule, like cron




Build on a schedule, like cronPoll the git server, build if there is new content



Schedule

• In the “Schedule” box, enter

•H/5 13 * * *

• The “H” allows Jenkins to spread out jobs that might all be requested to run at the same time

• This job will run every five minutes during the 1300 hour - the boxes are in UTC


Adding Build Steps

• Since we’re freestyling, we can add whatever commands we need to the build steps

• Jenkins will take care of checking files out of git

• We’ll build a package using fpm


A quick aside

• fpm

• “f’ing package managers”!

• by Jordan Sissell

• https://github.com/jordansissel/fpm

• A ruby gem that creates various package types from a number of different content sources


https://github.com/jordansissel/fpm


The fpm Build Step/opt/chef/embedded/bin/fpm -s dir -t rpm -p /srv/repo \

-v $BUILD_NUMBER --prefix=/var/named -n "zonefiles" *.db

all on one line

fpm is going to:-s read files from a directory-t create an rpm-v use the BUILD_NUMBER from the jenkins env as the rpm version-p write the file to /srv/repo--prefix all the files will live in /var/named-n the package is named “zonefiles”


The location of fpm is an artifact of my building the lab boxes with chef and not wanting to deal with the system ruby. you could shorten this by adding “/opt/chef/embedded/bin/” to the PATH of the jenkins environment. that yak was being cranky

Add a Second Build Step

• Click “Execute shell” again

• This time, we’ll update the repository metadata

• The command is

•createrepo /srv/repo

• That’s the directory we told fpm to build the rpm into


Let’s Run It

• At the bottom of the page, click “Save”

• You’ll now see the project menu on the left side of the page


Build Now


Build when you want

Build Now


First Build

• All build output status will show up in the “Build History” box

• When the build fails, the ball will be red

• When the build succeeds, the ball is blue

• You can use the console output to figure out what happened to your build if things go wrong, or just check the output if it’s ok

• Click on the build datestamp, then “Console Output” on the left


Debugging Jobs

• Console output is your friend

• Part of the output will include where the build is happening

• All errors from included commands will show up in the output


Add More Steps to Our Job

• Let’s load the zonefiles we built onto our server, and restart bind

• Install the rpm we built into /srv/repo

• Since we built a repo, we could yum it from there, but I’ve not given you a web server

• The “jenkins” system user has full sudo on this host for cooking-show type magic


Build Steps

• In your “Bind” job, click “Configure” again

• At the bottom of the page, add these two execute steps

sudo rpm -Uhv /srv/repo/zonefiles-$BUILD_NUMBER-1.x86_64.rpm

sudo service named reload


The $BUILD_NUMBER parameter follows through the entire build, through all the steps. You can use it to refer into the artifacts created by the build.

Where things get hard is when passing parameters around among builds. you need additional plugins for that sort of work.

You’ll want to “Add Build Step”“Execute Shell”And then put these commands in the windows.

You can put both commands in one window, and string them together with &&, but that’s the whole point of having jenkins do this for you.

Additionally, if we were for-real building a repo available over yum, we could leave these pieces out of this job, and create a new job that went to the DNS servers and ran them on those hosts. for another day.

Build It!

• Save your changes

• Click Build Now

• Check your console output, you should see the output from rpm and the bind reload


Ok, Pause

• Shake it out.

• Now we’ve got some basics.

• What other stuff can Jenkins do for us?


Example 2: mysql Backup

• The mysql server has the sample employees database installed from http://dev.mysql.com/doc/employee/en/index.html

• We can back it up, load it into a new database, and validate the backup in a Jenkins job

• Other pieces could copy the backup to another host, push it to long term storage, etc


http://dev.mysql.com/doc/employee/en/index.html

http://dev.mysql.com/doc/employee/en/index.html

Create a New Job

• Create a new free style job, call it “MySQL Backup” or similar

• If you already have a mysqldump running on your servers, you could configure them to notify Jenkins when they’re done, and make this job “monitor external job”

• We’re not going to put a trigger on this job - we’ll just run it ourselves

• There are four steps


Step 1

mysqldump --no-create-db --password=rootpass -u root employees > /tmp/employees_dump-$BUILD_ID.sql


Step 2

mysql -u root --password=rootpass -e "create database employees_backup"


Step 3

mysql -u root --password=rootpass --database=employees_backup -t < /tmp/employees_dump-$BUILD_ID.sql


Step 4

mysql -u root --password=rootpass -t < /var/lib/mysql/scripts/test_employees_sha.sql


Build Now

• Save the build

• Click “Build Now”

• Hang out. This build takes a few minutes

• Jenkins will tell you it’s doing something

• You can watch the console output while the build is running


The barber pole lets you know that jenkins is busy doin stuff on your behalf.

What else?

• When this build is done, we could

• clean up the old dumpfile

• drop the back up database we used for testing

• have a beer, huzzah! we tested our backup for reals!


Jenkins Dashboard• Now that we’ve run some jobs, check out the dashboard

All ourjobs!

Last goodbuild

Last badbuild

Build time of

last build

Schedule a build


The weather mnemonic in the second column gives you an idea of how good or bad your build has been recently. Stormy weather means the most recent builds have failed. Sunshine means all the builds have been good. The first column lets you know that the most recent build was a success.

Let’s Add One More

• Different kind of project - multijob

• This is a plugin that I’ve added to your Jenkins server

• We build the child jobs first, then the parent job

• The jobs run in phases, and each phase can have multiple jobs that run in parallel


First Job: Build Server Docs

• I’ve borrowed the Chef documentation from our github repo

• https://github.com/opscode/chef-docs

• We use a python-based document processor called sphinx

• Sphinx can build a number of outputs from your source content using a plain old Makefile


New Job - Server Docs

• Create a new free style software project

• Name is “build server docs”

• It will have one execute step

cd /srv/docs/chefdocs; make server

• Save the job


New Job - Open-Source Docs

• Create another free style software project

• Name it “build open source docs”

• It has one execution step

cd /srv/docs/chefdocs; make open_source

• Save the job


Parent Job

• Now we build a multi-job project

• This project type is good for jobs that can be run parallel but still make sense to run together


you’ll be able to execute as many of the subjobs as you have executors for.

Parent Job

• Create one more new job

• This job will have a different type: Multijob Project

• Name it “build docs” or similar


Add build step

• This job’s build steps will belong to phases

• The first thing to do is click “Add build step” and select “Multijob Phase”

• This will change the screen a bit


Phases

These are the jenkins job names


name the phase, and then include the names of the other jobs we’ve created

Phase Success

• A single phase of multiple jobs can be successful when:

• all member jobs are successful

• member jobs are stable or unstable but not failed

• complete - always continue

• Our two build jobs will run in parallel, then the job will move on


Execute Shell• Pack the built docs into an rpm with our buddy fpm

• Add a build step -> Execute shell command/opt/chef/embedded/bin/fpm -s dir -t rpm -p /srv/repo \

-v $BUILD_NUMBER --prefix=/srv/site/docs -n "docs" \

/srv/docs/chefdocs/build/open_source \

/srv/docs/chefdocs/build/serverall on one line


Final Execute

• One more execute statement, rebuild your repo

•createrepo /srv/repo

• Save the job


Project Dashboard• The multi-job job has a dashboard of its own


All Successful


Things to note: you can only schedule a build for the parent project.

Console Output

Both document build jobs completed before Jenkins moved on


Taking Care of Jenkins• Jenkins itself is pretty straightforward

• All the jobs are stored in the “jenkins” system user’s homedir


Saving Jobs

• To back up or move a job, simply back up or move its directory

• Managing the access jenkins needs is also important

• SSH keys for getting around to other hosts

• Access to source code repositories

• sudo or privileged access to complete tasks

• Jenkins user must have a shell!


depending on your source for the jenkins things, your jenkins user might be installed with /bin/false as its shell. you’re going to want it to be able to do stuff with things, give it a shell.

Adding Capacity

• Jenkins is able to farm jobs out to remote nodes

• These nodes are “build slaves”

• They run an agent, not a full Jenkins server


Other Neat Stuff

• Jenkins jobs also have post-complete actions

• Work can be sent downstream from one job to another

• Plugins allow for parameterized triggered builds

• Send email notifications

• Talk to your change management system


Accessible by URL

• Jenkins has an included REST-like API

• Trigger jobs, check status, create new jobs, etc

• This makes it easy to integrate with other pieces of infrastructure

• Build Bind:

• http://$IP:8080/job/Bind/build


Post-Receive Hooks From Git

• The Jenkins plugin from git adds a trigger feature to builds

• It tells the Jenkins server that a new checkin has been made, so a new build should be run

• curl http://localhost:8080/git/notifyCommit?url=git@localhost:bindfiles.git

• Must be the same url you used in the set up of the job

Monday, October 14, 13curl http://yourserver/jenkins/git/notifyCommit?url=<URL of the Git repository>[&branches=branch1[,branch2]*]

git@localhost:bindfiles.git

http://localhost:8080/jenkins/git/notifyCommit?url=bindfiles




Building bindfiles on Checkin

• Log into your box over ssh

• User: velocity Password: velocityconf

• sudo vi /srv/git/bindfiles.git/hooks/post-receive

• curl http://localhost:8080/git/notifyCommit?url=git@localhost:bindfiles.git

• sudo chmod +x /srv/git/bindfiles.git/hooks/post-receive


The post-receive hook is a server-side only configuration. it is not writable from the user’s view of the repo





Check in a Change

• cd ~/bindfiles/

• edit db.local, add a new host

•wat!IN! A!192.168.1.22

• edit db.192, add the same host

•22!IN! PTR!wat.local.

• Update serial for correctness, save the files


git Checkin

• From the bindfiles directory

• git add .

• Tells git that new files are ready to be checked in

• git commit -m “added wat.local”

• Commits to your local repo

• git push origin master

• Sends the commit to the server


Watch Jenkins!

• You’ll have a new build running of your Bind job!

• You could go back and turn off the polling in the job config

• Jenkins will build every checkin


Dude.

We are so totally continuously deploying all the things.


Go Forth and Build

• Manage your configuration management

• See me at Office Hour tonight at 5pm if you want to talk Chef+CI

• Work better with your developers

• Speak the same language, use the same tools

• Don’t be bored

• Let Jenkins be your bot of all work for boring stuff


For the Archaeologists

• If you’re lost and just want to see the finished product of these labs:

• boot a box in AWS with the ami, log in with the creds in the deck

• sudo tar -xf .jobs.tar ~jenkins/


I know, these decks lose some of their flavor on the bedpost over night.

Thanks!

• More on Jenkins: http://jenkins-ci.org/

• More on fpm: https://github.com/jordansissel/fpm

• More on CI / CD: http://continuousdelivery.com/


http://jenkins-ci.org

http://jenkins-ci.org



http://continuousdelivery.com

http://continuousdelivery.com

Pre Stuff

• If you’ve got an AWS account, boot ami-0195c668

• http://bit.ly/velojenkins for notes

• The image is just ridiculously huge


http://bit.ly/velojenkins

http://bit.ly/velojenkins

at your service: using jenkins in operations

Technology