Optimizing Your CI: Lessons Learned from a Successful Jenkins Rebuild

Santa Clara, Us West, September 2015Jonathann Zenou

Footer

#jenkinsconf

Greetings fellow humans

• Hi, my name is Jonathann• Living in Tel Aviv, Israel• Originally from Marseille, France• Leading the DevOps effort at Ravello Systems• @Zenoujohn• jonathann.zenou@ravellosystems.com

#jenkinsconf

Footer

What is Ravello Systems ?

#jenkinsconf

Footer

Ravello Systems

• 30 developers • 7 major projects• Kanban like

#jenkinsconf

Footer

A part of our stack

#jenkinsconf

Footer

What are we going to talk about ?

#jenkinsconf

Footer

Old problems

#jenkinsconf

Footer

Jobs Leftovers

• Causes full disk• Workspace Cleanup Plugin• Can pollute your maven repo or npm cache• Solution needed : a new clean environment for every job

Footer

#jenkinsconf

Better resource distribution

Backend Team Frontend Team

Footer

#jenkinsconf

Better resource distribution

Backend Team Frontend Team

Footer

#jenkinsconf

Better resource distribution

Backend Team Frontend Team

Footer

#jenkinsconf

Better resource distribution

Backend Team Frontend Team

Footer

#jenkinsconf

Better resource distribution

Backend Team Frontend Team

“waiting for next available executor …”

Footer

#jenkinsconf

Better resource distribution

Backend Team/Frontend Team

Footer

#jenkinsconf

Better resource distribution

Backend Team/Frontend Team

Footer

#jenkinsconf

Better resource distribution

Backend Team/Frontend Team

Footer

#jenkinsconf

Frontend Team leader’s reaction

#jenkinsconf

Footer

Better resource distribution

• Solution needed : a rapid and scalable way to spawn dynamic environments

#jenkinsconf

Footer

Better resource distribution

#jenkinsconf

Footer

Using Docker in Jenkins

• Using Jenkins Docker plugin• Dedicated Docker host for each team• Hosts machine capabilities are a function of team size and

requirements • All images on all hosts• Problems solved : jobs leftovers and resource distribution

#jenkinsconf

Footer

Security

• No active directory or LDAP• Jim uses Dave’s account• Employee leaves the company

#jenkinsconf

Footer

Security

#jenkinsconf

Footer

Security

#jenkinsconf

Footer

Security

• Solution needed: third party dynamic authentication mechanism

#jenkinsconf

Footer

Security

• Using Jenkins Google Login plugin• Coupled with Matrix Authorization Strategy plugin• 2-Factor authentication• Open our Jenkins to the world

#jenkinsconf

Footer

Too much code in jobs

#jenkinsconf

Footer

Too much code in jobs

• Hard to maintain• Can be easily broken• “Who touched this ??”

#jenkinsconf

Footer

Too much code in jobs

• Solution:• Put your scripts in revision control (Github)• Using Jenkins Multiple SCM plugin

#jenkinsconf

Footer

Jobs Versioning

#jenkinsconf

Footer

Jobs Versioning• Solution:

• Using Jenkins JobConfigHistory plugin• Saves old configurations• Diff between jobs configurations• Revert job configuration

• Coupled with Jenkins SCM sync configuration plugin• Saves you whole Jenkins configuration in revision

control (Github)

#jenkinsconf

Footer

Jobs data sharing

Backend Job Frontend Job

Parameters:- Backend version: 1.1- Frontend version: stable

Parameters:- Frontend version: 1.1- Backend version: stable

#jenkinsconf

Footer

Jobs data sharing

• Solution needed: an interface/service that will store data that will be accessible from all the jobs

• Using Jenkins Global Variable String Parameter Plugin• Cannot alter them from a job• But you can using Groovy …

#jenkinsconf

Footer

Jobs data sharing

#jenkinsconf

Footer

Jobs data sharing

• Keep a global parameter for each of product stable version

• Let all the other jobs use these values• Change the global parameter value of a product stable

version automatically when this product passes a green CI• Gist link: https://gist.github.com/johnyzed/2af71090419af2b20c5a

#jenkinsconf

Footer

CI at Ravello Systems

build+

unit tests

environment creation

codedeployment test suites stop

environments

artifactory

log collection

#jenkinsconf

Footer

CI at Ravello Systems

environment creation

#jenkinsconf

Footer

Before digging further

#jenkinsconf

Footer

The pieces are ready• What Do Developers Want ?• Blackbox• No duplication• C.D.R.S• Single form, with a lot of triggers• Using Multijob Plugin

#jenkinsconf

Footer

Jobs sandbox• Improving a job without:

• Breaking it• Stopping other people using it

• Solution needed: a parallel Jenkins environment almost identical to our production Jenkins environment to test the new changes on

#jenkinsconf

Footer

Jobs sandbox• Jenkins environment blueprint on Ravello Systems • Using Ansible:

• git clone• minor changes

• Running regular jobs• All that orchestrated from a Jenkins job

#jenkinsconf

Footer

Jenkins sandbox flow

changes on Jenkins script

branch

create dynamic Jenkins

environmentrun jobs stop dynamic

environment

#jenkinsconf

Footer

Jobs sandbox

#jenkinsconf

Footer

Comparing Jenkins environment

Old Jenkins New Jenkins

Number of jobs 137 56Number of slaves 10 0-50Build and test duration 70 43

#jenkinsconf

Footer

Comparing Jenkins environment

#jenkinsconf

Footer

Take-aways

• Ask your developers how to make their life easier

• Don’t be afraid to break things• Follow @jenkins_release

#jenkinsconf

Footer

One last thing

• Ravello Repo• http://www.ravellosystems.com/repo/• http://www.ravellosystems.com/repo/blueprints/6039143

6

#jenkinsconf

Footer

Questions ?

#jenkinsconf

Footer

Please Share Your Feedback

• Did you find this session valuable?• Please share your thoughts in the

Jenkins User Conference Mobile App.

• Find the session in the app and click on the feedback area.

#jenkinsconf

Thank you