Drupal Continuous Integration With Jenkins – Deploy

Classic Graphics – Charlotte, NC

By: John W SmithLicense: Creative Commons CC0

1. In the following slides, we will be describing simple Jenkins jobs that can be used to automate the deployment your git repository based code (not specific to Drupal, although that is our purpose) to your Demo, Live, Staging and/or Testing environments.

First, we will configure a Jenkins job that has the simple task of deploying code to a single server / environment. Although this is not optimal, as it requires a separate Jenkins job for each site / server / environment, it is simple to setup and requires only basic scripting.

Second, we will introduce a couple of scripts that will allow deploying multiple repositories to their corresponding servers based on parameters passed to the Jenkins job. This goes back to our original recommendation to name your repositories, servers, Jenkins jobs, etc using a common simple format.

2. We assume you have already followed the previously released document “Drupal Continuous Integration with Jenkins” describing the Basic install, setup and configuration of Jenkins for Drupal Continuous Integration.

Description / Purpose

This document does NOT go into the complexities of securing your system from internal or external attackers, please ensure your system is secure whether or not it is accessible from the internet.Also, please consult with your organizations security and system administrators before making any changes to existing systems configuration or authentication models, and before introducing new systems into the development and / or production server environments.

Disclaimer

2GHz+ Multi-Core CPU 4GB+ RAM

250GB or more depending on the number of concurrent builds, and build history, artifacts, etc that you will keeping around for review.

Software needed: Jenkins 1.xxx (I used 1.433.x) Java JDK/JRE 1.x MySQL 5.x PHP 5.x Drush (latest version) Drupal 6.x Core SimpleTest patch file (included with

SimpleTest Drupal Module)

System Requirements

1. Points of Contact The Points of Contact (POC) for this

documentation are: Technical Lead: David Norman

(davidn@knowclassic.com)Project Lead: John Smith

(johns@knowclassic.com)or(JSmith@i1Technologies.com)

References

Jenkins is here to help. Specifically, Jenkins is an "extensible continuous integration (CI) server". From a 10,000ft view, Jenkins is a tool for offloading the deployment and automated testing of a software application. You write your code, push it into version control, and Jenkins will take over the task of grabbing that code, running an installation / deployment process, testing the application (if you provide it with a test environment), and reporting back to you those test results. ~ (http://thinkshout.com/blog/2010/09/sean/beginners-guide-using-hudson-continuous-integration-drupal)

There are a number of different CI tools out there, but we (Classic Graphics) have chosen Jenkins, as it seems to be the leader in the Drupal world. Jenkins, for us, provides a web-based user interface, easy integration via LDAP to our Windows AD Domain, a friendly way to run scripts, integration with CVS / Subversion / Git repositories, and of course it works seamlessly with multiple projects / builds simultaneously. Also, as an added bonus, Jenkins has a Debian repository that we can use to install it on our Ubuntu servers without worrying about clobbering other libraries and avoid any dependency nightmares.

Welcome!

Here, we will be creating a simple server specific deployment job for our Drupal environment, this job could then be used as the base for other deployment sites / servers. Since every shops environment, testing and verification requirements will differ, we leave the advanced deployment scripting to the user of this document.

The Jenkins job setup in this tutorial makes a few assumptions:1. You have setup pub-key based auth over ssh to your git repository and possibly the

destination web servers, depending on your configuration this type of setup may or may not work for you.

2. You have Drupal core, all required modules, and all custom modules located within your git repository. However, the script could be easily modified to retrieve the latest released version of Drupal and any other required modules using Drush if this is not the case in your organization.

3. Access to deployment account private and public keys (if required) and / or the accounts password. Deployment account MUST have SSH access to the destination web server(s) AND write permission to each sites “DocumentRoot” directory. Obviously this type of setup has security implications, please consult with your systems security officer / administrator.

Create a Server Specific Deployment Job

1. First, open up your favorite browser and point it to the Jenkins server (http://jenkins.ourdomain.com:8080). Click the “log in” link in the upper right-hand corner of the browser window.

2. Enter your login credentials and click “log in”.

3. For this job, we will need an additional plugin installed and enabled. Click the “Manage Jenkins” link on the left.

4. Click the “Manage Plugins” link.5. Scroll down to find the “SSH

Plugin”, click the check box to enable it. Scroll to the bottom of the page and click “Install without restart”.

6. Click the “Return to Dashboard” link in the left menu to return to the Jenkins Dashboard once the plugin has been installed.

7. Click the “Manage Jenkins” link in the menu.

8. Click the “Configure System” link.9. Scroll down to the SSH section, and

add the appropriate information for your destination server. We have the SSH Server advanced view shown in this screen shot which allows entry of server login / password information for the SSH account to be used.

10.Add as many entries as needed. You will need one for each server you plan to use as a destination.

11.Click “Save” when done.

12.Click the “New Job” link in the menu. Enter “Deployment Test” and select “Build a free-style software project”. Click “Ok” when done.

NOTE: When creating these jobs, in order to simplify the creation of the “git” post-receive bash script to launch a build remotely, you should consider naming your jobs the same as the name of your repository/repo-directory name. So, if you have your repos on your git server in the /var/git/reponame.git directory,then this job would be called reponame. As we progress through this tutorial, it will become apparent why this makes things easier.

13. Now let’s configure this simple job.14. Scroll down to the Build Environment and

we’ll enable the “Send files or execute commands over SSH after the build runs” option.

15. Select the server you will be deploying to from the Drop down list.

16. Enter the following text in the “Exec Command” text area without the quotes. “git-update.sh user.account branch” where:• git-update.sh – is a script defined later

in this document • user.account – is the user the script will

use to retrieve the git repository code (should be the same account as the SSH account in a perfect world)

• branch – the repository branch you want to deploy with this script (most likely master in all instances except the test server environment)

17. Enable the “Exec in pty” option.18. Click “Save”.

Here, we will be creating a generic deployment job for our deployment environment. Since every shops environment, testing and verification requirements will differ, we leave the advanced deployment scripting to the user of this document.

The Jenkins job setup in this tutorial makes a few assumptions:1. You have setup pub-key based auth over ssh to your git repository and the destination

web servers, depending on your configuration this type of setup may or may not work for you. Alternately, you could use an expect script to enter the login credentials for you, however this is somewhat insecure as the password would be available in the expect script.

2. You have code (Drupal core, all required modules, and all custom modules) located within your git repository. However, the script could be easily modified to retrieve the latest released version of Drupal and any other required modules using Drush if this is not the case in your organization.

3. Access to the deployment account private and public keys (if required) and / or the accounts password. Deployment account MUST have SSH access to the destination web server(s) AND write permission to each sites “DocumentRoot” directory. Obviously this type of setup has security implications, please consult with your systems security officer / administrator.

Create a Generic Deployment Job

1. First, open up your favorite browser and point it to the Jenkins server (http://jenkins.ourdomain.com:8080). Click the “log in” link in the upper right-hand corner of the browser window.

2. Enter your login credentials and click “log in”.

3. Click the “New Job” link in the menu. Enter “Deployment Test” and select “Build a free-style software project”. Click “Ok” when done.

NOTE: When creating these jobs, in order to simplify the creation of the “git” post-receive bash script to launch a build remotely, you should consider naming your jobs the same as the name of your repository/repo-directory name. So, if you have your repos on your git server in the /var/git/reponame.git directory,then this job would be called reponame. As we progress through this tutorial, it will become apparent why this makes things easier.

4. Now we need to fill in some basic configuration for our newly created Deploy-Test job as shown in the following screenshots.a. Enter a description for our job (not

required)b. If you would like to setup automatic

deletion of old builds, check the “Discard Old Builds” checkbox (not needed for this build, as nothing will be downloaded or created in this deployment scenario).

c. You will probably want to leave project based security enabled, add any additional users and set permissions as required.

d. We will configure 3 build parameters for this build.

e. The first parameter is “REPO” and will be used to select the git repository to deploy.

f. The second parameter is “BRANCH” and will determine the branch of the given repository to deploy. The “BRANCH” parameter should default to master in most cases.

g. The third parameter is “SRVENV” and will contain the server enviroment we want to deploy the git repository to (in our shop we use [test, demo, staging, and live environments] ).

h. Check the “Execute concurrent builds if necessary” option.

i. No Source Code Management needed for deployment jobs.

j. Check “Trigger Builds remotely” option, and enter a token that can be used to deploy the sites. For security reasons, only persons designated to deploy code to the production environment should be given this token.

k. Check the “Color ANSI Console Output” option. (not required)

l. Now we’ll add a BASH script to our job, shown to the right here. This will get you started, and can be expanded upon depending on your requirements.

m. That’s it for Jenkins using this deployment method. Scroll down to the bottom of the job creation screen and click “Save”.

n. We’ll now move on to the server side script that will be used with both of the above Jenkins deployment jobs.

#!/bin/bash -x######## This script is an example ONLY and should be customized for the site/modules you are working on...######

####### This is pretty straight forward actually, and can be accomplished a couple of different ways. This# script is designed to be used inside the Jenkins environment, with a job that can deploy a repository# to a matching server environment.## Requirements:# SSH Keys setup for the Jenkins user account# SSH authorized_keys on destination servers must contain the jenkins account pub key in the# destination servers deployment account# The generic version of git-update.sh script included with this job script.# The deployment account on each destination server needs to have SSH keys setup to allow read-# only access to the git repository server. This will enable the deployment account to retrieve the# appropriate repository and branch to deploy without the need for human interaction.## Jenkins job parameters:# REPO = GIT Repository to deploy# BRANCH = GIT Repostiry branch to deploy# SRVENV = Should be one of the following for CG's environment / configuration. Your setup may# differ.# test - test server environment# staging - staging environment for final QA# demo - demonstration server environment# live - production server environment######

### Configuration Variables #### Your deployment user accountDEPLOY_ACCOUNT="deploy_user"

# The base domain for all your servers in your test/production environment this could also be moved# up to the Jenkins job parameters section if you need to deploy to several different domainsDOMAIN="exampledomain.com"

# set the deployment server based on supplied informationif [ "x"${SRVENV%live} = "x" ]; then # deploying to the live / production environment DEPLOY_SERVER="${REPO,,*}.${DOMAIN,,*}"else # deploying to an environment other than live / production DEPLOY_SERVER="${REPO,,*}-${SRVENV,,*}.${DOMAIN,,*}"fi

# now we'll simply execute the git-update.sh script (installed on the deployment# destination server) with the appropriate parametersssh ${DEPLOY_ACCOUNT}@${DEPLOY_SERVER} git-update.sh ${DEPLOY_ACCOUNT} ${BRANCH}

1. The following script will need to be placed on each server that will be used as a destination for one of our Jenkins deployment jobs, and should be placed within the path and accessible / executable by the deployment account.

Generic git Deployment Script#!/bin/bash

script=`basename $0`;usage=$(cat <<EOF_USAGEUSAGE: $script <cgnum> [branch]EOF_USAGE);

if [[ $1 == '' ]]; then echo $usage >&2; exit 1;fi

gitconfig=$(cat <<EOF_USAGEYou need to configure your git identity and try again.\nThis information is required once for identifying who is\nresponsible for each tag this script creates during a deploy.\n\nExample:\n$ git config --global user.name “Git User"\n$ git config --global user.email “guser@ourdomain.com"\n\nTemplates to copy/paste:\n\ngit config --global user.name\ngit config --global user.email\nEOF_USAGE);

if [ ! -e "$HOME/.git/config" ]; then echo -e $gitconfig >&2; exit 1;Fi

HOSTNAME=`hostname`

if [ ${HOSTNAME:(-8)} == "-staging" ]; then REPO=${HOSTNAME%-staging} ENVIRON="staging“elif [ ${HOSTNAME:(-5)} == "-test" ]; then REPO=${HOSTNAME%-test} ENVIRON="test"

elif [ ${HOSTNAME:(-5)} == "-demo" ]; then REPO=${HOSTNAME%-demo} ENVIRON="demo“else REPO=${HOSTNAME} ENVIRON="live“fi

if [[ $2 == '' ]]; then BRANCH=“master”else BRANCH=$2fi

if [ -d "/tmp/clone/" ]; then echo "Removing clone directory...“ rm -rf "/tmp/clone/“fi

echo "Cloning $BRANCH...“mkdir "/tmp/clone/“

git clone -b "$BRANCH" ssh://$1@code.cgraphics.com/var/git/$REPO.git/ /tmp/clone/

cd "/tmp/clone/“

TAG=`date +%Y-%m-%d-%H-%M`HASH=`git log "$BRANCH" -n 1 --pretty=format:%h`

echo "Tagging local clone...“git tag -a -m "$BRANCH $HASH to $ENVIRON by $1" "$ENVIRON-$TAG“

echo "Pushing tag to origin...“git push origin "$ENVIRON-$TAG“

echo "Syncing the $BRANCH clone to public_html...“rsync -rl --exclude=private --exclude=".git" --exclude=".gitignore" /tmp/clone/ /var/www/

echo "Removing clone directory...“rm -rf /tmp/clone/

echo "Launch complete!“echo "Template for pasting to a related issue:“echo "Deployed $HASH with tag $ENVIRON-$TAG."

2. That’s it, just place this script in the path, setup one of the previous Jenkins jobs modified appropriately for your environment, and you will be on your way to automated deployment of your code via your Jenkins CI Server.

Drupal Continuous Integration with Jenkins - Deploy

Enjoy, and feel free to contact us with any suggestions, corrections or mistakes we may have made or that inadvertently slipped between the cracks.

John W Smith (johns@knowclassic.com) or (Jsmith@i1Technologies.com)