How to push to production a project with 100+ plugins inless than 10 minutes

Senior Consultant @ Liferay Brazil

Thiago Moreira

Agenda

•Motivation•About the project•Script workflow•Benefits•Challenges•References•Future

Motivation (1/2)

Back in 2010 I was working in another consulting project and we were getting ready to push to production a new version of an intranet based on Social Office when I had the following phone talk with Michael Han

Motivation (2/2)

•Thiago: Hey mike, I have to say that I’m confortable with the project implementation but I’m a little worried about the task of push to production… (waiting a response like: don’t worry everything will be fine) •Mike: Thiago, something will go wrong! The thing is, you have to be ready to fix it.•Thiago: Oh man!

About the project (1/5)

•1 ext•4 layout templates•121 portlets (21 projects with 1 portlet and 1 project with 100 portlets•12 hooks•6 themes•4 webs

•Total 148 plugins.

About the project (2/5)

•4 Production server in cluster•1 QA server•0 Development server•1 Jenkins server•1 Nexus server (shared with Jenkins)•Github as source code server

Environment profiles

Liferay plugins

Liferay patches

Tomcat and Liferay bundles

About the project (3/5)

Deployment host,

deployment user,

application user, JVM

settings, etc

Main script

Properties files

About the project (4/5)

•3 Jenkins jobs were configured to each of the 3 branches on git.

• Master -> development (every push to upstream)

• QA (homologacao) -> QA (every day at 7:00 am)

• Production -> production (push of a button)

About the project (5/5)

Jenkins

Production Environment

Development Environment

QA Environment

Script Workflow

Check Environment

Clean Build Tomcat

Build Liferay

Build Application

Configure Tomcat

Deploy to environment

Script Workflow

Check environment

•It is used to check if the environment is correct

1. It checks if the developer set the target.environment variable

2. It checks if the environment has the correct Ant version (configurable)

3. It checks if the environment has the correct Maven version (configurable)

Script Workflow

Clean

1.It is used to clean the main build directory e.g. the target directory.

Script Workflow

Build Tomcat1.Unpack a vanilla Tomcat from the ${basedir}/src/main/bundle directory to ${basedir}/target

2.Delete all webapps’ directory content3.Delete all *.bat files from bin directory4.Set execution permission on *.sh files from bin directory

Script Workflow

Build Liferay

1.Unpacks the ${basedir}/src/main/bundles/liferay-version.war to ${tomcat}/webapps/ROOT

2.Unpacks the ${basedir}/src/main/bundles/liferay-dependencies-version.zip to ${tomcat}/lib

3.Install and configure the patching-tool-4.zip into the right place (liferay home)

4.Copy the license to the deploy directory

Script Workflow

Build Application (1/2)

1.Build and direct-deploy the ext project Ant based2.Call Maven to build the other projects

• The sequence of build is defined at the pom.xml and it is • commons• webs• layouttpl• themes• hooks• portlets

Build Application (2/2)

3.Call direct-deploy on each of the projects built by Maven4.Copy the patches to the patching-tool home5.Copy plugins (downloaded from liferay.com) to deploy directory

Avoid Liferay to deploy the plugins

on startup (cold deploy)

Script Workflow

Configure Tomcat (1/2) Worker-Name1.Creates a MANIFEST.MF file with set a properties that identifies the current build

2. Configure portal-ext.properties with environment properties

3. Configure server.xml with environment properties

Can be accessed thought

www.domain.com/html/environment.t

xt

Configure Tomcat (2/2) Worker-Name4. Configure setenv.sh with the proper TOMCAT_HOME and

CATALINA_OPTS

Script Workflow

Deploy to environment (1/2)

1.Pack a tgz bundle with everything built2.Shutdown the remote Tomcat3.Remotely remove the old Liferay4.Copy the bundle to the remote machine5.Unpack the bundle 6.Change the ownership of liferay home to the application user and group

7.Configure *.sh to be runnable8.Apply the patches9.Start the remote Tomcat10.Wait 90 seconds (configurable) until start the next node

Deploy to environment (2/2)

Benefits

•We use the same script to build and deploy on developers machine as well QA and Production boxes. This ensure that the script is validated several times before run on production.•The whole process does not take more than 10 minutes to finish on any environment.

•We are able to rollback the version in production within a few minutes

Challenges

•Upgrade the version of Liferay from 6.0.11 to 6.0.12•Push changes that affects the database•If needed put the production environment in maintenance automatically through script. •Test cluster configuration directly on production

References

•Continuous Delivery: Reliable Software Releases Through Build, Test, And Deployment Automation by Jez Humble and David Farley

Future

•Short term

• Add jsp pre-compilation

• Replace the direct-deploy Ant task per a Maven goal

• Use a sudo user to remote deploy instead of root•Middle term

• Move the infra structure to a PaaS (Cloudbees)

• Script the process of update and restart the web server when needed

•Long term

• Create a Maven plugin to replace the Ant script

Thank you!

thiago.moreira@liferay.com

tmoreira2020 @ facebook | linkedin | plus | slideshare | twitter

Don’t forget to rate the presentation on our mobile app!