juc paris 2012 — jenkins @ nuxeo

Jenkins User Conference Paris, 17 April 2012 #jenkinsconf

Julien Carsique, DevOps

https://qa.nuxeo.org/jenkins

https://qa.nuxeo.org/jenkins


17/04/2012 2/25

The Nuxeo company — www.nuxeo.com

● Founded in 2000

● ~ 50 employees (mainly France & USA)

● Open Source Enterprise Content Management

● No-charge software licensing with LGPL

● Nuxeo Connect subscription offering for support, maintenance, consulting, training and services:

● Nuxeo Studio

● Nuxeo Marketplace

● Nuxeo Cloud

http://www.nuxeo.com/


17/04/2012 3/25

The Nuxeo productsDocument and Case Management


17/04/2012 4/25

The Nuxeo productsDigital Assets Management


17/04/2012 5/25

The Nuxeo productsWeb sites


17/04/2012 6/25

The Nuxeo productsPublic facing applications, RIA and much more …


17/04/2012 7/25

The Nuxeo productsOpen Source ECM platform, applications and services

● Nuxeo Platform● Java and standards based

● Component based architecture

● Extensible and configurable ECM services

● UI building blocks and High level frameworks

● A lot of integration levels and APIs

● Testing, benchmarking and packaging tools

● Development tools (Nuxeo Studio, Nuxeo IDE)

● Ready to use pre-packaged applications


17/04/2012 8/25

The challenge ofThe source code

● 200+ SubVersion, Mercurial and Git repositories

● 1M+ LoC, 500+ Maven modules

● 5 application modules and 60+ addons

● 500+ components in the Platform

● 2500+ unit tests, 2000+ Selenium and WebDriver tests

● 10+ languages (Java, XML, JavaScript, HTML, CSS, Python, Shell, SQL, Groovy, …) and tools (Maven, Ant, Shell, …)

● 30~50 active committers and contributors

● ~1000 commits/month

● 1 development branch, 3~5 maintenance branches


17/04/2012 9/25

The challenge ofThe human constraints

● Follow and maintain quality

● Save developers time● Doing more checks than they can

● Providing a safety net for their changes

● No need to build everything themselves

● Save testers time● Automate tests in various environments

● Warn on regression

● Easy to download latest deliverables

● Save integrators time● Work on Nuxeo development branch with more confidence

● Get guarantees on code quality


17/04/2012 10/25

The challenge ofThe deployment constraints

● Multiple deployment target● WAR, JEE or bare Java

● OS (Linux, Unix, Windows, MacOS)

● DB (PostgreSQL, Oracle, SQLServer, MySQL, …)

● BlobStore backend (FS, NAS, S3, SQLDB, ...)

● Scalability and performance● Cluster support

● Multi-DB scale-out (several repositories and/or DB)

● Raw performances (100M docs benchmark on a single server; 200 pages/s output on complex business apps)

● Easy maintenance● Backup, restore, upgrade

● export/import


17/04/2012 11/25

The history ofThe beginning

● Before 2007: BuildBot● 2007 — Nuxeo 5 switch to Java

● Trying CruiseControl, TeamCity, Continuum, … Hudson!

● Goals for Nuxeo 5.1● Code analysis (Quality Assurance)

● Build and unit test Nuxeo against default environment

– Linux, JBoss, Sun JDK 5, H2

– Smoke test at JBoss start

– Run Selenium suite● Automate Nuxeo release process● Automate generation and test of documentation

● Involve developers


17/04/2012 12/25

The history ofNuxeo 5.1 — CI architecture

UP

LOA

D

DO

WN

LO

AD

RELEASES

RELEASES

UP

LOA

D R

ELE

AS

ES

& S

NA

PS

HO

TS

RELEASES and SNAPSHOTSfrom outside

RELEASES

Public Internal Nexus Private Internal Nexus Nexus Preprod

Public Jenkins Master Private Jenkins Master

UP

LOA

D

DO

WN

LO

AD

UP

LOA

D

DO

WN

LO

AD

Maven.nuxeo.org


17/04/2012 13/25

The history ofNuxeo 5.1 — CI architecture and usage

● Speed up build time

● Organize teams resources

● Ensure publicly available artifacts quality

● Isolate internal and external build chains

● Enforce rules for developers● Early use of CI

● Think about what kind of build (Maven, Shell, Ant, …) and what must be checked for validation (unit tests, errors in logs, …)?

● Automated build: nothing specific or contextual to the slave

● Make the build self-testing

● Stay tuned, be aware of CI activity

● Never let a project in an unstable status


17/04/2012 14/25

The history ofNuxeo 5.1 to 5.3 — making heavy use of CI

● Release process improvement● Automated tasks

● Nightly / On demand

● Build promotion

● More and more tests and environments● More performance and functional tests

● Tests with Jetty, JBoss and Tomcat

● Matrix testing jobs with OS and databases

● Testing development and administration tools

● Seriously increasing number of builds● New packages and installers (Debian, VM, Windows, Multi-OS, …)

● New applications (Android, Nuxeo CMF, Nuxeo DAM, …)

● New plugins for Firefox, IE, OpenOffice/LibreOffice

● Accumulating versions to maintain


17/04/2012 15/25

The history ofNuxeo 5.3 to 5.4 (2010-2011) — growing CI issues

● CI performance issues● CPU, network and disk usage

● Slave usage strategy

● Full cycle duration

● CI usage issues● Build waiting time

● Feedback delay

● Feedback accuracy

● Jobs' dependencies


17/04/2012 16/25

The history ofNuxeo 5.4 (2011) — CI improvements, Jenkins analysis

● Build cycles, workflows and durations

● Use of Jenkviz (homemade tool), Dependency Graph View and (enhanced) Build Timeout plugins


17/04/2012 17/25

The history ofNuxeo 5.4 (2011) — CI improvements, solutions

● Work on source code● GWT optimizations (reduce build time)

● Review testing suites, prefer use of WebDriver over Selenium

● Work on CI● Improve slaves usage (using labels), local vs remote

● Give priority to jobs with low duration (earlier feedback)

● New short and permanent cycle

● Create independent jobs for specific purposes

● Create new views to help developers

● Add CPU and/or memory on some slaves

● Improve Nexus infrastructure (Maven build time)

● Fix jobs triggering issue


17/04/2012 18/25

The history ofNuxeo 5.5 — the current status


17/04/2012 19/25

The history ofNuxeo 5.5 — the current status

● Great coverage and good delays● 4 kinds of CI cycles per Nuxeo version maintained

– Triggered by commit (300 jobs)

– Full build and tests per day (10 jobs)– Independent cycle per 3 hours (11 jobs)

– Nightly build (40 jobs)● Less than 10 minutes to 3 hours max for main feedback

● Code and applications tested on multi-DB/OS/JDK/servers

● 6 hours for the longest build cycle

● About 450 jobs on 25 servers (2 masters), 200~?K builds/day

● 6 database engines tested on 5 operating systems

● 7 applications tested in all environments

● 50K~100K tests ran including all products and versions under CI


17/04/2012 20/25

The history ofNuxeo 5.5 — the current status, major CI cycles

CommitMercurial/GitHub

Sourcescheckout

Build & Unit tests

Job trigger

SCMtrigger

Packaging7 apps, 2 servers

Sourcescheckout

dailytrigger

Packaging multi-OS Smoke tests

Sourcescheckout

3htrigger Full build (no tests) Packaging all applications Functional tests

Permanent cycle

Merge « stable »branch (GitHub)

Sourcescheckout

Full build (no tests)Prepare release

Packaging all applications

Merge «5.6-SNAPSHOT»branch (GitHub)

Deploy «5.6-SNAPSHOT»Maven artifacts

Build installers (Debian, Win, multi-OS, ...)

Test and upload installers

Performance tests (référence)Perform release

Batch task

Push release tag (GitHub)

Deploy release artifacts

Functional testsmulti-DB/OS/JDK/servers, cluster

Functional tests multi-OS/DB

Performance tests multi-DB

Various addons functional tests

Full build & tests multi-JDK

Full build & tests multi-DB Linux

Full build & tests multi-DB Win

Nightly

Manual


17/04/2012 21/25

The history ofNuxeo 5.5 — main Jenkins plugins used

● Required● Subversion, Mercurial, MSBuild, GIT, GitHub, Android Emulator

● Useful for Jenkins management● Priority Sorter, Parameterized Trigger, Retry Failed Builds, Rebuilder,

Hudson Post build task, Matrix Reloaded, next-executions, Bulk Builder, Join, Promoted Builds, batch task

● Change Log History, Maven Deployment Linker, SCM Sync Configuration, Multiple SCMs, Static Analysis Utilities, Timestamper, build timeout, Hudson Setenv, java.io.tmpdir cleaner, Hudson global-build-stats, Node and Label parameter, Monitoring, SSH Slaves, Dependency Graph Viewer, Performance, Disk Usage

● Useful reports● Checkstyle, FindBugs, Task Scanner, PMD, Warnings, Hudson Seleniumhq, Selenium

HTML report, Cobertura, TextFinder, JUnit Attachments, Duplicate Code Scanner

● For end-users● Dashboard View, Claim, Radiator View, Hudson iPhoneView, Continuous

Integration game, instant-messaging, Jabber notifier, built-on-column, Compact Columns, Version Column, Email-ext, View Job Filters, Extra Columns, Simple Theme


17/04/2012 22/25

The history of Nuxeo 5.6 — the near future

● Better quality testing with Sonar

● Integration with Jira

● Automated merges from QA dedicated branches to the development branch

● Automated Jenkins infrastructure deployment on cloud

● Custom slaves on demand(Chef / Puppet, chroot / LXC, …)

● Work on Nuxeo Cloud & Compatible One R&D

● CloudBees

● Nuxeo CI (partial or full) migration to the cloud


17/04/2012 23/25

The history of Nuxeo 5.6 — the near future

● Jenkins enhancements (plugins)● More abstraction to avoid plugins compliance

● Multiple SCM & Git

● SCM change log feeder

● Matrix jobs improvements

● Node Label & Parameterized trigger

● Dependency Graph

● Infrastructure failures (Check job prerequisites++)

● Jobs workflow management

● …


17/04/2012 24/25

Conclusion

● Jenkins● Easy maintenance

● Good scalability

● Great community and plugins catalog

● Powerful extensibility

● Efficient contribution process

● Cloud capabilities

● Nuxeo● CI & QA on Nuxeo Platform frameworks

● Stable APIs with backward and forward compatibility

● Stable applications with good performances

● Automated processes with very few manual interventions

Jenkins User Conference Paris 17 April 2012 #jenkinsconf 17/04/201225

Thank You To Our Sponsors

Platinum Sponsor

Silver Sponsor

Gold Sponsor

juc paris 2012 — jenkins @ nuxeo

Technology

jenkins user conferenceparis

jenkinsconfthe nuxeo

nuxeo cmf

nuxeo dam

nuxeo ideready

buildbot2007 nuxeo

unit test nuxeo

nuxeo development branch