jenkins job builder: our experience

Jenkins Job BuilderOur experience

Open Source and Linux Lab

OSLL

Our setup

• Product to build and test: Maxscale – database proxy, lots of plugins (routers, filters, protocols, etc, etc, etc)

• Test environment:

– at least 9 VM (1 Maxscale server, 4 Master/Slave servers, 4-servers Galera cluster)

– libvirt/qemu, AWS virtual machines + Docker containers

– need to configure backend for every test run

– need to carefully clean up

• 12 Linux distributions, 3 major MariaDB version, 2 major MySQL version, a lot of Maxscale plugins – any combination should be testable

• dependency on VM provider: it can easily FAIL

What does it mean for Jenkins?

• Lots of parameters for every job

• Cascades of jobs: e.g. prepare backend, install Maxscale, run test, cleanup

• Error processing (e.g. backend VMs preparation failure)

• Matrix jobs: test a lot of combinations of distros, backedn MariaDB version, backend parameters

• very dynamic setup

We wanted to put all Jenkins jobs into GIT repository

Jenkins Job Builder

• A tool from OpenStack developers https://docs.openstack.org/infra/jenkins-job-builder/

• YAML

• Pipelines did not exist when we started

• our jobs: https://github.com/mariadb-corporation/maxscale-jenkins-jobs

What we get from JJB?

• All our jobs (54) are represented as a code (branches, forks, review, pull requests, …)

• Parameters values are in one place for all jobs (easy maintenance of parameters defaults and lists)

• Automatic validation of jobs takes place at every push to jobs repo

• very easy jobs deployment

• at the beginning: app. 20 jobs, we spent 1 week to convert to JJB code (1.5 persons)

- job: name: run_test description: 'This job perform integration testing of maxscale' parameters: - !include: './maxscale_jobs/include/boxes_all_incl.yaml' - !include: './maxscale_jobs/include/products_incl.yaml' - !include: './maxscale_jobs/include/versions_incl.yaml' .... - !include: './maxscale_jobs/include/test_branch.yaml' - !include: './maxscale_jobs/include/slave.yaml' scm: - git: # TODO parametrize this url url: https://github.com/mariadb-corporation/maxscale-system-test.git branches: - $test_branch builders: - !include: './maxscale_jobs/include/build_parser/create_env_vars.yaml' - !include: './maxscale_jobs/include/build_parser/inject_initial_env.yaml' - !include: './maxscale_jobs/include/build_parser/run_test_and_collect.yaml' - !include: './maxscale_jobs/include/build_parser/parse_build_log.yaml' - !include: './maxscale_jobs/include/build_parser/inject_build_results.yaml' - !include: './maxscale_jobs/include/build_parser/create_env_coredumps.yaml' - !include: './maxscale_jobs/include/build_parser/inject_coredumps_var.yaml' - !include: './maxscale_jobs/include/build_parser/write_build_results.yaml' publishers: - !include: './maxscale_jobs/include/build_parser_mail_subject_with_name.yaml' - !include: './maxscale_jobs/include/call_cleanup.yaml' wrappers: - !include: './maxscale_jobs/include/workspace-cleanup.yaml' - !include: './maxscale_jobs/include/timeout.yaml' concurrent: true

choice: name: box choices: !include: './maxscale_jobs/include/boxes_all.yaml' description: 'Virtual machine OS (name of Vagrant box)'

- centos_7_libvirt- centos_6_libvirt- centos_5_libvirt- ubuntu_wily_libvirt- ubuntu_wily_aws…- suse_13_libvirt

boxes_all_incl.yaml

boxes_all_incl.yaml

shell: '$HOME/build-scripts/test/run_test.sh | tee $WORKSPACE/build_log_$BUILD_ID; echo ${PIPESTATUS[0]} > result_$BUILD_ID'

boxes_all_incl.yaml

email-ext: recipients: !include-raw: "./maxscale_jobs/include/mail_recipients.yaml" reply-to: $DEFAULT_REPLYTO content-type: default subject: $DEFAULT_SUBJECT ($name) body: !include-raw: "./maxscale_jobs/include/build_parser_email_body" attach-build-log: false ...

boxes_all_incl.yaml

Problems: !includechoice: name: box choices: !include: './maxscale_jobs/include/boxes_all.yaml' description: 'Virtual machine OS (name of Vagrant box)'

boxes_all.yaml

!include: './maxscale_jobs/include/boxes_libvirt.yaml'!include: './maxscale_jobs/include/boxes_aws.yaml'

boxes_libvirt.yaml

- centos_7_libvirt- ubuntu_trusty_libvirt- debian_jessie_libvirt- suse_13_libvirt

boxes_aws.yaml

- ubuntu_wily_aws- centos_7_aws- sles_12_aws- rhel_7_aws

choice: name: box choices: !include: './maxscale_jobs/include/boxes_libvirt.yaml' description: 'Virtual machine OS (name of Vagrant box)'

choice: name: box choices: !include: './maxscale_jobs/include/boxes_aws.yaml' description: 'Virtual machine OS (name of Vagrant box)'

Problems: !includechoice: name: box choices: !include: './maxscale_jobs/include/boxes_all.yaml' description: 'Virtual machine OS (name of Vagrant box)'

boxes_all.yaml

- centos_7_libvirt- ubuntu_trusty_libvirt- debian_jessie_libvirt- suse_13_libvirt- ubuntu_wily_aws- centos_7_aws- sles_12_aws- rhel_7_aws

Problems: !include

- job: name: build description: 'This job builds Maxscale' parameters: - !include: './maxscale_jobs/include/value.yaml' - !include: './maxscale_jobs/include/source.yaml' - !include: './maxscale_jobs/include/boxes_all_incl.yaml' - !include: './maxscale_jobs/include/target.yaml' - !include: './maxscale_jobs/include/cmake_flags.yaml' - !include: './maxscale_jobs/include/do_not_destroy_vm.yaml' - !include: './maxscale_jobs/include/build_experimental.yaml' - !include: './maxscale_jobs/include/run_upgrade_test.yaml' - !include: './maxscale_jobs/include/old_target.yaml' - !include: './maxscale_jobs/include/cnf_file.yaml' - !include: './maxscale_jobs/include/maxadmin_command.yaml' - !include: './maxscale_jobs/include/slave.yaml' scm: - git: url: !include: './maxscale_jobs/include/maxscale_repo.yaml' branches: - 'develop' builders: - shell: '$HOME/build-scripts/prepare_and_build.sh' - shell: 'export new_target=$target; $HOME/build-scripts/upgrade_test.sh'

build.yaml

- job: name: build_all description: 'This job builds Maxscale' project-type: matrix execution-strategy: sequential: false axes: - axis: type: user-defined name: box values: !include: './maxscale_jobs/include/boxes.yaml' parameters: - !include: './maxscale_jobs/include/value.yaml' - !include: './maxscale_jobs/include/source.yaml' - !include: './maxscale_jobs/include/target.yaml' - !include: './maxscale_jobs/include/cmake_flags.yaml' - !include: './maxscale_jobs/include/do_not_destroy_vm.yaml' - !include: './maxscale_jobs/include/build_experimental.yaml' - !include: './maxscale_jobs/include/run_upgrade_test.yaml' - !include: './maxscale_jobs/include/old_target.yaml' - !include: './maxscale_jobs/include/cnf_file.yaml' - !include: './maxscale_jobs/include/maxadmin_command.yaml' - !include: './maxscale_jobs/include/slave.yaml' scm: - git: url: !include './maxscale_jobs/include/maxscale_repo.yaml' branches: - 'develop' builders: - !include: './maxscale_jobs/include/build_all/call_build.yaml'….

build_all.yaml

Problems

• No parametrized !include

• a bit tricky to put right number of spaces in the beginning of every line (YAML problem)

• can not delete job automatically (rename case)

• Jenkins plugins have to be installed manually

• Views have to be configured manually

Jenkins CLI can solve it!

Our additions

• simple wrapper to load jobs https://github.com/OSLL/jjg.git

– adds timestamps and git commit ID to job description

– easier way to manage Jenkins creds• Job Visualizer (not ready yet, but working)

https://github.com/OSLL/jenkins_job_builder_visualization

Authors:

Evgeny Bovykin rebovykin@gmail.comMark Zaslavskiy mark.zaslavskiy@gmail.com

Job Visualizer: Call Graph

● Do you call all the jobs correctly?

● Do you have cyclic dependencies?

Our additions

• simple wrapper to load jobs https://github.com/OSLL/jjg.git

– adds timestamps and git commit ID to job description

– easier way to manage Jenkins creds• Job Visualizer (not ready yet, but working)

https://github.com/OSLL/jenkins_job_builder_visualization

Authors:

Evgeny Bovykin rebovykin@gmail.comMark Zaslavskiy mark.zaslavskiy@gmail.com

Job Visualizer: Include Graph

If you change one file, which jobs you can break?

MariaDB Maxscale Continuous Integration

Open Source and Linux Lab

OSLL

in cooperation with

Open Source and Linux Lab

Open Innovations Association FRUCT

and

http://fruct.org/

https://osllblog.wordpress.com/

https://mariadb.com/products/mariadb-maxscale

timofey.turenko@mariadb.com

Mark Zaslavskiy <mark.zaslavskiy@gmail.com>

Sergey Balandin <Sergey.Balandin@fruct.org>