Blake Lusenhop & Bill Creager

Speeding up Jenkins Test Runs

Improving Test Automation

● Eventually codebase will become too large to run sequentially

● Releases/patches/hotfixes are all time sensitive

● Test has a greater responsibility to turn around finished code in a timely manner

● Must ensure that test results are accurate

Common Problems● Slow execution time

○ ex: 120m + regressions

● Unreliable test results○ Timing issues can cause test results to be

inconsistent

The Solution: Parallel Test Execution● Parallel test execution can decrease your

regression length drastically● Local parallelization vs Multi-System Setups

○ CPU core based vs Machine/VM based● Local Example: parallel_tests gem

○ https://github.com/grosser/parallel_tests○ Run separate process on each core: i5 could run up

to 4 simultaneous processes (2 physical and 2 virtual cores)

Parallel Test Considerations● Each individual test case must be able to be

run independently of other tests○ Tests are run randomly○ No tests that require pre-reqs to be run

● Be conscious of what tests are hitting what areas of the application○ Possibility of multiple tests trying to modify the same

data at the same time○ If possible, group tests by feature of the app so only

one test will hit that area at a time

Parallel Test Considerations Cont.● Do certain tests require test data to be

loaded before execution?● Do other tests require a clean DB install?● Think about what your individual tests

require and group them accordingly○ Example: testdb regression that uses a stock DB

when run vs non_testdb which uses a clean DB install

○ Use tags on your tests so that you can execute tests in groupings that you create

Using Reruns for Better Test Results● Automation can be very inconsistent ● Consider automatically rerunning failed tests

multiple times before logging a failed result● Prevents having to run an entire regression

just because of a single timing error

Log Individual Failures to Reduce Rerun Time

● Log each failing test● Gather the failures and pass back in as a

parameter for next test run● Re-run only failed examples

Test:Unit

Test:Functional

Test:Engines

Test:Integration

Jenkins: Slave

Jenk

ins:

Mas

ter

Problems● Physical constraints, always on● HUGE spinup time ● All or nothing ● Dirty data (database, indexes, file

system, cache)● Impossible to run locally (devs)

Traditional Jenkins Setup

Docker

● Acts as a slave to Jenkins● Fully baked environment

○ Postgres, Redis, Elasticsearch○ Code, git, jruby○ EVERYTHING!!!

● Multiple (jailed) copies can be provisioned● Disposable, portable, consistent containers

Dockerized Jenkins Setup

Test:Unit

Docker: Slave

Test:Functional

Docker: Slave

Test:Integration

Docker: Slave

Test:Engines

Docker: Slave

Jenk

ins:

Mas

ter

Docker solutions● Slaves spin up when needed, and

spin down when jobs complete● Short turnaround● Choose your own adventure● Run locally● CLEAN data

Quick Docker tips● VM vs Container● Script it, and zip it!

○ Dockerfile○ build.sh○ setup.sh

● Mount your data● Save your progress● Supervisord for multiple runnables● Clean up, and watch your cache!

Examples - build.sh, supervisord, Dockerfile

Examples - clean_instances, clean_images

Questions?