running a test harness in parallel using jenkins
TRANSCRIPT
Running Regression
Tests in Parallel
using Jenkins
Presented by Josh Shapiro & Dustin Masterson
SiteSpect Inc.
What is SiteSpect?
SiteSpect is website testing and optimization
It is a reverse proxy
Uses regex to apply changes to page
Control panel for creating tests, analyzing data
What is the Test Harness?
Sets up SiteSpect and a website
Verifies:
Using the control panel
Visitor assignment
Data collection
Diagram of the Test Harness
Harness VM
Headless Browser
Harness
Test Web Site
Control Panel
SiteSpect
Traffic Processing
1. Configure Test
Objects/Features
2. Stream Traffic
a. User
b. SiteSpect
c. Website
d. SiteSpect
e. User
Why change what works?
350 tests
550,000 subtests
Full run takes 7.5 hours
Requires a specific system configuration
What is Jenkins?
Distributes and manages external jobs
Works across multiple machines
1000+ plugins, good community
Runs in JRE, uses Groovy for internal scripting
Jenkins Components/Terms
● Jobso Series of steps, each of which can fail or succeed
o shell scripts, groovy scripts, email notifications,
start other jobs, etc.
● Buildo A single invocation of a given job
● Downstream/Upstreamo Jobs & builds that are called by another jobs or that call
another are downstream or upstream respectively
Summary of Harness Lab
1 master VM, 24+ worker nodes (~75 VMs)
Master hosts:
Jenkins’ Control Panel
MySQL
Custom Test Results Viewers
Worker hosts:
SiteSpect, Harness, Jenkins worker Agent
Diagram of Jenkins
Jenkins Workers
24 Harness Clusters
SVN
Are there new commits?
Configure Machines, Distribute Tests
Test Results to MySQL, Test Result TAP
General Outline of How it All Works
1. Find SiteSpect version and Harness version
2. Configures a single worker
3. Configures the rest of the workersa. Cannot configure multiple boxes in parallel without
some customizations
b. Matrix Project Plugin—It’s a TRAP!
c. NodeLabel Parameter Plugin
General Outline of How it All Works
5. Finds list of all tests in Harness brancha. Automatically creates Jobs for new tests
6. Queues all the tests
7. Adds results to the database
8. Creates a result page
9. Emails the responsible developer
Other Uses for Jenkins
● Building and testing our GeoIP and
Mobile Device databases (WURFL)
● iOS Testing
o OS X worker node
o appium testing with iOS simulator
● VM setup tool
Part III: Performance
At first, performance was a nightmare!
We could not:
Load Control Panel
Display Test Results
Go a single day without a reboot
Performance Issues and Solutions
Culprits
Heap size/Machine Power
A runaway restart worker call
Front-end issues
Using the file system
Test Results Searching
Harness API Viewer - Hackathon Project
Searching test results DB
See relationships between runs that are not
tracked by Jenkins
Much faster than Jenkins
Avoid the Jenkins UI
For us:
The Jenkins UI is slow
It doesn’t look good
It hurts performance
Still popular, and they are redesigning UI
Takeaways
Parallelizing your tests makes everyone happy
Now takes 30 minutes instead of 8 hours
Jenkins is powerful and flexible
It is a very quirky program
Always check the plugins
List of Useful Plugins
Parameterized Trigger
Groovy/Groovy Post-Build
Purge Build Queue
JobConfigHistory
Priority Sorter
Build Timeout
Build User Vars