running tests for every commit: gerrit, jenkins, docker, aws

Running tests for every commitGerrit, Jenkins, Docker, AWS

Alexander AkbashevDevOps Specialist16/04/2016

Nice to meet you

HERE

• employs 6,500 people in 55 countries.

• produces maps for almost 200 countries.

• enables four out of five in-car nav systems in North America and Europe

• enables mobile, web, and enterprise solutions for global industry leaders.

HERE is a leading location cloud providing superior mapping, services and location intelligence across the automotive, enterprise and internet industries.

55

Stack

• CI: Jenkins

• Tests: RobotFramework/Cucumber, GoogleTest, etc

• Target platforms: Windows 32/64, Linux 32/64, QNX, Android/iOS

Copyright © HERE 2016

66

Where we were

• Build every 40 minutes for several commits

• Review on-demand

• Each team could have own Jenkins

• Each team could have CI pipeline


77

Desired

• Build on every commit

• Review for every commit

• One Jenkins

• Unified CI pipeline


88

Numbers*

• 55 - 65k builds/day

• 5 - 6k builds/hour

• 1.8 - 2k commits/day

• 1100 -1200 executors

• 19 minutes

* - for the main productCopyright © HERE 2016

</cut>

1010

Big picture

Code Gerrit Jenkins Gerrit Repository


1111

Gerrit

• Git

• Open source

• Plugins

• Scale

• JVM


1212

Gerrit

• Code Review ( +2 )

• Test passed ( +1 )

Feedback time: 2 minutes for build fail, 19 minutes for success


1313

Jenkins

• OpenSource

• Plugins

• Community

• JVM


1414

AWS EC2 + EC2 Plugin

• Pool

• Maintenance

• Scalability

• Price


1515

Docker

• DRY

• Isolation

• Limits

• Repeatability


1616

Jenkins is not for

• Artifacts

• Statistics

• Managers

• Lazy people


1717

Pipeline plugin*

• Reduce monkey work

• Review

• Configuration as code

* - by default in Jenkins 2.0Copyright © HERE 2016

1818

In-house dashboard + Groovy Event Listener Plugin

• Nice UI

• Quick UI

• No action buttons

• Results for release scope


1919

AWS S3 + S3 plugin

• Keep artifacts longer

• Save money

• Faster


Room for improvements

2121

Divide et empera


2222

Divide et empera


2323

Divide et empera


2424

Profiling

• IO

• CPU

• Memory

• Time budget

• Cut explicitly


2525

Jenkins

• Yourkit

• GC logs

• FlightRecorder

“All magic comes with a price”


2626

Android

• TestDroid

• Battery Life

• OOM Killer

• Data updates


Stability

2828

Angry Devs

0

20

40

60

80

100

100 90 80 70 60 50 40 30 20 10 0


2929

Verification jobs

• Same tests

• Schedule

• Pass rate - 100%


3030

Dashboards

• Queue Size

• Chance to submit

• Feedback

• Executors

• Flaky tests/jobs

• etc


3131

Automatization

• Sanity checks

• Dead nodes

• Configuration


Summary

3333

Plugins

• S3 Plugin

• EC2 Plugin

• Build Failure Analyzer Plugin

• Groovy Event Listener Plugin

• Hierarchy Killer Plugin


3434

Summary

• Automate automatization

• Be strong

• Visibility

• Relax&enjoy


[email protected]

github: Jimilian

habrahabr: Jimilian

running tests for every commit: gerrit, jenkins, docker, aws

Software