running tests for every commit: gerrit, jenkins, docker, aws

Running tests for every commitGerrit, Jenkins, Docker, AWS

Alexander AkbashevDevOps Specialist16/04/2016

Nice to meet you

HERE

• employs 6,500 people in 55 countries.

• produces maps for almost 200 countries.

• enables four out of five in-car nav systems in North America and Europe

• enables mobile, web, and enterprise solutions for global industry leaders.

HERE is a leading location cloud providing superior mapping, services and location intelligence across the automotive, enterprise and internet industries.

55

Stack

• CI: Jenkins

• Tests: RobotFramework/Cucumber, GoogleTest, etc

• Target platforms: Windows 32/64, Linux 32/64, QNX, Android/iOS

Copyright © HERE 2016

66

Where we were

• Build every 40 minutes for several commits

• Review on-demand

• Each team could have own Jenkins

• Each team could have CI pipeline

Copyright © HERE 2016

77

Desired

• Build on every commit

• Review for every commit

• One Jenkins

• Unified CI pipeline

Copyright © HERE 2016

88

Numbers*

• 55 - 65k builds/day

• 5 - 6k builds/hour

• 1.8 - 2k commits/day

• 1100 -1200 executors

• 19 minutes

* - for the main productCopyright © HERE 2016

</cut>

1010

Big picture

Code Gerrit Jenkins Gerrit Repository

Copyright © HERE 2016

1111

Gerrit

• Git

• Open source

• Plugins

• Scale

• JVM

Copyright © HERE 2016

1212

Gerrit

• Code Review ( +2 )

• Test passed ( +1 )

Feedback time: 2 minutes for build fail, 19 minutes for success

Copyright © HERE 2016

1313

Jenkins

• OpenSource

• Plugins

• Community

• JVM

Copyright © HERE 2016

1414

AWS EC2 + EC2 Plugin

• Pool

• Maintenance

• Scalability

• Price

Copyright © HERE 2016

1515

Docker

• DRY

• Isolation

• Limits

• Repeatability

Copyright © HERE 2016

1616

Jenkins is not for

• Artifacts

• Statistics

• Managers

• Lazy people

Copyright © HERE 2016

1717

Pipeline plugin*

• Reduce monkey work

• Review

• Configuration as code

* - by default in Jenkins 2.0Copyright © HERE 2016

1818

In-house dashboard + Groovy Event Listener Plugin

• Nice UI

• Quick UI

• No action buttons

• Results for release scope

Copyright © HERE 2016

1919

AWS S3 + S3 plugin

• Keep artifacts longer

• Save money

• Faster

Copyright © HERE 2016

Room for improvements

2121

Divide et empera

Copyright © HERE 2016

2222

Divide et empera

Copyright © HERE 2016

2323

Divide et empera

Copyright © HERE 2016

2424

Profiling

• IO

• CPU

• Memory

• Time budget

• Cut explicitly

Copyright © HERE 2016

2525

Jenkins

• Yourkit

• GC logs

• FlightRecorder

“All magic comes with a price”

Copyright © HERE 2016

2626

Android

• TestDroid

• Battery Life

• OOM Killer

• Data updates

Copyright © HERE 2016

Stability

2828

Angry Devs

0

20

40

60

80

100

100 90 80 70 60 50 40 30 20 10 0

Copyright © HERE 2016

2929

Verification jobs

• Same tests

• Schedule

• Pass rate - 100%

Copyright © HERE 2016

3030

Dashboards

• Queue Size

• Chance to submit

• Feedback

• Executors

• Flaky tests/jobs

• etc

Copyright © HERE 2016

3131

Automatization

• Sanity checks

• Dead nodes

• Configuration

Copyright © HERE 2016

Summary

3333

Plugins

• S3 Plugin

• EC2 Plugin

• Build Failure Analyzer Plugin

• Groovy Event Listener Plugin

• Hierarchy Killer Plugin

Copyright © HERE 2016

3434

Summary

• Automate automatization

• Be strong

• Visibility

• Relax&enjoy

Copyright © HERE 2016

QAalexander.akbashev@here.com

github: Jimilian

habrahabr: Jimilian