scaling software builds with jenkins
DESCRIPTION
This talk revolves around the scalability strategies for Jenkins, the most used Continuous Integration server. In particular, the talk focuses on how to scale Jenkins on the cloud (Amazon EC2) and create a build matrix so that the code can be built on different OSes. Furthermore, we will also look at the Cloudbees offering.TRANSCRIPT
![Page 1: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/1.jpg)
Scaling Software Buildswith Jenkins
01
![Page 2: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/2.jpg)
About me
Name: Andrey Adamovich
Bio: Developer, coach, speaker, author
Company: Aestas/IT
E-mail: [email protected]
Linkedin: http://www.linkedin.com/in/andreyadamovich
•••••
02
![Page 3: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/3.jpg)
Software development process
03
![Page 4: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/4.jpg)
Why scale?04
![Page 5: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/5.jpg)
Feedback loop
05
![Page 6: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/6.jpg)
Reason 106
![Page 7: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/7.jpg)
Monolithicbuild
07
![Page 8: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/8.jpg)
Monolithic build
Time-consuming
Resource-consuming
Frustration-accumulating
•••
08
![Page 9: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/9.jpg)
Before youcan scale...09
![Page 10: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/10.jpg)
SPLIT!!!10
![Page 11: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/11.jpg)
Reason 211
![Page 12: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/12.jpg)
Too manythings to
build12
![Page 13: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/13.jpg)
Too many things to build
Too many components
Too many branches
Too many unit/integration tests
Too many validation rules
••••
13
![Page 14: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/14.jpg)
Reason 314
![Page 15: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/15.jpg)
Too manythings to test
15
![Page 16: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/16.jpg)
Too many test variations
Different operating systems
Different target platforms
Different markets
Different customers
••••
16
![Page 17: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/17.jpg)
Reason 417
![Page 18: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/18.jpg)
Lack ofresources
18
![Page 19: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/19.jpg)
Reason 519
![Page 20: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/20.jpg)
Lack of time!20
![Page 21: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/21.jpg)
Solutions?21
![Page 22: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/22.jpg)
Add morehardware!
22
![Page 23: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/23.jpg)
or...23
![Page 24: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/24.jpg)
Go to theCloud!
24
![Page 25: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/25.jpg)
Who is Jenkins?
25
![Page 26: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/26.jpg)
Architecture
26
![Page 27: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/27.jpg)
UI
27
![Page 28: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/28.jpg)
Master/Slave
28
![Page 29: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/29.jpg)
Labelling
29
![Page 30: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/30.jpg)
Slaves vs.Executors
30
![Page 31: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/31.jpg)
Executor contention
31
![Page 32: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/32.jpg)
Other types of contention
CPU contention
I/O contention
••
32
![Page 33: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/33.jpg)
Cloud slaves
33
![Page 34: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/34.jpg)
Executor scaling
34
![Page 35: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/35.jpg)
Amazon Web Services
35
![Page 36: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/36.jpg)
Elastic Compute Cloud
instances
storage
images
•••
36
![Page 37: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/37.jpg)
EC2 Console
37
![Page 38: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/38.jpg)
ALERT!!!38
![Page 39: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/39.jpg)
It's not free!39
![Page 40: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/40.jpg)
But...40
![Page 41: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/41.jpg)
There is aFree Tier!
41
![Page 42: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/42.jpg)
Cost calculations
Name Compute Units Memory Hourly Cost
t1.micro 2 0.60 GB $0.02
m1.small 1 1.70 GB $0.06
c1.medium 5 1.70 GB $0.14
m1.medium 2 3.75 GB $0.12
m1.large 4 7.50 GB $0.24
c1.xlarge 20 7.00 GB $0.58
m3.xlarge 13 15.00 GB $0.5042
![Page 43: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/43.jpg)
Labor cost
43
![Page 44: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/44.jpg)
Public access44
![Page 45: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/45.jpg)
Public Clouds
VMWare vCloud
Ubuntu Enterprise Cloud
Microsoft Azur
Google Cloud
Digital Ocean
Cloudbees Cloud
••••••
45
![Page 46: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/46.jpg)
Private Clouds
OpenStack
VMWare vSphere
Amazon VPC
•••
46
![Page 47: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/47.jpg)
Jenkins plugins
47
![Page 48: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/48.jpg)
Jenkins EC2 plugin
Allow Jenkins to start slaves on EC2 or Ubuntu Enterprise Cloud
(Eucalyptus) on demand, and kill them as they get unused. With this
plugin, if Jenkins notices that your build cluster is overloaded, it'll start
instances using the EC2 API and automatically connect them as
Jenkins slaves.
“
48
![Page 49: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/49.jpg)
Preparation49
![Page 50: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/50.jpg)
1. Configure access/secret key
50
![Page 51: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/51.jpg)
2. Configure key pair
51
![Page 52: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/52.jpg)
3. Configure security group
52
![Page 53: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/53.jpg)
4. Select machine image
Amazon base images
AWS market place
Create your own
•••
53
![Page 54: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/54.jpg)
5.1. Configure EC2 plugin
54
![Page 55: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/55.jpg)
5.2. Configure EC2 plugin
55
![Page 56: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/56.jpg)
5.3. Configure EC2 plugin
56
![Page 57: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/57.jpg)
Demo57
![Page 58: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/58.jpg)
Cloudbees
Company behind Jenkins
Kohsuke Kawaguchi (author of Hudson/Jenkins) works there
Jenkins Enterprise Edition
Jenkins Cloud = DEV@Cloud
••••
58
![Page 59: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/59.jpg)
DEV@Cloud
API & SDK
Pricing based on size (# masters/executors/slaves)
Slave OS variety
iOS coming soon
Windows Phones/Tablets coming soon
Manual or auto-scaling
Minute-based pricing
•••
••
••
59
![Page 60: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/60.jpg)
Free-tier
300 free build minutes a month
1 build executor
FOSS programme for open-source projects
•••
60
![Page 61: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/61.jpg)
Services Partner
61
![Page 62: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/62.jpg)
Questions?62
![Page 63: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/63.jpg)
GitHub projects
Jenkins setup:
https://github.com/aestasit/talks2013-devconfu-scaling-builds-setup
Slides:
https://github.com/aestasit/talks2013-devconfu-scaling-builds-slides
63
![Page 64: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/64.jpg)
Scaling Software Builds with Jenkins
Author: Andrey Adamovich
Company: Aestas/IT
E-mail: [email protected]
Linkedin: http://www.linkedin.com/in/andreyadamovich
••••
64
![Page 65: Scaling software builds with Jenkins](https://reader033.vdocuments.site/reader033/viewer/2022060109/5552a27bb4c9051f108b55f2/html5/thumbnails/65.jpg)
Feedback
65