mesos and marathon scaling jenkins on distributed ci · pdf filesingle git-based workflow...
TRANSCRIPT
![Page 1: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/1.jpg)
Distributed CI:Scaling Jenkins on
Mesos and Marathon
Roger Ignazio – Puppet Labs, Inc.MesosCon 2015 – Seattle, WA
![Page 2: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/2.jpg)
Roger Ignazio
QE Automation Engineer Puppet Labs, Inc.
@rogerignazio
About Me
![Page 3: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/3.jpg)
Mesos In Action
mesosinaction.com
Code: cftwmesos
![Page 4: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/4.jpg)
Agenda
● Puppet Labs’ testing environment● Conventional methods for scaling Jenkins● Motivations for re-architecting CI● Breaking up the Jenkins monolith● Demo, outcomes, and future work
![Page 5: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/5.jpg)
Audience Poll
![Page 6: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/6.jpg)
Intro to Mesos
● General-purpose cluster manager● Represent many machines as a single entity● Advertise resources directly to applications
![Page 7: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/7.jpg)
Intro to Marathon
● Mesos framework that provides private PaaS○ Manages long-running tasks
● Easily scale apps to N instances● Automatically restarts failed app instances
![Page 8: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/8.jpg)
Intro to Jenkins
● Distributed, open source CI tool● Repeatable build/test of software projects● Large community, rich plugin ecosystem
![Page 9: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/9.jpg)
Puppet Labs’Testing Environment
![Page 10: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/10.jpg)
Testing at Puppet Labs
● 4k to 5k builds/day across 75 platforms● 15 Jenkins clusters
○ Loosely based on team, project, function○ ~ 1,300 executors across ~ 240 build machines
● Tooling: Beaker, vmpooler
![Page 11: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/11.jpg)
Testing at Puppet Labs
● Configuration management○ Most job configs and scripts stored in Jenkins○ Infrastructure managed by Puppet
● Reporting○ Jenkins UI ( x15! )○ Clockin, Waylon
![Page 12: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/12.jpg)
Conventional Methods for Scaling Jenkins
![Page 13: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/13.jpg)
● Two common deployments○ Single Jenkins master with many jobs○ Master per team, project, or function
● Not highly available● Can’t load balance across masters● Static partitioning kills overall utilization
Scaling Jenkins
![Page 14: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/14.jpg)
Scaling Jenkins
FOSS Jenkins
. . .Jenkins Slave 1
Jenkins Slave 2
. . .Jenkins Slave N
PE Jenkins
Jenkins Slave 1
Jenkins Slave 2
. . .Jenkins Slave N
Project X Jenkins
Jenkins Slave 1
Jenkins Slave 2
. . .Jenkins Slave N
90% Utilized(Normal)
140% Utilized(40 builds in queue)
0% Utilized(Idle)
![Page 15: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/15.jpg)
Scaling Jenkins
What can we do about it?
On-demand resources!
![Page 16: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/16.jpg)
Scaling Jenkins
FOSS Jenkins . . .
Mesos Slave 1
Mesos Slave 4
Mesos Slave N
PE Jenkins
Mesos Slave 2
Mesos Slave 5
Project X Jenkins
Mesos Slave 3
Mesos Slave 6
. . .
Mesos Master
![Page 17: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/17.jpg)
Motivations forRe-architecting CI
![Page 18: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/18.jpg)
User Stories
As a $role,
I { want, need } $something,
so that $outcome
![Page 19: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/19.jpg)
User Stories
As a Developer, I want tests to be run against pull requests, so that I have confidence in the code
about to be merged
![Page 20: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/20.jpg)
User Stories
As a Developer, I don't want to worry about the underlying infrastructure of the CI system
![Page 21: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/21.jpg)
User Stories
As a CI consumer, I want a central location to view all CI activity, so that I don’t have to visit multiple URLs
![Page 22: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/22.jpg)
User Stories
As a QE, I want slaves to be on-demand, so that infrastructure resources are used more efficiently
![Page 23: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/23.jpg)
Motivation
● Reduce friction in dev workflows● Event (and data)-driven system● Improve reporting and user experience● Scale to meet growing demand
![Page 24: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/24.jpg)
Breaking UpThe Monolith
![Page 25: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/25.jpg)
The Jenkins MonolithJenkins Master
Web UI HTTP API
Disk
Configuration Management
Job Trigger (polling, cron)
Plugins
Scheduler Build Queue
Remoting (master / slave coordination)
Build Info and Results
Jenkins Slave(Executors)
GitHubProject Repo
![Page 26: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/26.jpg)
The Jenkins MonolithJenkins Master
Web UI HTTP API
Disk
Configuration Management
Job Trigger (polling, cron)
Plugins
Scheduler Build Queue
Remoting (master / slave coordination)
Build Info and Results
Jenkins Slave(Executors)
GitHubProject Repo
Devs
![Page 27: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/27.jpg)
Breaking Up The Monolith
● Job configurations● Build trigger● Build history
![Page 28: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/28.jpg)
GitHubProject Repo
Hook Processor
Jenkins
Mesos Master
Reporting Application HTTP API
Build Info and
Results
Job Config
Breaking Up The Monolith
Mesos Slave(s)
Jenkins Slave(s)
Builds
![Page 29: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/29.jpg)
Hook Processor
Jenkins
Mesos Master
Reporting Application HTTP API
Build Info and
Results
Mesos Slave(s)
Jenkins Slave(s)
Builds
Breaking Up The MonolithDevs
Build Info and
Results
GitHubProject RepoJob Config
![Page 30: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/30.jpg)
Jenkins on Marathon
● Marathon as a private PaaS● Scale Jenkins masters horizontally● Deploy updates, config changes, plugins● Continuous deployment of CI ?!
![Page 31: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/31.jpg)
Jenkins on Marathon Mesos
Marathon Marathon
Jenkins Master 1
Jenkins Master 2
Jenkins Master n
JenkinsSlave(s)
JenkinsSlave(s)
JenkinsSlave(s)
Redis + Elasticsearch, Logstash, Kibana (ELK)
![Page 32: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/32.jpg)
GitHub Webhook Processor
Marathon
Jenkins Master
Logstash andElasticsearch
Seed Job
Dynamic Job
Dynamic Job
Webhook Listener
Redis
Job Config
![Page 33: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/33.jpg)
Reporting
● Unique IDs for each event (in Redis)● Webhooks and build data (in Elasticsearch)● Query and visualize system activity (in Kibana)● Build our own reporting app?
![Page 34: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/34.jpg)
Reporting
![Page 35: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/35.jpg)
Demo
rji/mesoscon-2015-demo
![Page 36: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/36.jpg)
Outcomesand
Future Work
![Page 37: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/37.jpg)
Outcomes
● Single Git-based workflow● Standardized, stateless Jenkins masters● Jenkins slaves provisioned on-demand● Bonus: private PaaS (Marathon)
![Page 38: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/38.jpg)
Future Work
● Adoption● Single reporting dashboard (with API)● Intelligent job queueing and throttling● Job DSL plugin abstraction and templates
![Page 39: Mesos and Marathon Scaling Jenkins on Distributed CI · PDF fileSingle Git-based workflow Standardized, stateless Jenkins masters Jenkins slaves provisioned on-demand Bonus: private](https://reader031.vdocuments.site/reader031/viewer/2022030416/5aa208717f8b9ac67a8c9558/html5/thumbnails/39.jpg)
Q & A
@rogerignazio