![Page 1: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/1.jpg)
Continuous Integrationin the Cloudwith HudsonKohsuke KawaguchiJesse GlickSun Microsystems, Inc.Hudson committers
![Page 2: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/2.jpg)
Rise of Continuous Integration
> Offload from people, push to computers
2
$
time
computers
us
![Page 3: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/3.jpg)
Spend more CPU power to help you
> … even if it only helps a little
> First on your laptops and workstations IDEs are at the forefront
> And then to the servers a.k.a. “Continuous Integration” More frequent build/test executions Static code analysis tools And more to come
3
![Page 4: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/4.jpg)
Hudson
> Open-source CI server at java.net
> Emphasis on ease of installation and use “java -jar hudson.war” execution Configure everything from browsers
> Extensibility 140+ community-developed public plugins By 150+ contributors
> Estimated 13,000 installations4
https://hudson.dev.java.net/
![Page 5: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/5.jpg)
It basically does builds and tests
> Check out the source code Subversion, Perforce, Git, Mercurial, CVS, …
> Do builds and/or tests Ant, Maven, MSBuild, shell script, …
> Record results Binary, test results, code coverage, static analysis
> Notify people E-mail, IM, RSS, tray apps, IDEs
5
![Page 6: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/6.jpg)
Localized to 8 languages
6
![Page 7: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/7.jpg)
And hopefully more to come…
7
![Page 8: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/8.jpg)
8
Adoption in all kinds of businesses
![Page 9: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/9.jpg)
9
![Page 10: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/10.jpg)
Why Distributed Builds?
> You need to use multiple computers because… You need different environments You need isolation
> There’s only so much you can do with 1 computer
10
![Page 11: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/11.jpg)
Before we talk about clouds…
> Going virtual doesn’t solve… Software installation problem Node failure problem Remote maintenance problem …
> What does Hudson do to help you with these?
11
![Page 12: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/12.jpg)
Installing new slaves
> For first 20 or so slaves, we did it manually Insert CD, click, type, click, type, click, … But that doesn’t scale
> Then we automated Available as “Hudson PXE Plugin”
12
![Page 13: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/13.jpg)
Automated System Installations
13
> Slaves Power on, hit F12 PC boots from network (PXE)
> Hudson + PXE plugin ISO images of OS
![Page 14: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/14.jpg)
Automated System Installations
14
> Slaves Power on, hit F12 PC boots from network (PXE) Choose OS from menu Installs non-interactively
> Hudson + PXE plugin ISO images of OS
Your corporate IT guy & his DHCP server
![Page 15: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/15.jpg)
Automated System Installations
> Supports OpenSolaris, Ubuntu, CentOS, Fedora Trivial with most Linux
> Cooperate with Windows, too
> Quite useful outside Hudson, too No more broken CD drives No more CD-Rs
15
![Page 16: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/16.jpg)
Distributed builds with Hudson
> Master Serves HTTP requests Stores all important info
> Slaves 170KB single JAR Assumed to be unreliable Scale to at least 100
> Link Single bi-di byte stream No other requirements
16
![Page 17: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/17.jpg)
How master and slaves start talking
> For Unix slaves, via SSH Only need SSH and JRE on slaves We just need a host name
17
![Page 18: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/18.jpg)
How master and slaves start talking
> For Windows, DCOM We just need admin user name and password No manual intervention Works even from Unix masters
18
![Page 19: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/19.jpg)
How master and slaves start talking
> Via Java Web Start When master cannot see slaves A separate socket connection is made
19
![Page 20: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/20.jpg)
Automating JNLP launch
> Once started, can be installed as Windows service
20
![Page 21: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/21.jpg)
Automating JNLP launch
> Emulate the JNLP client headless
21
$ java -jar slave.jar -jnlpUrl URL
![Page 22: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/22.jpg)
Automated Tool Installation - JDK
> JDK from http://java.sun.com/ Hudson automatically chooses the right bundle Always up to date with new releases
22
![Page 23: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/23.jpg)
Automated Tool Installation - Apache
> Ant and Maven from Apache
23
![Page 24: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/24.jpg)
Automated Tool Installation - Custom
> Download arbitrary archive and unpack
24
![Page 25: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/25.jpg)
Automated Tool Installation - Custom
> Run arbitrary shell commands Can have variants by OS
25
![Page 26: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/26.jpg)
Automated Tool Installation - Extensible
> Write your own Simple Hudson extension Just write Java code to create tool on slave
> In progress: SCMs – Mercurial, … Install from Subversion
26
![Page 27: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/27.jpg)
Heterogeneous Cluster Challenge
> Your builds/tests need to run in specific environment
> Dependency on individual nodes hurts utilization
27
WombatWindows test
WombatWindows test
Hudson Windows test
Hudson Windows test
Windows #1
Windows #1
jobs slaves
GlassFishWindows test
GlassFishWindows test
Windows #2
Windows #2
Solaris#1
Solaris#1
Hudson Solaris test
Hudson Solaris test
![Page 28: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/28.jpg)
Labels to rescue
> Label is a group of slaves> Tie jobs to labels
28
WombatWindows test
WombatWindows test
Hudson Windows test
Hudson Windows test
Windows #1
Windows #1
jobs slaves
GlassFishWindows test
GlassFishWindows test
Windows #2
Windows #2
Solaris#1
Solaris#1
Hudson Solaris test
Hudson Solaris test
WindowsWindows
SolarisSolaris
Windows #3
Windows #3
![Page 29: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/29.jpg)
Forecasting failures
> Hudson monitors key health metrics of slaves Low disk space, insufficient swap Clock out of synch Extensible
> Slaves go offline automatically> Catch problems before they break builds
30
![Page 30: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/30.jpg)
Clean up mess after builds
> Kill runaway processes Daemons, background processes left by your build Works on Windows, Linux, Mac, and Solaris
31
![Page 31: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/31.jpg)
Load Statistics Monitoring
32
![Page 32: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/32.jpg)
When it’s time to add more slaves
33
![Page 33: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/33.jpg)
Hudson made this extensible
> Hudson detects excessive workload> Hudson notifies plugins> Plugins can provision more slaves
… assuming that you have that infrastructure
34
![Page 34: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/34.jpg)
35
![Page 35: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/35.jpg)
Amazon EC2: The Good
> Pay as you go (10¢/h or so) Loads on Hudson tend to be spiky
> Programmable API> Instances launch at machine-speed> EC2 instances are forgetful
36
![Page 36: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/36.jpg)
Amazon EC2: The Bad
> Your data is still inside your firewall Takes time to check out code … or to archive build artifacts Some data just can’t be moved
> EC2 instances are forgetful> Can your tests run in parallel?
37
![Page 37: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/37.jpg)
Hudson EC2 plugin
> Built on top of typica*> What does it do?
Automatically provisions slaves on EC2 on demand Picks the right AMI depending on demand Starts slave agent Shuts down unused instances
38* http://code.google.com/p/typica/
![Page 38: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/38.jpg)
Putting it all together
39
time
# of
exe
cuto
rs
capacity
usagequeue length
![Page 39: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/39.jpg)
Hudson “Appliance” on EC2
> Run the master in the cloud too, if you like Hudson on stock OpenSolaris AMI Data stored persistently in Elastic Block Storage
Dynamically expandable thanks to ZFS Online, too
> Packaged as a wizard
40
![Page 40: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/40.jpg)
41
![Page 41: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/41.jpg)
42
![Page 42: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/42.jpg)
> Hudson Hadoop plugin Just a few mouse clicks to install Turn every Hudson slave into a Hadoop node
> Distributed file system Automatic data replication (fault tolerant) Nice for storing old artifacts, logs, test records, …
> Map/reduce framework Large scale test results analysis / datamining More interesting work to be done in the future
43
![Page 43: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/43.jpg)
Hudson Selenium Plugin
> Selenium Tests webapps by scripting browsers
> Selenium Grid Runs Selenium over a grid of computers
> Allow Hudson labels to specify where to start browsers
> Hudson & Selenium both need heterogeneous cluster
44
![Page 44: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/44.jpg)
Selenium Grid
45
Hudson master(selenium hub)
Hudson slaves
![Page 45: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/45.jpg)
Conclusion
> CI is here to stay We’ll continue to push more workload to servers
> Hudson makes this easy for you> Reap the benefit of a cluster in multiple ways
46
![Page 46: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/46.jpg)
Resources
> http://hudson.dev.java.net/
> BOF-5105 “Hudson Community Meet up” Today 7:45pm same room
> Hudson booth inside Sun Pavilion
> Support Subscription [email protected]
47
![Page 48: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/48.jpg)
Agenda
> Quick Hudson introduction> Doing distributed builds> How Hudson make distributed builds easier> Taking infrastructure to cloud
49
![Page 49: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/49.jpg)
Setting up slaves
> Keeping slaves consistent is a good thing Particularly hard on heterogeneous environment
> General system administration tasks Network configuration Package installations for native tools Tools like Puppet or cfEngine are supposed to help
> Install build tools in the cluster Prepare tools on one file system rsync to everywhere
> This part of Hudson needs improvements50
![Page 50: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/50.jpg)
Hudson EC2 plugin usage
> Tell Hudson your AWS account information
51
![Page 51: Continuous Integration in the Cloud with Hudson](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f54550346895daa1596/html5/thumbnails/51.jpg)
Hudson EC2 plugin usage
> Tell Hudson what AMIs you want to start
52