php unconference continuous integration

Continuous Integration for Heterogeneous Environments

Nils Hofmeister, 11.09.2011PHP Unconference

Hello world!

2

• Nils Hofmeister

• 5+ Years experience in realtime 3D / C++

• 2+ Years experience in web-based games (php, javascript, Java, C++, C#)

• Lead Integration Architect @Bigpoint, Hamburg – “the SVN guy” ;)

Agenda

3

• What is continuous integration?

• Why do you want continuous integration for your project?

• How do we implement continuous integration at Bigpoint?

• What are our next steps?

4

What is continuous integration?

What is continuous integration?

5

The gossip

• A way to automize certain tasks, e.g.• code validation• build automation• packaging of products

• A system of cooperating hardware- and software components , e.g.• Source code management system (SCM)• Staging system + production servers• Build cluster with all kinds of compilers and frameworks on it

The idea behind this is much more important, so we do the technical stuff later.

What is continuous integration?

6

The idea

• Not brand-new: First known example: IBM OS/360, early 60s

• Current ideas and drivers closely related to agile movement and Extreme Programming

• Martin Fowler says:• software development practice • integrate […] frequently• multiple integrations per day• verified by an automated build • detect integration errors• significantly reduced integration problems• develop cohesive software more rapidly

Martin Fowler on Continuous Integration - 2006 version

Original article – 2000

What is continuous integration?

7

The philosophy

Jez Humble says in "Continuous Delivery":

• Create a repeatable, reliable process [...]• Automate almost everything• Keep everything in version control• If it hurts, do it more freqeuntly and bring the pain forward• Build quality in• Done means released• Everybody is responsible for the delivery process• Continuous improvement

“If something hurts, do it more often”

What is continuous integration?

8

The attitude

• Accept that CI is a standard tool. Just as compilers, SCMs or documentation are• Understand that failing builds must have highest priority - always!• Keep in mind that errors happen anyway. CI only makes them visible

immediately and reliably.

9

Why do you want continuous integration for your

project?

Why do you want continuous integration for your project?

10

Reduce risks

• Defects are detected and fixed sooner

• Health of software is measurable

• Reduce assumptions (environment variables, local setup, etc)

Why do you want continuous integration for your project?

11

Reduce repetitive manual processes

• Process runs the same way every time

• The process starts automatically

• This frees resources for higher-value work

Why do you want continuous integration for your project?

12

Generate deployable software

• Product is deployable always

• Paradigm-shift from crunch time before release to "always ready to deliver"

Why do you want continuous integration for your project?

13

Better project visibility

• Supports effective decisions

• Facilitates recognizing trends

Why do you want continuous integration for your project?

14

Greater confidence in your project

• CI provides a safety net to avoid releasing flawed software

• Developers get a better feeling for impact of changes

15

How do we implement continuous integration at

Bigpoint?

How do we implement continuous integration at Bigpoint?

16

The setting

• Websites, backend systems, clients

• PHP is one of many technologies required to assemble our games (an important one, don’t worry ;))

How do we implement continuous integration at Bigpoint?

17

The requirements

• Per artifact:• Check for defects• Measure quality

• Per project:• Assemble deliverable packages• Deploy these

How do we implement continuous integration at Bigpoint?

18

The solution space (subjective! and there is more of course)

Microsoft Team Foundation ServerIntegrates fine with Microsoft Tools. But with nothing else

TeamCityWell-known product by JetBrains. Feature set is fine, but no USP that justifies to pay for it right now.

CruiseControlThe classic invented by ThoughtWorks. Now an open source project. Delightfully old-school.

BambooQuite solid, full-featured product by Atlassian. No real USP

though that justifies the high price.

GoThe current CI solution by ThoughtWorks. The theory behind it is cool but it lacks a lot of features and configuration is no fun.

How do we implement continuous integration at Bigpoint?

19

The actual solution for us

• Hudson:• Created by Kohsuke Kawaguchi• "Duke's Choice Award" in 2008• Comes from the Java world, ships with CVS/SVN, Maven, JUnit support

• Jenkins:• Kohsuke was working for Sun till Oracle acquired Sun in 2010• Conflict between the Hudson Core developers and Oracle => Fork in 2011

How do we implement continuous integration at Bigpoint?

20

The actual solution for us II

• Trends after fork• Hudson => Focus on quality & stability• Jenkins => Core developers, faster pace

• Our decision• choose Jenkins as for us features > stability.• to compensate we use Jenkins LTS

How do we implement continuous integration at Bigpoint?

21

The actual solution for us III

We like the Jenkins community!(My colleague Marcelo and Kohsuke on their way to get their hands on some beer)

How do we implement continuous integration at Bigpoint?

22

Hamburg Jenkins cluster

There are more instances, but this is the biggest one…

• Setup• 1 fileserver as master (Debian)• 3 blades (Debian)• 4 windows workstations

• Toolchains• Java• Flash/Flex• Php (+ JS, html, CSS)• Unity/Mono• C++

• Highly customized• Home-grown housekeeping + backup• Custom plugins / actively develop Jenkins• All the stuff needed to deal with Kerberos…

How do we implement continuous integration at Bigpoint?

23

Hamburg Jenkins cluster II

Under the hood:

• Unity3D• Unity/Mono + Ant

• Java• Maven + some Ant• Artifactory/Nexus (memo: synergies…)

• Flash• Ant/Maven (Not sure who will win yet)

• C++• gcc/make via Ant/Shell

• PHP/web apps• All glory to Sebastian Bergmann – we more or less use his template…• …plus some mods, e.g. minifiers, CDN stuff,

How do we implement continuous integration at Bigpoint?

24

Bigpoint standard game build pipeline

Behold: join plugin, downstream builds, copy artifacts, parameterized builds.

We do not do deployment from within Jenkins but have a custom deployment tool. Interface is SVN tags.

How do we implement continuous integration at Bigpoint?

25

Bigpoint standard build pipeline II

• Oh noes, this is too slow…• Optimize hardware• Parallelize wherever possible• Where not possible, decompose and create additional pipelines

• But… this is a SPOF!• Backup _everything_ automatically• Fully automized CI server setup• Find some replication/failover setup (todo)• Remember: everything the CI server does must run on your machine as well!

• Wait a minute… no more Ninja-deployment? But we need that to be agile…• No you don’t

How do we implement continuous integration at Bigpoint?

26

The obstacles

• The vast amount of knowledge needed

• Understanding the Bigpoint platform (the Release Engineering team is “brand new”)

• The different requirements, setup and skills of about 25 groups of stakeholders

• Linux/IT-driven platform vs. Java toolchain

• Acceptance (everyone claims to like Software Engineering, but some people seem to have a different understanding of what that is )

We can handle much of this very well by now.

Our biggest problem remaining:• Technical debt (already!)• We need a build framework…

27

What are our next steps?

What are our next steps?

28

Win the SVN + Kerberos battle

• Custom SVN sub system for Jenkins + custom Ant task• Allows svn+ssh with Kerberos for our setup. _Massive_ speedup

• At some point we will contribute this to the Jenkins community…

What are our next steps?

29

The Build Framework

• Collect learnings and known requirements

• Derive a flexible framework

• Unify maintenance and interfaces to internal customers

Either we automize things further or we need tons of people doing frustrating stuff.

We don’t like frustrating stuff.

What are our next steps?

30

The canonical build pipeline in Jenkins

https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin

• Learn from the ThoughtWorks guys!

• Interesting: build pipeline plugin – Centrum Systems

31

“If something hurts, do it more often”

Find us on32

Bigpoint GmbH

Alexanderstraße 510178 BerlinGermany

Bigpoint Inc.

500 Howard StreetSuite 300San Francisco, CA 94105

Bigpoint Distribuição de Entretenimento Online Ltda.

Av. Brig. Faria Lima3729 cj. 52804538-905 São PauloBrazil

Bigpoint GmbHNils HofmeisterLead Integration Architect

Drehbahn 47-4820354 Hamburg Germany

Tel +49 40.88 14 13 - 0Fax +49 40.88 14 13 - 11

info@bigpoint.netwww.bigpoint.net

Contact us

Bigpoint International Services Limited

1 Villa ZimmermannTa’Xbiex TerraceXBX 1035 Ta’XbiexMalta

Find us on

33

Bigpoint GmbHNils Hofmeister

Lead Integration Architect

Drehbahn 47-4820354 Hamburg

Germany

Tel +49 40.88 14 13 - 0Fax +49 40.88 14 13 - 11

info@bigpoint.netwww.bigpoint.net