Continuous Integration & Continuous Delivery with OpenSource Tools

PHP Usergroup Frankfurt, 2014-11-20

Annotated Version

• The grey slides are additional slides to the presentation to make it easier to understand the presentation on Slideshare etc.

@kaktusmimi

Inspired by…

Jo

Dani

Paddy

Seb

Daniel

Christiane

Inspiration

• This talk was inspired a lot by the work of my colleagues at punkt.de

• And a presentation of Sebastian Helzle at FOSSASIA 2014

• http://www.slideshare.net/Sebobo/continuous-delivery-with-open-source-tools

Motivation

TestsVersion Control

JenkinsDeploymentConfiguration Management

Monitoring

Summary & Further Reading

Are you agile?

Agile Requirements

• From a Developers Point of View

• Delivering Increments often

• Being able to do that very fast

• Being able to be sure, the Increments work

Feedback

Improving through Feedback

• Feedback is great tool for learning

• Continuous Integration can give developers feedback on their code „automatically“

• Can come from tests, metrics, linting

• The more specific the longer it takes

• Feedback from different stages of CI

Safety

Safe Delivery

• Continuous Integration assures that your code is working as expected

• We don’t see bugs only in production, but during the CI cycle

• Let’s you deploy your code „with a better feeling“

① Git

First Step Towards Continuous Integration !

if you don’t use it yet

USE GIT

Source: http://nvie.com/posts/a-successful-git-branching-model/

Git Workflows• There are several Workflows for git

• Centralized Workflow

• Feature Branch Workflow

• Gitflow Workflow

• Forking Workflow

• Overview: https://www.atlassian.com/git/tutorials/comparing-workflows/

Git GUIs• „Not Nerdy“

• If you cannot manage to handle Git on a command line

• Use a GUI!

• Better, more concise commits

• Better overview of project history

• Eg. SourceTree, Tower, TortoiseGit

git pull --rebase

Image Source: Atlassian

Rebase instead of merge

• Gives you a clean Git commit history

• Sometimes it makes sense to still keep your feature branches and only rebase inside a branch and not when merging

Gitlab

• OpenSource Git Server

• Written in Ruby

• Available as a ready-to-run virtual machine on Bitnami

• https://bitnami.com/stack/gitlab/virtual-machine

Mind

the Seb!

Code Review with Gitlab and RSS Feeds

• See blog post of Seb

• http://www.mind-the-seb.de/blog/codereview-made-simple.html

Gitlab Server

git pull / push

Developer

Reviewer

RSS Feeds

② Tests

TDD is dead. Long live testing. !

Test-first fundamentalism is like abstinence-only sex ed: An unrealistic, ineffective morality campaign for self-loathing and

shaming.

Use the right Tests

• Unit Tests are a great tool for business logic

• Unit Tests don’t work for testing legacy code

• Functional Tests or testing „from outside“ might give you a lot more feedback with less effort

Functional Testing

• Test that your code is working

• Not how it is working

• More black-box testing

• Incorporate your Database

• End-to-End testing

/** @test */public function createCustomerCreatesExpectedCustomer() { $this->customerService->createCustomer('4711', 'MickeyMouse'); $this->persistenceManager->persistAll(); $this->assertSame( '4711', $this->customerRepository->findAll()->getFirst()->getId() ); } !

Behat• Behat can be used for different kinds of tests

• Unit-Tests, Functional Tests, UI Tests

• Enables non-technical people to write tests

• Let’s developers implement the required logic

• Uses Gherkin Language

• Easily extendible using Contexts

Feature: Language menu In order to switch the language on the website As a website user I need to be able to select the language in a menu Background: Given I am in "desktop" layout And I am on "/" Scenario: Switching the language from english to german When I follow "Language" And I wait for 500 milliseconds And I follow "Deutsch" Then I should be on "/de.html"

/** * Given I am in "desktop" layout * * @When I am in :layout layout */public function iResizeTheWindowToLayout($layout) { if (array_key_exists($layout, $this->screenSizes)) { $currentLayout = $this->screenSizes[$layout]; $this->getSession()->getDriver()->resizeWindow( $currentLayout['width'], $currentLayout['height'], 'current'); return TRUE; } throw new \Exception(sprintf('Layout "%s" not defined', $layout)); }

③ Jenkins

Jenkins

• Java Application

• Can be deployed into Tomcat or Standalone

• There are good reasons for Standalone!

• Many Plugins available

• Basically a Task Runner

Deployment Stage!!* Demo

Deployment * Production

Deployment

Acceptance Stage!!* Functional Tests * Frontend Tests

Commit Stage!!* Clone Repository * Build Project * Unit Tests

Triggeredby

Commit

Triggered by Scheduler

green green

Version Everything!

• Even Jenkins Configuration

• And most of all: Jenkins Jobs!!!

• Make them part of your project!

④ Surf

TYPO3 Surf

• A Remote Server Automation and Deployment Tool

• Written in PHP

• Based on the TYPO3 Flow Framework

• Can be deployed as a Flow Package or Standalone

Simple Deployment

• Build Locally (e.g. on Jenkins)

• Ship as Package / Files (e.g. with rsync)

• Don’t use git remotely

• Too many things can go wrong!

• Make sure to have a Rollback!

More sophisticated…

• Build Locally

• Run Tests

• Ship / Transfer

• Run Smoketests

• Only switch Release if Tests pass

Surf Concepts

$workflow !! ! set up HOW to deploy!

$node !! ! ! ! set up WHERE to deploy!

$application ! set up WHAT to deploy

$deployment !! glue it all together

<?php$workflow = new \TYPO3\Surf\Domain\Model\SimpleWorkflow(); $node = new \TYPO3\Surf\Domain\Model\Node('staging'); $node->setOptions(array( 'username' => '<username>', 'composerCommandPath' => '/usr/local/bin/composer')); $node->setHostname($host); $application = new \TYPO3\Surf\Application\TYPO3\Neos('<project_name>'); $application->setOptions(array( 'repositoryUrl' => '<git_remote_url>', 'keepReleases' => 5, 'packageMethod' => 'git', 'transferMethod' => 'rsync', 'updateMethod' => NULL)); $application->setDeploymentPath('<deployment_path>'); $application->addNode($node); $deployment->addApplication($application); $deployment->onInitialize(function() use ($workflow, $application, $project) { $workflow->afterStage('migrate', 'codecoon:importContent'); });

/var/apache/XXXXXX/staging/XXXXX/releases$ ls!!20141105100124!20141105102935!20141112095924!20141118055114!20141118072225!20141119041835!20141120045634!current -> ./20141119041835!previous -> ./20141118072225!next -> ./20141120045634!!!/var/apache/XXXXXX$ ls -la!!htdocs -> staging/XXXXXX/releases/current/htdocs

⑤ Chef&Vagrant

Developer Machine / Laptop

vagrant up

Virtual Machine

WorkspaceProject 1

chef-solo

Gitlab Server

gitpull / push

ssh

mysql

samba nfs

sambanfs

gitssh

httphttp

chef run

ssh

Our Vagrant Approach* We have VirtualBox and Vagrant running on our laptop * We start a virtual machine and run Chef inside this machine * Chef sets up

* Our services (Apache, PHP, MySQL, …) * Our projects / webspaces

* We can now use our familiar tools to work on our projects * It feels like „working locally“ although we have a Sandbox

* We cannot crash the host OS when crashing the Dev-Environment * Think about packages * Think about different software versions for different projects * Think about how long it takes to re-install your laptop…

WorkspaceProject 1

WorkspaceProject 2

WorkspaceProject 2

Prov

ision

Pro

ject

s

Prov

ision

Jenk

ins S

erve

r

Confi

gure

Apa

che

Confi

gure

PH

P&

MyS

QL

WorkspaceProject 1

WorkspaceProject 2

WorkspaceProject 2

Prov

ision

Pro

ject

s

Prov

ision

Jenk

ins S

erve

r

Confi

gure

Apa

che

Confi

gure

PH

P&

MyS

QL

We set up our Projects on Jenkins

WorkspaceProject 1

WorkspaceProject 2

WorkspaceProject 2

Prov

ision

Pro

ject

s

Prov

ision

Jenk

ins S

erve

r

Confi

gure

Apa

che

Confi

gure

PH

P&

MyS

QL

We set up our Projects on Jenkins

Automatically!!!

Chef & Jenkins

There shall be 2 Chef runs 1. The one that provisions our Jenkins Server [not yet finished…] 2. The one that provisions the projects inside our Jenkins Server Those Projects are „publicly“ available * We can use them as „normal“ Website (e.g. for Review and Manual Testing) * We can run UI tests on them !In the future we want to change this to a master/slave approach.

Tease

r

More on Chef & Vagrant

• There is another presentation on Chef & Vagrant

• http://de.slideshare.net/mimiknoll/vagrant-fossasia-2014

⑥ Monitoring

Monitoring Stage• Check for Website to be alive

• Use simple Tools

• I.e. Selenium has many Hiccups —> False Positives!

• Send Notifications in Case of Emergency

• Email, Jabber, SMS

• Ops: Nagios — Devs: Jenkins

• Devs can do more sophisticated Things with Tests

Performance Stage

• Check Realtime Performance of your Website

• Gatling is a great Tool for writing Performance Tests

• Use a Dashboard to visualize your Results

• Dashing can help you to create Dashboards

• Raspberry Pi is great Hardware to display Dashboards

Summary

Version Control Build

gitlab triggers

Unit TestsCommit Stage

build succeeds

UI TestsAcceptance Stage

DB Tests

tests pass

tests pass

Deployment Stage

tests pass

build succeeds

Build locally

Demo Stage

Production Stage

rsync

rsync

git push

Developer

Feedback

Gitlab Server

Jenkins Server

WorkspaceProject 1

WorkspaceProject 2

WorkspaceProject 2

prov

isio

n

Selenium Serverhttp / RESTful Services

RepositoryProject 1

WorkspaceProject 2

WorkspaceProject 3

git shell / ssh

http / UI Testing

git clone

Production 1

WebspaceProject 1

Production 2

WebspaceProject 2

Production 3

WebspaceProject 3

ssh / rsync

www.codecoon.comMake it fun to code again

Check it out

Questions

Some Questions

• „Do you still have time to write code?“

• Yes - this toolchain makes it a lot easier for us

• „This is to complex for me!“

• You can pick out modules that work for you

More about Chef?!?

DevOps Meetup Frankfurt, 2014-12-08

„Testing Infrastructure Code with Chef“

https://about.gitlab.com/

http://jenkins-ci.org/

https://phpunit.de/

https://www.getchef.com/

https://www.vagrantup.com/

http://www.seleniumhq.org/

http://typo3.org/additional-products/surf

http://www.sourcetreeapp.com/download/

http://dashing.io/

http://gatling.io/

http://www.martinfowler.com/articles/continuousIntegration.html

http://www.thoughtworks.com/insights

http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html

http://www.rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf

http://www.mind-the-seb.de/blog/codereview-made-simple.html

https://www.atlassian.com/git/tutorials/comparing-workflows/forking-workflow

http://nvie.com/posts/a-successful-git-branching-model/