rocket fuelled cucumbers

Rocket FuelledCucumbers

Joseph Wilk

Dealing with an Outbreak of Cucumbers

Joseph Wilk

No Cucumbers where harmed in the making of

this presentation.

Just mildly shaken up

Scenario: Happy Railsconf attendees Given you have a vague idea what Cucumber is When Joseph completes his presentation Then you should understand the scaling choices And you should feel better prepared to face them

Cucumber

• BDD framework

• Plaintext

• Promotes communication

Given /^I have trained monkeys$/ do @test_pilots << Monkeys.newend

Feature: Filling Cucumbers with rocket fuel

Scenario: Rocket fuel Given I have trained monkeys When I click the launch button Then the monkeys should not die

Your code makes it slow

Cucumber provides the venue

Shame on you

What are Acceptance Tests?

• Customer-facing

• Cut through the whole stack

• Examples of how the system is intended to be used

You mademe Cuke

1 minute ~ build

1 minute ~ build

Local builds

1 minute ~ build

AutotestLocal builds

1 minute ~ build

AutotestLocal builds

Growl

Don’tPanic

Slow testing problems

Development is slowed down by the time to run

focused Cukes

Running all the Cukes takes too long!

Development

Test Build

Types of Feedback

SpeedConfidence

-+

SpeedConfidence

+-

Happiness is a 10 min build time

Slow builds arethe enemy of continuous integration

Development Feedback

Its too slow to run the cukes.Just push the code and run away

Reuse - Spork$ spork cucumber$ cucumber --drb

require 'rubygems'require 'spork'

Spork.prefork do puts "I'm loading all the heavy stuff..."end

Spork.each_run do puts "I'm loading the stuff just for this run..." # Cucumber hooks must go hereend

git://github.com/timcharper/spork.git

Slow services

• Search

• Solr

• Sphinx

• Databases

• Mongo

• Redis

• Mysql

• Message Queues

• RabbitMQ

Before('@solr') do Solr.boot unless Solr.running?end

Feature: Searching

@solrScenario: Indexed search Given ... When ... Then ...

Run Just Enough Tests

# * Test files must be stored in test/# * Test files names must start with test_# * Test class names must start with Test# * Implementation files must be stored in lib/# * Implementation files must match up with a # test file named# test_.*implementation.rb

@analyzer = Rcov::CodeCoverageAnalyzer.new

Cucover

Autotest

http://vimeo.com/12500864



Slicing Features

Tagging

Filenames

$ cucumber --tags @media

Subsystems

$ cucumber visitor_*

$ cucumber features/admin/*

Profiles $ cucumber --profile admin

@media, @publicFeature: Visitor views artist’s media

@feed, @adminFeature: Logged in user views views their activity feed

Just enough Database

Just enough Database

city data

venue data

SELECT * FROM `cities` WHERE (`cities`.`id` = 105838)

INSERT INTO `cities` (`name`) VALUES ('Test name')

INSERT INTO `venues` ...

SELECT * FROM `venues`

WHERE ...

Cache

Cache

Test Build Feedback

Commit codeHave a 1 hour nap

Options

• Hardware

• Intelligent test selection

• Tighter focused tests

• Divide and Conquer

TestjourSongkick.com

254 Features, 1257 scenarios, 10807 steps

1 build server ~ 4 hours

5 build servers ~ 2 hours

120 sad minutes of my life...

$ git clone git://github.com/brynary/testjour.git$ rake gem install

$ testjour --help

$ git clone git://github.com/brynary/testjour.git$ rake gem install

$ testjour --help

testjour help:

Testjour

MasterWorkQueue

Slave

Redis

Slave

Slave

Worker

Worker

Worker

Worker

Worker

Worker

Testjour

MasterWorkQueue

Slave

Redis

Slave

Slaversync - code

rsync - codeWorker

Worker

Worker

Worker

Worker

Worker

Looking to the Cloud

EC2, Rackspace Cloud, Windows Azure




20 build servers ~ 11:40 minutes




20 build servers ~ 11:40 minutes

$2068.99 / $3000

Ec2

AMI

slim-sumo

Disc image

Ec2

AMILaunch

slim-sumo

Disc image

Ec2

AMILaunch

ec2 node

ec2 node

ec2 node

ec2 node

slim-sumo

Disc image

Ec2

AMILaunch

ec2 node

ec2 node

ec2 node

ec2 node

slim-sumo

BuildENV['EC2_URL'] = ec2_urlrequire 'slim-sumo'Sumo.new.running.map{|ec2_node| ec2_node[:hostname]}

Disc image

require 'rubygems'require 'hydra'require 'hydra/tasks'

Hydra::TestTask.new('hydra') do |t| t.add_files 'features/**/*.feature' t.verbose = false t.autosort = falseend

“Buy lots of hardware”


“more hardware please”


“more hardware please”

“Just a little bit more hardware”

Just run the tests that matter...

Failure probability matters

European Computing manufacturer



Automated test suite ~ 18 hours



Tests that were most ~ 2 hours likely to fail

Automated test suite ~ 18 hours

Tests that never fail


Tests which regularly fail

Daily Build



Daily Build Nightly Build



Daily Build Nightly Build



$cucumber --tags ~@nightly $cucumber --tags @nightly

Flickering Confidence


FAIL


FAIL

FAIL


FAIL

PASS

FAIL


FAIL

PASS

FAIL

Main Flicker@flicker

Divide and conquer

• Rails - Engines

• Service Orientated Architecture

1 hour

7 7 7

7 7 7

7 7 7

Divide and Concuquer

Divide and Concuquer

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

Divide and conquer

Rails


Divide and conquer

RelatedArtistsService

Rails


Divide and conquer

related(artist)


Rails


Divide and conquer

related(artist)


HTMLRails


Divide and conquer

related(artist)


HTMLRails


unit testunit test

Don’t use Acceptance tests

http://jamesshore.com/Blog/Alternatives-to-Acceptance-Testing.html

Heresy!





PairwiseScenario: Testing Cucumber against different environments Given I have a rails app <Rails version> And I’m using Ruby <Ruby version> And I am using the <Test Framework> And I am using the driver <Driver> Then Cucumber should install and play nicely Examples: | Rails version | Ruby version | test framework | Driver | | 3.0 | 1.8.7 | Rspec | Webrat | | 3.0 | 1.8.7 | Testunit | Capybara | | 3.0 | 1.8.7 | Rspec | Capybara | | 3.0 | 1.8.7 | Testunit | Webrat | | 3.0 | 1.9 | Rspec | Webrat | | 3.0 | 1.9 | Testunit | Webrat | | 2.3.1 | 1.9 | Rspec | Webrat | ....

| Rails version | Ruby version | test framework | Driver | | 2.3.1 | 1.8.7 | Rspec | Webrat | | 2.3.1 | 1.8.7 | Rspec | Capybara | | 2.3.1 | 1.8.7 | Testunit | Webrat | | 2.3.1 | 1.8.7 | Testunit | Capybara | | 2.3.1 | 1.9 | Rspec | Capybara | | 2.3.1 | 1.9 | Rspec | Webrat | | 2.3.1 | 1.9 | Testunit | Capybara | | 2.3.1 | 1.9 | Testunit | Webrat |

....

“most faults are caused by interactions of at most two factors”

Pairwise

http://www.screencast.com/t/NGI0NjVk



We should add some funky Ajax features

Just when you thought things were going well

JavaScript

In-memoryBrowser basedSelenium, Watir Celerity

Need for speedWebrat

Celerity

Watir

Selenium

0 3.75 7.5 11.25 15

12.9

12

12

0.8

start-up time (seconds)

CapybaraLike Webrat but bigger

envjsJohnson

spidermonkey

gem install johnson --prerelease

gem install envjs

sudo gem install capybara

Capybara-envjs

Find the right compromise for you


Slow test builds produce slow release cycles


Slow test builds produce slow release cycles

Your test builds are a goldmine of data

Thanks

Joseph Wilkhttp://blog.josephwilk.net

@josephwilk

Its been emotional

http://blog.josephwi




rocket fuelled cucumbers

Technology

filling cucumbers

cucumber iswhen joseph

rocket fuelgiven

vague idea

happy railsconf attendeesgiven

scaling choicesand

making ofthis presentation