Google Confidential and Proprietary

Succeeding with Behavior Driven Development (BDD) Testing and Automation

Seattle Area Software Quality Assurance GroupOct 18, 2012

Alan MyrvoldGoogle

Google Confidential and Proprietary

About me

Feb 2011 - now

2005 - 2011

1998 - 2005

1988 - 1998

GoogleTest Engineer, Ads

MicrosoftSDET in Office Security + Outlook

EntrustTest manager, development manager,security assurance manager

CognosTester, developer, test manager,development manager

Google Confidential and Proprietary

More about me

Google Confidential and Proprietary

Buzzword bingo

ATDD

BDD

Cucumber

Cucumber - JVM

Gherkin

Jasmine

SpecFlow

Google Confidential and Proprietary

Buzzword bingo

ATDD - acceptance test driven development

BDD - behavior driven development

Cucumber - a Ruby tool that supports BDD

Cucumber - JVM - a Java tool that supports BDD

Gherkin - the language used by Cucumber

Jasmine - a javascript tool for BDD

SpecFlow - a .NET tool for BDD

Google Confidential and Proprietary

Cucumber Example

Feature: AdditionIn order to avoid silly mistakesAs a math idiotI want to be told the sum of two numbers

Scenario: Add two numbersGiven I have entered 50 into the calculatorAnd I have entered 70 into the calculatorWhen I press addThen the result should be 120 on the screen

Google Confidential and Proprietary

Given / When / Then

Given - precondition

When - action

Then - assertion on expected result

And - same action as before

Google Confidential and Proprietary

Cucumber step implementation

@When("I have entered (.*) into the calculator")public void enterNumber(int number) {

...}

Google Confidential and Proprietary

Scenario Outline ExampleScenario Outline: Add two numbers

Given I have entered <x> into the calculatorAnd I have entered <y> into the calculatorWhen I press addThen the result should be <z> on the screen

Examples:| x | y | z || 2 | 3 | 5 || 0 | -1 | -1 |

Google Confidential and Proprietary

Why BDD?

Clarifying requirements by example

Demystifying automated tests by using English

Demystifying repeated manual tests by emphasizing why and what to verify.

Google Confidential and Proprietary

Some test types

Google Confidential and Proprietary

How I used BDD at Microsoft

Clarifying requirements in my test plan.

Google Confidential and Proprietary

How we are using BDD at Google

My group, DoubleClick Bid Manager, uses BDD for Java API-level system tests and repeated manual tests.

Other groups use BDD tests for Java WebDriver tests.

We share the same framework, developed internally.

Google Confidential and Proprietary

Tools support + Books

Tools:o Cucumber - Ruby http://cukes.info o Cucumber JVM https://github.com/cucumber/cucumber-jvm o SpecFlow - Binding business requirements to .NET code

http://specflow.org

Bookso The Cucumber Booko The RSpec Booko Cucumber Recipies (beta, scheduled 3/7/2013)

All books from Pragmatic Programmers, http://pragprog.com

Google Confidential and Proprietary

When / Then elsewhere

Mockito … a Java unit testing framework@Mock CalculationEngine engine;Calculator calculator = new Calculator(engine);

when(engine.add(2, 2)).thenReturn(4);calculator.parse("2 + 2 =");assertEquals("4", caclulator.getResult());

Compare the syntax to EasyMock:expect(engine.add(2, 2)).andReturn(4);

Google Confidential and Proprietary

describe / it

Rspec … a Ruby unit testing frameworkdescribe Calculator, "#basics" do it "return 4 for 2+2" do calc = Caculator.new calc.add(2, 2) calc.result.should eq(4) endend

Jasmine … a Javascript unit testing frameworkdescribe("calc", function() { it("2+2 is 4", function() { expect(calc(2, 2).toEqual(4); });});

Google Confidential and Proprietary

Failure mode #1 - Too implementation dependent

BAD

Given I have entered 50 into the calculator

And I have entered 70 into the calculator

When I press addThen the result should be

120 on the screen

BETTER

When I add 50 and 70Then the result is 120

Google Confidential and Proprietary

Failure mode #2 - Programmatic Scripts

BAD

When I set x to 1And while x < 50 And set row x to "empty"

BETTER

Then set the first 50 rows to "empty"

Google Confidential and Proprietary

Failure mode #3 - Too low level

BAD

When I go to the login pageAnd enter "bob" into the

username fieldAnd enter "pass123" into

the password fieldAnd click loginThen I am logged in as

"bob"

BETTER

When I log in as "bob"

Google Confidential and Proprietary

Failure mode #4 - Not exploring interesting cases

BAD

When I add 50 and 70Then the result is 120

BETTER

When I add 50 and 70Then the result is 120

When I add 1e90, 0.1, and -1e90

Then the result is 0.1

Google Confidential and Proprietary

Failure mode #5 - Not using regex

BAD

@When("Set the budget to 100")

public void setBudget(){ ...}

BETTER

@When("Set the budget to (.*)")

public void setBudget(int amount)

{ ...}

Google Confidential and Proprietary

Failure mode #6 - Complex parsing logic

BAD

@When("(.*) the (.*) to (.*)")

public void doAction(String action, String name, String value)

{ if (action.equals("set")

&& name.equals("budget") }

BETTER

@When("Set the budget to (.*)")

public void setBudget(int amount)

{ ...}

Google Confidential and Proprietary

Failure mode #7 - Leaking code details

BAD

When I click the BTN-REFRESH-ALL button

BETTER

When I refresh all

Google Confidential and Proprietary

Failure mode #8 - Bad test architecture

BAD

Calling entry points in code that are fragile, or disappear

BETTER

Using supported public or test API's

Google Confidential and Proprietary

Success tips from me

Use BDD for a small set of tests

Focus on human readability, as a domain expert using the feature would describe a test

Google Confidential and Proprietary

Success tips from The Cucumber Book, by Matt Wayne and Aslak Hellesoy

DAMP beats DRYDAMP: Descriptive and meaningful phrasesDRY: Don’t repeat yourself

Declarative better than imperativeDeclarative: Given I am logged inImperative: Log in as user “Bob”

Google Confidential and Proprietary

Questions?

http://testapprentice.com amyrvold@google.com