introduction to acceptance test driven development

Beginning with the End in Mind: Driving Development with Acceptance Tests Elisabeth Hendrickson

Quality Tree Software, Inc.

www.qualitytree.com

[email protected]

Last updated November 10, 2009

This work is licensed under the Creative Commons Attribution 3.0 United States License. View a copy of this license.

What is ATDD?

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Acceptance-Test Driven Development (ATDD) Cycle

!"#$%"&

'"(

)&""*+#",*

!$-./--0&1(/.2

0&1(/.23,.4#15

-21&6-21&6-21&6-21&6-21&6

!"%"#17

(Model developed with Pekka Klärck, Bas Vodde, and Craig Larman.)

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

ATDD: Discuss

!"#$%&'()&*+#,

-'#".-'#".-'#".-'#".-'#".

!"#$%&'(&%$)*#(&+(,

Concrete Examples with Expectations

(Slide developed in collaboration with Pekka Klärck.)

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

ATDD: Develop

!"#

$%""&'

(")& !"#$

%#$&

'()*+"&

%#$&

'*&+%","-."/,/

,-(*

.&/*/


Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

ATDD: Deliver

FeatureFeature Feature

Feature

Feature

FeatureFeature

Feature

Feature

Product


Introducing an Example

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Start with the Story

As an administrator, I want users creating accounts to be required to choose secure passwords so that their accounts cannot be hacked by someone using a password guessing program.

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Discuss

What does “secure” mean

to you? At least 6 characters with

at least one letter, one symbol, and one number.

And if a user provides an insecure password,

display an error message.

Who’s in the room?

Product Owner, Testers,

Developers, and anyone else who will

touch the story.

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Capture Concrete Expectations and Examples

Password Valid? “p@ssw0rd” Yes “p@s5” No “passw0rd” No “p@ssword” No “@#$%1234” No

Given a user is creating an account

When they specify an insecure password

Then they see a message, “Passwords must be at least 6 characters long with at least one letter, one number, and one symbol.”

Or can be expressed in tables

Can be expressed as

“Given - When - Then”

Or in other formats depending on the

Framework

Why ATDD? Reason #1: Drive Out Ambiguity and Clarify Expectations

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

This is not an Argument about a Bug

“Bug Triage Meeting” The Tuesday before release.

Whether or not it’s a bug, if we make a change we’ll blow the

schedule.

It’s a bug.

No it’s not.

Is too.

IS NOT.

IS TOO!

NOT NOT NOT!

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Acceptance Tests Define Scope

A Short Digression on ATDD-Friendly Tools

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Examples of ATDD-Friendly Frameworks

• Cucumber: a Ruby-based BDD tool that supports “Given-When-Then”

• Fitnesse: a table-driven framework that uses a wiki for displaying and editing tests

• Robot Framework: keyword-driven framework that supports text or tables

• Concordion: Java-based framework for expressing expectations in prose

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Test “Fixture”

Frameworks, Interfaces, and Drivers

Natural Language

Expectations

GUI

Created in collaboration.

Format defined by the

framework

Code written during development in a

programming language determined by the

framework.

Implementation

“Guts”

Public API

Other interfaces

GUI Driver (e.g. SeleniumRC)

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Characteristics of ATDD-Friendly Frameworks

• Support expressing expectations in a language and format that fits the context

• Support collaboration among the whole team including developers, testers, & the product owner

• Connect expectations to the system under test with a minimum of test code (“fixtures,” “libraries,” “steps”) to leverage expectations as executable requirements

• Play nicely with source control systems and continuous integration

• Pluggable to support a variety of interfaces

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Contrasting View: Traditional Test Automation

Written or recorded after the fact.

Expectations are translated, not

leveraged.

GUI

Implementation

“Guts”

Public API

Other interfaces

Automated Test Scripts:

Combination of business-facing expectations and implementation

details.

Back to the Example…

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Take the Acceptance Tests…


Given a user is creating an account

When they specify an insecure password

Then they see a message, “Passwords must be at least 6 characters long with at least one letter, one number, and one symbol.”

(Note that these expectations are

implementation-agnostic and express just the

essence of the expectation.)

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

…and Write the Code


Test “Fixture”

GUI

Implementation

“Guts”

Public API

Other interfaces

Why ATDD? Reason #2: Make progress visible.

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Are We There Yet?

Why ATDD? Reason #3: Leverage, Efficiency, and Executable Specifications

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Traditional Approaches

Requirements Management

System

Test Management

System Technical

Specifications

Traceability Matrix

?

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Test “Fixture”

Efficiency, Reusability, Maintainability

Natural Language

Expectations

The tests define the

requirements.

New interface? Just add a test

fixture

Implementation

(No reconciling multiple, duplicate

artifacts.)

…

New Interface

Implementation change? Make a

localized update to the test fixture.

(Preserve valid expectations.)

(Leverage relevant expectations.)

How does ATDD fit with the rest of the process?

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

ATDD: Part of an Agile Testing Strategy

(A variation on Brian Marick’s Agile Testing Quadrants as published in his essay “Agile Testing Directions”)

Business-facing

Code- facing

Support Critique

Acceptance Tests Exploratory Testing

Unit Tests Reviews, inspections,

pairing, code quality metrics

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Tests are Versioned with the Code

Natural Language

Expectations Test

“Fixture” Code

Unit Tests

Production Code

Source Control

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Tests Execute as Part of the Automated Build

Continuous Integration

Images courtesy Mike Clark, www.pragmaticautomation.com. Used with permission.

Why ATDD? Reason #4: No more bugs. (No, I’m not kidding. But yes, there is a catch.)

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Zero Tolerance for Bugs

!"#$%"%&'%()'*+,

!"#$%"%&'%()'*+,

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

But Not Everything is a Bug

In this context, a BUG

is behavior that violates the letter

or spirit of the Product Owner’s

expectations for the implemented story.

If the behavior does not violate expectations related to the implemented stories, it’s an item

for the backlog.

Given all that… Why not ATDD?

Copyr

ight

© 2

01

0 Q

ual

ity

Tre

e So

ftw

are,

Inc.

Resources

Adzic, Gojko (2009). Bridging the Communication Gap: Specification by Example and Agile Acceptance Testing. Neuri Limited.

Crispin, Lisa (2005). “Using Customer Tests to Drive Development.” Methods & Tools. Summer 2005 Issue. Available online at http://www.methodsandtools.com/archive/archive.php?id=23

Crispin, L., & Gregory, J. (2009). Agile Testing: A Practical Guide for Testers and Agile Teams (Addison-Wesley Signature Series (Cohn)). New York: Addison-wesley Professional.

Marick, Brian (2003). “Agile Testing Directions.” (An explanation of business-facing v. code- facing tests.) Available online at http://www.exampler.com/old-blog/2003/08/22/#agile- testing-project-2