living with acceptance tests: beyond write-once
DESCRIPTION
Presented at CukeUp! NYC 2014 in Brooklyn, NY. Session description: Acceptance tests have become a common part of building software for an agile process. However, I've observed a trend of growing build times - with teams saying "Our build is 15 minutes... well, actually it's 45 minutes because our Cucumbers took so long, so we parallelized the cukes." One reason for this may be that teams treat their acceptance tests as "Write-Once": create for a new story, get them to pass, and leave them be. In this talk we'll explore the technical and social forces that cause an acceptance test suite to grow, and discuss some ways to refactor and improve these tests. We'll touch on hexagonal architecture, declarative vs. imperative style, and alternate Cucumber step implementations.TRANSCRIPT
Living with!Acceptance Tests: Beyond Write-Once
Daniel Wellman
Twitter: @wellman!E-mail: [email protected]
www.intentmedia.com
Activity
Goal!Build the least helpful,
most frustrating acceptance test suite
Revise Your Tests With the Same Care as Your Production Code
Check for Duplication
Refactor with New Insights
Monitor for Problems
Problem:!Slow and Flaky Tests
Server
Database QueueTest
End-to-End Tests
New Project
End-to-End!Acceptance Test
New Project
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
End-to-End!Acceptance Test
How did it get this way?
Run Acceptance Tests in Parallel
Faster Build Same Risk
Failures are Hard to Diagnose
Problem:Flaky Tests
Don’t make every acceptance test run
end-to-end
Ports and Adapters!a.k.a. Hexagonal Architecture
Example: Fraud Protection for an Amazon-like store
Example: Fraud Protection for an Amazon-like store
Example from Gojko Adzic, “Specification by Example”
Example: Fraud Protection for an Amazon-like store
Example from Gojko Adzic, “Specification by Example”
Given a user with no previous transaction history, And the user’s account registration country is the UK, When the user places an order with delivery country U.S., Then the transaction is marked as suspicious, But the user sees order status as “Pending.”
AdaptersPorts
Domain
AdaptersPorts
Domain
GUI
DB
AdaptersPorts
Domain
GUI
DB
Memory
AdaptersPorts
Domain
GUI
DB
Memory
End-to-End Test
AdaptersPorts
Domain
GUI
DB
Memory
End-to-End Test
Test Using!Domain
Before: End-to-EndWhen(/^the user places an order with delivery country (.*),$/) do |country_name| !!!!!!!!!!end
Before: End-to-EndWhen(/^the user places an order with delivery country (.*),$/) do |country_name| !!!!!!!!!!end
# ... fill_in 'Address', with: ... select country_name, from: 'Country Selector’ click_button 'Ship To This Address' expect(page).to have_content 'Your order has been placed’ # ...
After: Using Domain ModelWhen(/^the user places an order with delivery country (.*),$/) do |country_name|
!!!!!!!!!end
After: Using Domain ModelWhen(/^the user places an order with delivery country (.*),$/) do |country_name|
!!!!!!!!!end
# ... country = Country.new(country_name) order = Order.new( ..., ship_to : Address.new( street: ..., country: country)) sales_clerk.accept(order) # ...
Service
UI
Unit
The Testing Pyramid
Service
UI
Unit
The Testing Pyramid
Create a Separate System Test Suite
Alternate Step Implementations
Alternate Steps: Use Conditionals
use_browser = ENV['USE_BROWSER_FOR_TESTS']
Alternate Steps: Use Conditionals
!!When(/^the user places an order with delivery country (.*),$/) do |country_name| if use_browser # drive a browser using Capybara; slower else # use domain objects; faster end end
use_browser = ENV['USE_BROWSER_FOR_TESTS']
Write Tests Without Specifying
Implementation Details
Imperative!vs.!
Declarative!Style
Imperative Style
Imperative Style
Given I am on the registration page, And I enter "UK" as my billing country, And I press the "Register" button, And I add an item to my cart, When I checkout And I enter "U.S." as my shipping country, And ...
Declarative Style
Declarative Style
Given a user with no previous transaction history, And the user’s account registration country is the UK, When the user places an order with delivery country U.S., Then the transaction is marked as suspicious, But the user sees order status as “Pending.”
Example by Gojko Adzic, “Specification by Example”
Declarative Style
Given a user with no previous transaction history, And the user’s account registration country is the UK, When the user places an order with delivery country U.S., Then the transaction is marked as suspicious, But the user sees order status as “Pending.”
Example by Gojko Adzic, “Specification by Example”
You can’t tell how these are implemented!
Use Declarative Style to Help Keep
Implementation Details Out
of Tests
Problem: I’m not sure what is actually
tested…
How did it get this way?
Periodically Evaluate Your Test Suite
Have you introduced a new domain concept?
Are you in a new stage of your product’s life?
Suggestions• Maintain your acceptance tests just like
production code
• Write tests which describe what, not how, so their implementations can easily evolve later
• Decide when you can start moving acceptance tests from end-to-end to using the domain directly
• Re-evaluate your product’s tests and testing strategy as your system grows
Resources
Photo Credits“HighLine_061409_2779” by Jessica Sheridan is licensed under CC BY 2.0
“Highline New York” by Rebecca Krebs is licensed under CC BY 2.0
“NYC: The Highline" by Wally Gobetz is licensed under CC BY-NC-ND 2.0
“Rot”, “Yellow Cart”, and “Many Hands Make Light Work” by Oliver Rich are licensed under CC BY-NC-ND 2.0
“highline-1010384.jpg” by keroism is licensed under CC BY-SA 2.0
“highline” by Elijah Porter is licensed under CC BY-NC-SA 2.0
“Highline_030” by Timothy Vogel is licensed under CC BY-NC 2.0
“201106073451_DSC_0014_1.JPG” by Leonel Ponce is licensed under CC BY-NC 2.0
“193” by verndogs is licensed under CC BY-NC 2.0
Living with!Acceptance Tests: Beyond Write-Once
Daniel Wellman
Twitter: @wellman!E-mail: [email protected]
www.intentmedia.com