creating maintainable automated acceptance tests
DESCRIPTION
There's a recording of this talk at Agile 2012 here: http://www.youtube.com/watch?v=v-L_2y6g5DI Creating automated end-to-end functional acceptance tests is hard. Maintaining them over time is harder. Some agilistas even claim that the cost outweighs the benefit. In this lecture we present five principles for creating valuable, maintainable acceptance test suites. We discuss practices such as layering acceptance tests to reduce coupling between the test harness, and talk about how teams should be organized in order to efficiently manage acceptance test driven development. The core of the talk discusses how to manage the evolution of acceptance tests by organizing them as scenarios rather than as suites of story tests. Finally we show how to manage data for acceptance tests.TRANSCRIPT
![Page 1: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/1.jpg)
http://thoughtworks-studios.com/
Maintainable Acceptance Tests
Badrinath JanakiramanJez HumbleAgile 2012 Dallas
@badrij | [email protected]@jezhumble | [email protected]
Thursday, August 16, 12
![Page 2: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/2.jpg)
what to expect
•Creating high quality acceptance tests•How to structure a maintainable acceptance test
suite•Patterns for effective teamwork•Managing test data
Thursday, August 16, 12
![Page 3: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/3.jpg)
what to take away
•Quality is everybody’s responsibility•High quality test suites are continuously curated
- by testers and developers working together•Test code needs to receive the same care as
production code•Exhaustive story-level testing is not a good
basis for maintainable acceptance suites
Thursday, August 16, 12
![Page 4: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/4.jpg)
di!erent kinds of tests
Functional acceptance tests
ShowcasesUsability testing
Exploratory testing
Unit testsIntegration tests
System tests
Non-functional acceptance tests
(performance, scaling, ...)
Business facing
Technology facing
Critiq
ue p
roje
ct
Support
pro
gra
mm
ing
AUTOMATED
AUTOMATED
MANUAL
MANUAL / AUTOMATED
Diagram invented by Brian Marick
Thursday, August 16, 12
![Page 5: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/5.jpg)
UI
Service
Unit
Mike Cohn: Succeeding with Agile
Thursday, August 16, 12
![Page 6: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/6.jpg)
End to EndBusiness Facing
Localized Technology Facing
Thursday, August 16, 12
![Page 7: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/7.jpg)
principle 0
Writing good acceptance tests is hard.
(good: when the tests are green, we know the so!ware works)
Thursday, August 16, 12
![Page 8: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/8.jpg)
mingle
• actual running time: 55 minutes• 7-10 times a day• for 6 years, across 4 o"ces now
200620 tests500 LOC2 minutes
20123000 tests50k LOC12 hours
Thursday, August 16, 12
![Page 9: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/9.jpg)
for the same reasons code does
we don’t pay enough attention to expressing intent
only testers care about maintaining tests
why do test suites decay?
Thursday, August 16, 12
![Page 10: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/10.jpg)
principles
Thursday, August 16, 12
![Page 11: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/11.jpg)
principle 1
Tests are #rst-class citizens of your project
Thursday, August 16, 12
![Page 12: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/12.jpg)
don’t repeat yourself
treat test code as production c0de
refactor relentlessly
use record-playback tools to build your suite
don’t repeat yourself
preventing decay in test code
Thursday, August 16, 12
![Page 13: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/13.jpg)
express the test as steps of a user's journey
given-when-then is insufficient
separate intention from mechanics
preventing decay of intention
Thursday, August 16, 12
![Page 14: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/14.jpg)
use natural language to express intentions
use a general purpose programming language to express test mechanics
use a tool that allows you to operate in either domain transparently
a solution
Thursday, August 16, 12
![Page 15: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/15.jpg)
Thursday, August 16, 12
![Page 16: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/16.jpg)
page object
public class LoginPage { private final SeleniumSession browser;
public LoginPage(Selenium browser){ this.browser = browser; } public HomePage loginAs(String user, String password){ browser.type('#login', login); browser.type('#password', password); browser.submit('#login-form'); return new HomePage(this.browser); }
public HomePage loginExpectingError(String user, String password){ browser.type('#login', login); browser.type('#password', password); browser.submit('#login-form'); return new LoginPage(this.browser); }}
https://gist.github.com/3345556
Thursday, August 16, 12
![Page 17: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/17.jpg)
Customer Tester
Developer Tester
Acceptance Criteria
Test implementation
Thursday, August 16, 12
![Page 18: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/18.jpg)
...advocates for the user and makes the quality of the system transparent
...is a role, not a person
...is not a failed developer
...should be focussed on exploratory testing & maintaining automated acceptance tests
...should not be primarily working on manual regression testing
tester / quality analyst
Thursday, August 16, 12
![Page 19: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/19.jpg)
passing acceptance tests are necessary (but insu"cient) for “done”
encapsulate!
the acceptance tests are owned by - and the responsibility of - the team
remember
Thursday, August 16, 12
![Page 20: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/20.jpg)
principle 2
always interact with the system under test the same way a user would
Thursday, August 16, 12
![Page 21: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/21.jpg)
ajax based tests?
"the test fails in CI, but when I run the app, everything seems #ne"
usually an indication that test mechanics and user interaction patterns di$er
JS heavy applications, which need non-zero processing time to modify the UI
browser based tests are unreliable
Thursday, August 16, 12
![Page 22: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/22.jpg)
Don’t use bare sleeps: poll
Test your application the way a user might use it.
Understand when behavior is asynchronous and account for it explicitly
If it’s hard to write the test, you need to have a conversation with the team
some solutions
Thursday, August 16, 12
![Page 23: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/23.jpg)
wait-utils (https://github.com/itspanzi/WaitUtils)
for ajax tests, if your JS framework provides a pre- and post-call hook, intercept those to count the number of active calls before proceeding
some solutions
Thursday, August 16, 12
![Page 24: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/24.jpg)
var AjaxTracker = { PENDING_REQUESTS: $A([]), onCreate: function(request){ if (!request.url.match(/gadgets\/js\//)) { this.PENDING_REQUESTS.push(request.url); } }, onComplete: function(request){ this.PENDING_REQUESTS = this.PENDING_REQUESTS.without(request.url); }, onException: function(request, exception){ try { this.onComplete(request); }catch(e){ if (Prototype.isFireBugsEnabled) { console.log("Got Exception on request: " + request.url); console.log(e); throw(e); } } }, allAjaxComplete: function(includeCardSummary){ var requests = this.PENDING_REQUESTS.reject(function(url) { return url.match(/cards\/card_summary/) || url.match(/also_viewing/); }); return requests.size() == 0; }};
Ajax.Responders.register(AjaxTracker);
https://gist.github.com/3315690
Thursday, August 16, 12
![Page 25: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/25.jpg)
make time to go back and refactor your tests
use layers and encapsulation: separate high level intent and low level mechanics
use page object to interact with SUT; run against service layer where possible
remember
Thursday, August 16, 12
![Page 26: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/26.jpg)
principle 3
continuously curate the structure of your test suites
Thursday, August 16, 12
![Page 27: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/27.jpg)
#1301As a... I want... So that...
Given... Given...When... When...Then... Then...
#1302As a... I want... So that...
Given... Given...When... When...Then... Then...
#1303As a... I want... So that...
Given... Given...When... When...Then... Then...
#1304As a... I want... So that...
Given... Given...When... When...Then... Then...
#1305As a... I want... So that...
Given... Given...When... When...Then... Then...
#1306As a... I want... So that...
Given... Given...When... When...Then... Then...
#1307As a... I want... So that...
Given... Given...When... When...Then... Then...
#1308As a... I want... So that...
Given... Given...When... When...Then... Then...
#1309As a... I want... So that...
Given... Given...When... When...Then... Then...
#1310As a... I want... So that...
Given... Given...When... When...Then... Then...
#1311As a... I want... So that...
Given... Given...When... When...Then... Then...
#1312As a... I want... So that...
Given... Given...When... When...Then... Then...
#1313As a... I want... So that...
Given... Given...When... When...Then... Then...
#1314As a... I want... So that...
Given... Given...When... When...Then... Then...
#1315As a... I want... So that...
Given... Given...When... When...Then... Then...
#1316As a... I want... So that...
Given... Given...When... When...Then... Then...
#1317As a... I want... So that...
Given... Given...When... When...Then... Then...
#1318As a... I want... So that...
Given... Given...When... When...Then... Then...
Thursday, August 16, 12
![Page 28: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/28.jpg)
Buy Product
Search product catalogueAdd product to cartCheck outCreate new accountProvide address detailsProvide credit card detailsComplete orderVerify order createdVerify credit card debitedVerify email sent
#1612As a customerI want a gift wrapping optionSo that I don’t have to wrap them and post them myself Buy Product
Search product catalogueAdd product to cartCheck outCreate new accountProvide address detailsProvide credit card detailsSelect gift wrapping optionComplete orderVerify order createdVerify gift wrapping optionVerify credit card debitedVerify email sent
journey tests
Thursday, August 16, 12
![Page 29: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/29.jpg)
most applications have very few distinct personas
identify user journeys
( journey: the path a persona takes through the application to achieve an end goal)
most stories in iterative development are enhancements to existing journeys
some solutions
Thursday, August 16, 12
![Page 30: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/30.jpg)
features
Basic shopping cart functionality
Searching for a product- searching for a word should bring up all products which have that word in their name- searching for a phrase should bring up all products which have any of the words in their name- searching for a quoted phrase should bring up all products which have all words in the the quoted phrase in their name
Paginating search results- return 25 results per page by default- if there are fewer than 25 results, do not show pagination links- provide a "previous" link on every page other than the first page of results- provide a "next" link on every page other than the last page of results- if user supplies a page number which is less than 1 in the URL, stay on the first page- if the user supplies a page number greater than the last page of results, stay on the last page
Gift-wrap option
Thursday, August 16, 12
![Page 31: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/31.jpg)
story tests
Story tests for search- test that searching for "friends" brings back 782 results-- results should include how to win friends and influence people
- test that searching for dead friends brings back 8900 results-- results should include <how to win friends and influence people>-- results should include <The Zombie Survival Guide: Complete Protection from the Living Dead>
- test that searching for "dead friends" brings back 57 results-- results should include <all my friends are dead>
Story tests for pagination- with a catalog of 3 products, I should see no pagination links- with a catalog of 25 products, I should see no pagination links- with a catalog of 26 products, I should see 1 link to page two, along with a next link but no previous link- with a catalog of 26 products, on page 2, I should see one product, with a link to page one, a previous link but no next link
Story tests for gift wrapping
Thursday, August 16, 12
![Page 32: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/32.jpg)
journey tests
Journey of user buying a book
- Login as user "bob"- Search for <"my friends" dead>- Make sure that 3 pages of results show- Verify that "All My Friends Are Dead" by "Avery Monson" is on the first page- Add two copies of the book to the shopping cart- Gift wrap one of them- Proceed to checkout
Thursday, August 16, 12
![Page 33: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/33.jpg)
do test the most likely path that the team, business and UX folk agree upon
extract journeys from your acceptance tests
make them fast and run them #rst
extract negative tests and edge cases into a regression suite which runs a!er your journey tests
do not test every possible path through the system
more solutions
Thursday, August 16, 12
![Page 34: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/34.jpg)
build quality in“Cease dependence on mass inspection to achieve quality. Improve the process and build quality into the product in the #rst place”
W. Edwards Deming
Thursday, August 16, 12
![Page 35: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/35.jpg)
feedback from test framework to system architecture
output of testing is not just bug reports
feedback from testing to product design
developers and testers share knowledge and skills
why cross-functional teams?
Thursday, August 16, 12
![Page 36: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/36.jpg)
principle 4
everyone owns acceptance tests
Thursday, August 16, 12
![Page 37: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/37.jpg)
Add a guard to prevent it happening again
Triage to #nd root cause1. There was an environmental problem2. There is a bug with the test3. An assumption changed4. The test actually caught a bug
Fix the problem
Optimise your process for time to #x tests
Optimise your test suite: detect failures fast
when acceptance tests break
Thursday, August 16, 12
![Page 38: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/38.jpg)
%aky tests are worse than useless
quarantine %aky tests - but not forever
http://martinfowler.com/articles/nonDeterminism.html
intermittent failures
Thursday, August 16, 12
![Page 39: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/39.jpg)
not all tests should call the external system
parameterize connections to external systems
Run integration smoke tests before full acceptance suite
external systems
Thursday, August 16, 12
![Page 40: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/40.jpg)
run integration smoke tests before acceptance suite
create a proxy from SUT to external system
cache results from integration smoke tests
only run acceptance suite if integration smoke tests pass!
periodically expire cache
impersonator pattern
Thursday, August 16, 12
![Page 41: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/41.jpg)
principle 5
acceptance tests are responsible for managing their own test data
Thursday, August 16, 12
![Page 42: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/42.jpg)
Application reference data
Test-speci#c data
Test reference data
Don’t use production data dumps (except for performance testing and staging)
Ensure tests are independent
test data
Thursday, August 16, 12
![Page 43: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/43.jpg)
recap
1. treat acceptance tests like production code
2. always interact with the SUT like a user would
3. continuously curate your user journeys
4. collective ownership of acceptance tests
5. acceptance tests own their data
Thursday, August 16, 12
![Page 44: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/44.jpg)
take-aways
•quality is everybody’s responsibility
•high quality test suites are continuously curated - by testers and developers working together
• test code needs to receive the same care as production code
•exhaustive story-level testing is not a good basis for maintainable acceptance suites
Thursday, August 16, 12
![Page 45: Creating Maintainable Automated Acceptance Tests](https://reader037.vdocuments.site/reader037/viewer/2022103016/554f5b6cb4c905524c8b5507/html5/thumbnails/45.jpg)
http://thoughtworks-studios.com/
questions
@badrij | [email protected]@jezhumble | [email protected]
http://continuousdelivery.com/
© 2011 ThoughtWorks, Inc.
Thursday, August 16, 12