nt Session 

 

Presented by: 

Matthew  

  

Brought to you by: 

  

340 Corporate Way, Suite   Orange Park, FL 32073 888‐2

T11 Concurre4/8/2014   2:00 PM     

“Balancing Exploratory and  Automated Testing in Agile” 

 

  Attaway

Perforce Software   

    

300,68‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com 

Matthew Attaway Perforce Software  

A jack of all trades and master of some, Matt Attaway has worked as a tester, developer, researcher, designer, manager, DevOps engineer, and elephant trainer. He currently manages the open source development group at Perforce Software, but prior to that he led a test team working on the highest profile projects at Perforce. When not tinkering with the latest software development pipeline tools, Matt enjoys experimenting with cocktails and playing games of all sorts—preferably simultaneously.

2/18/2014

1

Balancing Exploratory and Automated Testing in Agile

Matt Attaway | @mataway

2/18/2014

2

• Projects varied in size from 3-20 people

• Multiple languages, multiple stacks

My World

• GUIs, CLI tools, servers, and APIs

• Remote and co-located teams

• Scrum, Kanban, Scrumban, Waterscrumfallban

• Managed ~10 testers on 6 simultaneous projects

• Almost all "bet the company" projects

• Automation or bust!

My Constraints

• Exploratory testing a must

• Volatile projects; few design docs

• Extremely high bar for quality

• Executive demand for measurability and metrics

• Do more with less

2/18/2014

3

Must handle the constant flood of change

My Process Requirements

• Must handle the constant flood of change

• Must integrate automated testing without sacficing exploratory testing

• Must communicate progress both to the product team and the executive teamteam and the executive team

Must not die

2/18/2014

4

Ruthlessly optimize

Exploratory testing

T t l

Automated testing

Test plans

2/18/2014

5

Exploratory testing

T t l

Automated testing

Test plans

Helm th on Moltke the Elder

“No plan survives contact with the enemy.”

–Helmuth von Moltke the Elder

2/18/2014

6

Needed test plans per epic story feature and fix• Needed test plans per epic, story, feature and fix

• Easy report generation and APIs

• Easy access for testers

Cheap both in time and in money• Cheap, both in time and in money

Issue tracker

2/18/2014

7

Focused around epic story feature and fix

Strengths

• Focused around epic, story, feature and fix

• Built-in report generation and APIs

• Testers already lived in it

Already installed and paid for!• Already installed and paid for!

Test plans highly visible to the developers

Bonuses

• Test plans highly visible to the developers

• Naturally supports multiple editors

• Extremely easy to extend

• Notifications of updates via email and RSS with diffs• Notifications of updates via email and RSS, with diffs

2/18/2014

8

• Added a "Testing Notes" field to issues

Testers added test cases as they did their

Process

• Testers added test cases as they did their exploratory testing

• + used to flag completed test, - to flag pending test

• Anyone could add test cases to an issue

• Tweet sized test cases; should be clear for knowledgable tester

Example

- Attempt to create user with HTTP special characters- Attempt to create user with XML special characters+Attempt to create user that already exists

Example

2/18/2014

9

• Devs handed off work by merging to mainline

Process & Measuring

• Issue not closed until all test cases complete

• Outstanding test work = issues in 'fixed' state

• Test work displayed on burndown charts; easy to spot when test was having a hard time keeping up

2/18/2014

10

Automated testing

• Numerous automated test technologies and languages

CLI t t t l b t t t l d GUI t t t l• CLI test tools, web test tools, and GUI test tools

• Poorly written code; copy/paste gone wild

• Each product had one (or more!) test frameworks

• Too slow to add tests• Too slow to add tests

• Tests were dependent on each other

2/18/2014

11

2/18/2014

12

Given I am speaking at StarCanadaWhen I start my talkThen my heart rate will go up dramatically

• Forced code modularization

Why Cucumber?

• Test cases are legible and shareable

• Can drive any tool; can use most languages

• Tests are easy to add; frequently require no "real" code

• Works out of the box; huge development community

W it t t ti• Write tests anytime

• Everyone can automate

• Cucumber is, dare I say, fun

2/18/2014

13

• Hour meeting at the beginning of the sprint; more as

Bugstorming

needed during sprint

• Wrote Gherkin as a team coming up with different attacks based on slotted stories

• Helped to share system knowledge and automated testing knowledge

• Track your hit rate; failed automated test is a bug

T k h t t t d h t d 't ( d h !)

Automated testing advice

• Track what you automate and what you don't (and why!)

• Aim to be able to add a test case in 15 minutes

• Always Reset State

• Automate all the things• Automate all the things

• Plan for parallelized tests from day one

2/18/2014

14

Ruthlessly optimize

Avoid documents that do nothing(or make your documents do more)

2/18/2014

15

Embrace your issue tracker

Detach writing automated tests from implementing automated tests

2/18/2014

16

Regression tests == automated tests

The Cucumber Book by Aslak Hellesøy

Books

• The Cucumber Book by Aslak Hellesøy

• Cucumber & Cheese by Jeff Morgan

• Continuous Delivery by Jez Humble & David Farley

Exploratory Software Testing by James Whittaker• Exploratory Software Testing by James Whittaker