colm oheocha agile testing v1.2 · 25/11/2014’ 3 ’ scrumcycle’ 1day’ daily’standx...

25/11/2014

1

Preven-ng Defects over Finding Failures Topics on Agile Tes-ng

Colm O’hEocha, AgileInnova-on

www.eurostarconferences.com

Topics We’ll Cover

•  A (very) brief introduc-on to agile •  Cost of Delay – how delayed test creates waste •  Agile Testers Role – Requirements to Tests •  Test First – developing tests before implementa-on •  Test Automa-on – key points •  Exploratory Tes-ng – its all in the mind

25/11/2014

2


COMPLEXITY & UNCERTAINTY

Business/System Requirements

Requirements are Vague & Liable to Change

Requirements are Clear & Stable

Solu-on/Technology Design & Implementa-on

Well understood & Predictable

Many possible solu-ons, es-mates ‘finger in the air’

Chao-c & Emergent •  We don’t know what we

don’t know •  Answers understood only in

retrospect •  Probe and see what happens •  Experimental Management –

reinforce desirable paWerns •  Poli%cs, Childs Birthday Party

Complex •  We know what we don’t

know •  Mul-ple Right Answers •  Experts know Answers •  Value of making decision

might outweigh wai-ng for right answer

•  So3ware Development

Simple •  We know •  Answer Self-‐

Evident •  Command &

Control •  Manufacturing


Empirical Process Control CLOSED-LOOP

Empirical - Adaptive OPEN-LOOP

Deterministic- Predictive

Execute Execute

Inspect

Adapt Set Ini-al Target

Set Target

AIM & FIRE

FIRE & AIM

25/11/2014

3


Scrum Cycle

1 DAY Daily stand-‐up mee-ng

Product Backlog

1-‐4 WEEK SPRINT

Sprint Planning Mee-ng

Test, Develop, Test

Review

Retrospec-ve

Product Backlog

Refinement (Con-nuous)

Sprint Backlog

VISION +

TEAM

(poten-ally releasable) PRODUCT

+ TEAM


From Plan-‐Driven to Scrum

Poten-ally Releasable Product Increments

VISION +

TEAM

PRODUCT +

IMPROVED TEAM

Requirements Design Implementa-on Verifica-on Produc-on NEED +

RESOURCES

PRODUCT +

RESOURCES

25/11/2014

4


AGILE MANIFESTO + 1 “We are uncovering beWer ways of developing socware by doing it and helping others do it. Through this work we have come to value:

•  – Individuals and interac-ons over processes and tools •  – Working socware over comprehensive documenta-on •  – Customer collabora-on over contract nego-a-on •  – Responding to change over following a plan •  – Preven%ng defects over finding failures

“That is, while there is value in the items on the right, we value the items on the lec more.”

9.00


Economies of Speed

Cost of Delay

25/11/2014

5


Evolu-on: WaTerFall, Sprint+1, Mini-‐Waterfall, Agile

Problems with Non-‐Agile approaches: •  Too late to fix bugs

at end of sprint •  Unevenness in

workload (Code/Test)

•  Cost of Delay


What is ‘Cost of Delay’?

•  The difference in the value of doing something sooner vs. doing it later E.g. –  Releasing a Product this month vs. next month –  Delivering a feature 6 months acer it was specified –  Valida-ng a perceived user need before we build the feature vs. acer

–  Fixing a bug today vs. in two weeks -me

25/11/2014

6


What is ‘Cost of Delay’?

•  The difference in the value of doing something sooner vs. doing it later E.g.

Finite benefits horizon, and reduced peak due to late delivery

€ Be

nefits / W

eek

Delayed Product Delayed Feature Delayed Bug Fix


Cost of Delay in Tes-ng

CREATE MERGE RECREATE, FIND,FIX

SETUP V2 & V3

SETUP V1 & V2

RECREATE, FIND,FIX

RECREATE, FIND,FIX

SETUP V3 & V4

FIND

SETUP V1

REVIEW

LOG

Time To Find/Fix/Retest (2 Sprints) FIND

SETUP V1 & V2

REVIEW

LOG

FIND

SETUP V2 & V3

REVIEW

LOG

FIND

SETUP V3 & V4

REVIEW

LOG

25/11/2014

7


Test & Debug – Typical Ac-vi-es (Re)Create Environments of differing versions for Test, Debug, Retest Log the Bug in a -cke-ng system, incl. how to recreate Manage the bug (review mee-ng, priori-se, assign) Find the Bug (among all the changes made since it was injected) Fix the bug (that I created weeks ago) Merge fix with all required branches (incl. managing any environment dependencies) Find and Fix Regression Issues Retest Regression Fixes Merge Regression Fixes with any branches created since the bug was injected Remembering where I was in the code, and what I was doing Retes-ng a failure found by another tester, as they are now on holidays …

9.15


Exercise: Es-ma-ng the Cost of Delay On day 1, Susan injects a defect into her code. 3 days later, a3er adding further func%onality, she merges her code to the V2.23 branch. Jim, the tester, is busy retes%ng some fixes from the last release, and its not %ll day 13 he discovers the failure caused by Susan’s bug. He logs the incident and assigns it to the test manager, who, 5 days later, schedules it for debug. Another 5 days pass before Susan opens up and starts working on the %cket. Other developers have been working on this same code recently and she’s been working on V1.24, so it takes her a full day to find and fix the bug. 6 days later, Jim retests and finds the defect has been fixed. He closes the %cket. In groups of 5/6: • Calculate the Cost of Delay in the above scenario in man hours. What ac-vi-es would be typical. Include (re)crea-ng environments, logging bug details, regression tes-ng, merging code, managing branches, reviewing, priori-sing, assigning -ckets, etc. See the Worksheet. • Which ac-vi-es could be avoided if the en-re code/test/fix/retest cycle took just 1 day, rather than 30 days? • Assuming a ‘man hour’ costs €100, what’s the cost of the 30 day delay?

9.35

25/11/2014

8



Late Elabora-on and Specifica-on by Example

User Stories

25/11/2014

9


What are User Stories? •  User Centric – what’s important to your customer •  Story – The Power of Narra-ve

–  We understand, remember and pay much more aWen-on to stories than facts

–  Collabora-ng on Stories helps us focus on the users and business value •  User Stories help us define

–  The Actor/Persona/Role (Who) –  The Ac-on/Func-onality (What) –  The Result/Benefit/Goal (Why)

A brief statement of intent that describes something the system needs to do for the user


What are User Stories? •  User Centric – what’s important to your customer •  Story – The Power of Narra-ve

–  We understand, remember and pay much more aWen-on to stories than facts

–  Collabora-ng on Stories helps us focus on the users and business value •  User Stories help us define

–  The Actor/Persona/Role (Who) –  The Ac-on/Func-onality (What) –  The Result/Benefit/Goal (Why)

A brief statement of intent that describes something the system needs to do for the user

A ‘Place

-‐holder

for a Co

nversa-

on’,

INPUT t

o Specifi

ca-on,

not the

output!

25/11/2014

10


Valuable ‘Slices’ of Func-onality

•  Stories should represent a ver-cal slice through the system and should be completed in one itera-on – and can therefore be tested in the same itera%on


User Story Example High Roaming Spend Warning (Data) As a billpay customer with data roaming enabled I want to be made aware if I’m approaching my threshhold so that I can decide whether to incur out-‐of-‐plan charges

Acceptance Criteria: •  Ini-al warning at 80% threshhold. •  Follow Up at 95% threshhold and op-on

to cap at threshhold. •  Warnings issued by SMS and email

(where email address validated) •  If 80% threshhold achieved in first 25%

of billing cycle, send ‘possible fraud alert’ to customer service queue.

CONVERSATION (Customer, Tester, Developer):

•  How many warnings? •  How should we issue the

warning? •  What text do we use in the

warnings? •  Do we force the user to

acknowledge the warning? •  Can the user configure

warning points? •  If they go above their

threshhold, do we issue periodic reminders/warnings?

25/11/2014

11


A Specifica-on


User Story As a HomeOwner, I want to regularly trim my lawn so its neat and -dy.

23

25/11/2014

12


Build Innova-on In

Problem Space Customers End Users

Domain Experts Product Owners

Solu-on Space Developers Testers Architects

UI/UX Designers

Innova-on Space

Uncertainty Ambiguity

Conversa-on Social Objects

24

As a <role>

so that <result>

I want to <ac-on>


Specifica-on By Example Business Goal

Scope (User Stories)

Key Examples

Specifica-on with Examples

Executable Specifica-on

Living Documenta-on

Deriving scope from goals

Specify collabora-vely

Refine the spec

Automate Literally

Validate Frequently

25/11/2014

13


What are Specifica-ons By Example?

•  Thin slices of system behaviour •  that deliver business value •  described as concrete examples •  that are poten-ally automatable •  without transla-on •  to create executable specifica-ons •  captured in live documenta-on.


Advantages of SBE

•  Realis-c Examples contain Precise Informa-on & require Precise Answers

•  Concrete examples s-mulate discussion & shared understanding •  They focus on business func-onality •  Edge cases flush out unseen requirements – but we can’t be

exhaus-ve Tes%ng: Crea%ng Examples Checking: Running Examples

25/11/2014

14


Using Examples to Elaborate and Illustrate User Stories

•  Each example describes a scenario or path through the func-onality in a user story.

•  Examples can define several pre-‐condi-ons and post-‐condi-ons (mul-ple Given and Then sec-ons) for one or more ac-ons (When clauses).

Given <a precondi-on> When <an ac-on happens> Then <the following is expected>


A User Story with Mul-ple Examples User Story: In order to aWract customers As an online bookseller I want to give purchasers of 5 books or over free delivery

Example: Given I have ordered 6 books When I go to online checkout Then I expect free delivery

Given When I go to online checkout

Then

Order Size Delivery

5 Free

4 At cost

100 Free

25/11/2014

15


Worked Example -‐ Customer Type User Story: In order to retain customers As an online bookseller I want to reward returning customers with free delivery on orders of 3 books or more Example: Given I am a returning customer And I order 3 books When I go to the checkout Then I expect free delivery

Given When I go to the checkout

Then

Customer Type

Order Size

Delivery

New 1 At cost

Repeat 2 At cost

Repeat 3 Free

Repeat 100 Free

New 3 At cost


Trade System Example Source: Cian Bracken, AgileTour Dublin 2014

25/11/2014

16


Trade System Example

Book is not adding any value here as the Primary Book aWribute is driving the behaviour

In order to be testable and unambiguous, the content of each cell in the table needs to represent only one value

User story is not formaWed correctly – it has two value statements

Source: Cian Bracken, AgileTour Dublin 2014


User Stories & Examples

’describe, demonstrate and develop’

User Story (Breadth)

Exam

ples (D

epth)

Exam

ples (D

epth)

Exam

ples (D

epth)

Exam

ples (D

epth)

Exam

ples (D

epth)

Exam

ples (D

epth)

Exam

ples (D

epth)

Exam

ples (D

epth)

Exam

ples (D

epth)

Edge Cases

Typical U

se Excep

-on

Typical U

se Excep

-on

Typical U

se Example

‘SPINE’ Example

Typical U

se Example

Typical U

se Excep

-on

Typical U

se Excep

-on

Edge Cases

25/11/2014

17


Examples vs. Scripts & Specifica-ons 1.  Set Thold 1=80%,

THold2=95%, Cap=N, In-‐Plan Limit 1GB, Usage 799MB

2.  Create Billing Request for 1MB and post to the users account

3.  Check that Request is granted and Warning 1 Issued

•  On reaching Thold1, Billing Requests are granted and Warning 1 issued.

Subjec-ve and difficult to automate.

10:30


Exercise -‐ SBE User Story: As a billpay customer with data roaming enabled I want to be made aware if I’m approaching my Data Limit so that I can decide whether to incur out-‐of-‐plan charges Example: Given my data limit is 1GB, Warning Threshhold 900MB, current usage is 999MB When I aWempt to download data Then I get a warning ‘You will incur extra charges acer a further 1MB’

Given When Then

In-‐Plan Data Limit

Warning Threshold

Data Usage

I go to download data Ac-on (s)

1GB 900MB 999MB You will incur extra charges acer a further 1MB

In groups of 5/6, Create a table of Examples to Test this User Story (See Worksheet)

25/11/2014

18



Whole Team Approach

Test First

25/11/2014

19


Ever Heard This?

•  When I see what it looks like, I’ll decide how to test it! •  How could I know how to test something un-l I’ve seen how its been implemented?

Specify Design Code Test


Test First

Specify Design Code Test

Test Code Design Specify

Test Code Design Specify Test Code Design Specify

Specify Design Code Test Specify Design Code Test

Test

Code

Design

Specify

Test

Code

Design

Specify

Test

Code

Design

Specify

Tradi-onal Sequence

Test First

Incremental

Incremental Test First Agile – All at Once

25/11/2014

20


Ever Heard This?


Requirement (User Story)

Developers Interpreta-on

Testers Interpreta-on

≠ Code

Tests


Ever Heard This?


Requirement (User Story)

Collabora-on (3 Amigos – Customer, Developer, Tester) results in a Shared Understanding

‘Single Source of Truth’: Tests, then

Code

25/11/2014

21


Test Driven Development


What’s your experiences with TDD?

–  Do you write a new test before you create a new class or method?

–  Do you refactor acer you get it working? –  Do your tests some-mes break when you refactor? –  Is that what you expected to happen?

25/11/2014

22


extremeprogramming.org on Unit Tests

•  TDD is usually synonymous with UNIT tes-ng •  “you should test all classes in the system. Trivial geWer and seWer methods are usually omiWed”

•  “Unit tests enable refactoring…Acer each small change the unit tests can verify that a change in structure did not introduce a change in func-onality”


Refactoring – A Defini-on Mar'n Fowler •  noun: a change made to the internal structure of so3ware to make it easier to understand and cheaper to modify without changing its observable behavior

•  verb: to restructure so3ware by applying a series of refactoring's without changing its observable behavior

25/11/2014

23


What’s a ‘Unit’ in Unit Tes-ng?

•  What Units are we talking about? –  Objects –  Private Methods –  External Interfaces –  APIs/Services

•  Refactoring Changes Implementa-on –  For TDD a ‘Unit’ is not a unit of implementa-on –  It’s a unit of ‘Observable Behaviour’

OBJ3

OBJ1

OBJ2

OBJ5

OBJ4 }

} Observable Behaviour

Implementa-on


The TDD Cycle

Step Ac-on Objec-ve

RED Write a Test reflec-ng the behavior required – it Fails

Learn – about the problem to be solved, the func-onality to be built

Step Ac-on Objec-ve



GREEN Write the code to make the test pass – as fast (cheap) as you can (kludge, copy/paste, spaghe] code, whatever)

Learn – about the solu-on needed to pass the tests – including excep-ons, edge cases, unreasonable use,…

Step Ac-on Objec-ve



GREEN Write the code to make the test pass – as fast (cheap) as you can (kludge, copy/paste, spaghe] code, whatever)

Learn – about the solu-on needed to pass the tests – including excep-ons, edge cases, unreasonable use,…

REFACTOR Design and re-‐implement the code in the simplest, most parsimonious and most elegant way you can

Implement – the cleanest, best designed code possible -‐ based on what you’ve learned about the problem and the required solu-on

25/11/2014

24


Some Objec-ves of TDD •  Firstly its about Learning

–  Understanding the requirement/problem –  Understanding what func-onality is needed

•  Second its about Development –  Implement the simplest thing that would work –  Allows us ‘refactor’ safely – reduced cost of change –  Enables ‘Emergent Design’ – design when we understand the problem –  Encourages callable, testable, loosely coupled code

•  Thirdly its about Tes-ng –  We test Early – even before we implement –  We test Ocen (if we automate) – fast, on-‐going feedback –  Reduces Tes-ng Cost of Delay: Prevent Defects over Finding Failures

•  Never write a single line of code unless you have a failing test •  Eliminate Duplica%on


Some Objec-ves of TDD •  Firstly its about Learning

–  Understanding the requirement/problem –  Understanding what func-onality is needed

•  Second its about Development –  Implement the simplest thing that would work –  Allows us ‘refactor’ safely – reduced cost of change –  Enables ‘Emergent Design’ – design when we understand the problem –  Encourages callable, testable, loosely coupled code

•  Thirdly its about Tes-ng –  We test Early – even before we implement –  We test Ocen (if we automate) – fast, on-‐going feedback –  Reduces Tes-ng Cost of Delay: Prevent Defects over Finding Failures

•  Never write a single line of code unless you have a failing test •  Eliminate Duplica%on

Tes-ng and Development are Inter-‐dependent,

Concurrent Ac-vi-es Con-nuously feeding off

each other to maximise Learning

25/11/2014

25


TDD -‐ Summary •  TDD is Outside-‐In: Create tests to test func-onality, not implementa-on (BB) – Beck –  Unit Tests test a ‘Unit’ of implementa-on

•  Verifies the code implements the developers design –  TDD tests a ‘Unit’ of func-onality/behaviour

•  Verifies the code implements the behaviour required

•  BDD is the new ATDD is the new TDD •  TDD is about LearningèDevelopingèTes-ng


Test First and You In Groups of 5/6, discuss barriers and possible solu-ons to implemen-ng Test First in your organisa-on:

–  Ge{ng testers involved before/concurrently with developers •  Gaining a shared understanding of requirements

–  Ge{ng early agreement of acceptance criteria for each requirement •  Defining Test Criteria before deciding on the implementa-on

–  Crea-ng Concrete, precise examples as ‘executable requirements’ and tes-ng using these (and automa-ng them)

–  Including ‘Refactoring Time’ in the schedule

10 mins discussion, 10 mins sharing barriers/solu-ons

25/11/2014

26


Autonoma-on

Test Automa-on


Test Automa-on – An ‘An--‐Paeern’

25/11/2014

27


The Automa-on Pyramid

Unit/Component layer Developer Tests

e.g. JUnit

API/Service layer Acceptance Tests

e.g. Fitnesse, Cucumber

GUI layer e.g. Selenium

Automate at feature/workflow level

Automate at story level

Automate at design level

Based on Mike Cohn

Things to Consider: •  Cost to Produce (TDD/BDD Collateral) •  Cost to Maintain •  Ability to Test First/Early •  Quick to Run/Reduced EXE Overlap •  Ease of Pinpoin-ng Defect

Manual Tests e.g. exploratory

Derived from Mike Cohn


Checking vs. Testing •  Checking •  is the process of making

evalua%ons by applying algorithmic decision rules to specific observa%ons of a product.

•  human or machine •  determinis%c •  a type of tes%ng

•  Tes-ng •  is the process of evalua%ng

a product by learning about it through experimenta%on, which includes to some degree: ques%oning, study, modeling, observa%on and inference.

•  a human endeavor •  subjec%ve •  includes checking

Source: James Bach blog

25/11/2014

28


Warnings on Automa-on •  Tests find bugs, automa-on doesn’t •  Automa-on good for:

–  Checking: Confirma-on, Machine decidable, determinis-c –  Regression, test prepara-on, non-‐func-onal

•  Can be overused – “if you have a hammer everything looks like a nail”.

•  Understand Costs & Benefits (create, maintain, run, interpret, etc.)


The Spy that came in from the Cold

Exploratory Tes-ng

25/11/2014

29


Exploratory Testing Hendrickson: a style of testing in which you explore the software

while simultaneously designing and executing tests, using feedback from the last test to inform the next

Kaner: Simultaneous learning, design and execution, with an

emphasis on learning. Cunninghan: “Because the program always runs, it is always ready

to be explored.”


Why do exploratory tes-ng? •  Advantages:

–  FAST: •  Minimal prepara-on (e.g. a charter/objec-ve, -mebox) •  Test Cases/Procedure not recorded, only incidents

–  FOCUSED •  On a par-cular area to achieve coverage/confidence •  Risk based focus – re-‐evaluated as tes-ng proceeds

–  PRODUCTIVE •  Finds more defects per unit of effort than other methods

•  Disadvantages: –  Un-‐repeatable and not suitable for regression –  May require a skilled tester to execute, with knowledge of the product/domain –  Need to avoid tendency to revert to ‘unplanned’ tes-ng

25/11/2014

30


Example: Session based tes-ng •  1-‐2 hour sessions each star-ng with a charter and ending with a report/debrief •  Charter – iden-fies the goal of the session

–  What to test? –  How long for? –  Areas to explore –  Risky areas? –  Bug types to be inves-gated? –  Extreme values/tricky situa-ons?

•  In general exploratory can help you get familiar with product & target hotspots –  Try extremes, anything you like to break product –  When you find a bug, explore around it –  Look for paWerns, clues to other bugs –  Document the test that caused the failure; document other tests that cover 'interes-ng' situa-ons –  Report/Log an incident –  Move on to next interes-ng area


Exercise: Exploratory Tes-ng •  Individually :

•  Download coin flip free (iHandy) app if you have smart phone (or sit with someone who has) •  Start exploratory tes-ng to find bugs (Record on worksheet) •  Acer 5-‐10 mins discuss what you did with the person beside you and see if you can come up

with any more tests/bugs •  (15 mins total)

•  Discussion on results •  What bugs did you find? •  How did you find them? What guided your ac-ons?

Thanks to Ann-‐Marie CharreW for the idea of this exercise

25/11/2014

31


How agile changes test •  Whole Team Approach to Quality

•  QA means ‘Quality Assistance’, ‘Ques-on Asker’ •  Specialised Skills, not Demarcated Responsibility (Competency vs. Role)

•  Be Proac-ve: Can’t ‘Test In’ Quality, need to “Build In’ •  Design, Coding and Tes-ng are one process (‘All at Once’)

•  Testers need to understand ‘the system’ – e.g. architecture •  ‘T Shaped Testers’: e.g. SQL, Environments, Perf Tes-ng, Data Prep,

scrip-ng

•  Role of Testers is to Help Prevent, not to Detect


Q&A

–  Training –  Consul-ng –  Coaching –  Assessments

www.agileinnova-on.eu

colm oheocha agile testing v1.2 · 25/11/2014’ 3 ’ scrumcycle’ 1day’ daily’standx...

Documents