Exploratory Testing: Real world rapid testing in Agile teams

Ravi Shanker Principal Program Manager, Microsoft Corporation

GSJGD

Agenda

Focusing on customer value through the lifecycle

Exploratory testing – why care?

Debunking exploratory testing myths

The Agile Workflow

Customer value – where are you?

Customer value in backlog item

Tester = customer proxy

Is the backlog item definitive customer value?

Is this the best customer value to deliver this sprint?

Exploratory testing

Centered on customer value rather than spec compliance

Find bugs fast and early without overhead

Tests working software!

Exploratory testing – myths

1. Exploratory testing is ad-hoc, random testing

2. There is no way to measure exploratory testing

3. Bugs found through exploratory testing are hard to repro

What are you DOING?

Method in the madness – “Testing Tours”

Guidebook tour

Back alley tour

Supermodel tour

You can make up your own…

Dogfood IDC NAR

Sample tours – how we did it

The Reflex tour – tests responsiveness of the IDE

The Idiot tour – tests F1 inline help on each element

The Palatial tour – tests our multi monitors experience

The Footprint tour – tests virtual memory usage

The Budget tour – tests experience on a low end machine

Dogfood IDC NAR

Food for thought…

For the application that you are working on, build a “tour” that you think will be useful for exploratory testing.

Dogfood IDC NAR

Where are we?

Measuring exploratory testing

Right metrics are like a fingerprint of your team: no silver bullet

Distinguish between app quality metrics and metrics that direct exploratory testing

Metrics – how we did it

Code coverage across tours

Story complexity to bugs ratio

Session duration for stories

Session roll up per tour

Heat maps – how we did it

Food for thought…

For the project that you are working on, think of what metrics are most meaningful to direct your exploratory testing

Dogfood IDC NAR

Finding reproducible bugs

Bug filing with exploratory testing

3 components of time spent

during exploratory testing

(courtesy Jon Bach)

• S = Setup time

• T = Testing time

• B = Bug filing time

S

T

B

S

T

B

“Cut the BS and get more T !!”

Rich bugs – how we did it

Automatic repro steps

Screenshots

Desktop recording

System information

Intellitrace logs

Test documentation – how we did it

Test cases automatically created while doing exploratory testing

Can save artifacts from exploration like expected image

With each sprint, regression suites are automatically ready

Conclusion

Testing for customer value is paramount throughout the product lifecycle

Exploratory testing done with right tools enables you to:

Define and vector testing as appropriate for your app

Communicate your testing goals and progress to the team

Test more frequently and file comprehensive bugs faster

Go on…be a fearless explorer!

Q&A