Download - Exploratory Testing Is Now in Session
TI AM Tutorial
4/30/13 8:30AM
Exploratory Testing Is
Now in Session
Presented by:
Jon Bach
eBay, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Jon Bach
Jon Bach works for eBay (San Jose) as a QA director for the Buyer Experience team. Formerly a manager of corporate intellect and senior test consultant at Quardev, Jon has been in testing since 1995 with experience that includes managing teams at Microsoft, HP, and LexisNexis. The co-inventor (with his brother James) of session-based test management, Jon is an award-winning speaker on test management and exploratory testing. He’s mostly known for his “half-baked” ideas about how to inspire innovation and create a learning culture for test teams. Find Jon on Facebook and Twitter @jbtestpilot. View his presentations at quardev.com/articles and his blog at jonbox.wordpress.com.
eBay Inc. confidential
Exploratory Testing: Now in Session
Jon Bach
QE Director, eBay
STAR East 2013
eBay Inc. confidential
Testing is about DISCOVERY…
eBay Inc. confidential
…finding emerging context
eBay Inc. confidential
enhanced by conference…
eBay Inc. confidential
...affected by inference
eBay Inc. confidential
…framed by reference
eBay Inc. confidential
“The” ET Definition
A style of software testing… that emphasizes the personal freedom… and responsibility of the individual tester… to continually optimize the quality of his/her work… by treating testrelated learning… test design… test execution… and test result interpretation… as mutually supportive activities… that run in parallel… throughout the project.
Cem Kaner, 2006
eBay Inc. confidential
The agile nature of exploration and the ability of testers to rapidly apply their skills and experience make exploratory testing a widely used test approach—especially when time is short.
But exploratory testing is often dismissed by project managers who assume that exploratory testing is not reproducible, measurable, or accountable.
eBay Inc. confidential
Why this tutorial?
1) Exploratory testers want respect: When testers explore during testing, they find great bugs. However, since they often don’t know how to describe their thinking, it’s considered to be dismissed as “playing around”.
2) The documentation dilemma: Project managers may insist that all testing be documented, but Agile does not emphasize that. So how to balance time spent documenting with time spent testing?
3) Your work might be scrutinized: You may have to give a report someday about something you did that was exploratory – like attending this conference.
eBay Inc. confidential
A boring (Boeing) story
eBay Inc. confidential
On October 30, 2006, Alaska Airlines flight 61, a Boeing 737200, N740AS, took off from runway 34 right at SeattleTacoma International Airport, Seattle, Washington.
There were no injuries to the 71 passengers or 5 crewmembers and there was no damage to the airplane.
The airplane was operating under the provisions of 14 CFR Part 121 and continued uneventfully to its destination of Juneau International Airport, Juneau, Alaska.
eBay Inc. confidential
The Report
eBay Inc. confidential
Why the report?
1. The government is collecting statistics on runway incursions, which are potentially deadly.
2. Since the pilots or ground controllers are subject to reprimand or dismissal for this sort of thing, a legal record that it happened is necessary.
3. Ground controllers or signage may have caused the problem. You need a record of the event, and investigation, to discover if there is another accident waiting to happen.
eBay Inc. confidential
If you’re not free to think,
or learn,
and adapt
while you test,
it’s not exploration.
Key Idea
eBay Inc. confidential
My missions for you
1) Discuss — and practice — a way to manage and measure exploratory testing.
2) See some free tools that support sessionbased exploration.
3) Learn why exploratory testing can be thoughtful, structured, and measurable.
4) Have some fun.
eBay Inc. confidential
A structure for exploration
Mission
Tactics
Story
eBay Inc. confidential
Session report
CHARTER-----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. #AREAS .LINKOS | Windows 2000Menu |Strategy | Function TestingStrategy | Functional AnalysisSTART-----------------------------------------------5/30/00 03:20 pm TESTER-----------------------------------------------Jonathan BachTASK BREAKDOWN----------------------------------------------- #DURATIONshort #TEST DESIGN AND EXECUTION65 #BUG INVESTIGATION AND REPORTING25 #SESSION SETUP20
My session writeup from exploratory testing for
Agile2010…
eBay Inc. confidential
Exploratory exercise
Test this…
eBay Inc. confidential
A story of my exploration
eBay Inc. confidential
Translation
Test Design and Execution
Bug Investigation
(and Reporting)
Session Setup
eBay Inc. confidential
How could you accurately report your testing?
unscripted
unrehearsed
improvisational?
??
?
?
?
eBay Inc. confidential
One Solution:
1) Time Box2) Reviewable Result3) Debriefing
The “Session”
eBay Inc. confidential
Time Box: focused effort, fixed duration
– Brief enough:• for accurate reporting• to allow flexible scheduling• to allow course correction
– Long enough:• to get solid testing done• for efficient debriefings
Short: 60 minutes (+-15)Normal: 90 minutes (+-
15)Long: 120 minutes (+-15)
eBay Inc. confidential
The deliverable: a session report
• Charter– #AREAS
• NotesCHARTER-----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. #AREASOS | Windows 2000Menu | et-jsb-010417-c.sesViewStrategy | Function TestingStrategy | Functional Analysis SampleSTART-----------------------------------------------5/30/00 03:20 pm TESTER-----------------------------------------------Jonathan Bach TASK BREAKDOWN----------------------------------------------- #DURATIONshort #TEST DESIGN AND EXECUTION65 #BUG INVESTIGATION AND REPORTING25 #SESSION SETUP20
Metrics– #DURATION– #TEST DESIGN AND EXECUTION– #SESSION SETUP– #BUG INVESTIGATION AND REPORTING– #CHARTER / OPPORTUNITY
Issues– #ISSUE
Bugs– #BUG
eBay Inc. confidential
Debriefing: a talk with the lead
The tester answers questionsSession metrics are checkedCharter may be adjustedSession may be extended New sessions may be chartered
eBay Inc. confidential
Anticipating Scrutiny
Feelings: How are you feeling about what happened?
Past: What kinds of things did you do?
Results: What were your findings?
Obstacles: What slowed your mission?
Outlook: Where do we go from here?
Agenda: “PROOF”Past Results Obstacles Outlook Feelings
eBay Inc. confidential
Are you accountable?
Was this mission reasonable?
What did you find?
How did you spend your time?
Did you need some help / tools?
Do you think there’s more to do here?
Agenda: “PROOF”Past Results Obstacles Outlook Feelings
eBay Inc. confidential
The real message
What’s being asked What they may be thinking
What was your mission? Remind me what I told you to do.
Why did you do that?How did it go? Were you careful or reckless?
What should I be worried about?How far did you get? Are we closer to shipping?
Can you help me know our status?Need anything? Can I speed this along?
Do you need more of my time?When will you be done? Will I get my bonus?
I have a new task for you…
eBay Inc. confidential
eBay sample
CHARTER-----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. #AREAS LINKMenu | et-jsb-010417-c.sesViewStrategy | Function TestingStrategy | Functional AnalysisSTART-----------------------------------------------5/30/00 03:20 pm TESTER-----------------------------------------------Jonathan Bach TASK BREAKDOWN----------------------------------------------- #DURATIONshort #TEST DESIGN AND EXECUTION65 #BUG INVESTIGATION AND REPORTING25 #SESSION SETUP20"C:\Documents and Settings\jobach\Desktop\Live Site\israt_sessions\Session_Listing_Promo.txt"
eBay Inc. confidential
Exercise:
An “exploration and analysis” session
eBay Inc. confidential
When to use SBTM
Getting support: “We have thousands of test cases and a huge regression suite, but some of our best bugs come from ad hoc testing. Why is that?”
Improving skill: “How she finds those great bugs without test cases, I’ll never know. I guess some people are just natural explorers you either have it or you don’t and I just don’t have her knack for it.”
Earning trust and autonomy: “The triage team needs more information about these 3 bugs, can you give me that by the end of the day?”
eBay Inc. confidential
Are you accountable?
Was this mission reasonable?
What did you find?
How did you spend your time?
Did you need some help / tools?
Do you think there’s more to do here?
eBay Inc. confidential
Mission brainstorm
As a London Underground rider,I want to enter my ticket into the kiosk,Because I want to go to Heathrow.
What are the possible outcomes?
http://www.faqs.org/faqs/uk/transportlondon/section6.html
eBay Inc. confidential
London Underground
Are these all of the possible states?
http://www.faqs.org/faqs/uk/transportlondon/section6.html
eBay Inc. confidential
Sources of charters
• Documentation / Specifications
• Web forums
• Previous products
• Team members
• Competing products
• Your expertise
• Help files
• Manuals
eBay Inc. confidential
High-level sources
Test Plan Evaluation Modelhttp://www.satisfice.com/tools/tpemodel.pdf
Test Planning Checklisthttp://www.satisfice.com/tools/buildtheplan.pdf
Heuristic Test Strategy Modelhttp://www.satisfice.com/tools/satisficetsm4p.pdf
eBay Inc. confidential
Exploratory testing
is a combination of
Imagination
and
Discovery
Key Idea
eBay Inc. confidential
CIDTESTDSFDPOTCRUSSPICSTMPLFDSFSCURA
How to Discover with Your Imagination
StructureFunction
DataPlatform
OperationsTime
Function testingDomain testingStress testingFlow testing
Scenario testingClaims testingUser testingRisk testing
Automatic testing
CustomersInformation
Developer relationsTeam
Equipment & toolsScheduleTest Items
Deliverables
CapabilityReliabilityUsabilitySecurity
ScalabilityPerformanceInstallabilityCompatibilitySupportability
TestabilityMaintainability
PortabilityLocalizability
eBay Inc. confidential
Charters
• General charters may be necessary at first:
• “Analyze the Insert Picture function”
• Specific charters provide better focus, but take more effort to design:
• “Test clip art insertion. Focus on stress and flow techniques, and make sure to insert into a variety of documents. We’re concerned about resource leaks or anything else that might degrade performance over time.”
eBay Inc. confidential
Some sample session charters
• Installation: When installed, does Triangle! put any files in the wrong places? Does it leave any files for the uninstall? Check the registry keys, use InCtrl to see what changes are made. Installation is new, so we want to be sure it’s clean.
• Boundary testing: We got word from customer support that there are runtime errors when using integers over 32000 but no one can repro it. Best recon is on Win XP Pro with Office 2003 running in the background. Sam K. in CSS says you can use his machine, and he also has customer specs.
• Ship drill: Start Triangle! right out of the box. For example, is the readme ready to go? We’re waiting from word on Legal as to the License Agreement, but that shouldn’t hold you up. Also make sure you hit Vista and see what issues arise there.
• Claims testing: Triangle is meant for first graders, but we plan to ship a version to General Dynamics in a few months. Try some usability profiles or personas to see what functions become more or less risky. Also, discover the algorithm by which Triangle! reports its results. Is it way off from what a user would expect. Does it cause the user to lower their confidence?
eBay Inc. confidential
Open-Book Charters for eBay
• What item for sale is nearest to you location?Submitted the most and sold the least?What item appears most with no bids?What are the most recent items posted for any given category?Which category has the most items?Which search strings create null queries?What are 3 examples of broad queries vs 3 examples of ambiguous queries?What queries create invalid results?How do you find "sold" items?What is the most common item?What is the most bizarre item?What is the most expensive item?Is it possible to restrict results to one category without the categories field?Which seller has listed the most items which have not sold?What search string takes the longest to return results?Is there a search that works differently on different browsers?What was the highest bid for an item in support of any charity?Find an item that the seller has no business selling!Find an item you find only on eBay, not Amazon.What item could you find that had the most bids?How many eBay stores are there?What is the oldest object (not item) for sale on ebay?Are results different if you are logged in?Try a search on ebay.com and then try the same on on your mobile. What are the 3 major differences?What item has the greatest difference between the highest and second highest bids?What user has bid the most but won the least?What item has been on eBay the longest? Is there a way to tell the age of the item ID?Who has sold the most items?Is there a way to find the most commonly misspelled item?What item can be found in the most categories?
eBay Inc. confidential
The session report
• Charter– #AREAS
• NotesCHARTER-----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. #AREASOS | Windows 2000Menu | et-jsb-010417-c.sesViewStrategy | Function TestingStrategy | Functional Analysis SampleSTART-----------------------------------------------5/30/00 03:20 pm TESTER-----------------------------------------------Jonathan Bach TASK BREAKDOWN----------------------------------------------- #DURATIONshort #TEST DESIGN AND EXECUTION65 #BUG INVESTIGATION AND REPORTING25 #SESSION SETUP20
Metrics– #DURATION– #TEST DESIGN AND EXECUTION– #SESSION SETUP– #BUG INVESTIGATION AND REPORTING– #CHARTER / OPPORTUNITY
Issues– #ISSUE
Bugs– #BUG
eBay Inc. confidential
Notes from a recent conference
eBay Inc. confidential
Styles I’ve seen
1) Novel Narrative
2) Flight Data Recorder
3) ESPN Highlight
eBay Inc. confidential
Clark’s notes…
Captain Clark, River Dubois opposite the mouth of the Missouri River, 13 May 1804
“I dispatched an express this morning to Captain Lewis at St. Louis. All our provisions, goods, and equipage on board of a boat of 22 oars [party], a large pirogue of 71 oars [in which 8 French], a second pirogue of 6 oars [soldiers], complete with sails, &c. Men completed with powder cartridges and 100 balls each, all in health and readiness to set out. Boats and everything complete, with the necessary stores of provisions and such articles of merchandise as we thought ourselves authorized to procurethough not as much as I think necessary for the multitude of Indians through which we must pass on our road across the continent.”
eBay Inc. confidential
What to write while exploring
Historical Explorer Tester
Observations(To the degree you think they are relevant to stakeholders)
• drawings of flora / fauna• descriptions of indigenous people• landmarks
• feature model• text from log files• text from dialogs
Conjectures(Inferences based on experiences. After I test, I think I know something)
• what is this thing?• where should we go today?• how do we get there?• new orders from HQ?• are those people hostile?
• test ideas• questions• product and project issues• concerns• risks
Project information (Independent of observer)
• mission • supplies and staff• latitude / longitude• death and disease• supply status
• charter• test actions• config info• build details• tools used
eBay Inc. confidential
Scrutiny: what happened here?
“(Orderly Book) Camp Mouth of the Kansies June 29, 1804Ordered A Court Martiall will Set this day at 11 oClock... for the trial of John Collins and Hugh Hall...John Collins Charged "with getting drunk on his post this Morning out of whiskey put under his charge as a Sentinal, and for suffering Hugh Hall to draw whiskey out of the Said Barrel intended for the party."...The commanding Officers approve of the Sentence of the Court and orders that Punishment take place at half past three this evening, at which time the party will Parrade for inspection.”
June 29, 1804William Clark
eBay Inc. confidential
Jon Bach, 2001
Sample test notes from actual exploratory testing :
etjsb010416a.ses
etjsb010416b.ses (for Issues)
etjsb010416c.ses (a map for more ideas)
etjsb010417a.ses (Claims Testing)
etjsb010417b.ses (for Bugs)
etjsb010417c.ses (for referenced notes)
etjsb010418c.ses (more Claims)
eBay Inc. confidential
How much detail is enough?
1) Sufficient benefits
2) No critical problems
3) Benefits outweigh problems
4) All things being equal, further documentation (or time, effort, testing, editing, etc.) is more harmful than helpful
All 4 conditions must be met to know when you’re done, (From brother James’ Rapid Software Testing course)
But ask: “good enough for who, what and when?”
eBay Inc. confidential
Metrics and Mechanics
Gathering empirical data about the object of your study; collecting different kinds of data, or data about different aspects of the object. Designing experiments and establishing lab procedures.
eBay Inc. confidential
The session report
• Charter– #AREAS
• NotesCHARTER-----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. #AREASOS | Windows 2000Menu | et-jsb-010417-c.sesViewStrategy | Function TestingStrategy | Functional Analysis SampleSTART-----------------------------------------------5/30/00 03:20 pm TESTER-----------------------------------------------Jonathan Bach TASK BREAKDOWN----------------------------------------------- #DURATIONshort #TEST DESIGN AND EXECUTION65 #BUG INVESTIGATION AND REPORTING25 #SESSION SETUP20
Metrics– #DURATION– #TEST DESIGN AND EXECUTION– #SESSION SETUP– #BUG INVESTIGATION AND REPORTING– #CHARTER / OPPORTUNITY
Issues– #ISSUE
Bugs– #BUG
eBay Inc. confidential
What to measure
Test Design and Execution
Bug Investigation
(and Reporting)
Session Setup
eBay Inc. confidential
Reporting the metrics
• Test, Bug, and Setup are orthogonal categories, but all we’re only tracking interruptions to testing
• Nearest 5% or 10% is good enough
• If activities are done simultaneously, report the highest precedence activity: T, B, then S
eBay Inc. confidential
TBS Report
eBay Inc. confidential
Sample project
HTML status report
eBay Inc. confidential
Tools
1) Scan Tool (James & Jon Bach)
2) Session Tester (Jonathan Kohl)
3) Rapid Reporter (Shmuel Gershon)
eBay Inc. confidential
No n S e s s io n6 1 %
T e s t2 8 %
B u g4 %
O p p o rtu n ity1 %
S e tu p6 %
Diagnosing productivity
• Do these proportions make sense?
• How do they change over time?
• Is the reporting protocol being followed?
0 .0
5 0 .0
1 0 0 .0
1 5 0 .0
2 0 0 .0
2 5 0 .0
3 0 0 .0
5 /2 6 6 /9 6 /2 3 7 /7 7 /2 1 8 /4 8 /1 8
eBay Inc. confidential
Estimating a Test Cycle
1. How many perfect sessions (100% on-charter testing) does it take to do a cycle? (let’s say 40)
2. How many sessions can the team (of 4 testers) do per day? (let’s say 3 per day, per tester = 12)
3. How productive are the sessions? (let’s say 66% is on-charter test design and execution)
4. Estimate: 40 / (12 * .66) = 5 days
5. We base the estimate on the data we’ve collected. When any conditions or assumptions behind this estimate change, we will update the estimate.
eBay Inc. confidential
Testing is like looking for words…
???
(10 minutes)
eBay Inc. confidential
Debriefing
All groups: How did you feel about your mission?
Mission A: What happened during the session?Mission B: How many words did you find?
Mission C: What obstacles did you face?
Mission D: Do you think you found all the words?
Agenda: “PROOF”Past Results Obstacles Outlook Feelings
eBay Inc. confidential
3 Key Ideas
Awareness
Guidance
Evaluation
eBay Inc. confidential
Awareness
• Where do charters come from? What are the domains?
• How to pay attention to the questions you have, second by second
• Questioning as a way to learn product modeling and test planning
• Paired testing sessions: teach collaboration and test technique
• Teamwide debriefing: teaches testers what test managers expect
• Testers can write charters for each other
eBay Inc. confidential
Guidance
• Types of charters – using several types of questions to demonstrate different paths or contexts through a product
• Helps explain critical thinking when a charter is vague or has several approaches
• Using personas can help frame your strategy
• Acquainting testers with both an intellectual “workspace” where certain answers are expected, and a “playspace” where initiative, creativity, and exploration is encouraged
• Sessions as a way to orient testers with a piece of software used in class
eBay Inc. confidential
Evaluation
• How do you approach the charters? • What paths did you take?
• What energy do you bring to the charter?
• What kinds of abilities emerged?
• What resourcefulness is demonstrated (i.e. what kinds of literal resources are being consulted)?
• Are your notes and narratives sufficient to convey answer(s)?
eBay Inc. confidential
More info
• ET Dynamics:
http://www.satisfice.com/articles/etdynamics.pdf
• Test Heuristics and Planning
http://www.satisfice.com
• ContextDriven Software Testing
http://groups.yahoo.com/group/softwaretesting
• Center for Software Testing Education and Research
http://www.testingeducation.org/BBST
• Books related to Exploratory Testing skills and tactics
http://www.testingreflections.com/node/view/3190
• Scenario testing examples
http://www.testingeducation.org/a/scenario2.pdf
eBay Inc. confidential
More info
• SBTM article – www.quardev.com
• Scan tool www.quardev.com
• James Lindsay http://www.workroomproductions.com/papers/STAREast_AiSBT_slides.pdf
• More about Exploratory Testing – http://www.satisfice.com/articles.shtml
eBay Inc. confidential
Next steps?
1) Tell the “tactics” story: Practice using terminology for the skills used during exploration. It has helped managers understand the sapience and discipline of exploration.
2) Try a threeweek SBTM pilot: Frame your explorations in sessions (timeboxed effort with a report) and see how your stakeholders react.
3) Use PROOF to anticipate scrutiny: Testing ourselves is just as important as testing software. It has won testers more credibility, autonomy, and respect.