or how to find bugs faster…. what is exploratory testing? why are we doing black box testing? ...
TRANSCRIPT
Exploratory Testing
Or how to find bugs faster…
What is Exploratory Testing? Why are we doing black box testing? Session Based Testing Where do we test? Types of testing How outlines can help How flow charts could help
Summary
Exploratory testing is an approach to software testing that is concisely
described as simultaneous learning, test design and test execution. Cem Kaner, who coined the term in 1983,[1] now defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."[2]
While the software is being tested, the tester learns things that together with experience and creativity generates new good tests to run. Exploratory testing is often thought of as a black box testing technique. Instead, those who have studied it consider it a test approach that can be applied to any test technique, at any stage in the development process. The key is not the test technique nor the item being tested or reviewed; the key is the cognitive engagement of the tester, and the tester's responsibility for managing his or her time.[3]
- http://en.wikipedia.org/wiki/Exploratory_testing
What is Exploratory Testing?
is not just adhoc and saying adhoc is a turn off
to people who don't understand exploratory testing. Why? True adhoc is literally monkey pressing buttons.
… is a learning process. The code will change as the project
progresses. Doing the same thing over and over again has diminishing returns
Not a replacement for doing/running test cases
Exploratory Testing
Is not a replacement for automation testing It’s a supplement to reduce the risk of the
unknown quantity of bugs
Comment : I would implore various strategies of depth and breadth type testing in trying to reduce the risk of an unknown quality and severity of bugs to make them known.
Exploratory Testing
Because human thoughts and behavior
changes from individuals as well as over time. We don’t necessarily do the same thing over and over again like automation testing does
Bottom line : people think. And people think differently.
Why do we do black box testing?
Be aware of yourself and why you may miss
bugs: http://viscog.beckman.illinois.edu/media/
mistakes.html
http://viscog.beckman.illinois.edu/flashmovie/15.php
Human Cognitive Behavior
Be a scientist! Poke around! Try to figure things out Go beyond just doing the test case… Talk to each other about it, learn from each other If you don’t understand the bug, then how do you expect other
people to? It’s like chess, try to think in steps ahead Test cases aren’t the rule book; they are a guideline
The more you test the more experience you will gain. You can learn from any source; I often learn from people I have mentored as much as I’ve learned from the people who have taught me!
Testing
Session-based testing is a software test method
that aims to combine accountability and exploratory testing to provide rapid defect discovery, creative on-the-fly test design, management control and metrics reporting. The method can also be used in conjunction with scenario testing. Session-based testing was developed in 2000 by Jonathan and James Bach.
Highly recommend looking at this… this is what top blackbox testers do in finding bugs as well as other things discussed later on.
Session Based Testing
New Code Change
Looking at code change Trends in bugs Talking to Dev; what might they be worried about? Talking to other QA; what seems fragile? Risk areas
What’s dangerous if it fails for human safety Embarrassment (functionality) Complexity untested
Where to test?
MVC = Model View Controller
UI Testing Functional/Business Logic testing Data/Input Testing
UI Testing Does user flow make sense? (Usability) Culturally effective? Which culture? Does it look cluttered?
Types of testing
Functional Testing (Positive Testing) Negative Testing Boundary Testing Race Conditions Environment conditions Device specific Localization Initialization Load/Endurance 508 Compliance
Types of testing
Functional Testing (Positive Testing)
Use Cases and specifications Negative Testing
Error handling, like bad email and incorrect website address
Boundary Testing Max characters, or with integers -1, 0, 1, max -1,
max, max + 1 Race Conditions
Running music app, video app, and getting a phone call at the same time
Examples
Environment conditions
Rural area testing of telephony vs urban Device specific
Battery running low, graphics issues Localization
Date format, starting day, fitting in ui, punctuations
Initialization Start up of device, FTE, start up of apps versus
OTA and continual usage
Examples
Load/Endurance
Loading tons of facebook, versus keeping music app running for days on end
508 Compliance Handicap ie : blind, deaf user interfaces
Examples
From 1.0
https://wiki.mozilla.org/B2G/QA/Gaia_Test_Plan/Browser
https://wiki.mozilla.org/Releases/Firefox_25/Test_Plan
https://wiki.mozilla.org/QA/Fennec/Milestones/2.0/TestPlan:Virtual_Keyboard/Android_VKB
https://wiki.mozilla.org/QA/Fennec/24/Beta/8
How outlines/test plans can help
https://docs.google.com/spreadsheet/ccc?key
=0Arku3jleCA0UdF9ZZWtyYm9iV1NLOGRTWkQzWHFsS2c#gid=5
B2G Specific Example
Mapping out user flow
Demo
How flowcharts can help
QA conferences
They can be pricey and advertisey QA newsgroups/forums/websites QA magazines/books/articles
http://en.wikipedia.org/wiki/Software_testing http://www.amazon.com/exec/obidos/ASIN/047
1358460/softwarqaandtest
Resources
https://wiki.mozilla.org/B2G/Architecture https://github.com/sotaroikeda/firefox-diagrams/wiki/Firefox
-Diagrams
Git Hub: https://github.com/mozilla-b2g/gaia/commits/master Compare to sources.xml file :
https://pvtbuilds.mozilla.org/pub/mozilla.org/b2g/nightly/mozilla-central-unagi/latest/sources.xml
Tool for searching code https://mxr.mozilla.org/
B2G Architecture