exploratory testing and automated testing: create a healthy relationship
TRANSCRIPT
T24 Special Topics 10/6/16 15:00
Exploratory Testing and Automated Testing: Create a Healthy Relationship
Presented by:
Dawn Jardine
Razorfish
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.starwest.techwell.com/
Dawn Jardine An associate quality assurance manager for Razorfish in Toronto, Dawn Jardine has worked in several roles in her career. After years in the teaching profession, Dawn changed direction and entered the software field as a front-‐end web developer for an eLearning software company. Dawn expanded her skills and expertise to include instructional design, eventually joining the QA testing team. With more than fifteen years of experience, she has participated in the launch of several eLearning and eCommerce projects, including Canadian Police Knowledge Network, Canadian Tire Corporation, Walmart Canada, and American Express Membership Rewards. Passionate about quality process and women in technology, Dawn can be found on Twitter @dawnjardine.
9/21/16
1
Exploratory & Automation Testing:
Create a Healthy Relationship
Dawn Jardine
2
Introduction
9/21/16
2
3
Toronto, Canada
4
1
2
3
4
The Role of Test Automation
Exploratory Testing and Its Role
The Mutual Relationship
How We Make it Work
Agenda
9/21/16
3
5
The Role of Test Automation
6
Why use automation?
9/21/16
4
7
Automation can provide better coverage • for a large code base • when a live site is in maintenance mode • when it supplements manual testing
8
Automation can find defects earlier • finds defects introduced during sprint development • supports process of continuous test-and-fix in lower life cycle • reduces the cost of rework
9/21/16
5
9
Automation speeds delivery to client • continuous integration and delivery during sprint • achieve 100% regression test case execution • release small and release often
10
When do you implement automation?
9/21/16
6
11
The decision to use automation • current code base is growing • addition of new features and enhancements are increasing • completion time of manual test case execution is expanding • releases are occurring frequently
12
What do you automate?
9/21/16
7
13
Candidates for automated tests • repeated tests within sprint development cycles • regression tests that are executed pre-release • sanity tests that are being performed during releases to
production
14
How do you implement automation?
9/21/16
8
15
Planning for automation • choose a valuable platform • involve your current QA team • hire QA developer(s) • keep the process agile
16
Exploratory Testing and its Role
9/21/16
9
17
Where does exploratory testing fit?
18
“Insanity is doing the same thing over and over again and expecting different results” ~Albert Einstein
9/21/16
10
19 James Bach, Exploratory Testing Explained
“An interactive process of simultaneous learning,
test design, and test execution.”
20
Does the QA team need specialized skills?
• experience is an asset but not a requirement
• critical thinking and problem solving • autonomy and independence • creativity
9/21/16
11
21
Can you “explore” if you already know the way?
• YES! • it’s not User Acceptance testing • there are many ways of navigating from Point A to Point B
22
The Mutual Relationship
9/21/16
12
23
“Assumptions are the termites of relationships” ~Henry Winkler
24
Build on the strengths of your team • trust • communication and collaboration • empowerment • confidence
9/21/16
13
25
How We Make it Work The challenges and successes
26
The story • online retail e-commerce solution in production • automation in place for pre-release regression • an established agile process in development and delivery • QA test coverage was still the bottleneck
9/21/16
14
27
Management approval for change • respect and value of the QA team • don’t be afraid to innovate! • plan for smarter test coverage
28
Shifting the mindset of the QA team • information sessions about exploratory testing • review of existing manual regression tests • created user stories for high level scenarios
9/21/16
15
29
Be creative, be organized and be brave! • mind-mapping sessions for the high level scenarios • manual and automation team participation • absence of test steps • traceability by recording the results • organized a trial run for the regression cycle
30
Gain stakeholders support • business wants return on
investment • transparency is key • continually inform • provide metrics
9/21/16
16
31
Sample of metrics for status reports ManualExploratoryTests
DesktopBrowsers OS/Devices
Total
Desktop
Chrome
FF
IE11
MAC
Safari
IOS8Chrom
e
IOS9Safari
IOS8Safari
AndroidCh
rome
#ofTestCases 91 9 15 10 12 14 9 12 10Pass 91 9 15 10 12 14 9 12 10Fail 0 Blocked 0 0 0 0 0 0 0 0 0Unexecuted 0 0 0 0 0 0 0 0 0Executed 91 9 15 10 12 14 9 12 10PercentageExecuHonComplete 100% 100% 100% 100% 100% 100% 100% 100% 100%
32
Continued collaboration • ongoing audits of current and
future release work • review recorded results from
exploratory tests • Repeat, Repeat, Repeat!
9/21/16
17
33
Benefits of testing style • the QA team’s confidence level grows • the project management can stick to release timelines • the business benefits by staying competitive
34
May 2014
• 261 manually executed step-by-step test cases
• No automation in place
• 6 days x 4 QAs
January 2016
• 49 Exploratory Test cases
• 1140 total automated tests covering 4 browsers
• 3 days x 2 QAs with focus on Mobile
vs
Rewards & Results
9/21/16
18
35
What’s next? • automate the mobile web and mobile app regression tests • streamline current exploratory tests • change the format of our sprint functional tests
36
Conclusion
9/21/16
19
Thank you
Q & A