sasqag powerpoint slides
TRANSCRIPT
Software Testers: The Next Generation
Bj Rollison, Test Architect
Engineering Excellence Group
Microsoft, Inc.
Microsoft® Copyright © Bj Rollison, 2005
Overview
Today’s testers State of tester’s knowledge Tester performance
The proof (Lies, damn lies and statistics) Empirical evidence
Why the current way isn’t good enough anymore Influences causing change
Tomorrow’s testers Here's what’s happening at MS Here's what you should/could do personally to help
yourself
Microsoft® Copyright © Bj Rollison, 2005
Tester training (or lack thereof) “Less than 10% of testers have formal training in
test techniques” – Dorothy Graham “…few testers or developers have received any
training in formal methods, especially test techniques.” – Marne Hutcheson (Software Testing Fundamentals)
The average software developer reads less than one professional book per year and subscribes to no professional magazines. - Steve McConnell, DeMarco and Lister, Peopleware, 2d Ed, 1999.
Microsoft® Copyright © Bj Rollison, 2005
Tester’s Body of Knowledge The average tester has read one book on
software testing Anecdotal evidence suggests the majority of testers
have NOT read more than one book on testing 97% read Testing Computer Software (Kaner et. al) 4% read How to Break Software (Whittaker)
Less than 25% have a degree in computer science or other engineering field
More than 75% unable to effectively code using a modern programming language
Microsoft® Copyright © Bj Rollison, 2005
The problem is...
Universe of functionality
&bugs
Actualtestingeffort
Under-testing
Over-testing
Average30%
Microsoft® Copyright © Bj Rollison, 2005
The venerable triangle case study“A program reads three (3) integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program displays a message that states whether the triangle is scalene, isosceles, or equilateral.” – G. Myers
Suggested answersG. Meyers – 65P. Jorgensen – 185R. Binder - 65K. Beck – 6R. Collard – 4
All answers are rightbecause there is noreal context!
Microsoft® Copyright © Bj Rollison, 2005
Establishing the baseline
if (Side A) or (Side B) or (Side C) != int16 (signed 16 bit int) if input > maxint then error message for overflow condition else if input ! = integer value then error message for format exception
if Side A or Side B or Side C <= 0 then error message sides must be > 0
if (Side A + Side B <= Side C) or (Side B + Side C <= Side A) or (Side A + Side C <= Side C) then input does not equate to valid triangle
if (Side A = = Side B) and (Side B = = Side C) then triangle is equilateral
else if (Side A = = Side B) or (Side A = = Side C) or (Side B = = Side C) then triangle is isosceles
else triangle is scalene
*Aside from the obvious conditions above additional tests include 1 for max val for all 3 sides, 1 for min val for all 3 sides, and 3 to test for BO of sum of 2 sides
Microsoft® Copyright © Bj Rollison, 2005
Testing the testers
Generate a set of tests using error guessing and exploratory testing methods that will: adequately* evaluate the triangle algorithm functionality implemented in C# against the specification
(Time limit 15 minutes)
Microsoft® Copyright © Bj Rollison, 2005
Case study demographics
Total number participants = 400 0% ever had formal training in software testing
Avg. years testing experience = 2¼ years 53% – < 1 year experience (20% STE) 12% – 1 - 2 years experience (60% STE) 10% – 2 - 4 years experience (70% STE) 25% – > 4 years experience (100% STE)
Approx. 50% C/C++/C# coding skill
Microsoft® Copyright © Bj Rollison, 2005
Case study synopsis 31% of tests covered baseline functionality
28% of the tests did not add significant value
Redundant coverage, results would not prove or disprove anything not covered by previous tests
14% of the tests missed the primary objective
These tests did not focus on the specific triangle functionality of the program
12% of the tests were incorrect assumptions
Wrong or incorrect results expected
Microsoft® Copyright © Bj Rollison, 2005
Overall Results
0
4
8
12
16
20
24
28
Triangle Effective Redundant Off Target Avg. # Tests
31% of tests metthe primary objective
> 50% of test effort was redundantor not focused on primary objective
Microsoft® Copyright © Bj Rollison, 2005
Results by Experience
0
4
8
12
16
20
24
28
Triangle Effective Redundant Off Target Avg Number
1-12 mon 13 - 24 mon 25 - 48 mon > 49 mon
Microsoft® Copyright © Bj Rollison, 2005
STE Results by Experience
0
4
8
12
16
20
24
28
Triangle Effective Redundant Off Target Avg # Tests
1-12 mon. 13-24 mon. 25-48 mon. > 49 mon.
Microsoft® Copyright © Bj Rollison, 2005
SDET Results by Experience
0
4
8
12
16
20
24
28
Triangle Effective Redundant Off Target Avg # Tests
1-12 mon. 13-24 mon. 25-48 mon.
Microsoft® Copyright © Bj Rollison, 2005
Technical skill comparison
0
2
4
6
8
10
12
14
Triangle Effective Redundant Off Target
STE (Gp. 1) SDET (Gp. 1) STE (Gp. 2)
SDET (Gp. 2) STE (Gp. 3) SDET (Gp. 3)
Microsoft® Copyright © Bj Rollison, 2005
Detailed Analysis
40% of the tests covered only 4 tests for the triangle (equilateral, scalene, isosceles, invalid) STE 56% more likely to execute 4 or less tests 20% STEs (1-12 month) did not test for triangle 24% of STEs did not test for invalid triangle input
Only 54% tested for specific boundary conditions Only 5% tested for overflow conditions
Less than 1% of STEs tested for overflow conditions
Microsoft® Copyright © Bj Rollison, 2005
Skill Comparison
STE 4 times more likely to write an incorrect / invalid test
STE 1.5 times more likely to execute a redundant test
SDET 2 times more likely to exercise specific boundary conditions and overflow errors
SDET tests 2 times more likely to be an effective test
Microsoft® Copyright © Bj Rollison, 2005
Case study summary
Black box testing is less than 35% effective Testers who lack formal training in testing
techniques are most likely to under test boundaries, exception handling routines, and critical functional areas
More than 50% of the testing effort by untrained testers results in redundant or ineffective* tests
Testers without computer/programming knowledge are less capable of improving test effectiveness efficiently
Microsoft® Copyright © Bj Rollison, 2005
So, what’s changing?
Market pressures Increasing demand for higher quality Customer’s more aware of capabilities
Complex solutions Increasing complexity (esp. integration)
Drive quality upstream Unit testing, test driven development, etc “Good Bugs harder to find
Cost cutting Hot-fixes, service packs, shelf-life (10 years) Less testers with greater skills & knowledge
Microsoft® Copyright © Bj Rollison, 2005
Changes at Microsoft
Hiring standards Computer science and other engineering background
Training 40 hours of hands on training for new test engineers Study groups, focus groups, etc. External conferences and seminars
Retention in testing discipline Stop the “brain-drain” Remove the “glass ceiling” for non-management roles Provide greater challenges and scope of influence
Microsoft® Copyright © Bj Rollison, 2005
What should I do? Increase professional knowledge
Formal training, conferences, etc. Books, magazines, industry white papers
Software Testing Techniques 2nd Ed. – Boris Beizer The Art of Software Testing .– Glenford Myers Testing Object Oriented Systems – Robert Binder A Practitioners Guide to Software Test Design – Lee
Copeland
Increase technical knowledge Modern programming language
Automation Domain expertise System expertise
Questions?
Testing is our profession; Quality is our passion!™