claus brabrand, itu, denmark feb 03, 2009testing testing claus brabrand [ [email protected] ] (...
Post on 21-Dec-2015
219 views
TRANSCRIPT
Claus Brabrand, ITU, Denmark Feb 03, 2009TESTING
Testing
Claus Brabrand[ [email protected] ]
( “FÅP” First-year Project Course, ITU, Denmark )
[ 2 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Outline
Motivation Why bother with testing?
The Psychology of Testing What is the goal of testing?
The Testing Process What is relation to other programming-related tasks?
Bugs Important properties of bugs
Implementation vs. Specification: Formal definition of a bug
[ 3 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Learning & Exam Goals
”Product”:
”Oral Exam”:
[ 4 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Software Errors
Therac-25 Radiation Therapy ’85-’87
Massive overdoses (6 deaths / amputations)!
Patriot Missile Guidance System ’91 (Gulf War 1.0)
Accumulating rounding errors deaths
Ariane V ’96 (one of the most expensive bugs, ever)
Conversion from 64-bit float to 16-bit signed int
[ 5 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
zzzz
Software Errors (cont’d)
Train Control System ’98 (Berlin)
Train cancellations
Mars Pathfinder July ’97
Periodic resets
Win95/98 w/ 3rd-Party Device Drivers late ’90es
Dysfunction (“blue screen of death”)!
...on mars!
[ 6 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Software Errors (cont’d)
Mobile Phones ’00-…
Freeze and odd behaviors (really annoying)!
Cruise Control System Model ’86 (Grady Booch)
Accellerated after car ignition car crashes
Baggage Handling System ’94-’95 (at Denver Int’l Airport)
$ 360,000,000 USD
[ 7 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
…and what about?!
Surgical Laser Control System
Oops…!
Air Plane Control System
Dysfunction (plane crash)!
Nuclear Powerplant Control System
Core melt-down (“China-syndrome”)!
[ 8 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Outline
Motivation Why bother with testing?
The Psychology of Testing What is the goal of testing?
The Testing Process What is relation to other programming-related tasks?
Bugs Important properties of bugs
Implementation vs. Specification: Formal definition of a bug
[ 9 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Testing Incomplete Process
A program has: many possible valid inputs many possible invalid inputs
Hence:
88
Testing can never prove absence of errors!
Testing is an incomplete process!
[ 10 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
The Psychology of Testing
X
Goal: test a program to show that it works …we might as well stop before we even start! :-(
Homo Sapiens are ”goal oriented” (steer towards goal)! :-)
”Testing is the process of demonstrating that errors are not present.”
”The purpose of testing is to show that a program performs its intended functions correctly.”
[cf. ”The Art of Software Testing” (Chap. 2), Glenford J. Myers, 1979]
”Testing is the process of establishing confidence that a program does what it is supposed to do.”
[ 11 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
successfultest case
unsuccessfultest case
Problematic terminology:
successfultest case
unsuccessfultest case
Much better terminology:
Psychology of Testing (cont’d)
Goal: find as many errors as possible Note: realistically assumes errors are present
Constructive goal (actually destructive)
”Testing is the process of executing a program with the intent of finding errors.”
vs
[cf. ”The Art of Software Testing” (Chap. 2), Glenford J. Myers, 1979]
[ 12 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Constructive vs. DestructiveThinking Constructive thinking:
(e.g., programming) ”Test-to-pass”
Often not a good idea to ”test” your own code :-(
Destructive thinking:(e.g., testing) ”Test-to-fail”
Often better to test/break someone else’s code :-)
FAAP Recommendation: Have another group help you test your FAAP group ”product”
[ 13 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Tester’s Goal
The goal of a software tester is:
Testers (therefore) often aren’t the most popular members of a project team
- 1) “to find bugs”; - 2) “find them as early as possible”; and - 3) “make sure they get fixed”
[R.Patton, “Software Testing”, p. 19]
[cf. ”Software Testing”, R.Patton, p.19]
[ 14 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Outline
Motivation Why bother with testing?
The Psychology of Testing What is the goal of testing?
The Testing Process What is relation to other programming-related tasks?
Bugs Important properties of bugs
Implementation vs. Specification: Formal definition of a bug
[ 15 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
ISO9126
Maintainability
ReliabilityUsability
Portability
How robust is the SW wrt.external network failures, ’^C’, ...?
How easy is the SW to understand?…and use?
How easy is it to transfer and adapt SW to new environment / platform?How easy is it to modify the SW?
And fix errors?
Software QualityInternational evaluation standard
Functionality Efficiency
ISO 9126
Does the SW do what it’s supposed to?
Does it work as intended?
How much time/memory/space/- bandwidth/… does the SW consume?
[ 16 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Testing vs. Debugging?
Testing vs. Debugging?:
Functionality: Efficiency:
Quality Assurance:(Functionality)
Testing(Performance)
Testing
Diagnosis: Profiling
Purpose:
Regarding:
Debugging
[ 17 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Testing vs. Debugging (cont’d)
Evaluate test resultsFix problem(reprogram)
FunctionalityTesting
Quality assurance:
Perform (functionality) test
Debugging
Diagnosis:
Determine problem?
01101021Program:
01101011
(greater confidence!)
SYSTEMATIC
Document test results
(confidence?!?)
Re-
01101011
[ 18 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Performance Testing vs. Profiling
Evaluate test resultsImprove program(reprogram)
Performance Testing
Quality assurance:
Perform (efficiency) test
Profiling
Diagnosis:
Determine problem?
01101021Program:
01101011
(greater confidence!)Document test results
(confidence?!?)
Re-
SYSTEMATIC
01101011
[ 19 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
White-box vs. Black-box Test
White-box Testing: (aka., ”structural testing”) (aka., ”internal testing”)
Test focus: source code
Black-box Testing: (aka., ”behavioral testing”) (aka., ”external testing”) (aka., ”input-ouput testing”)
Test focus: specification (or intention)
Complementary Approaches!!!
n = in();
n = n/2;
odd(n)
n = 3*n+1;
out(n);
tt ff ~?program spec
[ 20 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Exercise: which is (usually) ”best” - and why?
A) white-box testing ; black-box testing ; (i.e., white-box testing first)
B) black-box testing ; white-box testing ; (i.e., black-box testing first)
Answer: ’B’ Settle overall impl ~ spec problems first Before zooming in on details of impl
Exercise
…or…
[ 21 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
”Software Testing” (R. Patton)
Background reading: ”Software Testing”, Ron Patton, Sams Publishing, 2006 Part II (pp. 53 – 123); ”Testing Fundamentals”:
”Examining the Code”(chapter 6)
”Examiningthe Spec.”(chapter 4)
”Testing w/Blinders On”
(chapter 5)
”Testing w/X-ray Glasses”
(chapter 7)
Dynamic(running program)
Static(w/o running prg)
-testing -testingTime:
Type:
[ 22 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Outline
Motivation Why bother with testing?
The Psychology of Testing What is the goal of testing?
The Testing Process What is relation to other programming-related tasks?
Bugs Important properties of bugs
Implementation vs. Specification: Formal definition of a bug
[ 23 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Definition: ”bug”Main entry: 2bug
Pronunciation: /’bəg/
Function: noun
Etymology: origin unknown
Date: 16221 a: an insect or other creeping or crawling invertebrate (as a spider or centipede)
b: any of several insects (as the bedbug or cockroach) commonly considered obnoxious
c: any of an order (Hemiptera and especially its suborder Heteroptera) of insects that have sucking mouthparts, forewings thickened at the base, and incomplete metamorphosis and are often economic pests —called also true bug
2: an unexpected defect, fault, flaw, or imperfection <e.g., “the software was full of bugs”>
3 a: a germ or microorganism especially when causing disease b: an unspecified or nonspecific sickness usually presumed due to a bug
4: a sudden enthusiasm
5: enthusiast <a camera bug>
6: a prominent person
7: a crazy person
8: a concealed listening device
9: a weight allowance given apprentice jockeys
[ 24 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
”The Harvard Mark II Bug”
Photo of firstactual ”bug”:
“The first documented computer bug was a moth found trapped between points at Relay # 70, Panel F, of the Mark II Aiken Relay Calculator while it was being tested”
Harvard University, Sep. 9, 1947
[ 25 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Example of a Bug :-)
Hej Claus,
A propos test og datoberegninger:
Fredag morgen mente min PDA kalender (Microsoft Windows CE) at vi skrev lørdag 1. marts. Jeg stillede selvfølgelig tålmodigt uret tilbage til fredag 29. februar 2008 (dette fandt sted på et møde hos Microsoft i Vedbæk ;-) Lørdag formiddag startede den så med at vise min (ret tomme) kalender for august 2049. Det indbyggede ur stod på 1. januar 1601. Ahem.
Peter
[ 26 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Terminology (for ’Bugs’)
”Severe conditions”: Fault Failure Defect
”Unintended operation”: Anomaly Incident Variance Feature (sounds intended)
”Generic terms”: Problem Error Bug
Typically imply blame; as in:- ”it was his fault”
Famous quote:- ”It’s not a bug, it’s a feature”
we’ll use these terms
[ 27 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Bugs
Bugs often occur in groups: If you find one, chances are there are more nearby This property useful when testing program parts:
i.e., figuring out where to concentrate testing efforts
Not all bugs will be fixed: Too costly? Too risky? Not ”cost-effective”? Cause unknown?
[ 28 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Cost of (Fixing) Bugs
Cost of bugs increases exponentially (over time):
(log)
spec design code test release
$1$100
$10,000
$1,000,000
$100,000,000
[cf. ”Software Testing”, R.Patton, p.18]
cost
phase
[ 29 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
”The Pesticide Paradox”
”The pesticide paradox”: ”The more you test a software,
the more immune it becomes to your tests”
B.Beizer, “Software Testing Techniques”, 1990
[ 30 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Bug Kinds (diff. kinds of errs)
Syntactic errors: Mal-formed program:
Semantic errors: Symbol errors Type errors Other semantic errors:
(e.g. uninitialized vars)
Logical errors: Compiler: ”no errors”
int square(int x) { return x*x} *** syntax error at line 2
’;’ expected
int square(int x) { return n*n;} *** symbol error at line 2
undefined variable ”n”
int square(float x) { return x*x;}*** type error at line 2 function returns float, not int
int square(int x) { return x+x;} no errors found!!!
[ 31 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Outline
Motivation Why bother with testing?
The Psychology of Testing What is the goal of testing?
The Testing Process What is relation to other programming-related tasks?
Bugs Important properties of bugs
Implementation vs. Specification: Formal definition of a bug
[ 32 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Definition: ”Bug”
A ”bug” is a relation between: Specification & Implementation
Whether or not specification is: Explicit or Implicit Written or Oral Formal or Informal
(e.g., ”product spec” vs. ”back-of-envellope”)
implementation(aka., ’program’)
specification(aka., ’spec’)
~bug
[ 33 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Specification
A spec states things an impl…: Should do! Shouldn’t do! Unspecified? (’unclear’, ’unmentioned’, or ’left open’)
Should do!
Shouldn’t do!
Unspecified?
S
Specs are often intentionally under-specified. It’s often better to not ”prematurely
commit” to a particular solution (by specifying exactly how a task should be done) –
and instead just state which overall tasks the system should do.
[ 34 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Formal Definition: ”Bug”
A (software) bug occurs when: 1. Impl doesn’t do sth spec says it should:
2. Impl does sth spec says it shouldn’t:
3. Impl does sth spec doesn’t mention:
4. Spec doesn’t mention sth, but should:
5. Impl is hard to understand/use by user(s):
~
~?!
(or does it incorrectly)IS
I S
S I
?!I
S
[cf. ”Software Testing”, R.Patton, p.15]
!~ideal
[ 35 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Consequences…
Consequently, …: ”Additional functionality”?
e.g., a calculator which also does square root (but wasn’t supposed to)
”Easter egg”? e.g., hit [Alt+Shift+2] in Solitaire to win game
[ 36 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Exercise
Exercise:
A) R.Patton interprets as (”spec := impl”):
trivially no bugs in impl ! (according to [own] definition #1,#2,#3)
B) More sensible to interpret as ”no spec”: entire impl is essentially one big bug !
(…according to definition #3+#4)
”Because the software is already complete, you have the perfect specification
– the product itself” [R.Patton, ”Software Testing”, p.31]
”an impl w/o a spec” ?
bug~
Claus Brabrand, ITU, Denmark Feb 03, 2009TESTING
Exercise on Testing
Intro to test cases
[ 38 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Testing…:
Testing is easy: (e.g., ”random experimentation”)
Testing well is not easy; requires: A SYSTEMATIC approach Test case…:
production evaluation documentation
[ 39 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Representative?
Comprehensive?
Quality?
Quantity?
…?
Appropriate Test Cases?
[ 40 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Myers’ 10 Testing Principles
1) expected output is part of a test case 2) avoid testing own program 3) avoid testing own (organization’s) program 4) thoroughly inspect result of each test case 5) also write ”invalid / unexpected” test cases 6) also check: doesn’t do what not supposed to 7) avoid throw-away test cases 8) never assume no errors when making test cases 9) Prob(more errors) ~ Prob(#errors already found) 10) Testing is creative+intellectually challenging task
[cf. ”The Art of Software Testing” (Chap. 2), Glenford J. Myers, 1979]
a) destructive vs. constructive modeb) can be overall misunderstandings
also wrt. program parts
instead: systematic”regression testing”!
a) …b) …
[ 41 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Exercise: Triangle Test Test Equilateral triangle (T-3):
All three sides have equal length Isosceles triangle (T-2):
Two sides have equal length Scalene triangle (T-1):
All sides have different length
Q: Which test cases should we use? E.g., (1,1,1), (2,2,2), (3,3,3), (4,4,4), (5,5,5), …?
enum Triangle { EQUILATERAL, ISOSCELES, SCALENE }Triangle isTriangle(int a, int b, int c);
The program reads three integer values from an input dialog. (The three values represent the lengths of the sides of a triangle.) The program displays a message that states whether the triangle is: - equilateral, isosceles, or scalene.
ligebenet
ligesidet
atypisk trekant(?)
Claus Brabrand, ITU, Denmark Feb 03, 2009TESTING
Groups
FAAP’09 Project Groups
[ 43 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
Considerations (Group Formation)
Considerations; e.g.: a) Balance Introverts and Extroverts:
Too many intros few discussions / limited exchange Too many extros Blah blah blah…
b) Avoid ”strategist-strategist” clashes: More likely to ”clash”
c) Balance heterogeneously wrt. ”MBTI” Many different inputs
d) Avoid ”old group mates” in same group Learn to work with ”new people” Avoid sub-groups (aka, cliques)
[ 44 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
FAAP’09 Groups
ISTJ( Inspector )
ISTP( Crafter )
ESTP( Promoter )
ESTJ( Supervisor )
ISFJ( Protector )
ISFP( Composer )
ESFP( Performer )
ESFJ( Provider )
INFJ( Counselor )
INFP( Healer )
ENFP( Champion )
ENFJ( Teacher )
INTJ( Mastermind )
INTP( Architect )
ENTP( Inventor )
ENTJ( Field Marshal )
S N
T F T
I
E
P
J
J
[ 45 ]Claus Brabrand, ITU, Denmark TESTING Feb 03, 2009
White: Anna Charlotte Jesper Kasper Peter P.M
Yellow: Frederik J. Mik Mikkel Mohammad Peter A.N.
Orange: Janne Mads T.P. Oliver Sebastian Thuy
Red: David Frederik W. Jakob Mads H.J. Pelle
Green: Bo Dorte Erla Ida Kristian K.M
Blue: Asger Lasse Niels Søren Torben
Brown: Christian T. Joakim Johannes Nikolaj D.L. Simon
Black: Jon Mads B.A. Nikolaj S. Nikolas Rene
Groups (FAAP’09)