claus brabrand, itu, denmark mar 04, 2008black-box testing black-box testing claus brabrand [...
Post on 20-Dec-2015
218 views
TRANSCRIPT
Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Black-Box Testing
Claus Brabrand[ [email protected] ]
( “FÅP”: First-year Project Course, ITU, Denmark )
[ 2 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Reminder: Learning & Exam Goals ”Product”:
”Oral Exam”:
[ 3 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Outline
Introduction: ”Bugs” and ”Black-box testing”
Testing the Specification: [”static black-box testing”]
Testing the Code: [”dynamic black-box testing”]: ”Equivalence partitioning”
Other techniques: Model-checking & Static-analysis
Exercises: Work on your group exercises
[ 4 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Definition: ”bug”Main entry: 2bug
Pronunciation: /’bəg/
Function: noun
Etymology: origin unknown
Date: 16221 a: an insect or other creeping or crawling invertebrate (as a spider or centipede)
b: any of several insects (as the bedbug or cockroach) commonly considered obnoxious
c: any of an order (Hemiptera and especially its suborder Heteroptera) of insects that have sucking mouthparts, forewings thickened at the base, and incomplete metamorphosis and are often economic pests —called also true bug
2: an unexpected defect, fault, flaw, or imperfection <the software was full of bugs>
3 a: a germ or microorganism especially when causing disease b: an unspecified or nonspecific sickness usually presumed due to a bug
4: a sudden enthusiasm
5: enthusiast <a camera bug>
6: a prominent person
7: a crazy person
8: a concealed listening device
9: a weight allowance given apprentice jockeys
[ 5 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Example of a Bug :-)
Hej Claus,
A propos test og datoberegninger:
Fredag morgen mente min PDA kalender (Microsoft Windows CE) at vi skrev lørdag 1. marts. Jeg stillede selvfølgelig tålmodigt uret tilbage til fredag 29. februar 2008 (dette fandt sted på et møde hos Microsoft i Vedbæk ;-) Lørdag formiddag startede den så med at vise min (ret tomme) kalender for august 2049. Det indbyggede ur stod på 1. januar 1601. Ahem.
Peter
[ 6 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Terminology (for ’Bugs’)
”Severe conditions”: Fault Failure Defect
”Unintended operation”: Anomaly Incident Variance Feature (intended)
”Generic terms”: Problem Error Bug ( we’ll use this term)
Typically imply blame; as in:- ”it was his fault”
Famous quote:- ”It’s not a bug, it’s a feature”
[ 7 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
”The Harvard Mark II Bug”
Photo of firstactual ”bug”:
“The first documented computer bug was a moth found trapped between points at Relay # 70, Panel F, of the Mark II Aiken Relay Calculator while it was being tested”
Harvard University, Sep. 9, 1947
[ 8 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Definition: ”Bug”
A ”bug” is a relation between: Specification & Implementation
Whether or not specification is: Explicit or Implicit Written or Oral Formal or Informal
(e.g., ”product spec” vs. ”back-of-envellope”)
implementation(aka., ’program’)
specification(aka., ’spec’)
~bug
[ 9 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Specification
A spec states things an impl…: Should do! Shouldn’t do! Unspecified? (’unclear’, ’unmentioned’, or ’left open’)
Should do!
Shouldn’t do!
Unspecified?
S
Specs are often intentionally under-specified. It’s often better to not ”prematurely
commit” to a particular solution (by specifying exactly how a task should be done) –
and instead just state which overall tasks the system should do.
[ 10 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Formal Definition: ”Bug”
A (software) bug occurs when: 1. Impl doesn’t do sth spec says it should:
2. Impl does sth spec says it shouldn’t:
3. Impl does sth spec doesn’t mention:
4. Spec doesn’t mention sth, but should:
5. Impl is hard to understand/use by user(s):
~
~?!
(or does it incorrectly)IS
I S
S I
?!I
S
[cf. ”Software Testing”, R.Patton, p.15]
!~ideal
[ 11 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Consequences…
Consequently, …: ”Additional functionality”?
e.g., a calculator which also does square root (but wasn’t supposed to)
”Easter egg”? e.g., hit [Alt+Shift+2] in Solitaire to win game
…is a bug !
…is a bug !
[ 12 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Question
Question:
A) R.Patton interprets as (”spec := impl”):
trivially no bugs in impl ! (according to [own] definition #1,#2,#3)
B) More sensible to interpret as ”no spec”: entire impl is essentially one big bug !
(…according to definition #3+#4 [p.15])
”Because the software is already complete, you have the perfect specification
– the product itself” [R.Patton, ”Software Testing”, p.31]
”an impl w/o a spec” ?
bug~
[ 13 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Often due to: complexity tight schedule under-specification under-documentation
Number one Cause of Bugs…[cf. ”Software Testing”, R.Patton, p.17]
(NB: no percentages given)
spec
…is the specification:
design
impl
oth
er
[ 14 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Cost of (Fixing) Bugs
Cost of bugs increases exponentially (over time):
log
spec design code test release
$1$100
$10,000
$1,000,000
$10,000,000
[cf. ”Software Testing”, R.Patton, p.18]
[ 15 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Tester’s Goal
The goal of a software tester is:
Testers (therefore) often aren’t the most popular members of a project team
- 1) “to find bugs”; - 2) “find them as early as possible”; and - 3) “make sure they get fixed”
[R.Patton, “Software Testing”, p. 19]
[cf. ”Software Testing”, R.Patton, p.19]
[ 16 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
”The Pesticide Paradox”
”The pesticide paradox”: ”The more you test a software,
the more immune it becomes to your tests”
B.Beizer, “Software Testing Techniques”, 1990
[ 17 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Bugs
Bugs often occur in groups If you find one, chances are there are more nearby
Not all bugs will be fixed Too costly? Too risky? Not ”cost-effective”? Bug unknown :-)
[ 18 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Programming vs. Testing
Constructive thinking: ”Test-to-pass”
Often not a good idea to ”test” your own code :-(
Destructive thinking: ”Test-to-fail”
(aka., ”error-forcing”)
Often better to test/break someone else’s code :-)
Advice:
”test-to-pass” before ”test-to-fail”
(i.e., test ”normal usage” first)
[ 19 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Black-Box Testing
The goal of black-box testing is: Make sure impl solves problem it is supposed to:
i.e.,
Point of departure: spec, not impl
not a particular program which ”claims” to solve problem testing w/o insights of code
impl ~ spec
Static(test spec):
Dynamic(test impl):
output
input
bug~
[ 20 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Outline
Introduction: ”Bugs” and ”Black-box testing”
Testing the Specification: [”static black-box testing”]
Testing the Code: [”dynamic black-box testing”]: ”Equivalence partitioning”
Other techniques: Model-checking & Static-analysis
Exercises: Work with your group exercises
[ 21 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Spec ”Warning Words” (I/III)
Unconditionals (universally): ’Always’, ’for every’, ’for each’, ’for all’, …
Try to violate (i.e., find exemptions from rule)!
Unconditionals (never): ’None’, ’never’, ’not under any circumstances’, …
Try to violate (i.e., find exemptions from rule)!
Tautologies: ’Trivially’, ’certainly’, ’clearly’, ’obviously’, ’evidently’, …
Check assumptions (that nothing’s swept under the rug)!
[cf. ”Software Testing”, R.Patton, p.61]
[ 22 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Spec ”Warning Words” (II/III)
Unspecified conditionals: ’Some(-times)’, ’often’, ’usually’, ’ordinarilly’, ’mostly’, …
Unclear spec (under which circumstances)?
Continuations: ’Etcetera’, ’and so forth’, ’and so on’, …
Check that spec is comprehensively unambiguous?
Examples: ’E.g.’, ’for example’, ’such as’, …
Is example representative (what about other examples)?
[cf. ”Software Testing”, R.Patton, p.61]
[ 23 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Spec ”Warning Words” (III/III)
Positive adjectives: ’Good’, ’fast’, ’efficient’, ’small’, ’reliable’, ’stable’, …
Subjective (needs objectification if to be used for testing)!
Alegedly completed: ’Handled’, ’processed’, ’taken care of’, ’eliminated’, …
Is something hidden?
Incompleted: ’Skipped’, ’unnecessary’, ’superfluous’, ’rejected’, …
Is something forgotten?
[cf. ”Software Testing”, R.Patton, p.61]
[ 24 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Spec ”Warning Words” (if-then)
Finally, watch out for: ”If … Then” (with missing ”Else”):
Check what happens in the ”Else-case”
[cf. ”Software Testing”, R.Patton, p.61]
IF … THEN … ELSE … ?!
[ 25 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Question: which is ”best” (…and why)?
A) white-box testing ; black-box testing ; (i.e., white-box testing first)
B) black-box testing ; white-box testing ; (i.e., black-box testing first)
Answer: ’B’ Settle overall impl ~ spec problems first Before zooming in on the impl [as in white-box testing]
Test Sequencing
…or…
~
[ 26 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Outline
Introduction: ”Bugs” and ”Black-box testing”
Testing the Specification: [”static black-box testing”]
Testing the Code: [”dynamic black-box testing”]: ”Equivalence partitioning”
Other techniques: Model-checking & Static-analysis
Exercises: Work on your group exercises
[ 27 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Test ”Boundaries”
Programs are vulnerable ”around the edges”: e.g. testing legal inputs (time, in hours):
e.g. testing legal inputs (dates, in April):
PropertyMinimum-1MinimumMiddle repr.MaximumMaximum+1
Input-1 011 (e.g.)2324
Expected outputinvalidvalidvalidvalidinvalid
Actual output
PropertyMinimum-1MinimumMiddle repr.MaximumMaximum+1
Input00/401/412/4 (e.g.)30/431/4
Expected outputinvalidvalidvalidvalidinvalid
Actual output
[ 28 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Test ”Powers-of-Two”
Programs vulnerable ”around powers-of-two”: e.g. years of age (assume held in a byte):
e.g. #game-spectators (assume held in a 16-bit word):
PropertyMinimum-1MinimumMiddle repr.MaximumMaximum+1
Input-1 027 (e.g.)255256
Expected outputinvalidvalidvalidvalidinvalid
Actual output
PropertyMinimum-1MinimumMiddle repr.MaximumMaximum+1
Input-1 012345 (e.g.)6553565536
Expected outputinvalidvalidvalidvalidinvalid
Actual output
[ 29 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Test ”Empty Input”
Default / empty / blank / null / zero / none / : e.g., ’any program’:
PropertyNo input
Input
Expected outputError message
Actual output
[ 30 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Test ”Invalid Input”
Invalid / illegal / wrong / garbage / bogus data: e.g., calculator:
PropertyInvalid inputBogus data!!!
Input+*31#$+~´?=!
Expected outputError messageError message
Actual output
Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Equivalence Partitioning
Partitions and
Equivalence Relations
[ 32 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Testing: Infinite process
Recall: ”testing is an incomplete process” (i.e., ”testing can’t prove absence of bugs”)
There are infinitely many possible inputs: (hence, testing will take an infinite amount of time)
sum(x,y)
(0,0)
0
sum(x,y)
(1,0)
1
sum(x,y)
(0,1)
1
sum(x,y)
(-1,0)
-1
sum(x,y)
(0,-1)
-1
sum(x,y)
(1,1)
2
sum(x,y)
(-1,-1)
-2
…
Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Crash course on Relations
Relations
Equivalence Relations
[ 34 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Relations
Example1: “even” relation: Written as: as a short-hand for: … and as: as a short-hand for:
Example2: “equals” relation: Written as: as a short-hand for: … and as: as a short-hand for:
Example3: “dist-btwn” relation: Written as: as short-hand for:
|_even Z
|_even 4
|_even 5
4 |_even
5 |_even
2 3 (2,3) ‘=’
‘=’ Z Z
(2,2) ‘=’2 = 2
‘’ CITY Z CITY
Aarhus Copenhagen310
(Aarhus, 310, Copenhagen) ‘’
[ 35 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Equivalence Relation
Let ‘~’ be a binary relation over set A: ‘~’ A A
~ is an equivalence relation iff: Reflexive:
Symmetric:
Transitive:
xA: x ~ x
x,yA: x ~ y y ~ x
x,y,zA: x ~ y y ~ z x ~ z
[ 36 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Exercise: Eq.Rel.
Which relations are equivalence relations?: …and which are not (and why not)?:
a) The "less-than-or-equal-to" relation: ' ' { (n,m) | n,m Z, n m }
b) The ”almost-total-relation-on-integers”, (relating all numbers except 42, but relating 42 with 42):
{ (n,m) | n,m ( Z \ {42} ) } { (42,42) } c) The "is-congruent-modulo-three" relation:
{ (n,m) | n,m Z, (n % 3) = (m % 3) } d) The "have-talked-together" relation:
{ (p,q) | n,m PEOPLE, p and q have talked together } e) The "is-in-the-same-group-as" relation:
{ (p,q) | n,m PEOPLE, p and q are in same FÅP group }
[ 37 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Eq. Rel. Partition
Equivalence relation ’~’ = { (A,A), (B,B), (A,B), (B,A),
(P,P), (X,X), (Y,Y), (Z,Z), (X,Y), (Y,X), (X,Z), (Z,X), (Y,Z), (Z,Y) }
E.g.: A ~ B, P ~ P, X ~ X, X ~ Z A ~ P, B ~ Y, P ~ Z
Partition ’~’:
A
B
P
X
Y
Z
”Canonical representatives”: [A] = [B] = { A, B } [P] = { P } [X] = [Y] = [Z] = { X, Y, Z }
Capture the same information;
i.e., notion of ”equivalence”
[ 38 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Testing: Infinite process
Recall: ”testing is an incomplete process” (i.e., ”testing can’t prove absence of bugs”)
There are infinitely many possible inputs: (hence, testing will take an infinite amount of time)
sum(x,y)
(0,0)
0
sum(x,y)
(1,0)
1
sum(x,y)
(0,1)
1
sum(x,y)
(-1,0)
-1
sum(x,y)
(0,-1)
-1
sum(x,y)
(1,1)
2
sum(x,y)
(-1,-1)
-2
…
[ 39 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Equivalence Partitioning
Partition input:
Finitary partition: If finite # categories (aka. ”equivalence classes”)
Here 3: { ”zero”, ”pos”, ”neg” }
We can now test all equivalence classes Using representative elements from each category
neg pos
0
zero
1 2 3 …… -3 -2 -1
[ 40 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Test Sum (cont’d)
We can now test all equivalence classes Using representative input from each category
Sum (testing all equivalence classes):PropertyPos, PosNeg, PosZero,PosPos, NegNeg, NegZero,NegPos, ZeroNeg, ZeroZero,Zero
Input(1,2)(-3,4)(0,5)(6,-7)(-8,-9)(0,-10)(11,0)(-12,0)(0,0)
Expected output315-1-17-1011-120
Actual output
[ 41 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Frequent Partitions for Testing
Numbers: Positive, Negative, Zero Zero, One, Two, Many (aka. ”Greenlandic Numbers”)
Lists: Length-0, Length-1, Length-2, Length-3+ Ascending-elements, Descending-elements, AscDesc-elements
…Advice: - consider how problem might be solved
- partition into qualitatively different
categories such that:
(likely: ”same category same error”)
[ 42 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Outline
Introduction: ”Bugs” and ”Black-box testing”
Testing the Specification: [”static black-box testing”]
Testing the Code: [”dynamic black-box testing”]: ”Equivalence partitioning”
Other techniques: Model-checking & Static-analysis
Exercises: Work on your group exercises
Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Model-checking
Modelling
Verification
Model-checking
[ 44 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Program worldModel world
ConcreteAbstract
P M
Program vs. Model World
abstraction
concretization
(Java)(modeling lang)
[ 45 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Methodology: Model-based Design Design abstract model
Decompose model
Reason/Test/Verify model individual parts and whole
Recompose insights make model safe
Impl. concrete program
concretize
REAL PROBLEM
SAFE MODEL
SAFE PROGRAM
abstract
? ?
?
?
MODEL
testreason
verify
[ 46 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Program worldModel world
ConcreteAbstract
P M
1. P ? 3. M ?
4. verify!!
5. M !7. P !
2. abstract
6. concretize
Model-checking
(Java)(modeling lang)
If models are “finite”, we can have
a computer test all possibilities.
“Never send a human to do a machine’s job”-- A.Smith (’99)
Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Static Analysis
Static Analysis
Undecidability
[ 48 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Undecidability
Most interesting properties are undecidable: e.g., can program P have a type error (when run):
Compilers use approximations (computed via ”static analyses”):
correct incorrect
?
correct incorrect
safe (over-)approximation
no program can ”decide” this line automatically (in all cases)
[ 49 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Undecidability (self-referentiality)
Consider "The Book-of-all-Books": This book contains the titles of all books
that do not have a self-reference (i.e. don't contain their title inside)
Finitely many books; i.e.: We can sit down & figure out whether to include or not...
Q: What about "The Book-of-all-Books"; Should it be included or not?
"Self-referential paradox" (many guises): e.g. "This sentence is false"
"The Bible"
"War and Peace"
"Programming Languages,
An Interp.-Based Approach"
...
The Book-of-all-Books
[ 50 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Termination Undecidable!
Assume termination is decidable (in Java); i.e. some program, halts: PROG BOOL
Q: Does P0 loop or terminate...? :)
Hence: "Termination is undecidable" ...for Java, C, C++, Pascal, -Calculus, Haskell, ...
bool halts(Program p) { ... }
Program p0 = read_program("P0.java");if (halts(p0) == “halts”) loop();else halt();
-- P0.java --
bool halts(Program p) { ... }
[ 51 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Example: Type Checking
Will this program have type error (when run)?
Type error <EXP> evaluates to true
void f() { var b; if (<EXP>) { b = 42; } else { b = true; } /* some code */ if (b) ...;}
Undecidable
(in all cases)
i.e., what <EXP>
evaluates to (when run)
[ 52 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Example: Type Checking (cont’d)
Hence, languages use static requirements:
All variables must be declared And have constant type (throughout the program)
void f() { bool b;
/* some code */ if (<EXP>) { b = 42; } else { b = true; } /* some more code */}
Static compiler error:
Regardless of what <EXP>
evaluates to when run
[ 53 ]Claus Brabrand, ITU, Denmark Mar 04, 2008BLACK-BOX TESTING
Outline
Introduction: ”Bugs” and ”Black-box testing”
Testing the Specification: [”static black-box testing”]
Testing the Code: [”dynamic black-box testing”]: ”Equivalence partitioning”
Other techniques: Model-checking & Static-analysis
Exercises: Work on your group exercises