starwest - code coverage is for testers too - tutorial
DESCRIPTION
Projects often set a goal of 80%, 90%, or even 100% code coverage. So, what is code coverage anyway, and why can it mean many different things? How can testers help the team reach its coverage goals? And why should we care about code coverage anyway? From a tester’s perspective, Julie Gardiner explores the variety of code coverage metrics, discusses the advantages and disadvantages of each, and gives examples of how these metrics are used in real projects. Through demonstrations and exercises, learn about a plethora of coverage metrics—statement, branch, decision, branch condition combination, and more. Julie describes the automated tools teams need to calculate and evaluate code coverage. Leave with the confidence to ask about your project’s current coverage levels, the knowledge to set realistic coverage goals, and the ability to guide testers and developers to improve your project’s code coverage.TRANSCRIPT
![Page 1: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/1.jpg)
Jonathon Wright Email: [email protected] Twitter: @Jonathon_Wright Julie Gardiner Email: [email protected] Twitter: @CheekyTester
![Page 2: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/2.jpg)
White Box e.g.
• Statement
• Decision
• Branch
Condition
Combination
• etc.
Black Box e.g.
• Classification Trees
• Pairwise
• Boundary Value
• etc.
• Heuristics
• Exploratory
• Defect Based
• Fault Attacks
![Page 3: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/3.jpg)
20 “things”
tests
our tests have covered 10/20 “things” = 50% “thing” coverage
“things” can be: lines of code = statements
decision outcomes =
decisions
branches
modules
menu options
functions
…
component
level
integration level
system level
![Page 4: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/4.jpg)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Declare variables. integer Counter, Quantity float Amount, Price # Program begins here. MAIN Amount = 0.0 Read Quantity # Determine total cost. FOR Counter = 1 to Quantity Read Price Amount = Amount + Price ENDFOR Print “Total Cost is “ & Amount ENDMAIN
1
2
3
4
5
6
7
Declaration section
(declares variables)
Executable section
(comprises executable
statements)
Comments
Labels (optional)
1 Line numbers
1 Statement numbers
![Page 5: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/5.jpg)
![Page 6: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/6.jpg)
![Page 7: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/7.jpg)
Example control flow chart
Make a list of items to sell For each item in the list Calculate cost of postage Write a description of this item If a photograph is needed Photograph this item Transfer photograph to PC Endif Submit details of item to eBay Endfor
End
end of IF
sequential
statements
single statement
decision
statement (FOR)
sequential
statement
decision
statement (IF)
sequential
statements
end of FOR
end of program
![Page 8: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/8.jpg)
executable statements
![Page 9: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/9.jpg)
− a measurement technique
− percentage of executable statements exercised by a test suite number of statements exercised
total number of statements
− example: − program has 10 statements
− tests exercise 8 statements
statement coverage =
=
Statement coverage
is normally measured
by a software tool
8/10 = 80%
1. statement
2. statement
3. IF….
4. then 6. else
5. … 7. …
8. ENDIF
9. statement
10. statement
test
![Page 10: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/10.jpg)
5
2
1
3
4
Example of statement coverage
read(n)
IF n > 6 THEN
n = n * 2
ENDIF
print n
1
2
3
4
5
Statement
numbers
5
2
1
3
4
True
False
Test Path Statement
Case Taken Coverage
A 1, 2, 3, 4, 5 100%
1
2
3
4
5
Test
Case Input Expected
Output
A 7 14
![Page 11: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/11.jpg)
True
False IF
![Page 12: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/12.jpg)
Decision coverage
is normally measured
by a software tool
½ = 50%
= 1. statement
2. statement
3. IF….
4. then 6. else
5. … 7. …
8. ENDIF
9. statement
10. statement
test
![Page 13: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/13.jpg)
B 1, 2, 4, 5 False 50% 80%
Example of decision coverage
read(n)
IF n > 6 THEN
n = n * 2
ENDIF
print n
1
2
3
4
5
Statement
numbers
B 3 3
5
1
3
4
True
False
Test
Case Input
Expected
Output
A 7 14
Test Path Decision Decision Statement
Case Taken Outcome Coverage Coverage
A 1, 2, 3, 4, 5 True 50% 100%
True
2 False
![Page 14: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/14.jpg)
5
1
3
4
True
False 2
False
1
2
3
4
5
![Page 15: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/15.jpg)
Read A
IF A > 0 THEN
Print “A positive”
ENDIF
Print “A positive”
FALSE
ENDIF
TRUE A>0
Read Read A
IF A > 0 THEN
ENDIF
2
1
![Page 16: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/16.jpg)
IF Age < 18 THEN
Print “No alcohol”
ELSE
Print “Have an alcoholic drink”
ENDIF
Print “Have an alcoholic drink”
FALSE
no alcohol
ENDIF
TRUE A<18
IF Age < 18 THEN
ENDIF
Print “No alcohol”
ELSE
alcohol
2
2
![Page 17: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/17.jpg)
Read stock level for item
IF Item in stock THEN
Get from stores
ELSE
Order from supplier
Print “Item back-ordered”
ENDIF
Print “Item processed”
Print “Item processed”
FALSE
Get
ENDIF
TRUE In?
Read stock level for item
ENDIF
Get from stores
ELSE
Order from supplier
IF Item in stock THEN
Read
Print “Item back-ordered”
Order
Draw the
diagram
in your
notes
2
2
![Page 18: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/18.jpg)
TRUE
Get
ENDWHILE
Check
Read order line
WHILE more items ordered DO
Check availability
Get from stores
ENDWHILE
Print “Finished processing”
Print “Finished processing”
1 1
Read order line
WHILE more items ordered DO
Check availability
Get from stores
ENDWHILE
FALSE
Read
?
![Page 19: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/19.jpg)
Read Questions
Result = 0
Right = 0
WHILE more Questions
IF Answer = Correct THEN
Right = Right + 1
ENDIF
END WHILE
Result = (Right / Questions)
IF Result > 60% THEN
Print "pass"
ELSE
Print "fail”
ENDIF
Pseudo-code:
2 2 end if
end if
do
if r=r+1
Init
if
result
pass
fail
end
while
![Page 20: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/20.jpg)
![Page 21: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/21.jpg)
Control flow graph structures
Graph notation as used in Bill Hetzel’s “The Complete Guide to Software Testing”
Node:
a single statement or
sequence of sequential
statements (that are not
decisions or branches).
Branch or Edge :
a logical link between
two nodes, indicates the
next statement(s) that
can be executed.
Decision Statement
or Branching
Statement
IF condition THEN
do this
ENDIF
IF condition THEN
do this
ELSE
do that
ENDIF
WHILE condition
do this
ENDWHILE
FOR condition
do this
ENDFOR
DO
this
UNTIL condition
Statement(s) executed
zero or more times
Statement(s) executed
one or more times
![Page 22: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/22.jpg)
Producing a control flow graph
− identify sequential statements − combine a series into one node
− draw one branch coming out from each node
− add a node for each decision statement − draw two branches coming out from each decision
node
− add a node for the end of a decision structure − i.e. where lines join back together
− complete graph with an end node − do not leave a branch pointing into space!
![Page 23: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/23.jpg)
Example control flow graph
Make a list of items to sell For each item in the list Calculate cost of postage Write a description of this item If a photograph is needed Photograph this item Transfer photograph to PC Endif Submit details of item to eBay Endfor
End
end of IF
sequential
statements
single statement
decision
statement (FOR)
sequential
statement
decision
statement (IF)
sequential
statements
end of FOR
end of program
![Page 24: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/24.jpg)
![Page 25: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/25.jpg)
ENDIF
print b
b = a * 2
read(a)
IF a > 6 THEN b = a * 2
read(a)
IF a > 6 THEN
ENDIF
print b
b = a * 2
read(a)
IF a > 6 THEN
ENDIF
print b
read(a)
IF a > 6 THEN
branch coverage counts
all branches
decision coverage counts only
conditional branches
Branch/decision coverage differences
Basic blocks
TRUE
FALSE
Conditional
branch
Unconditional
branch
ENDIF
print b
b = a * 2
![Page 26: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/26.jpg)
1 7 14
Test case Input
Expected output
2 4 ?
1 67%
2 B1 -> B3 33%
Test case 2
could reveal
a fault (‘b’
has not been
defined)
B1 -> B2 B2 -> B3
Test case Path
Branch coverage
Decision coverage
50%
50%
Totals: 100% 100%
read(a)
IF a > 6 THEN
b = a * 2
ENDIF
print b
B1
B3
B2
TRUE
FALSE
![Page 27: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/27.jpg)
number of boolean operand values executed
total number of boolean operand values
IF (a < b) AND (c > d) AND (count < 100) THEN
=
Operand
values
Condition coverage does not guarantee decision coverage
![Page 28: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/28.jpg)
number of boolean operand value combinations executed
total number of boolean operand value combinations
Multiple Condition coverage guarantees decision coverage
IF (a < b) AND (c > d) AND (count < 100) THEN
=
![Page 29: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/29.jpg)
IF (a < b) AND (c > d) AND (count < 100) THEN
number of boolean operand value (independently affect the decision)
total number of boolean operands
=
![Page 30: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/30.jpg)
number of executed LCSAJ’s
total number of LCSAJ’s =
![Page 31: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/31.jpg)
Result = 0
Right = 0
DO While more questions
IF Key = Answer then
Incr Right
END DO
Result = (Right / Questions) x 100
IF Result > 50 Then
Print "pass"
ELSE
Print "fail"
END
LCSAJ's
From To Target
1 3 7
1 4 6
1 6 3
3 3 7
3 4 6
3 6 3
6 6 3
7 8 10
7 10 12
10 12 end
12 12 end
etc.
1
2
3
4
5
6
7
8
9
10
11
12
![Page 32: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/32.jpg)
? ?
1 2
? ?
1 2 3 1 2 ?
?
1 3 2 4
![Page 33: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/33.jpg)
?
4 5 6 7 8 ….
for as many times as it
is possible to go round
the loop (this can be
unlimited, i.e. infinite)
2 1 3
![Page 34: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/34.jpg)
All Paths
L
B
S
BCC
MCDC
BC
Where:
All Paths = Path Coverage
L = LCSAJ
B = Branch/Decision Coverage
S = Statement Coverage
BCC = Branch Condition
Combination Coverage
MCDC = Modified Condition
Decision Coverage
BC = Branch Condition
Coverage
Source: BS 7925-2
![Page 35: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/35.jpg)
structural coverage can measure completeness or
incompleteness of a set of specification or
experienced based techniques
code coverage tools capture the structural coverage achieved by the tests
any gaps to achieve 100% are identified
very useful for demonstrating compliance to standards
especially safety-critical systems
![Page 36: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/36.jpg)
![Page 37: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/37.jpg)
![Page 38: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/38.jpg)
![Page 39: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/39.jpg)
Manual Testing
Layer Diagram
UML Modeling
Load Testing Web Testing
Test Case Management IntelliTrace™
Architecture Explorer Fast Fwd for Manual Test
Logical Class Designer
Cloud Development
Office Development
Windows Development
New WPF Editor Customizable IDE
Multi-core Development Silverlight Tools
Web Development
SharePoint Development
Generate from Usage
Static Code Analysis
Database Deployment
Code Metrics
Database Unit Testing Test Data Generation
Test Impact Analysis UI Test Automation
Code Coverage
Performance Profiling
Database Change Mgmt.
Visual Studio – Products
![Page 40: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/40.jpg)
![Page 41: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/41.jpg)
Fully supported platform
Partial solution. Further work
required to complete
Best efforts. May work with
known issues. No major
ongoing investment
Currently no support but on
the roadmap for future
releases
Currently no support and
none planned for now.
Platform Native CodedUI 3rd Party Notes
IE7/8/9+ HTML/AJAX
ASP.NET MVC 3/4
WPF 3.5+
Windows Win32
Windows Forms 2.0+
FF3 – HTML/AJAX
Citrix/Terminal Services
SharePoint
Silverlight 3.0/4.0
Java/Java Applets
Office Client Apps
IE 6 (Legacy)
Chrome/Opera/Safari
Flash/AIR
SAP
![Page 42: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/42.jpg)
generalist specialist
manual
testing
some
scripting
creates scripts
to set up lab,
create data
scripting skills
some
coding skills
strong coding
develops fully
automated
testing
procedures
expert
coding skills
black box testing
white box testing
API testing
![Page 43: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/43.jpg)
coded UI test
unit testing web performance test
load test
test runner
test case management
virtual lab management
data diagnostic adapters (video, intelliTrace, action log, event log etc.)
team foundation server with reporting
(defects, requirements, user stories, code coverage, source control & build information)
generalist specialist
![Page 44: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/44.jpg)
![Page 46: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/46.jpg)
Maintainability Index
Cyclomatic Complexity
Depth of Inheritance
Class Coupling
Lines of Code
![Page 50: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/50.jpg)
![Page 52: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/52.jpg)
TDD
BDD
UI
Automation
Manual
(Assisted)
bla
ck b
ox te
stin
g
wh
ite
bo
x te
stin
g
AP
I te
stin
g
component level
ATDD
Automation
Fakes & Stubs
system level
integration level
![Page 53: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/53.jpg)
![Page 54: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/54.jpg)
![Page 55: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/55.jpg)
![Page 56: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/56.jpg)
Dynamic Diagnostic Adapters (BPM Assist, IntelliTrace, Amber (Action & Event Logging))
A1
B1
Home
Add.Book
Shopping.Cart
Search.Book
Submit.Order
D1
C1
B2B4
C3B3C2
B5
D3C4 D2
D4 D5
C5
= High
= Medium
= Low
Test Control Variable
Confidence Level
![Page 57: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/57.jpg)
www.microsoft.com/visualstudio/11/
www.microsoft.com/visualstudio/11/en-US/downloads#tfs-group
www.windows.microsoft.com/en-US/windows-8/download/
www.specflow.org
![Page 58: STARWest - Code Coverage is for Testers too - Tutorial](https://reader034.vdocuments.site/reader034/viewer/2022042715/55946fb81a28aba00e8b461d/html5/thumbnails/58.jpg)