couverture erts2012
Post on 01-Dec-2014
8.279 Views
Preview:
DESCRIPTION
TRANSCRIPT
Formalization and Comparisonof mcdc and Object Branch
Coverage Criteria
Cyrille Comar, Jerome Guitton, Olivier Hainque, Thomas Quinot
1 / 17
Coverage for Level A
Considering the highest level of certification for aircraft (A inDO-178B):
I Test coverage goal: mcdc
I Structural (source) coverage
I More than statement coverage...I ...but not all execution paths (too costly)I e.g. for a decision C1 and C2 and . . . and CN :
I 2N execution pathsI N + 1 tests in mcdc
2 / 17
Coverage for Level A
Considering the highest level of certification for aircraft (A inDO-178B):
I Test coverage goal: mcdc
I Structural (source) coverage
I More than statement coverage...I ...but not all execution paths (too costly)I e.g. for a decision C1 and C2 and . . . and CN :
I 2N execution pathsI N + 1 tests in mcdc
2 / 17
Coverage for Level A
Considering the highest level of certification for aircraft (A inDO-178B):
I Test coverage goal: mcdc
I Structural (source) coverage
I More than statement coverage...
I ...but not all execution paths (too costly)I e.g. for a decision C1 and C2 and . . . and CN :
I 2N execution pathsI N + 1 tests in mcdc
2 / 17
Coverage for Level A
Considering the highest level of certification for aircraft (A inDO-178B):
I Test coverage goal: mcdc
I Structural (source) coverage
I More than statement coverage...I ...but not all execution paths (too costly)
I e.g. for a decision C1 and C2 and . . . and CN :I 2N execution pathsI N + 1 tests in mcdc
2 / 17
Coverage for Level A
Considering the highest level of certification for aircraft (A inDO-178B):
I Test coverage goal: mcdc
I Structural (source) coverage
I More than statement coverage...I ...but not all execution paths (too costly)I e.g. for a decision C1 and C2 and . . . and CN :
I 2N execution pathsI N + 1 tests in mcdc
2 / 17
Coverage for Level A
Considering the highest level of certification for aircraft (A inDO-178B):
I Test coverage goal: mcdc
I Structural (source) coverage
I More than statement coverage...I ...but not all execution paths (too costly)I e.g. for a decision C1 and C2 and . . . and CN :
I 2N execution paths
I N + 1 tests in mcdc
2 / 17
Coverage for Level A
Considering the highest level of certification for aircraft (A inDO-178B):
I Test coverage goal: mcdc
I Structural (source) coverage
I More than statement coverage...I ...but not all execution paths (too costly)I e.g. for a decision C1 and C2 and . . . and CN :
I 2N execution pathsI N + 1 tests in mcdc
2 / 17
Some issues with mcdc
I source coverage criteria means language-dependent
I no tool for Ada 2005 when Couverture started...
I instrumenting source code? intrusive...
I unbounded execution traces
I Use object coverage instead?
3 / 17
Some issues with mcdc
I source coverage criteria means language-dependent
I no tool for Ada 2005 when Couverture started...
I instrumenting source code? intrusive...
I unbounded execution traces
I Use object coverage instead?
3 / 17
Some issues with mcdc
I source coverage criteria means language-dependent
I no tool for Ada 2005 when Couverture started...
I instrumenting source code? intrusive...
I unbounded execution traces
I Use object coverage instead?
3 / 17
Some issues with mcdc
I source coverage criteria means language-dependent
I no tool for Ada 2005 when Couverture started...
I instrumenting source code? intrusive...
I unbounded execution traces
I Use object coverage instead?
3 / 17
Some issues with mcdc
I source coverage criteria means language-dependent
I no tool for Ada 2005 when Couverture started...
I instrumenting source code? intrusive...
I unbounded execution traces
I Use object coverage instead?
3 / 17
Object coverage to assess mcdc
I Assumption that object branch coverage (obc) is strongerthan mcdc
I widespread industrial practise
I language-independent
I bounded traces
4 / 17
Object coverage to assess mcdc
I Assumption that object branch coverage (obc) is strongerthan mcdc
I widespread industrial practise
I language-independent
I bounded traces
4 / 17
Object coverage to assess mcdc
I Assumption that object branch coverage (obc) is strongerthan mcdc
I widespread industrial practise
I language-independent
I bounded traces
4 / 17
Object coverage to assess mcdc
I Assumption that object branch coverage (obc) is strongerthan mcdc
I widespread industrial practise
I language-independent
I bounded traces
4 / 17
Using obc to achieve mcdc ? short-circuit operators...
(A mod B = 0) and then (C = 0)
A mod B = 0
F
F
C = 0
F
F
T
T
T
B = −1
A > 0
B < 0
F
R = 0
F
F
C = 0
F
F
T
T
T
F
T
F TT
A mod B = 0
5 / 17
Using obc to achieve mcdc ? short-circuit operators...
(A mod B = 0) and then (C = 0)
A mod B = 0
F
F
C = 0
F
F
T
T
T
B = −1
A > 0
B < 0
F
R = 0
F
F
C = 0
F
F
T
T
T
F
T
F TT
A mod B = 0
5 / 17
Using obc to achieve mcdc ? short-circuit operators...
(A mod B = 0) and then (C = 0)
A mod B = 0
F
F
C = 0
F
F
T
T
T
B = −1
A > 0
B < 0
F
R = 0
F
F
C = 0
F
F
T
T
T
F
T
F TT
A mod B = 0
5 / 17
What do certification standards say about that?
DO-248C FAQ #42: Object coverage can be used as long asanalysis can be provided which demonstrates that the coverageanalysis conducted at the Object Code will achieve a comparablelevel of coverage assurance as that conducted at the Source Codelevel.
6 / 17
The sad truth...
I The assumption is wrong: obc is not stronger than mcdc!
I A counterexample in DOT/FAA/AR-07/20, Jun 2007: ObjectOriented Technology Verification Phase 3 Report - StructuralCoverage at the Source Code and Object Code Levels
I (A and then B) or else C can be covered for obc withoutachieving mcdc
7 / 17
The sad truth...
I The assumption is wrong: obc is not stronger than mcdc!
I A counterexample in DOT/FAA/AR-07/20, Jun 2007: ObjectOriented Technology Verification Phase 3 Report - StructuralCoverage at the Source Code and Object Code Levels
I (A and then B) or else C can be covered for obc withoutachieving mcdc
7 / 17
The sad truth...
I The assumption is wrong: obc is not stronger than mcdc!
I A counterexample in DOT/FAA/AR-07/20, Jun 2007: ObjectOriented Technology Verification Phase 3 Report - StructuralCoverage at the Source Code and Object Code Levels
I (A and then B) or else C can be covered for obc withoutachieving mcdc
7 / 17
Having a closer look...
Alloy model: check conjectures, generate counterexamples...
8 / 17
Pathological case
Alloy helped finding a impressive counterexample:
I Decision with an arbitrary high number of conditions N...
I ...that needs N+1 tests to be mcdc-covered...
I ...and only 3 tests to be obc-covered!
Here obc is much weaker than mcdc!
9 / 17
Pathological case
Alloy helped finding a impressive counterexample:
I Decision with an arbitrary high number of conditions N...
I ...that needs N+1 tests to be mcdc-covered...
I ...and only 3 tests to be obc-covered!
Here obc is much weaker than mcdc!
9 / 17
Pathological case
Alloy helped finding a impressive counterexample:
I Decision with an arbitrary high number of conditions N...
I ...that needs N+1 tests to be mcdc-covered...
I ...and only 3 tests to be obc-covered!
Here obc is much weaker than mcdc!
9 / 17
Pathological case
Alloy helped finding a impressive counterexample:
I Decision with an arbitrary high number of conditions N...
I ...that needs N+1 tests to be mcdc-covered...
I ...and only 3 tests to be obc-covered!
Here obc is much weaker than mcdc!
9 / 17
Pathological case
Alloy helped finding a impressive counterexample:
I Decision with an arbitrary high number of conditions N...
I ...that needs N+1 tests to be mcdc-covered...
I ...and only 3 tests to be obc-covered!
Here obc is much weaker than mcdc!
9 / 17
Now what?
I Sure, in some cases, obc does not imply mcdc...
I ...and in some pathological cases, the two criteria divergesquite badly...
I ...but past experience has shown that it works in many cases
I ...so are there conditions that would allow this implication?
I could be enforced by a coding standardI could be an optimization for coverage tools
10 / 17
Now what?
I Sure, in some cases, obc does not imply mcdc...
I ...and in some pathological cases, the two criteria divergesquite badly...
I ...but past experience has shown that it works in many cases
I ...so are there conditions that would allow this implication?
I could be enforced by a coding standardI could be an optimization for coverage tools
10 / 17
Now what?
I Sure, in some cases, obc does not imply mcdc...
I ...and in some pathological cases, the two criteria divergesquite badly...
I ...but past experience has shown that it works in many cases
I ...so are there conditions that would allow this implication?
I could be enforced by a coding standardI could be an optimization for coverage tools
10 / 17
Now what?
I Sure, in some cases, obc does not imply mcdc...
I ...and in some pathological cases, the two criteria divergesquite badly...
I ...but past experience has shown that it works in many cases
I ...so are there conditions that would allow this implication?
I could be enforced by a coding standardI could be an optimization for coverage tools
10 / 17
Now what?
I Sure, in some cases, obc does not imply mcdc...
I ...and in some pathological cases, the two criteria divergesquite badly...
I ...but past experience has shown that it works in many cases
I ...so are there conditions that would allow this implication?
I could be enforced by a coding standard
I could be an optimization for coverage tools
10 / 17
Now what?
I Sure, in some cases, obc does not imply mcdc...
I ...and in some pathological cases, the two criteria divergesquite badly...
I ...but past experience has shown that it works in many cases
I ...so are there conditions that would allow this implication?
I could be enforced by a coding standardI could be an optimization for coverage tools
10 / 17
Cases where obc implies mcdc
Theorem
If there is only one execution path to each condition, then obcimplies mcdc.
C1 and then C2
C1
F
F
C2
F
F
T
T
T
11 / 17
Cases where obc implies mcdc
Theorem
If there is only one execution path to each condition, then obcimplies mcdc.
C1 and then C2
C1
F
F
C2
F
F
T
T
T
11 / 17
Cases where obc implies mcdc
Theorem
If there is only one execution path to each condition, then obcimplies mcdc.
C1 and then C2
C1
F
F
C2
F
F
T
T
T
11 / 17
Cases where obc implies mcdc
Theorem
If there is only one execution path to each condition, then obcimplies mcdc.
C1 and then C2
C1
F
F
C2
F
F
T
T
T
11 / 17
Cases where obc does not imply mcdc
Theorem
On the contrary, if there exists a condition that can be reached bymore than one execution path, obc does not always imply mcdc.
(A and then B) or else C
A
B
T C
T F
12 / 17
Cases where obc does not imply mcdc
Theorem
On the contrary, if there exists a condition that can be reached bymore than one execution path, obc does not always imply mcdc.
(A and then B) or else C
A
B
T C
T F
12 / 17
Cases where obc does not imply mcdc
Theorem
On the contrary, if there exists a condition that can be reached bymore than one execution path, obc does not always imply mcdc.
(A and then B) or else C
A
B
T C
T F
12 / 17
Cases where obc does not imply mcdc
Theorem
On the contrary, if there exists a condition that can be reached bymore than one execution path, obc does not always imply mcdc.
(A and then B) or else C
A
B
T C
T F
12 / 17
human-readable characteristic
Dec1 or else (Dec2 and then . . . )
OR ELSE
Dec
1
AND THEN
Dec
2
[. . .
]
13 / 17
human-readable characteristic
Dec1 or else (Dec2 and then . . . )
OR ELSE
Dec
1
AND THEN
Dec
2
[. . .
]
13 / 17
human-readable characteristic
Dec1 or else (Dec2 and then . . . )
OR ELSE
Dec
1
AND THEN
Dec
2
[. . .
]
no and then
13 / 17
human-readable characteristic
Dec1 or else (Dec2 and then . . . )
OR ELSE
Dec
1
AND THEN
Dec
2
[. . .
]
no or else
13 / 17
Experimental results
In the industrial applications that we looked at, 99 % of thedecisions are such that obc implies mcdc
configuration App. 1 App. 2 GNATcoverage
#decisions 869 37324 1026
#non-tree BDD 7 (0.8 %) 141 (0.4 %) 4 (0.4 %)
14 / 17
Experimental results
In the industrial applications that we looked at, 99 % of thedecisions are such that obc implies mcdc
configuration App. 1 App. 2 GNATcoverage
#decisions 869 37324 1026
#non-tree BDD 7 (0.8 %) 141 (0.4 %) 4 (0.4 %)
14 / 17
Experimental results
In the industrial applications that we looked at, 99 % of thedecisions are such that obc implies mcdc
configuration App. 1 App. 2 GNATcoverage
#decisions 869 37324 1026
#non-tree BDD 7 (0.8 %) 141 (0.4 %) 4 (0.4 %)
14 / 17
Experimental results
I Evaluating the impact of this optimization on the qualificationtestsuite of GNATcoverage;
I compute the coverage of GNATcoverage in 3 differentconfigurations:
I obc: as a baseline;I mcdc 1: historical traces on branches of all decision;I mcdc 2: historical traces only when there are conditions
reachable by several paths...
I ...and compare the size of the generated traces.
15 / 17
Experimental results
I Evaluating the impact of this optimization on the qualificationtestsuite of GNATcoverage;
I compute the coverage of GNATcoverage in 3 differentconfigurations:
I obc: as a baseline;I mcdc 1: historical traces on branches of all decision;I mcdc 2: historical traces only when there are conditions
reachable by several paths...
I ...and compare the size of the generated traces.
15 / 17
Experimental results
I Evaluating the impact of this optimization on the qualificationtestsuite of GNATcoverage;
I compute the coverage of GNATcoverage in 3 differentconfigurations:
I obc: as a baseline;I mcdc 1: historical traces on branches of all decision;I mcdc 2: historical traces only when there are conditions
reachable by several paths...
I ...and compare the size of the generated traces.
15 / 17
Experimental results
configuration obc mcdc 1 mcdc 2
#branches to trace 0 1788 22
size of traces 1.33G 5.06G 1.37G
I The optimization removes 99 % of historical traces
I The overead compared to obc is marginal
16 / 17
Experimental results
configuration obc mcdc 1 mcdc 2
#branches to trace 0 1788 22
size of traces 1.33G 5.06G 1.37G
I The optimization removes 99 % of historical traces
I The overead compared to obc is marginal
16 / 17
Experimental results
configuration obc mcdc 1 mcdc 2
#branches to trace 0 1788 22
size of traces 1.33G 5.06G 1.37G
I The optimization removes 99 % of historical traces
I The overead compared to obc is marginal
16 / 17
More information...
I Resources: Couverture public repository on the Open-Do forge(https://forge.open-do.org/projects/couverture/)
I Alloy modelsI Proofs of theorems comparing obc and mcdcI other results about mcdc
I Couverture project gave birth to an industrial tool:GNATcoverage
17 / 17
More information...
I Resources: Couverture public repository on the Open-Do forge(https://forge.open-do.org/projects/couverture/)
I Alloy models
I Proofs of theorems comparing obc and mcdcI other results about mcdc
I Couverture project gave birth to an industrial tool:GNATcoverage
17 / 17
More information...
I Resources: Couverture public repository on the Open-Do forge(https://forge.open-do.org/projects/couverture/)
I Alloy modelsI Proofs of theorems comparing obc and mcdc
I other results about mcdc
I Couverture project gave birth to an industrial tool:GNATcoverage
17 / 17
More information...
I Resources: Couverture public repository on the Open-Do forge(https://forge.open-do.org/projects/couverture/)
I Alloy modelsI Proofs of theorems comparing obc and mcdcI other results about mcdc
I Couverture project gave birth to an industrial tool:GNATcoverage
17 / 17
More information...
I Resources: Couverture public repository on the Open-Do forge(https://forge.open-do.org/projects/couverture/)
I Alloy modelsI Proofs of theorems comparing obc and mcdcI other results about mcdc
I Couverture project gave birth to an industrial tool:GNATcoverage
17 / 17
top related