iv&v facility pi: katerina goseva – popstojanova students: sunil kamavaram & olaolu...

IV&V Facility

PI: Katerina Goseva – Popstojanova

Students: Sunil Kamavaram & Olaolu Adekunle

Lane Department of Computer Science and Electrical Engineering West Virginia University, Morgantown, WV

[email protected]

Real-World Software Reliability Assessment

(WVU UI#7: Sensitivity of Software Reliability to Operational Profile Errors:

Architecture-Based Approach)

2

IV&V Facility

What we are doing?

Anyone can see a fire

What we need are smoke

detectors

But what about the sensitivity and accuracy of the alarms ?

3

IV&V Facility

Problem statement & Our goal

Traditional view: Point estimate of software reliability computed from the model using point estimates of input parameters

Problem: Estimation of a trustworthy operational profile is difficult IV&V information on operational profiles - limited, may be inaccurate Single operational profile could not be sufficient to describe the use by different

users Software systems evolve - operational profile may change

Our goal: Reliability “sensitometer” that enables us to answer the question “How parameters uncertainty propagates into overall application reliability?” Develop an architecture-based methodology for uncertainty analysis of

software reliability & apply it on case studies

4

IV&V Facility

What we can do?

Benefits to IV&V Software reliability assessment throughout the life cycle (keeping track

of the software evolution) Allocation of testing efforts Software certification

00.20.40.60.8

11.21.41.6

Reliabilityfrequency chart &distribution fitting

Certainty bands(percentiles)

Entropy asa measure of uncertainty

Execution rates &uncertainty of components

Certainty Bands - (Percentiles)

Centered on Medians

0.5000

0.6250

0.7500

0.8750

1.0000

Reliability

95%

75%

50%

25%

10%

Trend Chart

Frequency Chart

Certainty is 99.63% from 0.5000 to 1.0000

.000

.007

.014

.020

.027

0

67.5

135

202.5

270

0.5000 0.6250 0.7500 0.8750 1.0000

10,000 Trials 9,963 Displayed

Forecast: Reliability

5

IV&V Facility

Architecture - based methodology for uncertainty analysis

Uninformed Approach

(maximum entropy)

Uninformed Approach

(maximum entropy)

Intended Approach

(historical data, UML)

Intended Approach

(historical data, UML)

Informed Approach

(component traces)

Informed Approach

(component traces)

1-p23

1-p12

11

22

EE

33

p23

p12

1

Fault injection

Fault injection

Non-failed executions

Non-failed executions

Growth models

Growth models

R1

R2

R3

Uncertaintyanalysis

6

IV&V Facility

Methods for uncertainty analysis

Uncertainty analysis

Sensitivity studies

Entropy

Confidenceintervals

Probability distributions

Analytical

Monte Carlo simulationMethod of moments

Perturbationanalysis

7

IV&V Facility

Choice of the method

Choose the method using the following criteria Data requirements & ability to collect data Reliability measures Accuracy of the solution Scalability with respect to the number of components

Our goal: fill the tableMethod

Data requirements

Reliability

measuresAccuracy of the solution

Scalability

8

IV&V Facility

Construction of the software architecture model

11

22

EE

331-p23

1-p12

p23

p12

1

Structural phase – establishment of static software architecture

Software specifications Architectural design Parser-based or lexically based tools (SIAT tool - Titan Systems Corporation)

Statistical phase – estimation of the relative frequencies of component interactions, that is, transition probabilities

Uniform distribution – maximum entropy approach Historical data Software specification (e.g. UML use case & sequence diagrams) Component traces from profiles or test coverage tools

(Testing tool for JSC AERCam project - Dr.Yann-Hang Lee, ASU)

9

IV&V Facility

European Space Agency case study

Informed Approach

(component traces)

Informed Approach

(component traces)

1-p23

1-p12

11

22

EE

33

p23

p12

1

Fault Injection(real faults)

Fault Injection(real faults)

R1

R2

R3

Two faulty versions were obtained reinserting the real faults discovered during the integration testing and operational usage

Component traces obtained during testing were used for constructing software architecture &estimating transition probabilities

Almost 10.000 lines of C code The program has been extensively

used after the last fault removal without failures; this gold version is used as an oracle

10

IV&V Facility

Parameter estimation

Two versions Version A: faulty components 1&2, fault-free component 3 Version B: faulty components 2, fault-free components 1&3

Transition probabilities where is the number of times control was transferred from component i to component j, and

Component reliability

where is the number of failures and is the number of executions of component i in N randomly generated test cases

i

ijij n

np

ijn

j iji nn0.68660.7364B

0.77040.5933A

p23p12Version

Version R1 R2 R3

A 0.8428 0.8346 1

B 1 0.8346 1

ifin

i

i

ni n

fR

i

lim1

11

IV&V Facility

Construction of the architecture – based software reliability model

FF

1-R1

1-R2

1-R3

EE

33 (1-p23)R2

11

22

p23 R2(1-p12)R1

p12 R1

R3

CC1

12

IV&V Facility

Traditional View: Point estimates of software reliability

Actual reliability of the software

where F is the number of system failures in N randomly generated test cases

Estimated reliability from the model

Results

NF

RNlim1

Version Actual reliability

Estimatedreliability

Error

A 0.7393 0.7601 2.81%

B 0.8782 0.8782 0%

3212312212312112 1)1( RRRppRRppRpR

13

IV&V Facility



Sensitivity studies

Entropy

Confidenceintervals


Analytical



14

IV&V Facility

Sensitivity of software reliability to variations in operational profile

Version A reliability Version B reliability

Rmax = 0.8414Rmin = 0.7048

Rmax = 0.9983Rmin = 0.8363

15

IV&V Facility



Sensitivity studies

Entropy

Confidenceintervals


Analytical



16

IV&V Facility

Uncertainty study based on entropy

Entropy quantifies the uncertainty present in a stochastic source

where represents the usage distribution and the transition probabilities

Higher entropy implies an exponentially greater number of statistically typical paths

Maximum entropy – all transitions that are exit arcs from each state are equiprobable

j

ijiji

i ppH log

i ijp

17

IV&V Facility

Uncertainty of the operational profile

Hmax = 0.5514Hmin = 0.0404

Operational profile A (H=0.4707) is more uncertain than operational profile B (H=0.4604)

Software systems that have uniform operational profile are more uncertain and thus would require more testing

Hmax = 0.5514Hmin = 0.0404

18

IV&V Facility

Uncertainty of software reliability

Operational profile

Considering software failure behavior increases the uncertainty for both versions compared to the uncertainty due to operational profile

Version B, which is more reliable, is less uncertain than version A

Version A uncertainty Version B uncertainty

Version A reliability Version B reliability

19

IV&V Facility

Uncertainty of components for the operational profile

Uncertainty of component i is estimated using the conditional entropy

Uncertainty of component i will be higher if it transfers the control to more components and the transition probabilities are equiprobable

j

ijiji ppH log

Componen

t 1

Componen

t 2

Componen

t 3

State

E

Execution rate

Uncertainty

00.20.40.60.8

11.21.41.6

Execution rate

Uncertinty

00.20.40.60.8

11.21.41.6

Version A Version B

20

IV&V Facility

Uncertainty of components for the software reliability model

Uncertainty of component 1 version B remains the same because For all other components uncertainty increases due to Components that have higher expected execution rate, higher

component uncertainty, and moderate reliability should be allocated more testing effort

11 R1iR

Componen

t 1

Componen

t 2

Componen

t 3

State

E

State

FExecution rate

Uncertainty

00.20.40.60.8

11.21.41.6

Componen

t 1

Componen

t 2

Componen

t 3

State

E

State

FExecution rate

Uncertainty

00.20.40.60.8

11.21.41.6

c

Version BVersion A

21

IV&V Facility



Sensitivity studies

Entropy

Confidenceintervals


Analytical



22

IV&V Facility

Uncertainty study based on the method of moments

Method of moments involves the following steps1.Obtain the expression for the system reliability using the

architecture-based software reliability model

2.Expand the expression for system reliability using Taylor series

3.Determine the moments of the components reliabilities

4.Estimate the mean and the variance of the system reliability using the parameter moments and Taylor series coefficients

23

IV&V Facility

First order Taylor series

First order Taylor series expansion

where is the mean component reliability, and

Mean reliability is

Variance of the reliability is

where is the variance of the component reliability

0aRE 2

1

22i

n

iiR a

n

iiii RaaR

10 )(

);,2

,1

(n

fao

ii RVar2

),2

,1

(n

RiR

Ri

a

ii RE

24

IV&V Facility

Second order Taylor series

jjii

n

i

i

jijii

n

i

n

iiiiiiR RRRR aaaa

1

1

1

2

1 10 2

1 Second order Taylor series expansion

),,,(

2

),,,(

2

2

),,,(

210

212121

,),,,,(

nnn Rjiij

Riii

Riin RR

Rand

R

R

R

Rfwhere aaaa

Mean reliability is

Variance of the reliability is

n

iiiiaaRE

1

20 2

1

n

iiii

n

iiiiiiii

n

iiij

n

i

i

jiiji

n

iiR

aEaaEaaa RR1

222

1

34

1

22

1

1

1

222

1

22

4

1

4

1

25

IV&V Facility

Method of moments for the case study

Second order approximation does not improve accuracy significantly

First order Taylor series

Second order

Taylor series

Mean reliability 0.7601 0.7601

Version A Standard deviation 0.0825 0.0825

Variance 0.0068 0.0068

Mean reliability 0.8782 0.8782

Version B Standard deviation 0.0589 0.0589

Variance 0.0035 0.0035

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.95

1

Version A Version B

Re

liab

ility

Version B is more reliable with less variance of the reliability

26

IV&V Facility



Sensitivity studies

Entropy

Confidenceintervals


Analytical



27

IV&V Facility

Uncertainty study based on Monte Carlo simulation

Monte Carlo simulation involves the following steps1. Obtain the expression for the system reliability using the

architecture-based software reliability model 2. Assign probability distributions to the transition

probabilities and components reliabilities3. Sample the distributions 4. Compute the reliability of the system using the sampled

values5. Repeat steps 3&4 until the desired number of values of

system reliability has been generated6. Calculate the moments, frequency chart and percentiles

for the system reliability, do the distribution fitting

28

IV&V Facility

Variation of the operational profile:Frequency chart and distribution fitting

Distribution Fitting

.000

.005

.011

.016

.021

0.7053 0.7326 0.7600 0.7873 0.8146

Weibull DistributionLoc. = 0.7021Scale = 0.0648Shape = 3.00

Reliability

Ov erlay ChartFrequency Chart

.000

.004

.008

.012

.015

0

38.5

77

115.5

154

0.7060 0.7332 0.7603 0.7874 0.8146



Mean 0.7600

Standard deviation (Spread of the distribution) 0.0210

Variance (Spread of the distribution) 0.0004

Skewness (Lean of the distribution) 0.2072

Kurtosis (Peakedness of the distribution) 2.6047

29

IV&V Facility

95% certainty band shows the range of values in which reliability has 95% chance of falling

Variation of the operational profile:Percentiles

75%95%


Centered on Medians

0.7000

0.7375

0.7750

0.8125

0.8500

95%

75%

50%

25%

10%

Trend Chart

30

IV&V Facility

Convergence of the mean

The estimation of the mean reliability converges after around 3000 iterations

Mean reliability =0.7600

0.7550

0.7575

0.7600

0.7625

0.7650

1 1017 2033 3049 4065 5081 6097 7113 8129 9145

Number of Iterations

Mea

n R

elia

bilit

y

31

IV&V Facility

Reliability is more sensitive to p1E; the variance is positive

Reliability is also sensitive to p12; the variance is negative

Variation of the operational profile:Sensitivity measured by contribution to

variance

Target Forecast: Reliability

P1E 60.6%

P12 39.4%

P3E 0.0%

P23 0.0%

100% 50% 0% 50% 100%

Measured by Contribution to Variance

Sensitiv ity Chart

32

IV&V Facility

Version A Version B

Variation of the operational profile and component reliabilities: Frequency charts

Frequency Chart

.000

.009

.018

.028

.037

0

92

184

276

368

0.5000 0.6250 0.7500 0.8750 1.0000



Frequency Chart

.000

.007

.013

.020

.026

0

65.25

130.5

195.7

261

0.5000 0.6250 0.7500 0.8750 1.0000



Version A Version B

Mean 0.7589 0.8780

Standard deviation (Spread of the distribution) 0.0860 0.0660

Variance (Spread of the distribution) 0.0074 0.0044

Coefficient of variation (Relative measure of spread)

0.1493 0.0752

Skewness (Lean of the distribution) -0.5190 -0.9646

Kurtosis (Peakedness of the distribution) 3.1367 4.2254

33

IV&V Facility

Version A Version B

Variation of the operational profile and component reliabilities: Distribution fitting

& percentiles


.000

.009

.018

.028

.037

0.5000 0.6250 0.7500 0.8750 1.0000

Beta DistributionAlpha = 20.1525Beta = 2.7208Scale = 0.9965

Reliability

Ov erlay Chart


.000

.007

.013

.020

.026

0.5000 0.6250 0.7500 0.8750 1.0000

Beta DistributionAlpha = 17.5014Beta = 5.3662Scale = 0.9916

Reliability

Ov erlay Chart


Centered on Medians

0.5000

0.6250

0.7500

0.8750

1.0000

95%

75%

50%

25%

10%

Trend Chart


Centered on Medians

0.5000

0.6250

0.7500

0.8750

1.0000

95%

75%

50%

25%

10%

Trend Chart

34

IV&V Facility

Making a choice

Method Data

requirementsReliability measures

Accuracy of the solution

Scalability

Sensitivity Point estimates Sensitivity of the point estimate

Exact analytical solution Large systems

Entropy Point estimates NA Exact analytical solution Large systems

Method of

moments

Moments of the parameters

Moments Approximate solution: accuracy may be increased by higher order Taylor series

Small to medium systems

Monte Carlo simulation

Distribution functions of the parameters

Generation of random numbers

Distribution Moments

Approximate solution: accuracy may be increased by increasing the sample size

Sampling errors may be involved in case of long tail distributions

Large systems

35

IV&V Facility

Accomplishments

Architecture-based methodology for uncertainty analysis of software reliability was developed

Four different methods already developed These methods were illustrated on the European

Space Agency software

36

IV&V Facility

Future work

Develop other methods for uncertainty analysis Complete “Make a choice” table Apply & validate all methods using NASA case

studies SIAT tool - Titan Systems Corporation Testing tool for JSC AERCam project - Dr.Yann-Hang

Lee, ASU

iv&v facility pi: katerina goseva – popstojanova students: sunil kamavaram & olaolu...

Documents