FFault ault TToleranceoleranceBBenchmarkingenchmarking

2

Owerview

• What is Benchmarking?

• What is Dependability?

• What is Dependability Benchmarking?

• What is the relation between Dependability & Fault tolerance Benchmarking?

• Tsai (A Fault tolerance Benchmark)

/ 20

3

What is Benchmarking?

• Benchmarking is testing of several computer systems or peripheral devices with the aim to compare their performance-cost relationship.

• Benchmarking means measuring something In an automated, reproducible, portable, and truthful way, and this can only probably be achieved by relying on direct results.

(not by model analysis).

/ 20

4

What is a Benchmark ?

• A specification of all elements necessary for performing the benchmark tests.

What elements?

See later!

/ 20

5

Benchmarking Context

Definition of benchmark elements depends on: the system under benchmark the benchmark scope (internal/external) the application area the phase of system life cycle in mind

designImplementationintegrationtestproductionmaintenance

/ 20

6

• Dependability is the trustworthiness of a computer system such that reliance can justifiably be placed on the service it delivers.

• Dependability is normally described by a set of dependability attributes.

What is Dependability?

/ 20

7

Dependability Attributes Reliability

Measure of continuous correct service delivery(dependability with respect to continuity of service).

AvailabilityMeasure of correct service delivery with respect to the alternation of correct and incorrect service (dependability with respect to readiness for usage).

SafetyMeasure of continuous delivery of either correct service or incorrect service after benign failure (dependability with respect to the non-occurrence of catastrophic failures).

SecurityDependability with respect to the prevention of unauthorized access and/or handling of information.

RobustnessThe degree to which a system or component can function correctly in the presence of invalid inputs or stressful environment conditions.

/ 20

8

Why Benchmark Availability?

System availability is a pressing problem– modern applications demand near-100% availability

e-commerce, enterprise apps, online services, ISPs at all scales and price points

– we don’t know how to build highly-available systems!except at the very high-end

Few tools exist to provide insight into system availability– most existing benchmarks ignore availability

focus on performance, and under ideal conditions– no comprehensive, well-defined metrics for availability

/ 20

9

Availability metrics

Traditionally, percentage of time system is up– time-averaged, binary view of system state (up/down)

This metric is inflexible– doesn’t capture degraded states

a non-binary spectrum between “up” and “down”

time-averaging discards important temporal behavior– compare 2 systems with 96.7% traditional availability:

system A is down for 2 seconds per minutesystem B is down for 1 day per month

/ 20

10

Dependability Attributes

• How do we achiev these attributes?– start by understanding them– figure out how to measure them– evaluate existing systems and techniques– develop new approaches based on what we’ve

learned– and measure them as well!

• Dependability Benchmarks make these tasks possible!

/ 20

11

What is Dependability Benchmarking?

• Dependability benchmarking is performing a set of tests to quantify computer dependability.

• This quantification is supported by the evaluation of dependability attributes (e.g., reliability, availability, safety) through the assessment of direct measures related to the behavior of a computer in the presence of faults.

• Examples of these direct measures are failure modes, error detection coverage, error latency, diagnosis efficiency, recovery time, and recovery losses.

/ 20

12

Dependability Benchmarking

• The goal of dependability benchmarking is to quantify the dependability features of a computer or a computer component in a truthful and reproducible way.

• Unlike functionality and performance features, that are normally available to the customers or can be certified (functionality) or measured (performance), component or system dependability cannot be easily assessed today.

• The objective of dependability benchmarking is to change this picture, especially for COTS components and COTS-based systems.

/ 20

13

What is expected from a Software Dependability Benchmark?

Software characterization with respect to:– Internal faults– External faults

other software component(s)hardware

Aims– Properties + quantification of some specific measures– Avoid undesirable behavior → “environment” modification– Enhancement

CorrectionWrapping

/ 20

14

Availability Benchmark

/ 20

15

Software Testing Aims

– Activate / identify faults correction– Characterize software behavior

Various kinds of tests for characterization– Functional (statistical / operational) testing

• Validate functionality under ordinary operating conditions (typical load)

• Software reliability evaluation– Load testing

• Performance under heavy load (peak, worst case)– Robustness / stress testing

• Pushes the software beyond its specified limits(invalid inputs, stressful environmental conditions)

/ 20

16

Software Reliability

Aim– Evaluate software reliability measures (MTTF, failure

intensity) Means

– Failure data collection and processingFrom functional test

AdvantageEarly estimation

LimitationRepresentative operational profileDuration accuracy

/ 20

17

Software Performance Benchmark

• Most of the timePerformance benchmark = system benchmark

• Aims of software performance benchmark• Measure system performance (or price / performance) to

compare:– different software products on the same machine

Competition between software vendors– different releases/versions of the same product on the same machine

Performance improvement?• Two categories of benchmarks

– Coarse-grain benchmarks => execution times of the entire application

– Fine-grain benchmarks => execution time (rate) of specific operations

/ 20

18

Fault Tolerance & Robustness Benchmarks

Fault tolerance benchmarks– Ability to tolerate faults– Effectiveness of error detection and recovery mechanisms– Performance degradation due to faults

Robustness benchmarks– Ability to tolerate / resist to unexpected conditions caused

by:Hardware failuresOther user programs (system call with illegal parameters)

– Failure modes= Robustness test + repeatable

Fault injection

/ 20

19

The Fault Tolerance Benchmark(Tsai et al., 1996)

This fault tolerance benchmark uses a two-phase procedure:1) Determine whether the system tolerates the faults that it is

intended to tolerate, and evaluate the effect these faults have on the fault tolerance mechanisms.

2) Evaluate the reaction of the system to faults that it is not designed to handle(a demonstration of the degree of fault tolerance beyond what is expected).

For phase 1 injections, three types of measures are obtained:– Error/fault ratio: the ratio of the number of errors detected to the

number of faults injected.– Performance degradation: two times are measured:

The time to execute the benchmark with faults. The time to execute the benchmark without faults.

– Number of catastrophic incidents.

/ 20

20

Conclusion

• Dependability benchmarks rely on:development / validation process & field information+ additional specific work

• What about COTS? well tested BUT information not available all the time

/ 20