reliability and fault tolerance
Post on 03-Jan-2016
88 Views
Preview:
DESCRIPTION
TRANSCRIPT
Reliability and Fault Reliability and Fault ToleranceTolerance
Setha Pan-ngumSetha Pan-ngum
IntroductionIntroduction From the survey by American Society for From the survey by American Society for
Quality Control [1]. Quality Control [1]. Ten most important Ten most important product attributesproduct attributes
AttributeAttribute Ave. Ave. ScoreScore
AttributeAttribute Ave. Ave. ScoreScore
performanceperformance 9.59.5 Ease of useEase of use 8.38.3
Last a long Last a long time time (reliability)(reliability)
9.09.0 AppearanceAppearance 7.77.7
ServiceService 8.98.9 Brand nameBrand name 6.36.3
Easily repaired Easily repaired (maintainabilit(maintainability)y)
8.88.8 Packaging/Packaging/displaydisplay
5.85.8
warrantywarranty 8.48.4 Latest modelLatest model 5.45.4
IntroductionIntroduction
Embedded system major Embedded system major requirementsrequirements– Low failure rateLow failure rate– Leads to fault tolerance designLeads to fault tolerance design– Gracefully degradableGracefully degradable
Failures, errors, faultsFailures, errors, faults
Fault – defects that cause malfunctionFault – defects that cause malfunction– Hardware fault e.g. broken wire, stuck Hardware fault e.g. broken wire, stuck
logiclogic– Software fault e.g. bugSoftware fault e.g. bug
Error – unintended state caused by Error – unintended state caused by fault. E.g. software bug leads to fault. E.g. software bug leads to wrong calculation wrong calculation wrong output wrong output
Failure – errors leads to system Failure – errors leads to system failure (opearates differently from failure (opearates differently from intended)intended)
Causes of FailuresCauses of Failures
Errors in specification or designErrors in specification or design Component defectsComponent defects Environmental effectsEnvironmental effects
Errors in specification Errors in specification or designor design Probably the hardest to detectProbably the hardest to detect Embedded system development:Embedded system development:
– SpecificationSpecification– DesignDesign– ImplementationImplementation
If specification is wrong, the If specification is wrong, the following steps will be wrong. E.g. following steps will be wrong. E.g. unit compatibility of rocket example.unit compatibility of rocket example.
Component defectsComponent defects
Depends on deviceDepends on device Electronic components can have Electronic components can have
defects from manufacturing, and defects from manufacturing, and wear and tear.wear and tear.
Operating Operating environmentenvironment StressesStresses TemperaturesTemperatures MoistureMoisture vibrationvibration
Classification of Classification of failuresfailures Nature Nature
– Value – incorrect outputValue – incorrect output– Timing – correct output but too late.Timing – correct output but too late.
Perception – as seen by usersPerception – as seen by users– Persistent – all users see same results. Persistent – all users see same results.
E.g. sensor reading stuck at ‘0’E.g. sensor reading stuck at ‘0’– Inconsistent – users see differently. Inconsistent – users see differently.
E.g. sensor reading floats (say between E.g. sensor reading floats (say between 1-3V, and could be seen as ‘1’ or ‘0’). 1-3V, and could be seen as ‘1’ or ‘0’).
Called malicious or Called malicious or Byzantine failuresByzantine failures
Classification of Classification of failuresfailures EffectsEffects
– Benign – not serious e.g. broken tvBenign – not serious e.g. broken tv– Malign – serious e.g. plane crashMalign – serious e.g. plane crash
OftennessOftenness– Permanent – broken equipmentPermanent – broken equipment– Transient – lose wire, processors Transient – lose wire, processors
under stress (EMI, power supply, under stress (EMI, power supply, radiation)radiation)
– Transient occurs a lot more often!Transient occurs a lot more often!
Example of transient Example of transient failurefailure From report on fire control radar of From report on fire control radar of
F-16 fighters [3]F-16 fighters [3]– Pilot noticed malfunctions every 6 hrsPilot noticed malfunctions every 6 hrs– Pilot requested maintenance every 31 Pilot requested maintenance every 31
hrshrs– 1/3 of requests can be reproduced in 1/3 of requests can be reproduced in
workshopworkshop– Overall less than 10% of transient Overall less than 10% of transient
failures can be reproduced!failures can be reproduced!
Types of errorsTypes of errors
Transient Transient – Regularly occurs. E.g. electrical Regularly occurs. E.g. electrical
glitches causes temporary value glitches causes temporary value errorerror
PermanentPermanent– Transient fault can be kept in Transient fault can be kept in
database, making it permanent.database, making it permanent.
Classifications of Classifications of faultsfaults NatureNature
– By chance – broken wireBy chance – broken wire– Intentional – virusIntentional – virus
PerceptionPerception– PhysicalPhysical– DesignDesign
BoundaryBoundary– Internal – component breakdownInternal – component breakdown– External – EMI causes faultsExternal – EMI causes faults
Classifications of Classifications of faultsfaults OriginOrigin
– Development e.g. in program or deviceDevelopment e.g. in program or device– Operation e.g. user entering wrong Operation e.g. user entering wrong
inputinput PersistencePersistence
– Transient – glitches caused by lightningTransient – glitches caused by lightning– Permanent faults that need repairPermanent faults that need repair
Definitions Definitions
Reliability R(t)Reliability R(t)– Probability that a system will perform its intended Probability that a system will perform its intended
function in the specified environment up to time t.function in the specified environment up to time t.
Maintainability M(t)Maintainability M(t)– Probability that a system can be restored within t Probability that a system can be restored within t
units after a failure.units after a failure.
Availability A(t)Availability A(t)– Probability that a system is available to perform Probability that a system is available to perform
the specified service at tthe specified service at tdtdt. (% of system working). (% of system working)
Reliability [4]Reliability [4]
> R(0) = 1, R(R(0) = 1, R(> Failure density f(t) = -dR(t)/dtFailure density f(t) = -dR(t)/dt> Failure rate Failure rate (t)(t) = f(t)/R(t) = f(t)/R(t) (t) (t) dt dt is the conditional probability is the conditional probability
that a system will fail in the interval that a system will fail in the interval dt, provided it has been operational dt, provided it has been operational at the beginning of this interval at the beginning of this interval
> When When (t) = constant then R(t) = e(t) = constant then R(t) = e--tt
= MTTF = MTTF (Mean Time to Failure)(Mean Time to Failure)
Failure rateFailure rate
(t)
Real-time
Period of constant Failure Rate
Earlyfaillures
Latefaillures
Burn-in Wear-out
Failure rate vs Costs Failure rate vs Costs [4][4]
(t)
Cost of System
US Air Force:Failure rate of electronic systemswithin a given technologyincreases with increasing system cost.
1919
MaintainabiMaintainabilitylity
> Mesured by Repair-rate Mesured by Repair-rate > When When (t) = constant then M(t) = e(t) = constant then M(t) = e--tt
= MTTR = MTTR (Mean Time to Repair)(Mean Time to Repair)> Preventive maintenace:Preventive maintenace:
– If If increases in time, then it makes increases in time, then it makes sense to replace the aging unit.sense to replace the aging unit.
– If If of different units evolves of different units evolves differently, preventive maintenace differently, preventive maintenace consists in replacing the “Smallest consists in replacing the “Smallest Replaceable Units” with growing Replaceable Units” with growing
2020
Reliability vs. Reliability vs. MaintainabilityMaintainability> Reliability and maintainability are, to Reliability and maintainability are, to
a certain extent, conflicting goals.a certain extent, conflicting goals.> Example: ConnectorsExample: Connectors
> Inside a SRU, reliability must be Inside a SRU, reliability must be optimizedoptimized
> Between SRU’s, maintainability is Between SRU’s, maintainability is importantimportant
PlugPlugSolderSolder
ReliabilityReliability
badbadgoodgood
MaintainabilityMaintainability
goodgoodbadbad
2121
AvailabiliAvailabilityty> A = MTTF / ( MTTF + MTTR )A = MTTF / ( MTTF + MTTR )
> Good availability can be achieved Good availability can be achieved eithereither– by a high MTTFby a high MTTF– by a small MTTRby a small MTTR
A high system MTTF can be achieved A high system MTTF can be achieved by means of fault tolerance: the by means of fault tolerance: the system continues to operate properly system continues to operate properly even when some components have even when some components have failed.failed.
Fault tolerance reduces also the MTTR Fault tolerance reduces also the MTTR requirements.requirements.
2222
Fault toleranceFault toleranceobtained through obtained through redundancyredundancy(more resources assigned to a task than (more resources assigned to a task than strictly required)strictly required)
REDUNDANCYREDUNDANCY can be used forcan be used for
– Fault detectionFault detection– Fault correctionFault correction
can be implemented at various can be implemented at various levelslevels– at component levelat component level– at processor levelat processor level– at system levelat system level
2323
RedundancyRedundancy at component at component level level
Error detection/correction in memoriesError detection/correction in memories
Error detection by parity bit.Error detection by parity bit.
Error correction by multiple parity bits.Error correction by multiple parity bits.
2424
RedundancyRedundancy at component at component level level Stripe Sets with Parity (RAID)Stripe Sets with Parity (RAID)
Disk 1Disk 1 Disk 3Disk 3Disk 2Disk 2
= XOR of two other disks= XOR of two other disks
2525
RedundancyRedundancy at component at component level level
Error detection in an ALUError detection in an ALU
ALUALU
proofproofby 9by 9 Error !Error !
2626
Redundancy in Redundancy in componentscomponents Error detectionError detection– to correct transient errors by retryto correct transient errors by retry– to avoid using corrupted datato avoid using corrupted data
Error correctionError correction– to correct transient errors on the to correct transient errors on the
flyfly– to remain operational after to remain operational after
catastrophic component failurecatastrophic component failure– Scheduled maintenance instead of Scheduled maintenance instead of
urgent repair.urgent repair.
2727
Fault detection at Fault detection at Processor LevelProcessor LevelCC
PPUU11
CCPPUU22
==
ErrorError
2828
Fault correction at Fault correction at Processor LevelProcessor Level
CCPPUU11
CCPPUU33
CCPPUU22
Voting LogicVoting Logic
2929
Replica Replica DeterminismDeterminism A set of replicated RT objects is A set of replicated RT objects is
“replica determinate” if all “replica determinate” if all objects of this set visit the same objects of this set visit the same state state at about the same timeat about the same time..
““At about the same time” At about the same time” makes a makes a concession to the finite precision concession to the finite precision of the clock synchronizationof the clock synchronization
Replica determinism is needed forReplica determinism is needed for– consistent distributed actionsconsistent distributed actions– fault tolerance by active redundancy fault tolerance by active redundancy
3030
Replica Replica DeterminismDeterminism Lack of replica determinism makes Lack of replica determinism makes
voting meaningless.voting meaningless. Example: Airplane on takeoffExample: Airplane on takeoff
Lack of replica determinism Lack of replica determinism causes the faulty channel to causes the faulty channel to win !!!win !!!
System 1:System 1:System 2:System 2:System 3:System 3:
Majority:Majority:
Take offTake offAbortAbort
Take offTake off
Take offTake off
Accelerate EngineAccelerate EngineStop EngineStop Engine
Stop Engine (fault)Stop Engine (fault)
Stop EngineStop Engine
3131
Fault Correction at System Fault Correction at System LevelLevelHot Stand-ByHot Stand-By
SSYYSSTTEEMM11
SSYYSSTTEEMM22
Error DetectionError Detection
3232
Fault Correction at System Fault Correction at System LevelLevelCold Stand-ByCold Stand-By
SSYYSSTTEEMM11
SSYYSSTTEEMM22
Error DetectionError Detection
Common MemoryCommon Memory
3333
Fault Correction at System Fault Correction at System LevelLevelDistributed Common Distributed Common MemoryMemory
SSYYSSTTEEMM11
SSYYSSTTEEMM22
Distributed Common MemoryDistributed Common Memory
In fact, each processor has access to theIn fact, each processor has access to thememory of the other to keep a copy of thememory of the other to keep a copy of the
state of all critical processesstate of all critical processes
Error DetectionError Detection
3434
Fault Correction at System Fault Correction at System LevelLevelLoad SharingLoad Sharing
Common MemoryCommon Memory
SSYYSSTTEEMM11
SSYYSSTTEEMM11
SSYYSSTTEEMM11
SSYYSSTTEEMM11
3535
Safety Critical Safety Critical systemssystems
SSYYSS11
Voting LogicVoting Logic
SSYYSS22
SSYYSS44
SSYYSS33
Fail once, still operational, fail twice, still safe.Fail once, still operational, fail twice, still safe.
3636
Safety Critical Safety Critical SystemsSystems
ButBut
What happens in case What happens in case
of a Software Bug ???of a Software Bug ???
3737
Space Shuttle Computer Space Shuttle Computer systemsystem
SSYYSS11
Voting LogicVoting Logic
SSYYSS22
SSYYSS44
SSYYSS33
SSYYSS55
ReferencesReferences
1.1. Ebeling C, An introduction to reliability and Ebeling C, An introduction to reliability and maintainability engineering, McGraw-Hill, maintainability engineering, McGraw-Hill, 19971997
2.2. Krishna C, Real-time systems, McGraw-Hill, Krishna C, Real-time systems, McGraw-Hill, 19971997
3.3. Kopetz H, Real-time systems design Kopetz H, Real-time systems design principles for distributed embedded principles for distributed embedded applications, Kluwer, 1997applications, Kluwer, 1997
4.4. Tiberghien J, Real-time system fault Tiberghien J, Real-time system fault tolerance, Lecture slidestolerance, Lecture slides
top related