FAULT INJECTION TECHNIQUES AND TOOLS FOR EMBEDDED SYSTEMSRELIABILITY EVALUATION

FRONTIERS IN ELECTRONIC TESTINGConsulting Editor

Vishwani D. AgrawalBooks in the series:

Fault Injection Techniques and Tools for Embedded Systems ReliabilityEvaluation

A. Benso & P. PrinettoISBN: 1-4020-7589-8

High Performance Memory Memory TestingR. Dean AdamsISBN: 1-4020-7255-4

SOC (System-on-a-Chip) Testing for Plug and Play Test AutomationK. ChakrabartyISBN: 1-4020-7205-8

Test Resource Partitioning for System-on-a-ChipK. Chakrabarty, Iyengar & ChandraISBN: 1-4020-7119-1

A Designers’ Guide to Built-in Self-TestC. StroudISBN: 1-4020-7050-0

Boundary-Scan Interconnect DiagnosisJ. de Sousa, P.CheungISBN: 0-7923-7314-6

Essentials of Electronic Testing for Digital, Memory, and Mixed Signal VLSI CircuitsM.L. Bushnell, V.D. AgrawalISBN: 0-7923-7991-8

Analog and Mixed-Signal Boundary-Scan: A Guide to the IEEE 1149.4 Test StandardA. OsseiranISBN: 0-7923-8686-8

Design for At-Speed Test, Diagnosis and MeasurementB. Nadeau-DostiISBN: 0-79-8669-8

Delay Fault Testing for VLSI CircuitsA. Krstic, K-T. ChengISBN: 0-7923-8295-1

Research Perspectives and Case Studies in System Test and DiagnosisJ.W. Sheppard, W.R. SimpsonISBN: 0-7923-8263-3

Formal Equivalence Checking and Design DebuggingS.-Y. Huang, K.-T. ChengISBN: 0-7923-8184-X

Defect Oriented Testing for CMOS Analog and Digital CircuitsM. SachdevISBN: 0-7923-8083-5

Reasoning in Boolean Networks: Logic Synthesis and Verification Using Testing TechniquesW. Kunz, D. StoffelISBN: 0-7923-9921-8

Introduction toS. Chakravarty, P.J. ThadikaranISBN: 0-7923-9945-5

Multi-Chip Module Test StrategiesY. ZorianISBN: 0-7923-9920-X

Testing and Testable Design of High-Density Random-Access MemoriesP. Mazumder, K. ChakrabortyISBN: 0-7923-9782-7

From Contamination to Defects, Faults and Yield LossJ.B. Khare, W. MalyISBN: 0-7923-9714-2

FAULT INJECTION TECHNIQUESAND TOOLS FOR EMBEDDED SYSTEMSRELIABILITY EVALUATION

Edited by

ALFREDO BENSOPolitecnico di Torino, Italy

and

PAOLO PRINETTOPolitecnico di Torino, Italy

KLUWER ACADEMIC PUBLISHERSNEW YORK, BOSTON, DORDRECHT, LONDON, MOSCOW

eBook ISBN: 0-306-48711-XPrint ISBN: 1-4020-7589-8

©2004 Springer Science + Business Media, Inc.

Print ©2003 Kluwer Academic Publishers

All rights reserved

No part of this eBook may be reproduced or transmitted in any form or by any means, electronic,mechanical, recording, or otherwise, without written consent from the Publisher

Created in the United States of America

Visit Springer's eBookstore at: http://www.ebooks.kluweronline.comand the Springer Global Website Online at: http://www.springeronline.com

Dordrecht

Contents

Contributing Authors

Preface

Acknowledgments

PART 1: A FIRST LOOK AT FAULT INJECTION

Chapter 1.1: FAULT INJECTION TECHNIQUESIntroduction

The Metrics of DependabilityDependability FactorsFault Category

Fault SpaceHardware/Physical FaultSoftware Fault

Statistical Fault Coverage EstimationForced CoverageFault Coverage Estimation with One-SidedConfidence IntervalMean Time To Unsafe Failure (MTTUF)[SMIT_00]

An Overview of Fault InjectionThe History of Fault InjectionSampling ProcessFault Injection Environment [HSUE_97]

1.1.11.21.3

1.3.11.3.21.3.3

1.41.4.11.4.2

1.4.3

2.2.12.22.3

xiii

1

3

5

7789

101011121314

16

1718192020

vi FAULT INJECTION TECHNIQUES AND TOOLS FOREMBEDDED SYSTEMS RELIABILITY EVALUATION

Quantitative Safety Assessment ModelThe FARM Model

Levels of Abstraction of Fault InjectionThe Fault Injection Attributes

Hardware-based Fault InjectionAssumptionsAdvantagesDisadvantagesTools

Software-based Fault InjectionAssumptionsAdvantagesDisadvantagesTools

Simulation-based Fault InjectionAssumptionsAdvantagesDisadvantagesTools

Hybrid Fault InjectionTools

Objectives of Fault InjectionFault Removal [AVRE_92]Fault Forecasting [ARLA_90]

Further ResearchesNo-Response FaultsLarge Number of Fault Injection ExperimentsRequired

Chapter 1.2: DEPENDABILITY EVALUATION METHODSTypes of Dependability Evaluation MethodsDependability Evaluation by AnalysisDependability Evaluation by Field ExperienceDependability Evaluation by Fault Injection TestingConclusion and outlook

Chapter 1.3: SOFT ERRORS ON DIGITAL COMPONENTSIntroductionSoft Errors

Radiation Effects (SEU, SEE)SER measurement and testingSEU and technology scaling

2.42.5

2.5.12.5.2

3.3.13.23.33.4

4.4.14.24.34.4

5.5.15.25.35.4

6.6.1

7.7.17.2

8.8.18.2

1.2.3.4.5.

1.2.

2.12.22.3

2124252528292930303132323233333334343435353536373738

39

414142454647

494951515354

FAULT INJECTION TECHNIQUES AND TOOLS FOREMBEDDED SYSTEMS RELIABILITY EVALUATION

vii

Trends in DRAMs, SRAMs and FLASHsTrends in Combinational Logic andMicroprocessorTrends in FPGA

Other sources of Soft ErrorsProtection Against Soft Errors

Soft Error avoidanceSoft Error removal and forecastingSoft Error tolerance and evasionSOC Soft Error tolerance

Conclusions

PART 2: HARDWARE-IMPLEMENTED FAULT INJECTION

Chapter 2.1: PIN-LEVEL HARDWARE FAULT INJECTIONTECHNIQUES

IntroductionState of the Art

Fault injection methodologyFault injectionData acquisitionData processing

Pin-level fault injection techniques and toolsThe Pin Level FI FARM model

Fault model setActivation setReadouts SetMeasures set

Description of the Fault Injection ToolAFIT – Advanced Fault Injection ToolThe injection process: A case study

System DescriptionThe injection campaignExecution time and overhead

Critical Analysis

Chapter 2.2: DEVELOPMENT OF A HYBRID FAULT INJECTIONENVIRONMENT

Dependability Testing and Evaluation of Railway ControlSystemsBirth of a Validation EnvironmentThe Evolution of “LIVE”

2.3.12.3.2

2.3.32.4

3.3.13.23.33.4

4.

1.2.

2.12.1.12.1.22.1.3

2.23.

3.13.23.33.4

4.4.14.2

4.2.14.2.24.2.3

5.

1.

2.3.

54

555556575757585859

61

6363646464656565666767676868687373747778

81

818286

viii FAULT INJECTION TECHNIQUES AND TOOLS FOREMBEDDED SYSTEMS RELIABILITY EVALUATION

Two examples of automationExample applicationConclusions

Chapter 2.3: HEAVY ION INDUCED SEE IN SRAM BASEDFPGAS

IntroductionExperimental Set UpSEEs in FPGAs

SEU and SEFISupply current increase: SEL?SEU in the configuration memory

Conclusions

PART 3: SOFTWARE-IMPLEMENTED FAULT INJECTION

Chapter 3.1: “BOND”: AN AGENTS-BASED FAULT INJECTORFOR WINDOWS NT

The target platformInterposition Agents and Fault InjectionThe BOND Tool

General Architecture: the Multithreaded InjectionThe Logger Agent

Fault Injection Activation EventFault Effect Observation

The Fault Injection AgentFault locationFault typeFault durationThe Graphical User Interface

Experimental Evaluation of BONDWinzip32Floating Point Benchmark

Conclusions

Chapter 3.2: XCEPTION™ : A SOFTWARE IMPLEMENTEDFAULT INJECTION TOOL

IntroductionThe Xception Technique

The FARM model in XceptionFaultsActivations

3.14.5.

1.2.3.

3.13.23.3

4.

1.2.3.

3.13.2

3.2.13.2.2

4.4.14.24.34.4

5.5.15.2

6.

1.2.

2.12.1.12.1.2

889293

9595969999

103106107

109

111111112113114115115117117117118119119120121122123

125125126127127128

FAULT INJECTION TECHNIQUES AND TOOLS FOREMBEDDED SYSTEMS RELIABILITY EVALUATION

ix

ReadoutsMeasures

The XCEPTION TOOLSETArchitecture and key features

The Experiment Manager Environment (EME)On the target sideMonitoring capabilitiesDesigned for portability

Extended XceptionFault definition made easyXtract – the analysis toolXception™ on the field – a selected case study

Experimental setupResults

Critical AnalysisDeployment and development timeTechnical limitations of SWIFI and Xception

Chapter 3.3: MAFALDA: A SERIES OF PROTOTYPE TOOLSFOR THE ASSESSMENT OF REAL TIME COTSMICROKERNEL-BASED SYSTEMS

IntroductionOverall Structure of MAFALDA-RTFault Injection

Fault models and SWIFICoping with the temporal intrusiveness of SWIFI

Workload and ActivationSynthetic workloadReal time application

Readouts and MeasuresAssessment of the behavior in presence of faultsTargeting different microkernels

Lessons Learnt and Perspectives

PART 4: SIMULATION-BASED FAULT INJECTION

Chapter 4.1: VHDL SIMULATION-BASED FAULT INJECTIONTECHNIQUES

IntroductionVHDL Simulation-Based Fault Injection

Simulator Commands TechniqueModifying the VHDL Model

2.1.32.1.4

3.3.1

3.1.13.1.23.1.33.1.4

3.23.33.43.5

3.5.13.5.2

4.4.14.2

1.2.3.

3.13.2

4.4.14.2

5.5.15.2

6.

1.2.

2.12.2

129129129130131131132133133134134135136136138138138

141141143145146147149149150151151153155

157

159159160161162

x FAULT INJECTION TECHNIQUES AND TOOLS FOREMBEDDED SYSTEMS RELIABILITY EVALUATION

Saboteurs TechniqueMutants Technique

Other TechniquesFault ModelsDescription of VFIT

General FeaturesInjection PhasesBlock diagram

Experiments of Fault Injection: Validation of a Fault TolerantMicrocomputer SystemConclusions

Chapter 4.2: MEFISTO: A SERIES OF PROTOTYPE TOOLSFOR FAULT INJECTION INTO VHDL MODELS

IntroductionMEFISTO-L

Structure of the ToolThe Fault AttributeThe Activation AttributeThe Readouts and MeasuresApplication of MEFISTO-L for Testing FTMs

MEFISTO-CStructure of the ToolReducing the Cost of Error Coverage Estimation byCombining Experimental and Analytical TechniquesUsing MEFISTO-C for Assessing Scan-ChainImplemented Fault Injection

Some Lessons Learnt and Perspectives

Chapter 4.3: SIMULATION-BASED FAULT INJECTION ANDTESTING UNSING THE MUTATION TECHNIQUE

Fault Injection Technique: Mutation TestingIntroductionMutation TestingDifferent mutations

Weak mutationFirm mutationSelective mutation

Test generation based on mutationFunctional testing method

MotivationsMutation testing for hardware

2.2.12.2.2

2.33.4.

4.14.24.3

5.

6.

1.2.

2.12.22.32.42.5

3.3.13.2

3.3

4.

1.1.11.21.3

1.3.11.3.21.3.3

1.41.5

1.5.11.5.2

162164167167168168169170

173176

177177178179181182183184185185

187

189191

195195195196199199200200201203203203

FAULT INJECTION TECHNIQUES AND TOOLS FOREMBEDDED SYSTEMS RELIABILITY EVALUATION

xi

The Alien ToolThe implementation tool

General presentation of the toolALIEN detailed description

Experimental workBefore enhancement of test dataAfter enhancement of test dataComparison with the classical ATPGs

ConclusionApproach robustness

Robustness with regard to the differenthardware implementationsRobustness with regard to the differenthardware fault models

Limitations and Reusability

Chapter 4.4: NEW ACCELERATION TECHNIQUES FORSIMULATION-BASED FAULT-INJECTION

IntroductionRT-Level Fault-Injection CampaignFault Injection

Checkpoints and SnapshotEarly stopHyperactivitySmart resumeDynamic Equivalencies

Workload Independent Fault CollapsingWorkload Dependent Fault CollapsingDynamic Fault CollapsingExperimental ResultsConclusions

References

2.2.1

2.1.12.1.2

2.22.2.12.2.22.2.3

3.3.1

3.1.1

3.1.2

3.2

1.2.3.

3.13.23.33.43.5

4.5.6.7.8.

207207207208210211212212213213

213

214214

217217219221221222223223224224225226227229

231

Contributing Authors

Joakim Aidemark, Chalmers Univ. of Technology, Göteborg, SwedenJean Arlat, LAAS-CNRS, Toulouse, FranceAndrea Baldini, Politecnico di Torino, Torino, ItalyJuan Carlos Baraza, Università Polytecnica de Valencia, SpainMarco Bellato, INFN, Padova, ItalyAlfredo Benso, Politecnico di Torino, Torino, ItalySara Blanc, Università Polytecnica de Valencia, SpainJérome Boué, LAAS-CNRS, Toulouse, FranceJoao Carreira, Critical Software SA, Coimbra, PortugalMarco Ceschia, Università di Padova, Padova, ItalyFulvio Corno, Politecnico di Torino, Torino, ItalyDiamantino Costa, Critical Software SA, Coimbra, PortugalYves Crouzet, LAAS-CNRS, Toulouse, FranceJean-Charles Fabre, LAAS-CNRS, Toulouse, FranceLuis Entrena, Universitad Carlos III, Madrid, SpainPeter Folkesson, Chalmers Univ. of Technology, Göteborg, SwedenDaniel Gil, Università Polytecnica de Valencia, SpainPedro Joaquín Gil, Università Polytecnica de Valencia, SpainJoaquín Gracia, Università Polytecnica de Valencia, SpainLeonardo Impagliazzo, Ansaldo Segnalamento Ferroviario, Napoli, ItlayEric Jenn, LAAS-CNRS, Toulouse, FranceBarry W. Johnson, University of Virginia, VA, USAJohan Karlsson, Chalmers Univ. of Technology, Göteborg, SwedenCelia Lopez, Universitad Carlos III, Madrid, SpainTomislav Lovric, TÜV InterTraffic GmbH, Köln, GermanyHenrique Madeira, University of Coimbra,Portugal

xiv FAULT INJECTION TECHNIQUES AND TOOLS FOREMBEDDED SYSTEMS RELIABILITY EVALUATION

Riccardo Mariani, Yogitech SpA, Pisa, ItalyJoakim Ohlsson, Chalmers Univ. of Technology, Göteborg, SwedenAlessandro Paccagnella, Università di Padova, Padova, ItalyFabiomassimo Poli, Ansaldo Segnalamento Ferroviario, Napoli, ItlayPaolo Prinetto, Politecnico di Torino, Torino, ItalyMarcus Rimén, Chalmers Univ. of Technology, Göteborg, SwedenChantal Robach, LCIS-ESISAR, Valence, FranceManuel Rodríguez, LAAS-CNRS, Toulouse, FranceFrédéric Salles, LAAS-CNRS, Toulouse, FranceMathieu Scholive, LCIS-ESISAR, Valence, FranceJuan José Serrano, Università Polytecnica de Valencia, SpainJoao Gabriel Silva, University of Coimbra,PortugalMatteo Sonza Reorda, Politecnico di Torino, Torino, ItalyGiovanni Squillero, Politecnico di Torino, Torino, ItalyYangyang Yu, Univ. of Virginia, VA, USA

Preface

The use of digital systems pervades all areas of our lives, from commonhouse appliances such as microwave ovens and washing machines, tocomplex applications like automotive, transportations, and medical controlsystems. These digital systems provide higher productivity and greaterflexibility, but it is also accepted that they cannot be fault-free. Some faultsmay be attributed to inaccuracy during the development, while others canstem from external causes such as production process defects orenvironmental stress. Moreover, as devices geometry decreases and clockfrequencies increase, the incidence of transient errors increases, andconsequently, the dependability of the systems decreases. High reliability istherefore a requirement for every digital system whose correct functionalityis connected to human safety or economic investments.

In this context, the evaluation of the dependability of a system plays acritical role. Unlike performance, dependability cannot be evaluated usingbenchmark programs and standard test methodologies, but only observingthe system behavior after the appearance of a fault. However, since theMean-Time-Between-Failures (MTBF) in a dependable system can be of theorder of years, the fault occurrence has to be artificially accelerated in orderto analyze the system reaction to a fault, without waiting for its naturalappearance.

Fault Injection emerged as a viable solution, and it has been deeplyinvestigated and exploited by both academia and industry. Differenttechniques have been proposed and used to perform experiments. They canbe grouped in Hardware-implemented, Software-implemented, andSimulation-based Fault Injection.

2 FAULT INJECTION TECHNIQUES

The process of setting up a Fault Injection environment requires differentchoices that can deeply influence the coherency and the meaningfulness ofthe final results. In this book we tried to collect some of the most significantcontributions in the field of Fault Injection. The selection process has beenvery difficult, with the result that a lot of excellent works had to be left out.The criteria we used to select the contributing authors were based on theinnovation of the proposed solution, on the historical significance of theirwork, and also on an effort to give the readers a global overview of thedifferent problems and techniques that can be applied to setup a FaultInjection experiment.

The book is therefore organized in four different parts. The first part ismore general, and motivates the use of Fault Injection techniques. The otherthree parts cover Hardware-based, Software-implemented, and Simulation-based Fault Injection techniques, respectively. In each of these parts threeFault Injection methodologies and related tools are presented and discussed.The last chapter of Part 4 discusses possible solutions to speed-upSimulation-based Fault Injection experiments, but the main guidelineshighlighted in the chapter can be applicable to other Fault Injectiontechniques as well.

Alfredo Bensoalfredo.benso@polito.it

Paolo Prinettopaolo.prinetto@polito.it

Acknowledgments

The editors would like to thank all the contributing authors for theirpatience in meeting our deadlines and requirements. We are also in debt withGiorgio Di Natale, Stefano Di Carlo and Chiara Bessone for their valuablehelp in the tricky task of preparing the camera ready of this book.