Towards Target-Level Testing and Debugging Tools For Embedded Software

Harry Koehnemann, Arizona State University

Dr. Timothy Lindquist, Arizona State University

Presented By:Sidharth Kumar

CSCI 589October 9th, 2007

Summary of Paper IntroductionIntroduction Software TestingSoftware Testing Embedded TestingEmbedded Testing Problems with Embedded TestingProblems with Embedded Testing Increasing Target FunctionalityIncreasing Target Functionality Architectural additionsArchitectural additions Run-Time System AdditionsRun-Time System Additions Strengths of the PaperStrengths of the Paper Weaknesses of the PaperWeaknesses of the Paper ConclusionsConclusions

The Race Begins….The Race Begins….

22

Introduction

Software Validation & TestingSoftware Validation & Testing Goal: To Identify and address the problems Goal: To Identify and address the problems

associated with test case execution for Embedded associated with test case execution for Embedded SystemsSystems

To propose solutions for making embedded testing To propose solutions for making embedded testing more effective at revealing errors.more effective at revealing errors.

Introduction – Cont. Testing:Testing: Executing a piece of software in order to Executing a piece of software in order to

reveal errorsreveal errors Development of test cases, generation and Development of test cases, generation and

executionexecution

Debugging: Debugging: Locating and correcting the cause of an error Locating and correcting the cause of an error

once it has been revealedonce it has been revealed Developer must recreate exact execution Developer must recreate exact execution

scenarioscenario All environmental variants must be accounted All environmental variants must be accounted

forfor

Introduction – Cont.

Constraints: Concurrent Designs Real-time constraints Embedded target environments Distributed hardware architectures Device control dependencies

Restriction of execution visibility and control

Target environment shortcomings

Embedded Systems

Software Testing

Module level testingModule level testing Integration testingIntegration testing System testingSystem testing Hardware/Software Integration testingHardware/Software Integration testing

The fourth phase is unique to embedded The fourth phase is unique to embedded systems.systems.

“If you can't make it good, at least make it look good.” – Bill Gates

Testing Concurrent Systems Concurrency increases the difficulty of software testingConcurrency increases the difficulty of software testing Unmanageably large set of legal execution sequences Unmanageably large set of legal execution sequences

that a concurrent program may takethat a concurrent program may take Subsequent execution with same input may yield Subsequent execution with same input may yield

different-yet correct resultsdifferent-yet correct results

Non-intrusive testing Embedded applications have strict timing requirementsEmbedded applications have strict timing requirements No intrusions on a test executionNo intrusions on a test execution InstrumentationInstrumentation Real time Embedded tools: ROM monitors, Emulators, Real time Embedded tools: ROM monitors, Emulators,

Bus MonitorsBus Monitors

Software Testing – Cont.

Impact of the Underlying System

Dealing with Abstraction Task objects enable high level communication With abstraction, implementation details become

buried

Software Testing – Cont.

Language Constructs

Hardware

TimeC

omplexity

Embedded Testing

Developed on custom hardware configurations, Developed on custom hardware configurations,

Tools and techniques of one not applicable on Tools and techniques of one not applicable on anotheranother

Ad Hoc approach to integration and system Ad Hoc approach to integration and system testingtesting

Environments: Host and the Target. Target has Environments: Host and the Target. Target has little support for software development tools.little support for software development tools.

Embedded Testing – Cont.

Current state of Embedded Testing

Incorrect handling of interrupts Distributed communication problems Incorrect ordering of concurrent events Resource contention Incorrect use of device protocols and timing Incorrect response to failures or transients

As space-system software grows in size and complexity, adequate testing becomes more difficult—and more critical.

Embedded Testing – Cont.

Hardware Solutions: Hardware Solutions: Attempt to gain execution visibility and program control.Attempt to gain execution visibility and program control. Bus monitors, ROM monitors, in-circuit emulators.Bus monitors, ROM monitors, in-circuit emulators. Minimal effectiveness for software development, Minimal effectiveness for software development,

Information gathering on low level machine dataInformation gathering on low level machine data

Software Solutions: Software Solutions: Attempts to reduce costs of testing on target.Attempts to reduce costs of testing on target. Factors: Level of criticality, Test platform availability, Factors: Level of criticality, Test platform availability,

test classification etc.test classification etc. Lead to extensive modifications and therefore hence Lead to extensive modifications and therefore hence

extensive retestingextensive retesting

Current Solutions

Problems with Embedded Testing

Expense of testing process

Level of functionality on target

Late discovery of errors

Poor test selection criteria

Potential use in advancing architectures

ncreasing Target Functionality

Tool support for embedded systems is Lacking.Tool support for embedded systems is Lacking. Current approaches will soon be obsolete for future Current approaches will soon be obsolete for future

architecturesarchitectures Add facilities to the underlying system to better Add facilities to the underlying system to better

support testing and debugging toolssupport testing and debugging tools

Underlying system: Hardware architecture and RTSUnderlying system: Hardware architecture and RTS

I

Increasing Target Functionality – Cont.

Model Debugging System

Symbol table information

At least two processes executing: Test program and the Debugger.

To realize non-intrusion: Execute code only at break point Run debugger as a separate process Provide separate execution unit to execute debugger

Architectural Additions

Hardware partitioning of memory: Each process should Each process should have its own protected address space that is not have its own protected address space that is not accessible to any other process, shared memory concept.accessible to any other process, shared memory concept.

Computational facilities for Debugger: The internal The internal debugger on the target processor runs as a regular debugger on the target processor runs as a regular process or architecture support be provided for process or architecture support be provided for execution.execution.

Hardware Break Points: Software break points are Software break points are intrusive and require more computation. Hardware intrusive and require more computation. Hardware Breakpoint Registers (BPR) checked by processor Breakpoint Registers (BPR) checked by processor against the operands of each instruction.against the operands of each instruction.

Architectural Support for Abstractions: Abstractions to be Abstractions to be incorporated into the instruction sets, architecture must be the incorporated into the instruction sets, architecture must be the basis for emulation capabilities and providing execution basis for emulation capabilities and providing execution visibility.visibility.

Dedicated Bus: Interface that allows the processor to Interface that allows the processor to communicate with the external world without interfering with communicate with the external world without interfering with the system under test. the system under test.

Architectural Additions – Cont.

System Additions

RTS requirements describe an interface between a tool and RTS requirements describe an interface between a tool and the underlying system: logical interface requiring the underlying system: logical interface requiring substantial hardware support.substantial hardware support.

Goal is to minimize the data and computational Goal is to minimize the data and computational requirements of the internal debugger as well as the requirements of the internal debugger as well as the required communication between the internal and external required communication between the internal and external debuggers.debuggers.

MRTSI and CIFO implement the abstractions.MRTSI and CIFO implement the abstractions.

Architectural additions are costly in time and space

-

The CIFO/MRTSI provide The CIFO/MRTSI provide abstractionsabstractions for: for:

Run-Time System Additions – Cont.

RTS additions are minimal

Processes Interrupt Management Time Management Memory Management Exception/Fault Handling

Conclusions

RTS and Architectural additions propose a solution to the problems in embedded system testing.

Testing and Debugging of embedded, real time software remains a “black art”, with ad hoc methods and techniques.

Evaluation all additions and determine their feasibility.

Strengths of Paper

Focus on the need for detecting Focus on the need for detecting errors earlyerrors early

Improvement in architectural Improvement in architectural capabilities for better performance of capabilities for better performance of Embedded SoftwareEmbedded Software

Emphasis on improving testing Emphasis on improving testing efficiency on target environmentefficiency on target environment

Weakness of Paper

Safety critical aspects on target environment Safety critical aspects on target environment

Additions improve but till what extent? Additions improve but till what extent?

When to stop Target-Level testing ?When to stop Target-Level testing ?

From the Author’s Desk…..

No, we did not address anything specific to Safety Critical systems. Our goal was to open an Real-Time OS (RTOS) (really back then an Ada RTS) to provide testing and debugging tools control of the execution. What those tests were was irrelevant to us. Safety Critical analysis would produce interesting test scenarios that would require some target-level control to obtain the right system states, and might require something like we proposed in our paper.

Regarding b) The architecture additions came about from work I was doing at Intel at the time. In practice none of those ideas ever became reality.

Regarding c) I think we are talking about different types of systems. The embedded systems we describe are targeted for devices with little external visibility to their execution - embedded controllers, cell phones, etc. Traditional desktop software do not have these problems. Target for us means an embedded board with limited debugging capabilities. FYI, since that time, the embedded community has made great strides. The RTOS vendors have consolidated to 2 - Grean Hills/VxWorks - and both provide very good target level debugging capabilities.

Harry Harry

References

http://csse.usc.edu/classes/cs589_2007/http://csse.usc.edu/classes/cs589_2007/ http://www.aero.org/publications/crosslink/http://www.aero.org/publications/crosslink/

fall2005/06.htmlfall2005/06.html http://www.embedded.com/http://www.embedded.com/

2000/0011/0011feat1.htm2000/0011/0011feat1.htm http://www.ece.cmu.edu/~koopman/des_s99/http://www.ece.cmu.edu/~koopman/des_s99/

sw_testing/presentation.pdfsw_testing/presentation.pdf www.google.comwww.google.com

The Race Never Ends…..The Race Never Ends…..