an industrial application of cleanroom software ... · 1 an industrial application of cleanroom...

10

Click here to load reader

Upload: lyhanh

Post on 29-Aug-2018

212 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: An Industrial Application of Cleanroom Software ... · 1 An Industrial Application of Cleanroom Software Engineering - Benefits Through Tailoring Robert Oshana Raytheon TI Systems

An Industrial Application of Cleanroom Software Engineering -Benefits Through Tailoring

Robert OshanaRaytheon TI Systems

[email protected]

Abstract

Cleanroom is a set of software engineering principles that support the development of reliable software.The Systems group at Raytheon TI Systems, a SEI level 3 organization, successfully adopted Cleanroominto a pilot CMM level 5 project. The successful introduction of this technology was a result of theprinciples of Cleanroom being based on fundamental computer science foundations. As with any othermethodology, a certain amount of tailoring was required for the technology to be most effective. Ourtailoring approach was based on our project needs, our schedule constraints, and how comfortable we feltwith the various components of the technology. The paper will describe the tailoring approaches used forthe insertion of Cleanroom and the pros and cons of the Cleanroom approach as described by practitionersusing the technology day to day. The result of our approach is a model for software development that wefeel is very effective at producing quality software.

Introduction.

Cleanroom software developmentmethodology is a method of developing softwareunder statistical quality control. It is a theorybased, team oriented engineering process.Cleanroom attempts to develop very high qualitysoftware by preventing errors through theapplication of a sound engineering disciplinerather than focus on error detection and removal.Cleanroom Software Engineering is based uponthe structured programming ideas of focusedthinking and the application of mathematicalreasoning to software in a methodical way [1,2].

Cleanroom Software Engineeringattempts to replace crafting of software withmathematically based software engineeringpractices. Crafting applies human creativity indesigning a solution to a technical problem andoften results in ad hoc solutions to technicalproblems. The processes and practices oftencome from personal experience. In engineering,the problem is solved using practices derivedfrom an appropriate science base. Cleanroomprovides uniform, scalable processes and

1

1060-3425/98 $10.01060-3425/98 $10.0

artifacts drawn from a science base for softwareengineering. These processes and artifactsexhibit properties important for effectivesoftware engineering, and provide a frameworkfor applying human creativity to the problem athand.

The Cleanroom development process isa set of stepwise refinements or transformationsfrom the requirements to the code where eachtransformation is verified to the previous level ofrefinement in order to minimize errors. Errorsare found earlier in the software developmentcycle which minimizes rework and speeds time tomarket.

Historical data shows that theapplication of Software Cleanroom practicesimproves delivered software in the followingways:

1. Improved quality; The Cleanroom processbuilds in quality which reduces the overallcost. The goal is to prevent errors ratherthan accommodating errors.

2. Increased productivity; The improvedquality of the software reduces the timespent in debugging and rework whichincrease productivity and reduces cycle time.

0 (c) 1998 IEEE0 (c) 1998 IEEE

Page 2: An Industrial Application of Cleanroom Software ... · 1 An Industrial Application of Cleanroom Software Engineering - Benefits Through Tailoring Robert Oshana Raytheon TI Systems

3. Improved software maintainability.Software developed using Cleanroomtechniques has clear, well-definedspecifications and a less complicated designwhich allows maintainers to learn andmodify the software faster with fewer errors.

Cleanroom Technology Components.

The main Cleanroom technology components are(Figure 1) [3];

1. Software specification; During thespecification phase, the software engineersuse a strict stepwise refinement andverification process using a box structuredapproach that allows precise definition ofrequired user function and system objectarchitecture. This approach scales tosupport large systems development.

2. Development; The main goal of thedevelopment team is to take a set of softwarespecifications including hardware andhuman behavior components, and design andverify those behaviors using data andprocesses that implement the givenspecifications [4]. Box structures,representing defined system behavior, areused in development. The goal is to developsoftware that can be verified for correctnessagainst its specification using structuredprogramming correctness proofs.

3. Correctness verification; Duringdevelopment the software is verified usingstrict correctness verification methods thatprove that the software meets thespecification. Verification reviews are heldby the team to formally or informally verifythe software using a set of correctness proofsthat are standard in a structuredprogramming environment. This methodallows the verification of programs of anysize by reducing the verification process to amanageable number of independentverification checks. Proof of softwarecorrectness is mostly done by directassertion. Correctness verification is donebefore the software is ever executed, so thedevelopers avoid a “debugging” mode ofoperation.

4. Certification; The Cleanroom softwareengineering methodology focuses oncertifying the product meets requirements

2

1060-3425/98 $10.01060-3425/98 $10.0

and determining the reliability of the productrather than attempting to test quality into theproduct. A separate certification team teststhe product using usage model basedstatistical testing. Usage data is used todevelop Markov models of the system thatallow significant analysis capability both preand post-test, and straightforward test casegeneration. This testing approach providesstatistically valid quantitative measures oftesting progress and software reliability.The quantitative measures provided by theusage-based statistical testing approachprovide mechanisms to aid management indetermining when testing can and should bestopped [5,6]

5. Incremental development; The softwareproduct is developed in a series of functionalincrements that sum to the final deliverableproduct. These increments representoperational user functions. The integrationof these increments is done top down.Incremental development allows earlyassessment of product quality and givescontinuous feedback to management as tothe progress of development. When thefinal increment is integrated and tested, thesoftware product is complete.

SpecificationFunction Usage

IncrementPlanning

Box structure Specification andDesign

Usage Modelingand Test CaseGeneration

StatisticalTesting

Quality Certification Model

FunctionalSpecification

UsageSpecification

IncrementalDevelopmentPlan

SourceCode

TestCases

Failure Data

Measurements ofOperational Performance

Improvement and Feedback

Specification Team

Development Team

Certification Team

Figure 1. Cleanroom SoftwareEngineering process

There were several motivating factorsthat led us to consider incorporating Cleanroommethodology into our process:

0 (c) 1998 IEEE0 (c) 1998 IEEE

Page 3: An Industrial Application of Cleanroom Software ... · 1 An Industrial Application of Cleanroom Software Engineering - Benefits Through Tailoring Robert Oshana Raytheon TI Systems

e

1. Our schedule required a lower defect rategoing into system integration and test. In thepast, we used a strict unit testing and branchcoverage approach to testing. The programspent too much time and money developingthe unit test environment, trying to get everyinstruction to execute, and not understandinghow each module fit into the big picture.

2. Our paradigm for software development wasthat defects should be avoided instead ofdetected and removed. Cleanroom focuseson the same goal.

3. A more formal approach to softwarespecification was desired. Instead of thestandard English language requirementsdocument, more formal and rigorouslydefined requirements were desired. In thepast changing and unstable requirementscaused a lot of rework and uncertainty.Cleanroom specification approaches allowstepwise decomposition and enumeration tomore fully and formally specify softwarerequirements. In the past, our softwarespecification was, effectively, a single unit(document). Customer changes (which seemto be ubiquitous) could easily wreak havocon the specification and developmentprocesses. In the past, we found ourselves ina mode of constantly updating documentsand sending new versions to the customer.In our current approach using incrementaldevelopment and design, increments arespecified and built with a stable set ofrequirements. Requirements changes arefolded into future increments. We arefinding this approach easier on ourselves andthe customer.

4. Any technology improvement must notrequire the software development team to gothrough a lengthy learning phase. We feltthat Cleanroom was an extension to thestructured analysis and structured designtechniques we had used in the past.Cleanroom is not an orthogonal approach toSA/SD, OO or any other methodology. It’ssimply an approach that uses fundamentalcomputer science principles in thedevelopment of software.

5. Cleanroom permits replacement of unittesting with static verification. Although wefelt that some unit testing was necessary, wfelt that static verification in the form ofcode reading and formal peer reviews andcode inspections were cost effective. Wedid not feel comfortable enough using the

3

1060-3425/98 $10.01060-3425/98 $10.0

formal techniques of mathematicalverification to completely replace othertesting techniques.

Project Organization.

Cleanroom has been incorporated intoour mature SEI level program using technologyinsertion guidelines put in place as part of ourSEI level 5 piloting effort [7].

Our program consists of multiplesoftware teams, each developing software at theCSCI (computer software configuration item)level in a teaming environment. Each team isdeveloping their software in a paradigm of“mini” Cleanroom projects. In order to maintainconsistency among the teams with respect toCleanroom artifacts, lessons learned, and bestpractices, a CSCI convergence team wasorganized. This team meets periodically to listento and act on concerns and issues that arise fromthe teams. This convergence team has provedhelpful in keeping all of the teams consistent inthe deployment of the methodology (Figure 2).

The software being developed is real-time embedded and distributed software writtenmainly in C and ADA. The software itselfconsists of control software running on singleboard computers, and algorithmic and digitalsignal processing software running on digitalsignal processors.

Mid way through the implementationphase and the initial certification phase, weassessed the current status of the program usingCleanroom methods. The results of our in housesurvey are described here.

Software Teams Software Teams

Software Convergence Team (Software Team Leads)

Issues and inconsistencies

Resolutions andGuidelines

Resolutions andGuidelines

Figure 2. Software convergence team

0 (c) 1998 IEEE0 (c) 1998 IEEE

Page 4: An Industrial Application of Cleanroom Software ... · 1 An Industrial Application of Cleanroom Software Engineering - Benefits Through Tailoring Robert Oshana Raytheon TI Systems

Survey results.

The survey focused on those Cleanroomprinciples software engineers liked most andthose that they did not see as being useful. Theparticipants were also asked what they thoughtwere the barriers to more ubiquitous Cleanroomuse and what best practices they could take fromthe Cleanroom process. Of the many Cleanroomconcepts we were introduced to, some were moreuseful and applicable in our industrial settingthan others. The usefulness of variousCleanroom concepts for our program are shownin Figure 3 (7 being the most useful).

Cleanroom Survey

0

1

2

3

4

5

6

7

Bou

ndar

yD

efin

ition

Seq

uenc

eB

ased

Enu

mer

atio

n

Bla

ck B

oxD

esig

n

Sta

te B

oxD

esig

n

Cle

ar B

oxD

esig

n

Usa

ge M

odel

ing

Too

ls

Cor

rect

ness

Ver

ifica

tion

Incr

emen

tal

Dev

elop

men

t

Tea

mD

evel

opm

ent

Pee

r R

evie

ws

Concept

Use

fuln

ess

Series1

Figure 3. Usefulness of variousCleanroom principles

The respondents, regardless of whetherthey were working in control related oralgorithmic software, thought that rigorousspecification using sequence based specificationand box structures was well worth the effort.Just about every engineer commented that theyfound problems in their designs during thisprocess that they otherwise would not have founduntil later in the development process. Ourobservations led us to believe that the rigorousspecification process effectively elevated thedesign skills of all of the engineers to a commonlevel. Consistent use of box structures acrossthe various teams led to easier to understanddesigns at the same level of detail for each of theteams. Formal specification tools like Z andVDM were not required or used.

4

1060-3425/98 $10.01060-3425/98 $10.0

Team peer reviews also ranked high.The whole teaming approach to developingsoftware was not new to us. We had been usinga teaming approach to software design for quitesome time. Teaming concepts were not new tothe engineers and no training was required. Thepeer reviews were a modified Gilb [8] approachto reviews. Aside from assigned roles and amoderator, we allowed open and usefuldiscussion to prove designs correct. This allowedus to gain more intellectual control of theproduct. As a requirement for the review, asoftware certification team member had to bepresent. We applied the informal structuredprogramming correctness verification questionsloosely [9]. In many cases, they were not usedduring the peer reviews. In several special cases,a formal proof of correctness was performed toverify algorithmic correctness. This was doneonly when required.

Many of the concerns about thedevelopment of box structures were focused onthe available tools. In particular, the boxstructures approach to design requires a lot oftime to develop the sequence based enumerationtables, the state boxes and the clear boxes. Wedeveloped an in-house tool using excel macros toautomate the process of generating enumerationsthat was very helpful. Still, creating the otherbox structures was time consuming to go throughthe first time and almost impossible to re-do lateron if the design changed and still hold schedule.The only commercial tool available at the time ofour design was for certification (a specificationtool will be available shortly). We found the boxstructures to be very useful, but if was difficult tokeep them updated.

Usage modeling received mixedreviews. Some respondents thought the conceptof testing with usage models to be very useful.Others had a hard time buying into this type oftesting as the sole source of testing. Indeed, thewhole testing approach was tailored in thisproject. Usage models were created for all of theoperational software and used as a base ofsoftware testing. Additional “hand crafted” testsand unit level tests were also used to test some ofthe algorithmic software. There was somedifficulty in developing the models at the rightlevel of abstraction. This was mainly due to thefact that this was the first attempt at developingthis sort of model and the learning curve affectwas obvious.

0 (c) 1998 IEEE0 (c) 1998 IEEE

Page 5: An Industrial Application of Cleanroom Software ... · 1 An Industrial Application of Cleanroom Software Engineering - Benefits Through Tailoring Robert Oshana Raytheon TI Systems

f

Most liked and least liked principles.

The survey group was asked thequestion of what they liked most and least aboutusing Cleanroom (Tables 1 and 2).

Table 1. Cleanroom principles most likedMost liked Cleanroom aspectsSequence based enumerations

Peer reviews of every step in the processSystematic approach to development

Rigorous specificationForcing thought about “impossible” events

Clarity of the processAbility to determine exact behaviorForethought before coding begins

Simplicity

Table 2. Cleanroom principles least likedLeast liked Cleanroom aspects

More automation and cleaner toolsBackground required for full understanding

Usage modelsProcess poor for data based software

Lots of re-work required before getting it right (re-enumeration time consuming)

Maintenance of the Cleanroom artifactsCleanroom artifacts not useful

Process not clearly definedClear box code not as efficient as expected.

Sequence based enumerations and peerreviews ranked near the top of the list inCleanroom principles most liked. Othersresponses such as systematic approach todevelopment, rigorous specification, forcingthought about “impossible” events, and ability todetermine exact behavior are closely tied tosequence based enumeration. All of theseresponses indicate the acceptance of the teams ospending more time up front to get the designright. Follow up interviews indicated thatregardless of whether the engineer had abackground in SA/SD, OO, of some otherdevelopment language, the process of rigorousspecification was new to them and they thoughtthe process had a big benefit to getting designsright the first time.

There were areas of concern as well.More automation and tools are necessary toallow design artifacts to be changed or modified

5

1060-3425/98 $10.01060-3425/98 $10.0

quickly. There were many cases where a simplechange would cause a lot of re-enumeration andbox structure changes that frustrated theengineer. Lower on the list is “Lots of reworkrequired...” and “maintenance of Cleanroomartifacts” which are closely tied to this sameissue. Usage models were not particularly likedby some of the teams. Much of this, we think, isdue to the fact that at the time of this survey,many of the teams were struggling withdeveloping useful usage models and a lot ofrework was being done. The other factor is thatwe were attempting to automate the testingprocess. This required that we map each stateand arc in the usage model into the respectivesystem stimuli and oracle (expected output) andgenerate these in an automated environment. Wewere limited to some extent by the design of thespecial test equipment (STE). Mapping a usagemodel into a STE environment and automatingthe whole test forced us, in some cases, todevelop usage models at an acceptable level ofabstraction for the STE, but not necessarily forthe software. This limitation in level of detail inthe usage models forced us to craft someadditional tests to get acceptable coverage.

Cleanroom worked well for both typesof software being developed, control andalgorithmic. There were fewer issues raised forthose working on the control based software.Control based software inherently allows foreasily defined stimuli and responses. Usagemodeling was also easier, given the well definedpaths in the software architecture. Thealgorithmic software, by the nature of itcomputations, had very few stimuli andresponses and was mainly a data drivenapplication. The box structures and state dataenumeration were simplified. This allowed us tospend more time understanding the complexity ofthe algorithms. Unit tests developed forprototype algorithm software were re-used whichmade the effort of producing and running unittests easier.

Several respondents commented on theprocess not being clearly defined. This is due tothe fact that this was the first attempt at doingCleanroom in a project of significant size with noprevious project experience to draw from. Therewere times when the teams would diverge inareas such as artifact templates, level of detail,etc. The convergence team was developed toaddress these problems and force someconsistency between the groups. Some of therework of Cleanroom artifacts was to align then

0 (c) 1998 IEEE0 (c) 1998 IEEE

Page 6: An Industrial Application of Cleanroom Software ... · 1 An Industrial Application of Cleanroom Software Engineering - Benefits Through Tailoring Robert Oshana Raytheon TI Systems

with the appearance of all of the other teamsartifacts. Much of this is typical in projects usingnew technology for the first time.

Barriers to ubiquitous Cleanroom use.

The survey group was asked tocomment on the barriers they perceived to moreubiquitous Cleanroom use. The responses areintriguing (Table 3). Some members do not yetfeel that software can be treated as anengineering discipline and therefore, theengineering approach to software is not worth theeffort. The majority of respondents, however,did not feel this way. The approach ofdeveloping software without a large unit testingphase appeared to be a leap of faith to some teammembers. Cleanroom permits unit testing,although it provides an opportunity to eliminateor reduce it, because experience shows thatverification is more effective in terms ofeliminating errors and reducing costs. We areallowing all levels and types of testing to beperformed on the software for this project, withthe statistical approach being the commonunderlying model for testing. One of the inherentlimitations of the usage modeling approach isdeveloping accurate probabilities to assign to thearcs in the usage model. We were able to obtainaccurate data for this and the issue was notconsidered a major one. Compiling software anddesk checking via code reading or simple unittest development is being allowed. There is not aformal unit test development phase and this doesnot appear to be causing problems. Some simpleexperiments are showing that we are getting highlevels of coverage using statistically generatedtest cases, but this study is not complete.

Table 3. Barriers to Cleanroom UseBarriers to Ubiquitous Cleanroom Use

Cleanroom assumes that software development canbe treated as an engineering discipline but it is not(yet).Developing software without testing it requiresleap of faith many people are not ready for.Coding using the Cleanroom method involves justturning the crank. No interesting aspects, cleverinventions, elegant solutions, etc.More up front design and detail frustrates someengineersDoes not address real time issues.

6

1060-3425/98 $10.01060-3425/98 $10.0

Algorithm intensive applications (little control s/w)harder to do.Overcoming the notion that Cleanroom is a newtechnology completely different from SA/SDProcess seems very tedious and “front-loaded”Lack of toolsFormal correctness verification exceeds thecapabilities of much of the staffCleanroom is overly process oriented.

Coding too easy?

One interesting response was theconcern over the coding phase requiring just“turning the crank”. No interesting aspects,clever or elegant solutions, or crafting isnecessary when the software if fully specified ateach level. This is exactly what we are trying todo with the Cleanroom process model; eliminatethe clever elegant solutions and replace themwith simple, easy to understand solutions.Coding in the Cleanroom model is meant to bejust turning the crank! A related response that“more up front design frustrates some engineers”and the process being too “front loaded” are, webelieve, related to the same concern over notgetting on with coding and worrying about thedesign later. This is exactly what Cleanroom issupposed to prevent but, based on theseresponses, it is going to take a while forengineers that are used to hacking and crafting tostart designing first.

Real-time issues.

Cleanroom does not fully address realtime issues and this is a concern for us. We areusing other approaches such as rate monotonicanalysis, VHDL simulation, and discrete eventsimulation to supplement the Cleanroom process(Figure 4). There are remaining issues. Forexample, the Cleanroom process leads to theproduction of well structured easy to understandsoftware. In our real time environment, we areoften forced to optimize the software (by re-structuring the C code to take advantage of theoptimizing C compiler and by writing assemblylanguage for certain algorithms). This leads tosoftware that now does not match the structure ofthe clear box. This leads to a mapping betweenspecification and code that becomes harder tounderstand. When assembly language is written,

0 (c) 1998 IEEE0 (c) 1998 IEEE

Page 7: An Industrial Application of Cleanroom Software ... · 1 An Industrial Application of Cleanroom Software Engineering - Benefits Through Tailoring Robert Oshana Raytheon TI Systems

there are two mappings required; one betweenthe clear box and the C code, and anotherbetween the C code and hand optimizedassembly.

Discrete EventSimulation

Data Rates,Execution Deadlines,Real Time Deadlines

Rate MonotonicAnalysis, Multi-NodeAnalysis

Timing Information

Cleanroomlower level boxstructures

Schedulability

SystemEmulation

FunctionalCorrectness

HardwareAvailabiliy

VHDL Simulation

CleanroomSpecification andtop level boxstructures

IntegratedProduct

Cleanroom developed code

Figure 4. Cleanroom processsupplemented with real time analysis andsimulation

More tools required.

Cleanroom lacks the tool base that othermethodologies enjoy. The tools that do exist areextremely useful. Other tools are coming on linein the near future (some of these are being betatested by us now) that should improve thissituation. The in house tools that we developeddid not require a lot of effort and have provedvery useful We have also gained leverage off ofexisting tools. For example, we used Cadre’sTeamwork to develop our boundary diagramsand finite state machines. We use Excel togenerate many of our box structure artifacts.

Lack of mathematical backgrounds inindustry.

Cleanroom teaches us to gainintellectual control over our software usinginformal or formal proofs of correctness. Theinformal proofs of correctness arestraightforward and easy to apply by mostpracticing engineers. However, the formalproofs of correctness scare even the wellseasoned engineers. Much of the knowledge of

7

1060-3425/98 $10.01060-3425/98 $10.0

formal predicate logic required to perform aformal proof of correctness does not exist inmany software engineering teams. We are notdoing any formal proofs of correctness, unlessabsolutely necessary. We understand the need toperform such proofs when warranted. In asample test, we gave a Cleanroom consultant aversion of a special fast fourier transform (FFT)that we thought was correct, to verify usingformal techniques. Problems were found in theimplementation using formal techniques. Theproblem was, they were found by a consultantwith a strong mathematical background. Manysoftware development teams lack the backgroundto perform such formal proofs of correctness.Possible solutions to this are to contract out thesmall sections of the design that require formalproofs of correctness, or have a person in housethat is able and willing to do this. We took thelater approach and, to date, have encounteredfew situations where formal proofs ofcorrectness were required.

Fundamental principles guide the way.

The last barrier mentioned summarizesone of the underlying problems with Cleanroomacceptance; the perception that it is a conceptcompletely different from SA/SD, OO, or otherdevelopment methodologies. It is not. The mostrefreshing aspect of Cleanroom is that the modelis based, in large part, on fundamental computerscience principles. For example, to performsequence based enumerations using stimulushistory requires a knowledge of regularexpressions and finite automata. State boxdesign is nothing more than a tabularrepresentation of a finite state machine. Indeed,often we used finite state machine diagrams toverify the state box design. The entire boxstructured approach supports the fundamentalcomputer science concepts of referentialtransparency, transaction closure, and reuse.Formal and informal proofs of correctnessrequires varying levels of knowledge of predicatelogic. Usage modeling requires knowledge ofMarkov processes, grammars, Monte Carlosimulation techniques, hypothesis testing, andstatistics. Nothing here is new and theseconcepts can be applied to any developmentmethodology ones chooses. One of the mainreasons we chose Cleanroom for our project isbecause many of us more experienced engineers

0 (c) 1998 IEEE0 (c) 1998 IEEE

Page 8: An Industrial Application of Cleanroom Software ... · 1 An Industrial Application of Cleanroom Software Engineering - Benefits Through Tailoring Robert Oshana Raytheon TI Systems

n

at

e

f

ased

that had gone through at least one softwaredevelopment effort before saw Cleanroom not asa new approach to software development, butrather a more disciplined, complete way of doingthe same job! At this point in our project, thatcontinues to be the case.

Static analysis and dynamic analysis.

Cleanroom places a heavy emphasis onstatic analysis techniques. Static analysis is thesystematic inspection and investigation of thetextual source of the software product withouthaving to execute the code. The goal is to findproblems and gather metrics without having tocompile and execute the code. The entry criteriato verification or testing is often a clean compilein many Cleanroom projects. Dynamic analysisinvestigates the behavior of the software byexecuting it. Software execution providesinformation that cannot be obtained by staticanalysis such as timing behavior and profiles andexecution traces. Because of the real time issueswe were facing in our project, we have chosen touse static and dynamic analysis. Our reasons forusing dynamic analysis techniques were todetermine the behavior of COTS software, todetermine the real time behavior of the software,to cover the areas that were not covered by usagemodeling, and to do additional path and branchcoverage. The control software was modeledvery effectively with usage modeling. Thealgorithmic software supplemented usagemodeling with unit tests to verify some of thealgorithmic properties of the software. Thetesting paradigm consisted of a layered approachof static analysis, automatic code analysis, unittesting, and statistical testing with usage models(Figure 5). This flexibility in testing illustratescustomization and tailoring of the Cleanroomprocess to meet specific project needs, whilemaintaining adherence to overall Cleanroomprinciples.

Statistical testing with usage models

Unit and other structural testing

Leak detectors and other code analyzers

Static analysis (code walkthrough)

Code compilation

8

1060-3425/98 $10.001060-3425/98 $10.00

Figure 5. Testing Paradigm forOperational Software

Good specification leads to good testing.

Having a good specification allowstesters to develop good test cases. Developingtest cases by looking at the code does not insurethat the code is doing what it is supposed to bedoing. Test cases developed this way only provethat the software does what we think it shoulddo. These types of tests prove nothing (otherthan the compiler is working correctly). Testingcode against a good specification is the correctway to verify software. Not only can errors bediscovered in the code, but errors areoccasionally, as we discovered, found in thespecification as well. In general, betterspecification allow better testing. WithCleanroom, we felt we were able to develop veryrigorous, precise specifications. Our testing wasbased on these specifications.

Usage models for positive and negativetesting.

Usage models enable testing ofspecified transitions defined in the model. Thistype of testing is often referred to as “positive”testing because they are designed to verify thatthe software does what it is supposed to do. Othe other hand, “negative” testing is focused onverifying that the software does not do things thit is not supposed to do. We created usagemodels that allowed us to perform both kinds oftesting. Our positive testing was based on usagmodels of “typical” or “normal” operation. Thenegative testing was based on usage models o“perverse” or “abnormal” software use.

Testing as a full time job.

One lingering concern with adopting theCleanroom approach was whether we couldconvince software engineers who, by educationand habit, are used to writing software as theirmain job, to do testing and certification. Testingas a career has not fared as well in this countryit has in other countries, and people are rewardmainly based on how well their software writing

(c) 1998 IEEE (c) 1998 IEEE

Page 9: An Industrial Application of Cleanroom Software ... · 1 An Industrial Application of Cleanroom Software Engineering - Benefits Through Tailoring Robert Oshana Raytheon TI Systems

e

skills are, not their testing skills. One solution tothis barrier was to institute a round robin type ofrole playing within some of the teams. For thosesoftware programs that had multiple increments,engineers were given the opportunity to bedeveloper on some increments, and certifier onother increments. So far, this has seemed to be agood middle of the road approach. Automationof the testing process will make the job ofstatistical testing easier (Figure 6).

Software Test Station

toolSETCertify

TestSequences

AutomatedCertificationToolset (ACT)

Test ResultsVerification

ExpectedResults

Output Data/Results

Oracle

UsageModels

CraftedTestCases

Script SW Test Support

Software(TSS)

SoftwareUnder Test

Control

Data

Control andSequencing ofScript

TestData/

Results

Teamwork

Output Data/Responses

Output Data/Responses

Certification team

Figure 6. Automation of the TestingProcess

Cleanroom seems to address some ofthe fundamental testing questions that haveconcerned us in the past:

• Do not let the programmers test their ownwork. We are using separate certificationteams to test software.

• Do not let the testers decide when enough isenough. If statistical tests using Markovmodeling techniques are run after all othertypes of testing, statistics can be used todetermine stopping criteria for testing (this isvery dependent on the level of abstraction inthe usage models and cannot be doneblindly).

Still, not everybody likes Cleanroom.That was evident by some of the surveyrespondents. And there are issues remaining tobe worked in some areas.

The formalism and up front design timewere the main issues. Most of the moreexperienced engineers, especially the ones thathad at least one tour of system integration andtest, however, clearly saw the benefit and need to

9

1060-3425/98 $10.01060-3425/98 $10.0

do a complete, consistent, concise, correct designup front.

We successfully tailored the Cleanroomprocess to fit within the guidelines of our projectin regards to required deliverable documents,schedule, formal reviews, etc. We did notaccept the process dogmatically, nor are webeing orthodox about the methodology. We havetailored the process based on our judgment andunderstanding of what we needed to get the jobdone. Our customer has been totally enamoredwith our efforts to date using Cleanroom, andviews much of what we are doing as a “breadthof fresh air” compared to some of the other workthey are reviewing. Reviews by outside,customer hired, consultants verify the sameconclusion. We are farther ahead at this pointthan the customer expected. The customer hascontinued concerns with testing and we areaddressing these issues mainly by adding unittesting in some areas.

One team had to abandon much of theCleanroom process because of schedulealignment issues. This team had to interface withcommercial off the shelf (COTS) software andhad a difficult time understanding what theCOTS was supposed to do (a case for gooddocumentation). Since this group had to havetheir product out before any other team (theproduct was an operating system that was goingto be used by several other teams) they wererequired to have the design completed first.Prototyping was required to determine how theinterfaces worked and this prototype code wasnot developed using Cleanroom. The prototypecode effectively became the operational code dueto the fact that they spent a great deal of schedulgetting the interfaces to work correctly with theCOTS software. The team did, however,develop a statistical test approach to verify thesoftware. The statistically generated test casesfound many problems in the software. The teamlead was very excited about the use of statisticaltests to verify his software. The team lead in thisarea confessed that many of the problems foundcould have been prevented by doing a completesequence based specification.

Summary.Cleanroom has been inserted and used

successfully in a mature process drivenenvironment. Many of the principles of thetechnology are based on fundamental computer

0 (c) 1998 IEEE0 (c) 1998 IEEE

Page 10: An Industrial Application of Cleanroom Software ... · 1 An Industrial Application of Cleanroom Software Engineering - Benefits Through Tailoring Robert Oshana Raytheon TI Systems

t

science foundations. As with any othermethodology, a certain amount of tailoring isrequired for the technology to be most effective.We have tailored the use of Cleanroom based onour project needs, our time and scheduleconstraints, and on how comfortable we felt withthe various components of the technology. Whawe ended up with is a model for softwaredevelopment that we feel is very effective atproducing quality software. We particularlyenjoyed the concepts of complete, correct,concise, and consistent specifications using a

10

1060-3425/98 $10.1060-3425/98 $10.

step wise refinement of sequence and state basedenumerations. We are having success with ourincremental approach to development ofsoftware. We see the statistical approach totesting as an excellent addition to some of theexisting techniques of software testing and areenthusiastically using these methods.

ach”,

am”,

References

[1] Dyer, Michael; The Cleanroom Approach to Quality Software Development , John Wiley and Sons, 1992.

[2] Mills, H.D., M.Dyer, and R.C. Linger, “Cleanroom Software Engineering”, IEEE Software, September, 1987,pp19-25.

[3] Linger, R.C., “Cleanroom Process Model”, IEEE Software, March, 1994, pp. 50-58.

[4] Hausler, P.A., Linger, R.C., Trammel, C.J.; “Adopting Cleanroom Software Engineering With a Phased ApproIBM Systems Journal, Volume 33, Number 1, 1994.

[5] Whittaker, J.A. and M.G. Thomason, “A Markov Chain Model for Statistical Software Testing”, IEEE Transactionson Software Engineering”, October 1994, pp. 812-824.

[6] Poore, J.H., H.D. Mills, and D. Mutchler, “Planning and Certifying Software System Reliability”, IEEE Software,January, 1993, pp 88-99

[7] Kelly, David P. and Robert S. Oshana; “Integrating Cleanroom Software Methods Into an SEI Level 4-5 ProgrCrosstalk, November 1996.

[8] Gilb, Tom, Principles of Software Engineering Management , Reading, MA, Addison-Wesley, 1988.

[9] Linger, R.C., Mills, H.D., Witt, B.I.; Structured Programming Theory and Practice , Addison-Wesley, 1979.

00 (c) 1998 IEEE00 (c) 1998 IEEE