lessons learned from rigorous system software development

ELSEVIER Information and Software Technology 39 (1997) 55 I-560

Lessons learned from rigorous system software development

Jeremy Dick’.“, Eoin Woodsb’*

‘B-Core (UK), Magdalen Centre, Oxford Science Park, Oxford OX4 4GA, UK ?Tybase (UK), Hillgate House, 26 Old Bailey, London EC4M 7HS. UK

Received 21 August 1996; accepted 19 March 1997

- Abstract

This article describes the rigorous application of formal methods of software development to a system software development project within a conventional industrial software development environment. This project applied formal methods to a small part (one subsystem) of the development of a release of a system software product. The formal methods VDM and B were used to specify, design and implement the administration subsystem of a product that was otherwise developed along conventional lines. Support tools were used to assist with the use of both methods throughout the development lifecycle. Metrics for effort and quality were collected from the project and compared with historical metrics for similar projects to allow the impact of formal methods on cost and quality to be assessed. This article briefly outlines the product being developed and the subsystem to which formal methods were applied. The environment in which this development was performed and the particular approach used are described. The results of this industrial application of formal methods and the lessons learned by the project are discussed at some length. 0 1997 Elsevier Science B.V.

Keywords: Formal method; Process improvement; Software development

1. Introduction

Formal methods can loosely be described as mathematically based approaches for the development of computer systems. Although research has been going on in this area for nearly 20 years, there has not been a widespread adoption of these approaches in industrial software development circles.

During 1994, a group of engineers in Bull’s System Software Development Unit, at Hemel Hempstead in the UK, decided to try to apply formal methods to the development of a software product. This was to see whether use of such technologies could have a beneficial effect on the quality of the products produced and the efficiency of their production.

It was decided to apply two well known and established formal methods, VDM and B, to the development of a small well bounded subsystem that needed to be developed as part of a new release of one of our products.

Metrics were collected from this project and compared with similar measurements from previous projects, allowing

* Corresponding author. Tel.: +44 I71 285 4080; fax: +44 I71 236 1967; e-mail: [email protected]

’ Both of the authors were with Bull Information Systems, Hemel Hempstead, UK when this work was performed.

0950-5849/97/$17.00 0 1997 Elsevier Science B.V. All rights reserved PII s0950-5849(97)00013-x

us to study the impact that this new technology had in our environment.

This article describes the work performed during the experiment; the results obtained in terms of quality and efficiency achieved; and the lessons learned during the project that are felt to be useful to those who attempt similar applications of formal methods in their own projects.

2. The development project

The project in question was the development of the second release of Groupe Bull’s FlowBus@ product. Flow- Bus [l] is an application integration produlct (of the type often known as “middleware”) that allows applications to communicate in a number of ways without being aware of the details of each other, only the interface that each provides. The primary function of FlowBuls is to provide distributed, multi-platform, inter-application message handling services. This involves providing applications with transparent administrator controlled message handling and routing functions.

In essence, FlowBus allows applications 1.0 communicate without explicit knowledge of each other’s existence, form or function. The only interaction that an application can

552 J. Dick, E. Woods/Information and Sofiware Technology 39 (1997) 551-560

have with FlowBus is to send messages to it and receive messages from it (via logical communication ports known as “delivery points”). All messages sent within the FlowBus are of a given type,and the FlowBus routes messages between applications according to the message type and possibly the message content. As well as simple message transmission, FlowBus is designed to be capable of message enrichment, transformation and conversion. Other FlowBus facilities include deferred message delivery (i.e. message queuing) and centralised administration facilities to allow potentially large FlowBus networks to be administered and controlled from a single workstation. The work performed during this development project was centred in this area of control and administration, particularly that of queues.

All the queues in the FlowBus system are to be capable of being administered centrally through a single software tool that can provide status reporting and administration functions. Such a tool must be able to list the queues in the system, monitor the status of each queue and report it on demand, list the messages within a queue, and view or update individual messages if required. If messages are updated, it must allow the administrator to re-route the message, given certain system wide integrity constraints. It must also be capable of generating alarms if certain types of message arrive on certain queues. Configuration of which message type/queue combinations should cause these alarms must be possible, and also potentially what sort of alarm should be raised in each case.

As can be seen from this very brief description, this area of functionality is potentially quite complex. It is technically challenging to build and its successful, reliable, error free operation is critical to the eventual operational deployment of large FlowBus systems to support business critical computer applications. For these reasons, it was decided to apply mathematically based formal methods to the specification, design and construction of this software subsystem in an attempt to develop near zero-defect software to support this critical function. It was decided from a very early stage that the software required for queue administration should be broken into two distinct sections, the underlying “engine” to perform the functions required for queue administration, and the “interface” required to allow interaction with the user (the user in this case being the FlowBus administrator). Performing this partitioning and defining a clean interface between the two sections allowed a clear separation of concerns and responsibilities between software subsystems and the teams of engineers constructing them. It also allowed formal methods to be applied to the “engine” section of the software where it was felt that real benefits could be gained. Although there have been attempts to apply formal methods to the development of graphical user interfaces, the project team were far less confident that significant benefits could be gained from the application of formal methods to the development of the user interface sections of the software.

3. The development environment

This project was developed in a relatively conventional, albeit mature, system software development environment. The department numbered some 60 software developers engaged in all aspects of the system software development process, developing, maintaining and supporting three Unix based system software products that comprise part of Bull’s Distributed Computing Model (DCM). The department had specific responsibility for Unix based transaction processing and distributed printing products.

The development process used was relatively mature for a Unix system software development process, having been certified as TickIT [2] (IS0 9001) compli,ant for its quality management system and operating at a point close to level 3 of the SE1 Capability Maturity Model [3]. Some use was being made of structured development approaches (specifi- cally the Yourdon Structured Method [4]), although much of the development was still utilising a more traditional approach using natural language augmented with informal diagrams as the primary notation. There was no general awareness or understanding of formal methods throughout the development staff.

The FlowBus project in particular had a staff of about 12 software developers, of whom three were involved directly with the development of the queue administration subsystem’s functional engine using formal methods. Of these three developers, one in particular had responsibility for the delivery of the finished component and worked full time on the development of the subsystem from initiation to delivery from unit testing. Two of the three developers who worked directly on the queue administration subsystem had previous experience of applying formal methods. The developer with delivery responsibility had worked with formal specifications on small projects before but had not used formal methods for product development or used formal approaches to the design and implementation of software before working on this project. In particular, they had no knowledge of the B method or its associated tools, which were critical to the delivery of the software component, before commencement of the project.

4. Development approach used

4.1. Overview

As mentioned in the previous section, the development environment in which this project was executed was a relatively mature one. As this project was a development project rather than a research exercise, the development process to be used for the queue administration subsystem had to be defined before the commencement of the project and, additionally, the process used had to be compliant with Bull’s Software Engineering Process

J. Dick, E. Woods/Information and Sofrware Technology 39 (1997) 551-560 5.53

Natural Language VDM

@=q?zsi*

Architecture Specification

B Method C Language

refinement wandan

Interfaces Specification B Specification

Animated Specificslion

manual proof -‘a.. _._.‘. mm proof & Inrpeclron

. . . . . &CL-:::. _ iProven Specification. Verified Speciticadon -............_.._._.....

I

Fig. I. The formal lifecycle used.

Architecture (SEPA), which is Bull’s organisational level software process framework.

It was decided to use two formal methods in the development of the subsystem; the Vienna Development Method’s Specification Language (VDM-SL) [5] and the B method [6]. Both formal methods were used in conjunction with software tools to support their use. In the case of VDM, the tool used was VDM through Pictures (VtP), a research tool from Bull Corporate Research, based on GNU Emacs and Software through Pictures,* which assists with the editing, understanding, checking and maintenance of VDM-SL specifications. The tool used with the B method was the B-Toolkit [7] from B-Core (UK), which is a commercial toolset providing a complete team working support environment for the B method. The decision to use these two methods was the result of knowledge about them that was gained from conversation with experts in both methods and detailed examination of the perceived strengths and weaknesses of the two approaches. It was felt that the VDM-SL formal specification language (which the developers did have experience with) was strongest as an abstract modelling language to allow the specification of the system to be expressed and considered. On the other hand, the B method (particularly when used, as in this case, with the B-Toolkit) was ideally suited to the design and implementation phases of development but much less suited to the

’ Trademark of Interactive Software Engineering Inc.

initial specification modelling. Thus, three languages would be used in the development process; VDM-SL for specification, B for design and C for implementation. As these three languages had to be used together, this provided an extra incentive to plan the development process carefully from the outset!

The development process is shown graphically in Fig. 1. This process allowed a controlled approach to the appli-

cation of formal methods within a conventional development framework which allowed valid comparisons to be made against earlier developments. This sort of process was also familiar to the staff involved with the project and minimised difficulties of communication and integration between this project and the larger product development of which it was a part. Being based upon. Bull’s SEPA, the process could be treated as a normal product development project in terms of management, coordination and control.

The work performed at each stage of the process is described in more detail in the following sections.

4.2. Specification

The specification of the system was performed in a number of steps to allow control of complexity through abstraction. The approach used was that of performing a very abstract specification step initially to get the fundamental aspects of the system identified and discussed, with more

554 J. Dick, E. Woods/information and Sojiware Technology 39 (1997) S-560

concrete specification steps following and adding detail in a controlled manner. This approach allowed intellectual control of the problem and ensured that each stage was built on a sound foundation.

The abstract specification was the first specification written. It was written in VDM-SL and was four or five pages long. It described the main abstract data structures in the system (known as “the state” in formal specification terminology) and the most important operations on them. This stage was very quick, taking only a few days, and immediately raised a number of questions about the requirements for the system, noting inconsistencies and revealing some “impossible” operations.

The architecture specification was a structuring exercise applied to the first specification. The state of the system was partitioned into subsystems that neatly and conveniently divided the system’s responsibilities and reflected the underlying technology that the system would be developed upon. In particular, a new area of functionality was added as a subsystem (an Event Recogniser), entirely because of the technical limitations of the underlying queuing system being used by FlowBus. The operations were then reworked to use this new state, so revealing inter-subsystem depen- dences. Creating this specification supported good modular encapsulated design work, allowed us to use incremental development, perform project planning more easily and define work tasks earlier.

The interfaces specification was simply a refinement of the architectural specification. Given the architecture specification, any design approach could have been used to produce an implementation of each of the subsystems in the specification and integrate them together. In this case, the B method was to be used for the design and construction activities of the project. This inevitably involved a translation from VDM to B, and it was desirable to make this translation as simple as possible to avoid the introduction of errors at this stage. The two main constraints that B introduces into the translation process are that operation signatures cannot be changed during design (that is, the signatures of an operation’s design must match its specification exactly) and that B subsystems cannot access each other’s state directly (that is, they are fully encapsulated and an operation must be used to access any part of the state of another subsystem). It was felt that this amount of translation from the architecture specification to B was too great to be done in one step, and so the interfaces specification was created, firstly to define the desired system API and secondly to define interfaces between subsystems.

The B specification was a translation of the interfaces specification from VDM to B’s specification and design notation, called Abstract Machine Notation (AMN), to provide a basis for the design work. The B method requires all designs to have a specification. Translating the VDM specification into AMN allowed the use of B-Toolkit facilities to help validate the specification.

Specification validation was performed on the B

specification to increase our confidence that it was correct. This was done in two stages, animation and proof. Anima- tion is where the B-Toolkit interprets the specification to “execute’ ’ it and allow it to be tested as a piece of real software might be tested. The proof step is a purely mathematical step where we attempted (with assistance from the B-Toolkit) to prove that the specification we had written was internally consistent. We performed both of these steps, finding quite a few problems very quickly with the use of animation and then a few more over a longer period with some proof work. Both these approaches can also be used with the VDM specification language, but as we did not have suitable tool support for this we just did some reviews of the VDM proof obligations. The B-Toolkit made animation possible and proof much more practical [2 11.

It should be stressed at this point that this application of formal methods attempted to be rigorous rather than formal. That is, rather than proving that everything was “correct”, we used proof as a tool to increase our level of confidence in our specification’s correctness - with the belief that we could have made our arguments fully formal should it have become required. In the circumstances, this was felt to be the most cost effective approach to the use of proof. It is one of the strengths of formal methods that the degree of formality can be varied as conditions require. With less formal methods, little real increase in formality (and so confidence) can be achieved without attempting proof-of- concept implementations (which by the fundamental nature of digital computers must be fully formal). Formal methods allow this increase in confidence to be achieved at an appropriate level of abstraction without needing to be concerned with implementation details.

4.3. Design

Within the B method, design is the process of providing algorithms and data structures (expressed in AMN) that will fulfill the definitions of each operation in the specification. This process involves constructing an implementation of the system state using reusable modules (or “machines” in B terminology) supplied with the B-Toolkit in its library and then using the operations offered by these modules to construct an implementation of the operations in the specification. One interesting point is that variables as such are not allowed in B designs (only in B specifications) and, when information must be stored in a B design, it must be stored in an instance of a library module. The modules supplied in the library include implementations of sets, lists, functions/relations, variables, string containers etc. Although not large, the library is well designed and is adequate for most needs. Projects can also create their own library modules to augment or replace those in the standard library. Once the library is mastered (and extended as required) most of the design stage is a pleasingly simple process of fitting library modules together and designing the algorithms specific to one’s system, as so many of the

J. Dick, E. Woods/Information and Software Technology 39 (1997) 551-560 555

problems that are encountered in design have been met and solved earlier in the process.

4.4. Coding

The coding stage when using B is a very simple process. When the design is complete and has been checked by the Toolkit, it can be translated into conventional source code (usually “C”) for normal compilation and unit test. This stage is effectively instantaneous and so, in effect, the traditional coding stage has been removed from the lifecycle as the low level design is “compiled” to produce the delivered system. Some criticisms can be made of the source code generated; in particular, the code produced is not very robust, as it presumes that it will be called by other generated source code; it is relatively simplistic and does not use all of the sophistication available in ANSI C; and it is less maintainable than conventional carefully written source code. All these shortcomings probably stem from the assumption underlying the B-Toolkit’s code generator that all of the source code will be called from other generated source code, hence the scant regard for human frailties! There are a number of possible solutions to this problem for B-Toolkit users including; writing a “wrapper” interface on top of the generated code to provide a robust, conventional interface for human programmers; replacing or enhancing the code generation facilities of the Toolkit in partnership with its suppliers B-Core (UK); and finally, using the generated C code as a prototype before hand coding another version for delivery from the design notation. In fact, each of these three approaches has been tried on at least one project; the first on this project, the second as reported in Ref. [8], and the third as reported in Ref. [93. Each of the approaches has its own merits and limitations, making each suitable for different types of environment.

4.5. Unit testing

The unit test process followed the coding stage, using a very conventional unit test approach. Although it is possible to harness formal specifications for use in automatic test case generation (for example, see Ref. [22]), this was not seriously attempted here. The unit testing here aimed for 100% functional black box test coverage and 100% branch level white box coverage. This was achieved by identifying test cases using techniques including equivalence partitioning, boundary value analysis and a judicious amount of error guessing. The fault metrics were collected during unit testing, a fault being considered to be a product feature that required correction before the unit could complete unit testing.

5. Results

Changing the technology used in an engineering design process such as software development may be considered

to have all sorts of results, as it may impact on many factors that have an effect on some aspect of the product or its development process. Some results are rela.tively easy to measure quantitatively, but others can really only be measured qualitatively. Any result may be difficult to interpret so that its cause is fully understood and useful conclusions can be reached with regard to whether the change in technology is desirable.

For this project, the team were particularly interested in the impact that the use of formal methods aplpeared to have on both the quality of the product produced and the efficiency of the process used to produce it.

There are many ways to measure and consider a change to the product produced and the development process used to produce it. A very simplistic view of measurement was taken and two well understood metrics were used; the number of faults found (per thousand lines of code) during unit testing to measure the change in the quality of the product developed; and the effort required to develop the product (again per thousand lines of code) to measure the impact on the development process. We relied on the fact that, if other important aspects of the product or process were to change significantly during the project, they would be noticed by the engineers concerned without a quantitative measure being required. If such factors were noticed, they could then be investigated in detail using a more formal approach.

We believe that the approach used here minimised the overhead of measurement while retaining as much accuracy as possible and as much precision as was required. The other important reason for choosing these two metrics was that comparable historical data were available from previous projects.

The metrics for this project and the historical projects are shown in Table 1. The project under discussion here is project P3 in the table.

To understand the comparison fully, a number of points need to be made about the information presented here.

1.

2.

3.

4.

5.

These figures are all from projects developed in the same environment over a period of about 3 years. Other than the development technology, the development process used from requirements capture to unit testing was very similar between all four projects. All these projects were developed by engineers from the same development group. All these projects are fragments of much larger developments. The ‘ ‘lines of code’ ’ (LOC) counts for projects P 1 and P2 are C language statements counted with an automated counter. The LOC figure for project P3 is a “normalised” count. All the code for project P3 was in fact generated by the B-Toolkit. In all, 8000 LOC were generated for the implementation of this subsystem. However, much of this was code to implement library components

556 J. Dick, E. Woods/Information and Software Technology 39 (1997) 551-560

Table 1 Comparative project metrics

Project title Type of development Development approach Size/LOG

Effort/man-days Faults in unit test Unit test faultskLOC EffortkLOC

Pl

INGRES integration System utility development Y ourdon 3ooa

65 27

9 21.5

P2

Event monitor System software development

VDM and Yourdon 1300

27 I 5.5

20.5

P3

FlowBus admin System software development VDM and B

3500 (2200) 43

3 0.9 (2.5)

I;!.5 (36)

(which are generated as needed by the B-Toolkit). The figure of 3500 lines of code is our estimate of the amount of code that a software engineer would have produced to implement the same functionality without attempting any reuse. This estimate was produced by examining the generated code and “factoring out” that code which implemented parts of reusable modules that were not utilised for this system (At the time of this development, the B-Toolkit would inc1ud.e all of the code for a reusable component, whether or not it was called by the application being developed. Later releases of the Toolkit are more efficient and generate code only for the parts of the reusable modules that are actually used.) The figure of 1200 lines of code, shown in the table in italics, is the number of lines of B design notation needed to describe the software so that the code could be generated by the B-Toolkit (and the fault and effort per kLOC figures in italic are those calculated with 1200 lines of code).

6. All the projects are new developments of a very similar nature.

7. None of the effort figures for these projects includes the learning and technology transfer time that is an inevita- ble part of applying new approaches.

For example: 4 Faults Found in Manual Proof stage which

Translation CO AMN educed in stage

. _.... Abstract Specification

Translation to AMk

Unit Test

Y

Fig. 2. A fault grid for project P3.

A more detailed view of the faults that we found throughout the lifecycle of this project (P3) are presented in the fault grid shown in Fig. 2 (the fault grid approach is explained in Ref. [ 111).

The fault grid shows that a relatively small number (28 in total) of faults were discovered. Assuming that the dis- covery of faults is proportional to the number introduced, it seems that fewer faults were introduced to the product on this project than on the previous projects. An interesting point is the tiny number of faults that were discovered during unit testing (three in total), probably because of the effort we put into the earlier stages of development. Also interesting is the relatively large number of faults introduced by the translation from VDM into AMN (19 in total), indicating the potential problems in store when this kind of manual activity is necessary.

The metrics for the earlier projects are not as detailed as these. However, projects prior to this one tended to find faults later in the lifecycle (primarily during low level design, code and unit test).

As can be seen from these figures, the results we obtained on this project in terms of faults found during unit testing were significantly better than we managed on previous very similar projects. Experience has shown that, in general, the level of faults found in unit test is proportional to the number of faults found later in validat ion and customer use. In particular, the ratio of faults found in unit test to faults found in validation test is approximately 5:l. As the only real change to the development process used was the application of formal methods (and the pe:ople involved and the type of product being developed stayed relatively constant), it is probably fair to say that the use of formal methods contributed significantly to this improvement. This hypothesis is also partially supported by the fact that, looking at the three projects as a group., there is a strong correlation between increasing formality and reduction of faults found in unit test.

From these figures, we can see that by any reasonable interpretation of the data the use of formal methods has significantly improved the quality of the product delivered at the end of unit testing.

The effect of this approach on effort expended is slightly less clear. When comparing effort in terms of source code produced, this approach appears to be very effective, being

J. Dick, E. Woods/information and Sojbvare Technology 39 (1997) 551-560 557

significantly less expensive than comparable efforts. How- ever, as mentioned above, comparing the generated source code from this approach with the manually written code of the previous projects may not be a fair comparison. In fact, this simply confirms the widely held belief that lines of code is not a good measure of software size! When the effort per thousand lines of design notation is considered, this approach appears to be rather more expensive than the approaches used in the previous projects. However, again this may not be a fair comparison. The authors believe that overall it is fair to suggest that the approach used in this project was achieved at a reasonable cost and that it is probably no more expensive than previous approaches for the requirements capture to unit test activity (and is probably cheaper from that point onwards). When measuring effort on other projects, a functional size measure (such as Mark II function point analysis or feature points) would make this sort of comparison rather easier.

A more detailed and quantitative analysis of the results obtained can be found in Ref. [lo].

We have two main concerns about these results. Scalability: these results were produced on small sub-

developments with a few development engineers producing fewer than 5000 C statements. There is obviously a concern whether or not similar results would occur when formal methods technology is applied on larger projects with more staff and for larger software components. Projects such as that reported in Ref. (121 will help to answer this concern and should be interesting to follow.

Lifecycle results: these results show the effect of formal methods on the product lifecycle from abstract specification through to unit testing. Although experience has suggested that the level of quality found at the unit testing stage is a good indicator of the quality of final delivery, we cannot be certain of this when new approaches are used. It is possible that certain classes of error are introduced by a formal methods development that the unit test approach used here does not identify effectively. The only way to answer this concern is to apply formal methods to industrial projects and follow their progress through the product lifecycle, comparing them with existing product components. Unfortunately, in the case of this project, owing to changes in the development organisation, the software was not released into the field and so no figures as to its reliability in customer situations can be collected.

These concerns aside, it is felt that these results are very encouraging and are certainly positive enough to suggest that judicious use of formal methods could yield significant benefits for software development groups.

There are a number of reasons why this formal approach appears to be successful. Firstly, formality (and so precision) is available throughout the development lifecycle and not just when coding. This assists greatly with the early identification, removal and prevention of defects. Secondly, the B method and the B-Toolkit encourage a great deal of automation and reuse. This allows designers to work

confidently and precisely at a much higher level of abstraction than is otherwise the case and so results in higher productivity and fewer errors. Finally, because formal methods allow reasoning about specifications, they strongly encourage investment in careful problem investigation and analysis early in the lifecycle. This helps to eliminate errors as early as possible and at minimum cost.

To summarise, this experiment has resulted in software developed in a conventional industrial setting at less cost with significantly fewer errors than with the previous approaches attempted. Although the measurelment approach can certainly be criticised for its relative naived and the process cannot yet be guaranteed to be totally scalable, this does appear to be a very encouraging result for the application of formal methods in an industrial context.

6. Lessons learned

Whenever a new technology or approach is used to develop a piece of software, a lot of lessons will be learned during the project, and this case was no exception. Two sorts of lessons were learned; negative lessons that were the result of problems and are aspects to try to avoid in the future; and positive lessons that were the result of strengths of the approach and are the reasons we wou.ld attempt the approach again.

The negative lessons learned were:

1. Tool support: the application of formal methods without effective and reliable tool support would be significantly more difficult than when tools are available, and in fact could call into question the practicality of such approaches. In the project as a whole thle tool support was both effective and reliable. However, the few problems we did encounter quickly revealed just how dependent we were on our support tools.

2. Code generation: the source code generated by the B-Toolkit may not be suitable for product development environments and there is no way in which the user may directly alter the translation process. (Having said this, it is probably as good as most of the code generated from CASE tools today.) The code produced was not really robust enough for our needs and was not particularly easy to interface to conventional C code. This has been addressed in other projects and each project must carefully consider its needs in this area.

3. Expert advice: the assistance of formal methods experts will be required throughout the first project and possibly (less frequently) beyond. Like any powerful technology, formal methods and their support tools are complex and require an in-depth knowledge for effective application. Engineers new to formal methods may often be bewildered by the choices available thoughout the development process, and expert assistance will ensure that the correct choices are made while the: engineers are gaining the experience needed to make these decisions

558 .I. Dick, E. Woods/Information and Software Technology 39 (1997) 551-560

themselves. (As Mike Hinchey and Jonathan Bowen note [13], ‘Thou Shalt Have a Formal Methods Guru On Call’!) Having said this, whether it is really more difficult than learning (for example), MVS, Unix, C++ or SQL Server is a debatable point! Complexity: as noted in the previous point, like any powerful technology, formal methods are complex, Both VDM and B have some complex and subtle aspects which must be mastered to avoid problems and gain maximum benefit from their application. A certain intellectual commitment will be required for this and, without it, it is unlikely that this approach would be entirely successful. Hybrid formal methods: this project used a “hybrid” approach, using both VDM and B in a single development. We expected some problems in this area (hence our use of the interfaces specification) and so were ready for the ones we found. Our experience was that a number of faults will inevitably be introduced by the translation between notations even if there are no other problems, and if performing such a lifecycle again we would attempt to find ways of minimising this. Proof: as explained in much of the formal methods literature [ 14- 161, there are varying degrees of formality at which formal methods can be applied. In this project we attempted to discharge the proof obligations associated with the specification (i.e. to prove the specification “correct”, or more properly, “internally consistent”). This is an expensive process and we found that it identified a relatively small number of faults, most of the faults in the specification being found by less formal techniques including inspection and animation. Our experience is that, from a business perspective, we could not justify a general use of proof, and so used a rigorous rather than a formal approach. Obviously in more critical domains widespread use of proof may well be easier to justify.

The positive lessons we learned were:

1. Practicality: it was found that the use of formal methods is practical in an industrial context. Both VDM and B are powerful and well developed formal methods that are relatively easy to apply to industrial software development. Although the notations used by the methods appear to be a problem at first (being based on set theoretic and first order predicate calculus notations), practical experience suggests that the approach soon becomes natural for engineers to use (being no more complicated than programming notation) and as such ceases to be a problem.

2. Tool support: the B-Toolkit is an example of effective tool support for formal methods. In particular, the B method is very fully supported by and tightly inte- grated with the B-Toolkit, providing a good working environment.

3. Fault identification: the use of formal methods allows

early identification of faults in the software lifecycle because of the precision that early formality allows and the detailed precise thinking that formal specification encourages. Costs: when the costs of adopting new technology on the first project are factored out, formal methods appear to be cost effective during the development stage and to offer the promise of significantly reduced ‘costs later in the product lifecycle. Quality: the use of a formal approach appeared to result in a much higher quality product than hlad been achieved using less formal approaches. This promises many future benefits, particularly in product development environments. Lifecycle: using formal methods throug,hout the lifecycle appears to encourage the identification of problems and the expenditure of effort earlier than: with traditional approaches. This should help to reduce the likelihood of critical “show stopping” problems late in the project.

From the experience of this project, the authors believe that none of the negative lessons provides significant or lasting barriers to the adoption of formal methods and the positive lessons are strong incentives to e:ncourage the use of such approaches.

7. Summary

This article has described an industrial application of formal methods, which involved applying a hybrid formal approach to the development of a subsystem which forms part of a larger system software product.

VDM-SL was used for specification, and the B method was used for design and implementation work, resulting in a machine generated C language implementation. A research tool called VtP, from Groupe Bull Corporate Research, was used to support the VDM related aspects of the development, and a commercial tool called the B-Toolkit, from B-Core (UK), was used to support the use of B.

The application of formal methods to this development appears to have been successful. It is certainly the percep- tion of the authors that the software developed benefited greatly from the use of formal methods. The metrics collected during the project seem to confirm this view.

The formal methods technology used was, for the most part, very effective. Some of it needs further investment to allow it to match the maturity of maindream structured method CASE products, but it appears to be perfectly fea- sible to apply formal methods in this manner to fairly large industrial product developments.

The use of a hybrid approach did cause some problems where the two notations met during development. However, the use of two approaches was successful and allowed each approach to be used where it is most efFective, given the technology available today.

J. Dick, E. Woods/Information and Software Technology 39 (1997) 551-560 559

In summary, this project has been a very encouraging started to rely more and more upon software as a system application of formal methods to industrial software component. As the systems they build are safety critical engineering. Its results certainly suggest that it is not only systems which need to be highly reliable, this has meant practical, but positively beneficial, to use formal approaches that the software within the systems must be verified to in the industrial development of software products. operate correctly.

8. Related work

There have been a number of attempts to apply the B method to industrial software engineering problems, of which this is one. Two significant attempts have been IBM’s work applying B to the development of CICS and GEC-Alsthom’s work with B in the area of railway signalling systems.

When developing this software, GEC-Alsthom use a “V” lifecycle (where every development stage has a corresponding verification step). Initial approaches to software development involved a normal “V” lifecycle, augmented with a code verification process that used Hoare verification (an approach based on preconditions and postconditions).

8.1. IBM

IBM Hursley Park Laboratories are the developers of the well known Customer Information Control System (CICS) transaction processing monitor. As part of the development of a release of CICS/ESA (the version of CICS for the MVS operating system), IBM attempted to develop a component of the release using the B method, supported by the B-Toolkit [8].

The initial use of B was to provide forma.l specification within the existing development lifecycle, so acting as a formal input to the Hoare verification process, replacing the informal specifications that had been used as inputs previously. This approach appears to have been successful at correcting errors, and later uses of B have extended to the use of the B method and B-Toolkit in the design stages of the lifecycle.

Hursley Park have been formal methods users for over 10 years, having used the Z [17] specification language to re-specify large portions of the CICS product [12]. What made this project different was that, in addition to the use of Z for specification, B was used to support the design and implementation activity.

When GEC-Alsthom have used B for des-ign, structured analysis is used to define the initial specification. The structured analysis then forms an input to a El specification step which is verified with the use of proof. B refinement is then used to implement the software (which is again verified using proof) and the code for the completed system is generated automatically.

In many ways, the IBM CICS use of B is extremely similar to the use of B described in this paper. Both are system software developments, using a formal specification language in conjunction with B, where animation and proof were both attempted to try to find errors early in the lifecycle. Many of the experiences were shared too. The CICS team found animation very valuable, as did we. They also found that many of the errors introduced were caused by a manual translation from the specification language to B as we, too, reported. Finally, both teams stressed that the form of the code generated by automated tools such as the B-Toolkit is important, as well as its “correctness’ ’ .

The experience within this environment is that there is a large reduction in unit and integration testing effort, as these stages have been largely elimited by the extensive use of proof within the design cycle. Functional telsting effort is unchanged, as the purpose of functional testing is to estab- lish conformance between the formal specifications for the system and the customer’s requirements.

9. B literature

At the time this work was performed, one of the problems encountered was a lack of suitable introductory and tutorial texts on the B method. Luckily this situation has been remedied by the publication of a number of good books on B, such as refs. [6,16,19,20]. These references are briefly described below to assist the reader with the selection of appropriate literature.

The apparent similarities between these two independent attempts to apply B and the number of shared results seem to bode well for the possibility of reproducing these results elsewhere.

The B Language and Method - a Guide to Practical Formal Development, by Kevin Lano [ 191, is a comprehen- sive introduction to AMN and how it can be used to support formal specification and development of high integrity systems.

8.2. GEC-Alsthom

GEC-Alsthom Transport in Paris have attempted to apply the B method and the B-Toolkit to the development of railway signalling system software [18]. As railway signalling has become more complex, GEC-Alsthom have

Software Engineering with B, by John Wordsworth [16], is a book giving a balanced coverage of the B method cover- ing the whole software lifecycle from specification through to the production of programs using the B-Toolkit. It uses many tutorial examples to show how the method can be applied to real-world software engineering. The author’s

560 J. Dick, E. Woods/information and Software Technology 39 (1997) X5-560

experience has grown out of applying formal methods to the development of CICS at IBM Hursley.

The B-Book - Assigning Programs to Meanings [6], by Jean-Raymond Abrial, is the complete reference book on the mathematical foundations of the B method and AMN by its inventor. It contains numerous examples and exercises, often more suitable for the theoretician than the engineer.

Specification in B: an Introduction Using the B-Toolkit [20], by Kevin Lano and Howard Haughton, provides a practical introduction to AMN and the B method suitable for both undergraduate and postgraduate practitioners. It covers all lifecycle stages including animation, proof, design and code generation. Two large case studies are provided.

Acknowledgements

This work was carried out as part of EC funded ESSI project number 10619 (“MafMeth”).

References

VI

[21

[31

[41

[51

Bull Information Systems SA, Distributed computing FlowBus product overview, Bull Open Software, 86 A2 51CD. Bull S.A. CEDOC. UK Department of Trade and Industry, TickIT: Guide to Software Quality Management System Construction and Certification using

1S09001/EN29001/BS5750 Part 1, February 1992, TickIT Project Office, 68 Newman Street, London WIA 4SE, UK. MC. Paulk, W. Curtis, M.B. Chrissis, C.V. Weber, Capability Maturity Model for Software, Version 1 .l, Carnegie Mellon Univer-

sity Software Engineering Institute Technical Report, CMU/SEI-93- TR-24, February 1993. Yourdon Inc, The Yourdon Systems Method: Model Driven Systems Development, Prentice-Hall, Englewood Cliffs, NJ, 1993. C.B. Jones, Systematic Software Development using VDM, 2nd edn.,

International Series in Computer Science, Prentice-Hall, Hemel Hempstead, 1990.

[6] J.R. Abrial, The B-Book - Assigning Programs to Meanings, Cambridge University Press, Cambridge, 1996

[7] J.R. Abrial, B-Technology Technical Overview, B-Core Ltd., Oxford.

1993. Contact author Dick for copies of this reference, or see http:// www.b-corecorn/.

[8] J. Hoare, The Formal Development of CICS with B, in Bull Informa- tion Systems SA, Distributed computing FlowBus product overview, Bull Open Software, 86 A2 51CD, Bull S.A. CEDOC.

[9] B. Ormsby, An Approach to Testing during Formal Development with

the B Method, Euromicro 96 Short Contributions, IEEE Computer Society Press, 1997, pp. 107-112.

[ 101 J.C. Bicarregui, J. Dick, E. Woods, Quantitative analysis of an appli-

cation of formal methods, in: Proc. FME 96: Industrial Benefit of Formal Methods, Lecture Notes in Computer Science, Vol. 1051, Springer Verlag, Berlin, 1996, pp. 60-73.

[l 11 D. Maisey, J. Dick, Measuring the quality of the development lifecycle process, Software Quality J. 5 (3) (1996) 199-210.

[12] B.P. Collins, J.E. Nicholls, I.H. Sorenson, Introducing formal

methods: the CICS experience with 2, IBM Hurjley Technical Report, TR12.260, 1987.

[13] J.P. Bowen, M.G. Hinchey, Ten commandments of formal methods, Computer 28 (4) (1995) 56-63.

[14] M. Hinchey, J.P. Bowen. Applications of Formal Methods, Inter- national Series in Computer Science, F’rentice-Hall, Hemel

Hempstead, 1995. [ 151 J.M. Wing, A specifier’s introduction to formal methods, Computer 23

(9)(1990)8-24. [16] J.B. Wordsworth, Software Development with B: An Introduction,

Addison Wesley Longman, Wokingham, 1996. [17] J.M. Spivey, The 2 Notation: A Reference Manual, 2nd edn., Inter-

national Series in Computer Science, F’rentice-Hall, Hemel Hempstead, 1992.

[18] B. Dehbonei, F. Mejia, Formal methods in the railway signalling industry, in: Proc. FME 96: Industrial Benefit. of Formal Methods, Lecture Notes in Computer Science, Vol. 1051, Springer Verlag, Berlin, 1996, pp 26-34.

[ 191 K. Lano, The B Language and Method - a Guide to Practical Formal

Development, Springer-Verlag. Berlin, 1996. [20] K. Lano, H. Haughton, Specification in B: an Introduction Using the

B-Toolkit, Imperial College Press, London, UK, 1996. [21] J.C. Bicarregui, J. Dick, B. Matthews, E. Woods, Making the most of

formal specification through animation, testing and proof, Sci. Com- put. Programming 28 (2) (1997). in press.

[22] J. Dick, A. Faivre, Automatic partition analysis of VDM specifications, Bull Corporate Research Centre Technical Report, RADI DMA/92027, October 1992.

lessons learned from rigorous system software development

Documents