SOFTWARE PROJECT PREDICTABILITY AND CONTROL

1. INTRODUCT ION

Software project predictability and control is critical to. the Air- Force for twomain reasons:

1. Software projects are inherently difficult to predict and control.

2. Software is necessarily on the critical path in many Air Force system

acquisitions. Thus, any software project slippages are directly trans­

lated into slippages in operational readiness for many Air Force weapon3

systems and C I systems.

The second section of this chapter discusses the nature of the' software predicta­

bi lity and control problem: what is it about software projects which makes them

difficult to predict and control? The third section discusses a number of tech­

niques for improving software predictability and control which the Study Group

found being applied successfully in some parts of the Air Force, including the

use of prototyping, incremental development, design-to-cost, competitive analysis

and design contracts, and software development capability /capacity reviews.

However, the Study Group found that these and other proven software risk

management techniques are not being used in many Air Force system acquisitions.

The Study Group concluded that a significant improvement in software project

predictability and control would be effected by thorough application of these

techniques, and that the most effective ways to get them used would be to

establish and ensure the application of two further Air Force policies:

1. A policy on software risk management;

2. A policy on software oversight management.

Section 4 of this chapter presents and discusses the proposed policy on software

risk management, presented as an additional chapter in the relevant Air Force

regulation: AFR 800-14, Vol. II, "Acquisition and Support Procedures for

Computer Resources in Systems. II Section 5 presents the corresponding recom­

mended policy on software oversight management.

2. THE SOFTWARE PROJECT PREDICTABiliTY AND CONTROL PROBLEM

Necessary Conditions for Predictability and Control

One way to develop an understandi ng of the software project predictabi lity and

control problem is to compare software projects with an area in which the Air

Force has achieved considerable success in predictability and control: the area

of guided missiles. This comparison (see Fig. 1) shows us that there are three

necessary conditions for successful predictability and control which are strongly

satisfied in the guided missile field, but only weakly satisfied in the software

project field:

1. Understanding and stabilizing the target location;

2. Understanding the dynamics of the delivery system;

3. Good navigation, guidance, and control.

Deficiencies in these three necessary conditions are the main reason that the tra­

jectory of a software project looks so frequently like that of an unguided missile.

Understanding and Stabilizing the Target location

In contrast to the relatively precise geodesy supporting the targeting of guided

missi les, software projects are frequently launched with only the vaguest idea of

their target. However, software budgets and schedules are generally determined

under the assumption that the project will have no breakage of software design

and code due to imprecise understanding of the target requirements.

In the guided missile field, it is well understood that it is harder to hit a target

which is able to take evasive action. One of the main reasons that software pro­

jects look so uncontrolled is that they are continually being reoriented to cover a

stream of new or changed requirements inserted during software development.

Thus, one of the major directions by which we can improve software project pre­

dictability and control is by applying techniques which provide a better early

understanding ·of software requirements, and which enhance the stability of

software requirements during system development.

Understanding the Dynamics of the Delivery System

Frequently, the dynamics of a software project are counterintuitive. If we want

a guided missile to reach its target sooner, our intuition tells us we should apply

more thrust, and our intuition is confirmed by the performance of the missile.

On software projects, however, one of the best-confi rmed phenomena on project

scheduling is Brooks' Law [Brooks, 1975]:

"Adding more people to a late software project wiII make it even later."

The main reason for this phenomenon is that the added people require a great

deal of context and training, which can only be supplied by diverting the people

already on the project from making progress on the job.

In other areas, the dynamics of software projects are still not well understood, so

that projects must rely on the intuition of a few scarce experts for guidance in

software project direction. To improve this situation, the Air Force needs to

invest in further R & D on software process dynamics, but most importantly needs

to increase its scarce supply of software-knowledgeable personnel.

Good Navigation, Guidance, and Control

Again, in contrast to the world of guided missiles, the world of software projects

is sparsely supported by techniques for tracking project status, and hampered

in controllability by numerous contracting and procurement complications. Further,

available and effective software project tracking and control techniques are not

being used on many Air Force software projects. To improve this situation, the

Air Force needs to establish policy guidance on the uniform use of software tracking

and control techniq ues, and to provide better information support capabi lities to

assist the SPO in software project tracking and control.

Software Prediction Uncertainties

As discussed above, our ability to predict the cost and schedule of a software

project depend~ strongly on our degree of understanding of the software product

to be developed. Let us look at this issue in more specific detail, and discuss its

implications with respect to current Air Force software acquisition practice.

Figure 2, from [Boehm, 1981], illustrates the accuracy within which software

cost estimates can be made, as a function of the software life-cycle phase (the

horizontal axis), or of the level of knowledge we have of what the software is

intended to do. This level of uncertainty is illustrated in Fig. 2 with respect

to a· human-machine interface component of the software.

When we first begin to evaluate alternative concepts for a new software applica­

tion, the relative range of our software cost estimates is roughly a factor of

four on either the high or low side. * This range stems from the wide range of

uncertainty we have at this time about the actual nature of the product. For

the human-machine interface component, for example, we don't know at this time

what classes of people (clerks, computer specialists, middle managers, etc.) or

what classes of data (raw or pre-edited, numerical or text, digital or analog)

the system will have to support. Until we pin down such uncertainties, a factor

of four in either direction is not surprising as a range of estimates.

The above uncertainties are indeed pinned down once we complete the feasibility

phase and settle on a particular concept of operation. At this stage, the range

of our estimates diminishes to a factor of two in either direction. This range is

reasonable because we still have not pinned down such issues as the specific type s

of user query to be supported, or the specific functions to be performed within

the microprocessor in the intelligent terminal. These issues will be resolved by

the time we have developed a software requi rements specification, at which point,we will be able to estimate the software costs within a factor of 1.5 in either

direction.

By the time we complete and validate a product design specification, we will have

resolved such issues as the internal data structure of the software product and

the specific techniques for handling the buffers between the terminal micropro­

cessor and the central processors on one side, and between the microprocessor

and the display driver on the other. At this point, our software estimate should

be accurate to within a factor of 1.25, the discrepancies being caused by some

*These ranges have been determined subjectively, and are intended to represent

80%confidence limits, that is, "within a factor of four on either side, 80%of thetime. II

remaining sources of uncertainty such as the specific algorithms to be used for

task scheduling, error handling, abort processing, and the like. These will be

resolved by the end of the d~tailed design phase, but there will still be a

residual uncertainty about 10%, based on how well the programmers really under­

stand the specifications to which they are to code. (This factor also includes

such considerations as personnel turnover uncertainties during the development

and test phases.)

Impact on Air Force Software Acquisition: An Example

An example illustrating these problems is given in Figure 3 [Devenny, 1976],

which shows the comparative progress of software cost estimates and actual project

expenditures over the development cycle of a recent U.S. Air Force command-and­

control software project. The specific project and customer are not significant,

as this job's history is little different from many others in government and industry

around the world, except perhaps that the Air Force has more carefully recorded

and analyzed its history in an attempt to avoid similar future problems.

The initial estimate for the software job was roughly $1,500,000. After a round

of competitive "best and final" estimates, the winning bidder's estimate for the

job was about $400,000, based more on optimistic assumptions and claims than on

any sound software cost-estimation rationale.

The subsequent history of the project was a continuing series of realizations --

at the $400- K level, at the $700- K level, at the $2500- K level, and at the $3200- K

level -- that the current budget was running out, but the job was not finished.

Each time this happened, an increased budget and schedule was negotiated, finally

resulting in a total project cost of about $3700 K, almost ten times the early opti­

mistic estimate. Even in the later stages of the project, the cost-to-complete

estimates were highly inaccurate.

Implications for Air Force Software Acquisition

The primary implications of Figure 2 for Air Force software acquisition are the

following:

o A manageable level of software predictability and control is not achieved

until the software Preliminary Design Review (PDR). The remaining 20-25%

'cost' uncertainty is generally within the scope of management accommodation.

o Most Air Force acquisition commitments are fixed before PDR. Fixed­

dollar software estimates are inserted early in the paM process, and

frequently committed to fixed-price software acquisition contracts on

the basis of an incomplete req uirements specification.

o System budget pressures force software estimates to the low end of the

uncertainty range. In general, software requirements lag the definition

of the hardware systems within which the software operates. Many pro­

jects, if faced with a clear hardware overrun and an uncertain range of

software costs, wiII push the software budget to the low end of its

uncertainty range, building in an almost certain software overrun. In

addition, as was shown in the example project, premature best-and-final

negotiations on software projects provide another major source of built­in overruns.

o A similar curve to Figure 2 holds for the estimation of software develop­

ment schedules, with even stronger adverse consequences for the timely

completion of software-intensive Air Force acquisitions.

o To some extent, the cost and schedule uncertainties can be addressed

by a design-to-cost approach. However, if the software resources are

too far underbudgeted, the resulting compromises in functionality can

cause major deficiencies in overall mission effectiveness.

Other Software Project Risk Factors

We have discussed the following sources of software cost and schedule risk:

o Inadeq uate understanding of software product to be developed;

o Continual changes in software product requirements;

o Shortage of Air Force personnel experienced in software acquisition;

o Premature commitment to fixed-price software contracts based on best­

and-final price negotiations;

o Inadeq uate information support for software project tracking and control.

In addition, there are some further sources of risk on Air Force software projects.

o Technical risk involved in pushing the state of the art in multiprocessors,

key algorithms, display capabi lities, real-time processing, etc.

o Inaccuracies in software cost estimation models and techniques. Even when

all the cost model inputs are accurate, a current state of the art cost model

is doing well if it can estimate within 20%of the project actuals, 70% of thetime.

o Slippages in related software systems under concurrent development. As

the Air Force moves toward more highly integrated weapon systems and

C31 systems, this source of risk will become more and more critical.

o Selection of a poorly-qualified software development contractor, frequently on

the basis of an aggressive bid price and an impressive-looking proposal

developed by people who will not be around for the actual project.

With all of these sources of risk, we have painted a fairly grim picture for Air

Force software predictability and control. On the positive side, however, the

Study Group found a number of effective risk management techniques which were

being applied successfully on some Air Force projects. These will be discussed inthe next Section.

3. EFFECTIVE SOFTWARE RISK MANAGEMENT TECHNIQUES

Prototypi ng

Prototyping involves the early development and exercise of critical computer re­

source components (user interfa~, network operating system resource manager,

key algorithm processor) for the purpose of identifying an'd eliminating sources ofrisk.

The major risk management benefits of prototyping are:

o A clearer understanding of requi rements, particularly as a result of exer­

cising a user-interface prototype with representative Air Force users.

o A clearer understanding of design options, and how they may be imple­mented in code.

o Resolution of high-risk technical issues in areas where the system may

be pushing the software state of the art.

Incremental Development

Incremental development involves organizing the development of a software product

into a series of increasing increments of functional capability.

T he major risk management benefits of incremental development are:

o Spreading risk across several smaller increments, rather than concen­

trating risk in a single-shot development.

o Ability to stablize requirements during the development of a given incre­

ment, by deferring proposed changes until later increments.

o Clearer understanding of the requirements for later increments, based

on the user's ability to gain experience with the earlier increments.

Design-to-Cost

Design-to-cost involves fixing the development cost of a system, and prioritizing

system requirements so that the developer can deliver the most high-priority

system capabilities possible for the given cost.

The major risk management benefits of the design-to-cost approach are:

o A clearer understanding of requirements via the necessity to determine

thei r priorities.

o Eliminating sources of inefficiency due to renegotiating budgets andschedules.

o Reducing the impact on other interdependent systems of cost and schedule

slippages in the subject system.

Multisource Cost and Schedule Estimation

Multisource cost and schedule estimation involves the use of multiple, independent

organizations, techniques, and models to estimate costs and schedules, including

the analysis and iteration of differences between the estimates.

Its major risk management benefits are:

o Reducing the inaccuracies in software cost and schedule estimates.

o Providing a sounder set of "should-cost" estimates as a basis of soft-

ware project control.

Competitive Analysis and Design Contracts

Competitive analysis and design contracts involve the employment of multiple con­

tractors to develop competitive analyses, designs, prototypes, and refined re­

quirements specifications before proceeding to selection of a development contractor.

Their major risk management benefits are:

o A clearer understanding of requirements from the multiple perspectives

provided.

o More thorough identification and resolution of high-risk elements.

o Deferral of development contract budget commitment until PDR.

o Reduced risk of selecting a poorly-qualified development contractor.

Software Development Capability /Capacity Reviews

These reviews, previously called Pre-Award Surveys, involve survey visits to

bidders' computer system development organizations to determine their readiness

and capability to perform on a contract.

Their major risk management benefit is to reduce the risk of selecting a poorly­

qualified software contractor. Their existence also provides a stimulus for all

prospective contractors to be better prepared for the development project.

Incentive and Award Fee Contracts

Incentive and award fee contracts involve basing a significant portion of a con­

tractor's fee on either a pre-negotiated incentive structure or on performance

ratings determined by the customer.

Their major risk management benefit is to stimulate the developer and customer

to operate in a· teamwork mode rather than adversary mode.

Cost /Schedule /Performance Tracking

This involves the use of such techniques as work breakdown structures, activity

networks, and earned value systems to determine and track the progress of a

project with respect to its plans, schedules, and budgets.

Its major risk management benefits are:

o Early identification of potential schedule slippages and cost overruns.

o Reducing the impact on other interdependent systems of cost and schedule

slippages in the subject system.

Risk Element Tracking

This involves the identification by a project manager of the project's highest-risk

development issues, and the tracking of progress on these issues through sub­

sequent progress reports.

Its major risk management benefits are similar to those of cost /schedule /performance

tracking above, plus the added benefits of identifying and maintaining a high level

of consciousness of the righ risk elements.

Risk Management Plan

A risk management plan states how the organization will identify and deal with

sources of risk, expressed in relation to the major computer resource life cyclemilestones.

The risk management plan combines the benefits of the other risk management tech­

niques, again with the added benefit of identifying and maintaining a high level

of consciousness of the high-risk elements.

4. RECOMMENDED USAF POLICY ON SOFTWARE RISK MANAGEMENT

The Study Group found that the software risk management techniques discussed in

the previous Section were being applied successfully on some Air Force projects,

but many more were not taking advantage of them.

The Study Group believes that the most significant step the Air Force could take to

reduce its software predictability and control problems is to ensure that these

effective risk management techniques are uniformly applied across all Air Force

software acquisitions.

In order to achieve this objective, the Study Group recommends that the Air

Force establish a policy on software risk management. The best mechanism

the Study Group could identify for establishing this policy in an early, effec-

tive, and enforceable way is to incorporate it as a new chapter in AFR 800-14,

Vol. II, IIAcquisition and Support Procedures for Computer Resources in Systems. II

A draft candidate for this policy is provided as Exhibit 1.

Risk Management Information Support

In order to support the effective implementation of this policy, the Study Group

further recommends that a high priority be given to providing a set of software

management support tools as an integral part of any Air Froce software supportenvironment. Tools which should be included in this set are:

o Software cost and schedule estimation tools;

o Software cost and schedule tracking tools;

o Software progress and status tracking tools;

o Earned value reporting tools;

o An Air Force standard software work breakdown structure;

o Requirements specification and tracking tools;

o Electronic mail support between the SPO, his Air Force interfaces, and

his contractor, including support for classified projects.

The proposed USAF Computer Technology Center is a good candidate organization

for collecting, developing I and supporting these tools.

5. RECOMMENDED USAF POLICY ON SOFTWARE OVERSIGHT MANAGEMENT

Discussion

The direct problems encountered by Air Force software projects are magnified by

the fact that these problems frequently develop without high-level notice. Some

of the major reasons for this are:

o Software is often not properly reported in large-system status reports;

o High-risk software is not highlighted in reports and reviews;

o Software status reports are too detailed to provide useful information I

or too costly to be prepared correctly;

o Higher levels of man~gement do not have the time or the software literacy

required to analyze and determine a response to potential problems

indicated in the status reports.

In order to improve high level Air Force visibility into potential software problems I

the Study Group recommends that the Air Force introduce a policy of oversight

management for all major software development programs.

Proposed Oversight Management Policy

All programs with major software components should be monitored and controlled

with the aid of oversight management techniq ues. Software oversight managers

should be appointed at the division level and at the Systems Command level.

A software oversight manager (SWOM) should be a senior level manager and

must be software literate. This does not imply that the SWOM is a computer

engineer /scientist or software manager I only that he have an appropriate level

of software knowledge to ask the right questions. The SWOM is not a technical

manager and should carefully avoid getting involved in any technical managementdetails without full concurrence of the SPO.

Duties of the Product Division Software Oversight Manager

The duties of the product division SWOM begin when the program is initiated.

The SPO must ensure that the Program Management Plan and the Computer Pro­

gram Development Plan are structured to facilitate summary status reporting to

oversight managers. T his reporting information must include:

a. An appropriately detailed PERT chart which includes a range of completion

time and a range of cost for each task and the critical paths based on these

ranges;

b. A list of the top ten high risk software elements consistent with the PERT

chart;

c. A hierarchica I decomposition of the functional capabi lities of the system.

A plan for reporting this information to the SWOM should be reviewed, under­

stood, and agreed upon by both the SPO and the division SWOM. All misunder­

standing should be resolved at this time.

During the duration of the program, the SPO will periodically provide -- nominally

on a weekly basis at the division level -- information to the product division SWOM

which describes the progress made during that reporting period. The reportshall include:

a. Total project costs incurred to the date of the report in consonance with

contractors reporting cycle;

b. Tasks, as defined in the program plan, completed during the reporting

period;

c. A progress Istatus report on each of the high risk software elements;

d. Any additions or deletions to the Iist of top ten high risk software elements;

e. Any changes in functional capability of the system which impact or are

impacted by the software~

T he report should be short -- a single page if possible -- and include the items

above and any others mutually agreed to by the SPO and the product divisionswor\'1.

The product division SWOM will review these weekly reports and watch for problem

areas including cost overruns, time-schedule slippage, changes in functional capa­

bilities, and problems with high risk software elements reported by the SPO. The

product division SWOM will normally not review the day-to-day technical details

of the program, however, the division SWOM will attend all major design reviews.

In addition to the above duties, the division SWOM wiII:

a. Report on the status of the program to the command SWOM. Using the available

information, the product division SWOM will report to the command SWOM pro­

gram level highlights and make appropriate recommendations, e. g., the program

is progressing satisfactorily or the program manager and lor other key personnel

should be replaced.

b. Be an advocate of the program at higher Air Force levels;

c. Have a technically competent person, not related to the program, review the

major technical documents and report to the product division SWOM any

problems which appear to be developing or any gross deletions which may

become apparent;

d. Act as SWOM for other programs in the product division;

e. Report to the SPO any political, monetary, technical or other information

which may impact the program.

The division SWOM will not probe into the technical details of the program without

agreement from the SPO.

Duties of Command Software Oversight Manager

The command SWOM will receive periodic -- nominally monthly -- reports on the

program status from the division SWOM plus intermediate exception reports if the

program goes critical. These reports wi II be primarily exception reports, that is,

what problems exist or appear to be developing, is the program in trouble, should

the program continue, should the program be reoriented, etc. Generally speaking,

the command SWOM should provide oversight for all software development programs.

The command SWOM will report his findings and recommendations to the commander

of AFSC.

The command SWOM will act as an advocate for software programs and provide

political, monetary, technical and other appropriate information to the division SWOMs.

Fig. 4 provides an example of the oversight management information flow.

Implementation of Oversight Management

A software oversight management system requires the following elements in order

to be implemented:

a. Organizational - The product division above any SPO with a software intensive

program and Systems Command should each have a software oversight manager

to review the progress of the program software.

b. Policy - A SWOM must be a software literate senior manager; appropriate 300/

800 series AF regulations wi II reflect this function.

c. Procedures - A plan must be developed prior to program initiation which

includes at least a cost schedule, a time schedule, a list of high risk soft­

ware elements and system functional capabilities. The SPO must report

information relative to program costs, program time-sche~ule, functional

capabilities and status of all program elements to the division on a periodic

(nominally weekly) basis. T he product division SWOM must report to the

command SWOM on a regular basis.

d. Information Support Capability - An automated management support capa­

bility should be developed which allows the SPO, the contractor, and the

two SWOMs to communicate. The SPO and the contractor shall provide

up-to-date program progress information.

e. Research and Development - Appropriate cost and time schedule models

should be developed. Risk analysis methods should be used in these develop­

ments. Appropriate tools for tracking the program progress should be

developed which are consistent with these schedules.

f. Education - Every SWOM should be trained in both computer software /hard­

ware engineering design concepts and practices and in oversight manage­

ment techniques.

More extensive background on the implementation of software oversight manage­

ment is given in [Ware-Patrick, 1983].

6. REFERENCES

[AFR 800-14, 1975]. Air Force Regulation 800-14, Volume II, "Acquisition and

Support Procedures for Computer Resources in Systems.," Hq. USAF,

Washington DC 20330, 26 September 1975.

[Boehm, 1981]. Boehm, Barry W. Software Engineering Economics, Prentice­

Hall, Englewood Cliffs, New Jersey, 1981, 767 pages.

[Brooks, 1975]. Brooks, Jr., F. P., The Mythical Man-Month, Addison-Wesley,

Reading, MA, 1975.

[Devenny, 1976]. Devenny, T. J. , "An Exploratory Study of Software Cost

Estimating at the Electronic Systems Division," Thesis No. GSM/SM/765-4,

Air Force Institute of Technology, Dayton, OH, July 1976.

[Ware-Patrick, 1983]. Ware, Willis H. and Robert L. Patrick, "Perspectives on

Oversight Management of Software Development Projects," Rand Note

N-2077-AF, Rand Corporation, July 1983, 50 pages.

NATURE OF PROBLEM

• USAF haa not achieved nece •• ary condition. for .ucc ••• ful.oftware project predictability & control

IfII'~",

4"" ......• •••• " ,,,,

c.s~~

1. Understanding, stabilizinq target location-Software requirements

2. Understanding delivery system dynamics- Software technology, personnel

3. Good navigation, guidanco, and control- Software program and oversight management

Figure 1.

Classes of people,

.....-data sources to support

Ouery types, data loads,intelligent-terminal tradeoffs,

response times

/ Internal data structure,

,/ butfer handling techniquesI/ Detailed scheduling algorithms,

,-' error handling/Prog;ar!1mer understanding

of specifications/'

Acceptedsoftware

Detailed

designspeci ficat ions

Product

designspecifications

Requirementsspecifications

Concept ofoperation

L::.

4)(

0.25 •.

2x

'.5)(

~ c:'.25)(::

•...•

)(0

c.:r.>.~ O.S ..•.it: ~c::0.67)(

0.5)(

Feasibility plilns ,ndrequirements

Productde5ign

Detailed

designDevelopment ilnd test

Philses and milestones

Figure 2. Software Cost Estimation Accuracy Versus Phase

3200

~

~ 2400'C'0-!:ic:

~ 1600

/"\

0

IActual

.r::.

II-

~cost

-800

4000

o 6 24 30

Figure 3.

~umbef of month, after contr.:t eward

Example Software Project Cost History

CONTRACTOR

PM

Status on

Top-1 0 potentia I

show-stoppers

SPO

Top 10

DIVISION

SWOM

Top 10

COMMAND

SWOM

Detailed Intermediate,IIntermediate

\Program level

programmer assign-

Detailed PERT,PERT,Highlights

ments, progress

Earned Value,Highlights,

Completion estimates

Major status,Major status

indicators

Iindicators

EXAMPLE OF OVERSIGHT MANAGEMENT INFORMATION FLOW

Figure 4.

Exhibit 1

Proposed Addition to AFR 800-14, Vol II: "Acquisition and Support

Procedures for Computer Resources in Systems"

CHAPTER II

RISK MANAGEMENT

11-1. Purpose. This chapter provides guidance on the management

of risk in computer resource acquisition and support. It discusses

the major sources of cost, schedule, and performance risk. It

identifies proven risk management techniques for computer resources,

such as prototyping, incremental development, and design-to-cost.

It provides guidance on the application of such techniques within

the system acquisition and support life cycle.

11-2. Risk Management. Risk management involves the application of

planning, analysis, engineering, and program management techniques

to identify and eliminate sources of cost, schedule, or performance

risk in the development and support of computer resources in systems.

11-3. Major Sources of Risk. The major sources of cost, schedule,

and performance risk in the acquisition and support of computer

resources in systems are the following:

a. The number and scope of the computer system functional and inter­

face requirements specified as necessary to support operational

missions.

b. The degree of stress in computer system performance requirements

(processing speed, accuracy, capacity, reliability, etc.) specified

as necessary to support operational missions.

c. The degr~e of definition and understanding of the computer

resource functional, interface, and performance requirements prior to

initiation of development or enhancement.

d. The degree of technical sophistication required to meet the

computer resource requirements.

e .. rhe degree of accuracy in the computer software cost and

schedule estimation techniques used.

f. The degree of mismatch between the personnel, technical, and

resource capabilities required for the job and the capabilities of

the developer and of the acquisition manager.

g. The degree of mismatch between the incentive structures of the

developer, acquisition manager, support organization, and user

during system development and support.

h. The effect of performance, interface, or schedule slippages of

support systems or interface capabilities required for system

development or enhancement.

i. The degree of stability of the computer resource functional,

interface, and performance requirements during development or

enhancement.

11.4. Risk Management Techniques. The primary techniques which

have been used successfully to identify and deal with sources of

risk in the acquisition and support of computer resources in

systems are given below.

a. Risk Management Plan. A plan which determines how the organiza­

tion will identify and deal with sources of risk, expressed in

relation to the major computer resource life cycle milestones.

b. System Engineering Techniques. The techniques discussed in

Chapter 4: functional analysis, simulation, mathematical modeling,

tradeoff analysis, etc.

c. Prototyping. The early development and exercise of critical

computer resouce components (user interface, network operating

system resource manager, key algorithm processor) for the purpose

of ~qentifying and eliminating sources of risk.

d. Multisource Cost and Schedule Estimation. The use of multiple,

independent organizations, techniques, and models to estimate costs

and schedules, including the analysis and iteration of differences

between the estimates.

e. Competitive Analysis and Design Contracts. The employment of

multiple contractors to develop competitive analyses, designs,

prototypes, and refined requirements specifications before proceeding

to selection of a development contractor.

f. Software Development Capability/Capacity Reviews. Survey visits

to bidders' computer system development organizations to determine

their readiness and capability to perform on a contract.

g. Incentive and Award Fee Contracts. Contracts basing a significant

portion of a contractor's fee on either a pre-negotiated incentive

structure or on performance ratings determined by the customer.

h. Incremental Development. Organizing the development of a system

into a series of increasing increments of functional capability.

Incremental development provides better opportunities to stabilize

requirements during system development, since proposed changes can

often be deferred to later increments.

i. Design-to-Cost. Fixing the development cost of a system, and

prioritizing system requirements so that the developer can deliver

the most high-priority system capabilities possible for the given cost.

j. Cost/Schedule/Performance Tracking. The use of such techniques

as work breakdown structures, activity networks, and earned value

systems to determine and track the progress of a project with

respect to its plans, schedules, and budgets.

k. Risk Element Tracking. The identification by a project manager

of the project's highest-risk development issues, and the tracking

of progress on these issues through subsequent progress reports.

11-5. Risk Management Implementation. This paragraph discusses risk

management implementation in its relation to the system and computer

program life cycle.

a. Conceptual and Validation Phases. During these phasesl the

following risk management techniques should be applied.

(1) System Specification Rationale Document. This document

should accompany the initial system specification produced during the

conceptual phase. It should summarize the results of system trade-off

studies for the information processing portion of the system, including

the results of prototypes or other risk-reduction investigations. It

should identify all the potential sources of risk in computer resource

acquisition, and indicate how these have been or will be resolved.

(2) Computer Resource Cost and Schedule Ranges. Multisource cost

and schedule estimation techniques should be used to determine the

likely cost and schedule ranges which will be required for computer

resource acquisition. These ranges should be used in preference to

point estimates in computer resource planning activities.

(3) Risk Management Plan. The Program Management Plan (PMP)

developed during the validation phase should include a risk manage­

ment plan. This plan should identify by phase the computer resource

risk management techniques which will be used during full scale

development, and assign responsibilities for the identification and

resolution of high-risk computer resource elements. The objective

of the plan should be to have identified and addressed all high-risk

issues by the computer program Preliminary Design Review (PDR).

b. Full Scale Development, Production, and Deployment Phases.

During these phases, the following risk management techniques should

be applied.

(1) Those risk management techniques identified in the PMP

Risk Management Plan. For mission-critical computer resources, these

techniques should nominally include prototyping, incremental develop­

ment, competitive analysis and design contracting, and incentive or

award fee contracts. Design to cost should be used optionally. The

use of fixed price contracts or best and final bids for computer

programs should be avoided unless the computer program requirements

are precisely understood.

(2) Software Development Capability/Capacity Reviews

(3) Development Risk Management Plan. This should be

developed as part of the Computer Program Development Plan. This plan

should identify by computer program development phase the risk

management techniques which will be used during computer program develop­

ment. It should include the use of cost/schedul~/performance tracking

techniques and risk element tracking techniques.

c. Computer Program Operations and Support Phase. During these

phases, the techniques applied during full scale development should be

applied for major computer program enhancement efforts.