Estimating Project SuccessEstimating Project Success

June Verner and Barbara KitchenhamEmpirical Software Engineering

NICTA

Outline

• Background• Data

– Factors– Data sets

• Methodology– Factor analysis– Logistic regression– Results summary

• Conclusions and further work

Background

“Billions of dollars are wasted each year on failed software projects... ….we have a dismal history of projects that have gone

awry” [Charettte IEEE Spectrum Sept 2005]

• Have been developing software since the 1960s but still have not learned enough to ensure project

success

• Most project failures are predictable & avoidable

• How can we identify these projects early enough to take action?

Failures

• Most organizations try to hide their failures– Not only monetary loss, but also lost opportunity

• A recent “Hall of Shame” includes (in $US millions) – FBI 100– UK Inland Revenue 33– Ford Motor Company 400– Sainsburys 527– Sydney Water Corp 32.2+

• Other recent Australian problems include:

– National Australia Bank AUD200 million write down on failed ERP project,

– RMIT’s Academic Management System– Victorian State’s Infrastructure Management System– Continued controversy over the Federal Government’s new sea

cargo import reporting system

Data - Factors

• Literature• Discussions with 90+ developers• Categories

– Sponsor– Customer and users– Requirements– Estimation and scheduling– The project manager– Project management– Development process– Developers

Data sets

• Mostly in-house software developments– North American financial Institution

• 42 projects • 45% success rate

– Other NE US projects• 79 projects• 71% success rate

– Sydney• 42 projects• 78% success rate

• Chile 200+– In house– Developments for third parties

Methodology

• Correlate all factors with project success• Consider only those at the 95% level

– Overall– By groups

• Remove factors with large number of missing values• Use factor analysis on reduced set of variables to develop a

new set of variables suitable for predicting project success• Use these variables to develop prediction equations on

entire data set and by groups– How do these equations compare?

• Take original reduced set of correlated factors and develop prediction equations overall and for each of the groups

• Compare the results with the equations developed with reduced set of factors.

Correlations with project success

• Only variables correlated at the 95% level across 3 groups and overall

• Sponsor - nil• Customer and Users

– level of confidence of customers in the project manager, team members

– customers had realistic expectations• Requirements

– were requirements completed adequately at some stage– good requirements overall

• Estimation and Scheduling– how good were the estimates?– staff added late to meet an aggressive schedule?

• Project manager– the PM communicated well with the staff?– how good was project manager?– how well did project manager relate to software development staff?

Correlations with project success

• Development process– Adequate time was allowed for each of the phases

• Development team– How well did the team members work together?– How high was the motivation of the team members?– What was the working environment like?

Not included

• Sponsor– Project manager given full authority to manage project– Sponsor commitment (2)

• Customer and Users– Level of customer involvement (1)– Customer turnover (1)– Large numbers of customers and users

• Requirements– Adequate time made available for requirements gathering (2)– Central repository (2)– Size impacted requirements gathering– Scope was well defined (1)

• Estimation and Scheduling– Estimate of delivery date used adequate requirements

information (2)– Developers were involved in the estimates– Adequate staff assigned to project (1)– Developers were involved in the estimates (1)

Not included

• Project manager– Project manager background– Years of experience– Experience in the application area– Project manager had a vision of what the project was to do for

the organization (2)• Project management

– Did the PM control the project? (1)– Staff were appreciated for working long hours (2)– Staff were rewarded for working long hours (2)

• Development process– Defined development methodology used (1)– Risks incorporated into project plan (2)– Requirements managed effectively (2)

• Developers– Total number of staff (1)– Team members consulted about staff selection (2)

Factor analysis

• 75% of variance explained with 3 factors• Factor 1- Project manager

– The PM communicated well with staff– How good was project manager?– How well did project manager relate to software

development staff?

• Factor 2 - Customers and requirements– Level of confidence of customers in the project manager

& team members– Customers had realistic expectations– Good requirements overall– How good were the estimates?

• Factor 3– Staff added late to meet an aggressive schedule

Logistic regression - overall

27 17 62%failed

7 83 92%succeeded

82%Overall

All three factors

Logistic regression-by group

Group 1 Factor2 17

3314

85%82%84%

FailuresSuccesses Overall

Group 2Factor2 & Factor3

62

940

40%95%81%

FailuresSuccesses Overall

Group3Factor2 5

0431

56%100%90%

FailuresSuccesses Overall

Marginally better than overall 24 wrong versus 23

Logistic regression-Original 12 variables-overall

Level of confidence of customers in the project manager, team members

Overall reqts were good

How good were the estimates

Staff added late to met an aggressive schedule

29

11

15

79

66%

88%

81%

Failures

Successes

Overall

Logistic regression-original 12 variables-groups

Group 1Reqts good overallHow good were the estimates?

186

111

95%64%81%

FailuresSuccesses Overall

Group 2Reqt good overall 6

2940

40%95%81%

FailuresSuccesses Overall

Group3Staff added late to meet aggressive schedulePM communicated well with staff

63

328

67%90%85%

FailuresSuccesses Overall

Factors not significant for any of the groups

• Project manager given full authority• The project began with a committed champion• The committment lasted right through the project• The sponsor was involved with decisions• Other stakeholders comitted and involved• Senior management impacted the project• Involved customers/users stayed throughout the project• Customers/users involved in schedule estimates• Problems were caused by large by the number of customers/users involved• Reqts were gathered using a particular method• Was the manager involved in estimate?• The project had a schedule• There was a project manager• Years of experience of the PM• Was project manager experienced in application area?• Was project manager able to pitch and help if needed?• The consultants reported to the project manager• Did all the key people stay throughout the project?• Rewards at end of project motivated team

Results - summary

• Did better with the extracted factors overall– 24 wrong versus 26

• Did better with the extracted factors with groups– 21 wrong versus 24

• Nearly 3 times as many failed project predicted incorrectly

• Data set with too few failures problemmatic

Conclusions and further work

• Would it help if we had values for:– Adequate time was allowed for each of the phases– How well did the team members work together?– How high was the motivation of the team members?– Working environment

• Next steps– How best to deal with the missing values?– Use Bayesian networks– What are the missing failure factors?

• Don’t believe that in all cases failure is the converse of success