metrics for project management
TRANSCRIPT
Member of the
Metrics for Project Management
An Experience Report from a Successful e-Government
Software Project
Project: UKSMA Conference Wolverhampton
Document–ID: Metrics for Project Management V03 d Version: V1.0-08
Issue of: 25. August 2003
Replaces: W0.3-02
Author: Dr. Thomas Fehlmann
Confidentiality: Publication
Distribution: ISBSG, UKSMA
Carbon Copy: SwiSMA
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 2 of 15 Copyright © 2003 Euro Project Office
Change Control
The following table documents the actual development stage of this document.
Every change made to this document requires a new issue.
Modification Notice Author Version Date
New document Dr. Thomas Fehlmann X0.1 14.5.2003
Added acquisition story Dr. Thomas Fehlmann X0.2 21.5.2003
Submitted to UKSMA conference Dr. Thomas Fehlmann W0.3 29.5.2003
Reviewed and finalized Dr. Thomas Fehlmann V1.0 25.8.2003
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 3 of 15 Copyright © 2003 Euro Project Office
Abstract
This case study explains the Six Step To Completion – method for progress tracking in projects. Progress tracking is a project management task, but it has specific effects in software projects. In this case it was the first time software people had to get used to metrics and that they draw some benefit out of it.
The critical success factor is to link software quality assurance to progress reporting. A task is no longer finished when the planned effort had been spent. We want the deliverables resulting from that task be reviewed and approved. Based on that data we track actual duration and effort spent in projects and compare with plans.
The effect is surprising: software projects suddenly behave “normal”. Teams stay focused on the planned tasks and keep to the deadline. Although bad effort estimation cannot be completely avoided, it is detected in the very early stages of the project. Delivery rates become apparent from the beginning.
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 4 of 15 Copyright © 2003 Euro Project Office
Table of Contents
Chapter 1: The Problem 5 1.1 e-WorkPermits – an e-Government application 5 1.2 The Project 5 1.3 The Dilemma 6
Chapter 2: The Approach Taken 7 2.1 The Taskforce 7 2.2 The Six Step To Completion – Metrics 7 2.3 Integrated Quality Assurance 8 2.4 Progress Metrics for Consensus Build-up 8
Chapter 3: The Successful Outcome 9 3.1 The Effects of Visualization 9 3.2 Why This Works 9
Chapter 4: Tools And Methods 10 4.1 The Project Database 10 4.2 Visualization using Excel 10 4.3 Data Capture with Personalized Tickets 12 4.4 Data Capture in the War Room 13 4.5 Automated Data Capture using a Project Repository 14 4.6 Relation to Other Software Metrics 14 4.7 Conclusion 15
Index of Figures Figure 1: The Six Step To Completion Model for Measuring Objective Evidence 7 Figure 2: Sample Project Management Overview 11 Figure 3: Sample Progress Track with Issue History 12 Figure 4: Sample Personalized Ticket 13 Figure 5: Wall Sized Print of the Progress Track as Paper Tool 14
UKSMA Conference Wolverhampton Publication Dr. Thomas Fehlmann Metrics for Project Management Version V1.0-13, 23 September, 2003
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 5 of 15 Copyright © 2003 Euro Project Office
The Problem
e-WorkPermits – an e-Government application
The Department for Labour and Economy of the Swiss canton of Zurich decided to undertake a bold step towards e-Government in 2002 when starting a project to ease the issue of work permits.
Especially in the ICT industry, but as well for banking and pharmaceuticals, the need to exchange international experts is apparent. In many other aspects of economical life, having the ability to transfer experts from one country to another is decisive for the attractiveness of a site. Thus setting up a web site that allows applying for a work permit through the Internet and – if the application is all–inclusive for the needed documents and the applicant fulfils the law’s require-ments – issuing the permits instantly is a major competitive advantage.
The Project
The trouble started at the very beginning. Having no expertise in setting up a project, the government department hired a consulting company to write an invitation to tender. This document did not mention project methodology or project management at all. Thus it was not required, and no details specified, therefore the government department had no means of discrimination.
A large consulting company won the contract and subcontracted several specialists for creating software and setting up the service on the Internet. The same system was to provide workflow support for the civil servants of the several agencies involved in the review and approval of the applications. Using the same data for external and internal users, the system should maximize effectiveness.
However, despite allocating around 30% of the total price for project manage-ment, the consulting company was not able to provide a skilled project manager. The first holder of the position left the project within the first two months, the second tried to manage the project by hiding the relevant information. There was no project plan, no project organisation, no roles and responsibilities, and for sure no quality management or quality plan. All he had was a Gantt chart, mentioning activities lasting two to three month, by a total planned project
UKSMA Conference Wolverhampton Publication Dr. Thomas Fehlmann Metrics for Project Management Version V1.0-13, 23 September, 2003
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 6 of 15 Copyright © 2003 Euro Project Office
duration of about seven month, and a To-Do list that an impressing tendency to grow in size.
Thus the project went the way all such projects must go: It hardly made the first milestone, already with a long list of To-Do’s, and completely missed the second. The author remembers his first project steering meeting where the project manager stated that one of these three-month tasks is “98% finished”. The author remarked that, when this statement had been true the day before the meeting, this task must have been finished by this very afternoon. Obviously it was not. In reality not even the hosting was prepared, and the software was buggy and very late.
The Dilemma
Obviously this was not state–of–the–art project management. The contract did stipulate, but not substantiate project management. It did neither mention a professional (PMI) certificate, nor any specific project management method. When the author joined the project and proposed to use progress metrics to the supplier, he did not accept the suggestion because it was not foreseen in the contract.
For the government department, there was no other way than to insist on the schedule as stated in the contract but they had to let the supplier decide how to meet its obligations. All they could do, and what they did, was to spend additional money for a second project manager who should take over after completion of the contract, and for independent testing, to avoid having to rely on the supplier’s testing records. We had to endure another three missed milestones before the supplier finally admitted that the delivery date had passed and the contract failed.
UKSMA Conference Wolverhampton Publication Dr. Thomas Fehlmann Metrics for Project Management Version V1.0-13, 23 September, 2003
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 7 of 15 Copyright © 2003 Euro Project Office
The Approach Taken
The Taskforce
Thus the government department installed a taskforce to take over and lead the project to its goal. The primary objective was that the supplier finished the delivery of the web application as soon as possible; thus it was not an option to stop the project. So how do you deal with people from whom you depend and who can listen neither to you nor to common lore nor even simply to good reason?
The Six Step To Completion – Metrics
The answer was a very simple metrics that was so simple those people from the supplier did not even understand that it is a metric indeed.
We started with a To-Do list because that was a terminology people could follow. However, each of the To-Do’s was connected to a distinctive deliverable, as every project manager does with tasks in his project plan. That deliverable uniquely identifies the To-Do and gives objective evidence for completion of the task. Professionals with an auditing background understand the term “objective evidence”. Instead of arguing if the task or To-Do had been completed or not, the evident status of the deliverable tells us without asking people’s opinion.
Figure 1: The Six Step To Completion Model for Measuring Objective Evidence
Draft versionDraft versionDraft version
OperationsOperationsOperationsApp-rovalAppApp--rovalroval
Review reportReview reportReview report ReleaseReleaseRelease DistributionDistributionDistribution
IdeaIdeaIdea
TemplateTemplateTemplateAssignmentAssignmentAssignment
Input
Output
DraftDraftDraft ReviewReviewReview FinalizationFinalizationFinalization
10% 30% 15% 20% 15% 10%
UKSMA Conference Wolverhampton Publication Dr. Thomas Fehlmann Metrics for Project Management Version V1.0-13, 23 September, 2003
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 8 of 15 Copyright © 2003 Euro Project Office
Integrated Quality Assurance
All we need is to provide a project repository where to post draft versions, review reports, released versions, and evidence for operational use such as distribution lists for documents and build logs for software.
We use the Six Steps to Completion – model to measure progress based on objective evidence for completion for each of the six steps. The idea stage accounts for 10% of the total completion metric; the draft stage for another 30%. Then comes the review stage with 15%, the finalization stage with 20%, approval gives 15% and when put into operational use the last 10%. The last stage is equivalent for making the deliverable available for developers who have to rely on it and does not necessarily refer to the end users.
Progress Metrics for Consensus Build-up
Originally these percentages were just a guess. However, in the very first project where such completion metrics had been used (1999) we discovered that these percentages correspond quite neatly with the total task duration (not the effort!) such that we started using the completion percentages for Gantt chart tracking.
The percentage values set forth in Figure 1: The Six Step To Completion Model for Measuring Objective Evidence seem to fit pretty well for creating documents, preparing workshops, training, or writing software. Thus, completion rates give an accurate estimate for the actual task duration. For example, if the completion rate is 40%, it means that we still need 60% of the total duration, i.e. 1 ½ times the duration that had already passed. If that was not foreseen by the project plan then you better update the plan right now to make it consistent with reality and manage expectations.
Only tasks such as meetings – with the minutes as deliverable – and testing may have somewhat different percentages.
We have since 1999 a record of around forty projects in a dozen different organizations that followed the same pattern with a variation below ten percent.
The grading of the steps varies in nine degrees between “just begun” and “completed”. Thus it is possible to acknowledge work in progress within the Six Stages To Completion. Also it often happens that review or testing activities and finalization go in parallel, thus the familiar pattern of two green stages followed by two yellow ones, see the last line in Figure 3: Sample Progress Track with Issue History. More questionable it is when deliverables are already in use while still under finalization, but it happens too, e.g. in object-oriented development.
The benefit of completion metrics based on reviews and approvals is that they reflect the degree of consensus reached for the deliverables. Reviews and finalization work still going on, or missing approvals clearly indicate missing consensus about the delivered results. Thus progress metrics are very important for management. This worked well in the e-Government project.
UKSMA Conference Wolverhampton Publication Dr. Thomas Fehlmann Metrics for Project Management Version V1.0-13, 23 September, 2003
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 9 of 15 Copyright © 2003 Euro Project Office
The Successful Outcome
The Effects of Visualization
You never forget when the supplier’s project manager first noted that his To-Do, which he thought having finished by “85%”, counted only for some 30%. The taskforce met then weekly in a room with the actual Six Steps To Completion displayed prominently at a wall. There was not a single date missed. And interestingly: even the taskforce cost remained within budget.
Why This Works
Obviously this was more than just a psychological trick. The reason is very simple: The Six Steps To Completion represents nothing else than ordinary quality planning. Every serious Quality Plan must provide exactly that: a reviewing method and an approval procedure for every single deliverable in the project.
Often quality planning is done at the beginning of the project (although not in this instance) but then not executed because of other priorities shovelling up. The effect of such missing quality assurance is not felt immediately, in contrary. It might even felt favourably for the moment because the QA resources are now free to do even more draft development. This seemingly positive experience will then be iterated again and again until it is too late and the quality problems start becoming apparent.
Exactly that happened in the early stages of the e-Government project, and was the reason why the supplier’s people were not able to listen.
The Six Steps To Completion – metrics makes this impossible. It short-circuits the quality plan to the project plan, or in this case, to the To-Dos. The team silently agrees on the necessary quality assurance being done at the root, causing significantly less bad surprises later on.
The web site www.workpermits.zh.ch went on-line 20 February 2003, three months late with regard to the original schedule, but three weeks earlier than expected in the revised schedule after the taskforce took over. All To-Dos including all the quality issues discovered earlier were resolved, the hosting strategies established, and the site started to receive very excited feedback.
UKSMA Conference Wolverhampton Publication Dr. Thomas Fehlmann Metrics for Project Management Version V1.0-13, 23 September, 2003
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 10 of 15 Copyright © 2003 Euro Project Office
Tools And Methods
The Project Database
Since 1998, Euro Project Office AG developed a tool1 based on MS–Project and Excel to integrate the Six Steps To Completion – method into project planning. It was developed and refined during the sequence of projects that the Euro Project Office supported.
The basic philosophy was usability. This included the need to let the project manager use his tool without any restriction. This led to the solution to use the database of MS–Project to store all relevant project data, such as deliverable specifications, quality planning, quality records, and issues, on top of the data already present in MS–Project such as planned and actual schedule, planned and actual effort and cost, resource allocation and usage. All the MS–Project functionality is thus available but with significant enhancements.
As already explained, the completion rates collected by the tracking process become in turn available again showing the actual completion percentage of the planned tasks.
Visualization using Excel
The choice of Excel for visualization makes it possible to easily adapt more functionality and analyze the project for later improvements. Excel as GUI is possibly not the best choice, but using VBA and the (loop-free) formula programmability of Excel allows for fast implementation at a high Function Points2 (FP) delivery rate. We make more than one FP per hour, or about six times more than Java or C++. Such a delivery rate outperforms some of the shortcomings of VBA.
1 The tracking software is Open Source and can be found in the /ch/open project methodology
(www.ch-open.ch, or www.swisma.ch). However, since it is embedded in MS Office, it does not install without proper support, and is not self-explanatory. At least, you need training, and you need a project office to run the tool.
2 Unadjusted Function Points according IFPUG 4.1
UKSMA Conference Wolverhampton Publication Dr. Thomas Fehlmann Metrics for Project Management Version V1.0-13, 23 September, 2003
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 11 of 15 Copyright © 2003 Euro Project Office
The available Excel reports include
Management overview of overall progress Specifications for each deliverable in the project Progress Track including issues per deliverable Quality Plan per deliverable Quality Records and access to deliverables Pricing per deliverable Test stories per deliverable Summary test results Efforts planned and actual per deliverable and per resource Summary planned and actual efforts per deliverable Progress data per project phase Costs and earned value per project phase Costs and earned value and expected trend per accounting period
Figure 2: Sample Project Management Overview
25. Januar 2003 Weight 10% 30% 15% 20% 15% 10%
Management Overview Idea
Dra
ft
Rev
iew
Fina
lizat
ion
App
rova
l
Rea
dy F
or U
se
Completion Rate 30% 41% 38% 34% 26% 19% 16%
Planned Completion 32%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Idea
Dra
ft
Rev
iew
Fina
lizat
ion
Appr
oval
Rea
dy F
or U
se
Overall Progress
Completion Rate
Schedule Trend
13.07.2002
01.09.2002
21.10.2002
10.12.2002
29.01.2003
20.03.2003
09.05.2003
28.06.2003
17.08.2003
06.10.2003
25.11.2003
13.0
7.2
002
01.0
9.2
002
21.1
0.2
002
10.1
2.2
002
29.0
1.2
003
20.0
3.2
003
09.0
5.2
003
28.0
6.2
003
17.0
8.2
003
06.1
0.2
003
25.1
1.2
003
Actual Dates
Plan
ned
Dat
es
Actual Date / Planned Date
These reports substitute all other project status reports. Because it is based on object evidence, such reports can be generated by the project office and need but comments by the project manager. Thus it replaces administrative work by a service, and saves cost in the project.
UKSMA Conference Wolverhampton Publication Dr. Thomas Fehlmann Metrics for Project Management Version V1.0-13, 23 September, 2003
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 12 of 15 Copyright © 2003 Euro Project Office
Figure 3: Sample Progress Track with Issue History
30% delivered 10% 30% 15% 20% 15% 10% 100%
Result ID Deliverable Resp.
Start Date Id
ea
Dra
ft
Rev
iew
Fina
lizat
ion
App
rova
lRea
dy F
or U
seCom
plet
ion
Rat
ePr
iority
Due Date D
elay
(Day
s)
1 Proposal Development 93%
1.1 Opportunity Memo AM 6-Jan 100% 1 6-Jan 3
1.2 Risk Analysis PM; RM; Arch
6-Jan 100% 4 16-Jan 6
1.3 Customer Value Analysis PM; RM; TPL1; TPL2; AM
6-Jan 100% 5 K 13-Jan 4
1.4 Quality Plan QM 13-Jan 100% 2 27-Jan 13
1.5 Project Plan PM & Team 17-Jan 100% 3 26-Feb 43
1.6 Statement of Work PM & Team 13-Jan 93% 4 25-Feb 46
1.7 Project Calculation PM 6-Mrz 70% 3 14-Mrz 58
1.8 Business Plan AM 13-Jan 62% 3 K 14-Jan -1
1.9 Contractor Qualification Buyer 25-Feb 100% 5 5-Mrz 44
1.10 Select Contractor Buyer 5-Mrz 100% 3 13-Mrz 50
1.11 Proposed Solution & Commercial Offer
PM; QM; Arch; AM
7-Mrz 100% 3 T 17-Mrz 46
2 Analysis & Design 70%
2.1 Requirements Engineering 58%
2.1.1 Clickable Prototype Dev2; Dev3; Arch
24-Mrz 76% 3 5-Mai 66
2.1.2 Business Process Analysis Dev1; Arch; Exp; TPL1
31-Mrz 45% 3 5-Mai 61
2.1.3 Use Case Analysis Arch; PM; Customer
5-Mai 52% 3 19-Mai 68
Issue Reporting
History Records
17.07.2002: Who is responsible for gathering marketing data?11.06.2002: Marketing department task force to provide available data until 20.1.
17.07.2002: Delay because of resource problems.Search for available skills. Result due by 20.3. Final Project Organziation must be communicated to customer.
17.07.2002: Some minor points need clarification by the subcontractorSubco will provide statement by 18.3. Consolidate with project plan.
17.07.2002: Too many open points.We conduct a 2nd review by 18.3. to get agreement on the open points.
17.07.2002: Project Calculation complete. Formal approval asap after project calculation is final.Get approval immediately after 2nd review on 20.3.
21.07.2002: Missing feature added12.07.2002: Prototype lacks the priority setting feature
08.04.2003: 2nd review planned for 11.4.2003
In the past project experience, it was possible to integrate with an ERP tool such as SAP, or with configuration management such as CVS.
The tool is also available as an option to SCODi4P from Triloga1, or as an add-on to Bugzilla2. Currently we investigate migration to a DBMS to improve the GUI.
Data Capture with Personalized Tickets
The progress-tracking tool is not only integrated with the project repository, but also with the reporting process of each team member. Every team member gets his own personalized ticket, either via the Intranet, the project home page or any other suitable communication tool (including paper).
The tickets integrate all of the reporting needed in a project, including time reporting.
1 Contact Triloga AG, Lucerne, Switzerland; www.scodi4p.com
2 Contact GMC Software Technology, www.gmc.net
UKSMA Conference Wolverhampton Publication Dr. Thomas Fehlmann Metrics for Project Management Version V1.0-13, 23 September, 2003
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 13 of 15 Copyright © 2003 Euro Project Office
Figure 4: Sample Personalized Ticket
17.3. Priority 10% 30% 15% 20% 15% 10% 100%Requirements Engineering
Result ID Deliverable Id
ea
Dra
ft
Rev
iew
Fina
lizat
ion
App
rova
lRea
dy F
or
Use
Com
plet
ion
Rat
e
Due date D
elay
2.1.1 24.3 76% 5.5 179
14-Nov-02 30. Okt 01. Nov 10. Nov 15. Nov 07. Nov
Quality MeasurementsRelease Identification Version Under Test
Delivery item identification Actual Vers Tool Improvements Function Pointsppt 12 45 36 8 11 0 130
Quality Assessment Re-Factoring Runs1 Reviews 1 Test Case 0 Regression Tests
Average Test Success Average Test CompletionGrade 0 - 5
Assessment Result Reviewed by Approved by
Date of Approval
Work Reported (in Hours)
Arch
V02
Clickable Prototype Dev2; Dev3;
Pilot Functionality
PMCustomer
100%
Usability Test passed. Found that requirements need to be refined for various user categories.
3.0
Prototype Solution
Cycle 1
Base
line
Wor
k Bud
get
Actu
al W
ork
Perfor
med
Perfor
med
Th
is P
erio
dEs
timat
e To
Com
plet
e Ex
pect
ed A
t Com
plet
ion
Wor
k Ove
rrun
Specification 72 82 8 7 97 25Working on this task
2 1 0 3Hours32 5 7 4448 2 0 50 8 h/d
0 5 d/w0 4 w/m
ArchDev2
Rapid prototype displaying the functionality needed.
Dev3
Data Capture in the War Room
Although electronic communication means are abundant and sometimes indispensable, we made excellent experience with a more traditional way for data capture. Instead of distributing and collecting electronic tickets, a wall-sized print of the progress track sheet (similar to Figure 3: Sample Progress Track with Issue History) serves the same purpose. The team members report progress using colored markers, which they glue to the wallpaper. It requires more physical presence of the project office that has to input the data captured on the wallpaper into the Excel sheet but minimizes disruption for the develop-
UKSMA Conference Wolverhampton Publication Dr. Thomas Fehlmann Metrics for Project Management Version V1.0-13, 23 September, 2003
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 14 of 15 Copyright © 2003 Euro Project Office
ers and maximizes their sense of responsiveness and ownership. This effect could be worth considering as a trade off.
Figure 5: Wall Sized Print of the Progress Track as Paper Tool
Automated Data Capture using a Project Repository
If a project repository is available, data capture is even more automated. The draft deliverables, quality records and approvals can be tracked in the project repository and progress track relies less on people’s judgement. Team members still have to update time and issue reporting on the tickets and may want to set the draft or the review to some intermediary progress grade (yellow marker). However they can no longer pretend be in finalization state without at least a preliminary review report being available in the project repository1.
Relation to Other Software Metrics
Bad effort estimation cannot be completely avoided. Using the Six Steps To Completion method, errors in estimation are detected in the very early stages of the project. Delivery rates become apparent from the beginning. 1 For a project repository that integrates well see itp.montana (www.itp-solutions.ch, in German)
UKSMA Conference Wolverhampton Publication Dr. Thomas Fehlmann Metrics for Project Management Version V1.0-13, 23 September, 2003
METRICS FOR PROJECT MANAGEMENT V10.DOC Page 15 of 15 Copyright © 2003 Euro Project Office
When the team starts understanding that the progress metrics are not a threat but a valuable help for them, their reluctance against using metrics vanishes.
The GMC team in Hradec Králové (Czech Republic) integrated the method in the open source tool Bugzilla. Thus tracking project tasks and bug fixes becomes the same process. Developers have only one tool that supplies them with new tasks, with their specifications, test cases, records test results and review finding, and tracks effort and progress. It provides progress metrics and bug statistics at once, with no pain.
Clearly we would like to add a software sizing metric to the tool. However this yields some specific problems. The work breakdown structure of a software project does not automatically coincide with the software boundaries. Some-times it is possible to group development tasks such that they fit with logical boundaries, but it make not always sense.
Nevertheless, such an approach is very promising and deserves further investigation.
Conclusion
The Six Steps To Completion – metric is easy to use, simple and straightforward. In the beginning, it is often not even perceived as a metric, it stirs only little opposition, and almost no fear against being measured.
On the other hand, the effects are tremendous. If planning and estimation was correct, the schedule will be respected. The metric establishes a short – circuit between plan and team. The project manager perceives unrealistic assumptions and estimations within a single reporting period. We most often use one week, sometimes two weeks as a reporting period. Thus corrective actions can be taken immediately when needed.
Metrics provide huge benefits for people and organizations. Progress metrics are the door opener for other metrics dearly needed for today’s complex tasks. Moreover, many more sophisticated metrics such as software sizing using Function Points or bug statistics provide higher value when combined with progress metrics. This is why we us no metrics without progress metrics in our software projects.