publishable article
TRANSCRIPT
Project Success Criteria for a South African Power Utility
PUBLISHABLE ARTICLE
Presented to the
Graduate School of Business Leadership
University of South Africa
In partial fulfilment of the
Requirements for the
MASTERS DEGREE IN BUSINESS LEADERSHIP
UNIVERSITY OF SOUTH AFRICA
by
N MOODLEY (30683475)
30 NOVEMBER 2004
2
1 Introduction
Eskom, a South African power utility, is undergoing a period of rapid growth due to
the inherent increase in electricity demand in the sub-Saharan region. Much of this
growth is directly linked to ensuring that the present plant is reliable and able to
sustain its position for the long term, so as to adequately support the ‘new build
capacity’ projects. This article focuses on one of the business units of Eskom known
as Peaking. Eskom Peaking assets cover the hydro electric, pumped storage, gas
and wind generating plants of Eskom. Most of the Peaking plant is in the latter stages
of its life cycle and numerous refurbishment projects are planned in the next decade
so as to ensure that peak power capacity is available and reliable.
Problem Statement
Due to the large capital investment in projects over the next five years, management
is faced with the challenge of evaluating the present level of project success and to
develop effective performance measures to ensure the success of projects in the
future. Of utmost importance are the identification of success areas that could be
replicated within the Eskom business and the detection of areas needing
improvement.
Although numerous projects have been completed with some perceived level of
success, management would like to have specific measures in order to quantify such
success. Presently, only a few measures are used to evaluate project success
formally, thus it is paramount to identify what other factors affect the success of
projects.
In this study, the following are a list of questions from management;
• What are key performance indicators for ensuring future project success?
3
• Are time, cost and quality the only important project success criteria?
• What other factors affect the success of projects?
• What are the areas of concern / focus?
• Does quality affect project success?
• Is adequate time allocated for project execution?
• Does meeting the user requirements have an effect on the success of projects?
Objectives
The aim of this article is to identify archetypal success criteria for engineering and
refurbishment projects so that adequate key performance indicators for project
success are available to management.
The outcome of this study must also highlight the importance or non importance of
relevant secondary project success criteria. The conclusions drawn from this study
should provide a guideline to other power utilities and industries who envisage similar
refurbishment projects or projects of a similar nature.
The final analysis of this study provides an investigation into the management focus
areas so that management can concentrate its efforts on salient issues regarding the
improvement of the project process. The main question posed by management is
whether the traditional time, cost and quality measures are the only criteria to
measure project success. This forms the basis for this study which is composed of
an investigation into the current success criteria identified from a literature review and
interviews. In these interviews, selected criteria are tested on Eskom Peaking
stakeholders to identify the most significant criteria that can be used to evaluate the
success of future projects. The analysis is based on data obtained from
4
questionnaires and is presented in the form of descriptive and statistical analyses to
identify correlations between the various success criteria and relevance to the
business unit.
Paper Proposition
The following hypothesis will be tested in this article; Time, cost and quality are the
only important success criteria for engineering and refurbishments projects.
2 Theory and Practice of Project Success
Project purpose
It is appropriate to first identify the purpose of a project. Most articles describe a
project as being a unique activity having a clear goal with definite start and finish
dates. The whole reason for creating and implementing a project is to satisfy very
specific objectives of the business. Projects are viewed as being the execution of all
the visions, missions and strategies of an organisation [1]. This concept is further
elucidated by the PMBOK� Guide (PMI, 1996), highlighting that “all projects should be
supportive of performing the organisation’s strategic goals”. This places emphasis on
the projects’ meeting and supporting the strategy and long term goals of the business
[2]. In the case of Eskom Peaking, refurbishment projects are created to ensure that
the plant maintains a high level of availability and reliability. This would, in the long
term, ensure that the peak power needs within South Africa and sub-Saharan Africa
are adequately satisfied with the least cost of production. These projects also allow
for ‘future build projects’ to be given a chance to develop without the strain of being
overly fast tracked. The question is: What are the criteria for ensuring that the project
objectives are met and how are these measured?
5
Success Criteria
Success criteria are the measures by which success or failure of a project or business
is judged [3]. It is a set of measures that are understood by the relevant stakeholders
and, if performed adequately, would guarantee some level of success. De Wit [3]
further emphasises that project success is measured against the overall objectives of
the project whereas project management success is measured against the traditional
measures of performance against time, cost and quality. He further highlights that by
using cost as a measure, one is measuring the progress made and not the level of
success. Baccarini [4] proposes that project success is composed of two distinct
components. One is project management success (time, cost, quality, process) and
the other is product success (effects of the project’s final product).
Other authors (deWit [3], Wateridge [5], McCoy [6], Pinto and Slevin [7], Saarinen and
Ballantine [8], Davies [9]) also share the view that time, cost and quality are the basic
measures, but the assessment of project success is not only limited to these criteria.
Some authors are of the opinion that time, cost and quality are measuring the delivery
stage of a project. The post delivery stage, which identifies success criteria set by the
stakeholders [5], also plays an important role.
Stakeholders
The concept of project success has instigated numerous debates on its definition and
scope due to the very nature of its purpose. Does one evaluate the impact of the
project, or the way the project was implemented? Different stakeholders have
different views on what criteria identify the project as being a success [10]. This
would be driven by the unique need that is being satisfied by the project. An opinion
that is clearly identified in the literature review is that project success criteria must be
6
clearly defined before the project commences so that all stakeholders fully
understand the requirements and can work towards a common goal [5]. According to
Roger Atkinson [11], one reason for projects being perceived as failures is that the
criteria for evaluation have changed, yet the same old criteria are still being used to
assess project success.
Different stakeholders tend to place focus on different project criteria. Struckenbruck
[12] has suggested that the four most important stakeholders to decide on the
success criteria are the project manager, senior management, customer-client and
the project team members. The project manager would be content if the project
duration and budget are met with reasonable quality. The customer of the project is
satisfied if the user requirements are met with no, or insignificant, post-project
defects. Senior management focuses on how the project has contributed towards
achieving the strategy of the business and its alignment to achieving the company
goals. Van Aken [13] defines project success as “the satisfaction of all stakeholders”.
Research on project success suggests that it is extremely difficult to identify generic
project success criteria that can be used for the evaluation of all projects. Even
though some projects can be grouped into common types, the stakeholders have the
final say on the performance of the project. Each stakeholder brings his/her unique
opinions and perceptions of project success. The culture of the organisation also
plays a pivotal role towards defining inherent project success perceptions [14].
Alternative Criteria
Studies have shown that there are identifiable factors that are critical to ensure the
success of projects [15]. These are: communication throughout the project, clear
objectives and scope, breaking the project into ‘bite sized chunks’ and using project
7
plans as working documents. Morris and Huge [16] have highlighted that the success
of a project depends on the following factors: a realistic goal, competition, client
satisfaction, a definite goal, profitability, third parties, market availability, the
implementation process and the perceived value of the project.
Shenhar, Levy and Dvir [17] have identified four independent dimensions of project
success. These are: project efficiency, impact on customer, business success and
preparing for the future. Project efficiency covers both the time and cost aspect of
the traditional measures. Impact on the customer encompasses the meeting of the
technical and operational specifications of the project, fulfilling the customer’s needs,
actual use by the customer, solving of the customers major problems and, finally, the
overall aspect of customer satisfaction. The business success criteria for internal
projects include measures such as yields, cycle times, processing steps and quality.
The fourth dimension, which is preparing for the future, includes a measure to ensure
that the internal project contributes effectively towards developing new technology,
skills or competencies.
From studies carried out by Wateridge [5], it was found that for all stakeholders
involved, the following were the success criteria in order of importance: meet user
requirements, happy users, achieves purpose, meets budget and meets time.
3 Research Methodology
This study employs a normative investigative style by sending a questionnaire with
statements ranked on Likert scales. The criteria listed in the questionnaire are those
that were identified and deemed appropriate from the literature research and other
criteria suggested by the stakeholders in preliminary interviews. The questionnaire
8
also allowed for an open ended question which required the respondent to list any
other criteria that they thought relevant for the measure of project success. The
questionnaire was divided into two sections and designed to allow comparisons to be
drawn between the relevant project stakeholders. Questionnaires were sent and
received via e-mail. This was the most convenient and quickest means of response
due to Eskom Peaking having six power stations that are geographically dispersed
throughout South Africa. The survey concentrated on respondents in the business
that were directly involved in, or affected by, projects implemented. The following
stakeholders were identified: senior/line management, production/client, commercial,
finance, project managers and system/project engineers. Questionnaires were
initially tested on five respondents to identify whether they were easy to understand
and complete. Eskom Peaking consists of 241 employees, of which 62 employees
were directly affected by the projects. Questionnaires were sent to those 62
employees and a response was received from 37 of them, with a response rate of
59.67%. This was considered adequate for this study.
The following criteria were identified as the most relevant for Eskom Peaking by
means of initial interviews with key stakeholders and by the literature review. These
were used in the questionnaire:
o Time - Completing the project in time
o Cost - Completing the project within budget
o Quality - In the context of Eskom Peaking, quality is defined as meeting the
technical specification, no defects after one month of installation and having all
documentation in place for implementation and at close out of the project.
o User requirements - Meeting the customer’s specifications and requirements
for the project.
9
o Customer satisfaction – Satisfying the needs of the customer. This is more
than just satisfying the user requirements.
o Satisfy business goals - Does the project satisfy the goals and strategy of the
business?
o Communication – This criterion identifies if there was adequate communication
throughout the project. It was deemed important by some clients who needed
to be kept informed of all aspects of progress on the projects implemented on
their plant.
o Impact on plant – This criterion identified the impact that the project would
have on the plant in terms of the unit performance, availability and reliability.
o BEE – This criteria was included since it was a performance criteria prescribed
by the Eskom corporate office to improve the level of support for black
businesses in the market. BEE stands for Black Economic Empowerment
companies.
o Contractor management – the success of the project was also linked to how
the contractor was managed during the implementation phase of the project.
This could have an impact on resources, planning and contractual issues
which was considered important enough to include as a success criterion.
4 Research Observations
Figure 1 summarises the number and percentage make-up of the respondents who
participated in the questionnaire survey. An adequate number of responses were
received from the relevant stakeholders making the survey representative of the
business.
10
Figure 1: Respondent Make-upRespondents No. %Senior/Line manager 7 19%System/project engineer 13 35%Production/Client 8 22%Project manager 4 11%Finance 2 5%Commercial 3 8%
Total 37 100%
Figure 2 summaries the responses on the Likert scale for all respondents where
averages above 4.00 are considered significant. The results indicate some level of
concurrence on the relevance of the criteria chosen from the literature review and
initial interviews, since all success criteria except BEE and contractor management
were rated above 4.00. The BEE rating was consistently rated low by all
stakeholders. The highest rating was quality, which was closely followed by impact
on plant, satisfy business goals and time. Customer satisfaction, user requirements,
communication and cost were the next set of criteria rated above 4.00.
Figure 2: Likert averages for all responses
4.34
4.03
4.63
4.20
4.26
4.34
4.06
4.50
3.06
3.91
0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00
Time
Cost
Quality
User Requirements
Customer Satisfaction
Satisfy business goals
Communicaion
Impact on Plant
BEE
Contractor Management
Comparisons made between the main stakeholders revealed important focus areas.
Figures 3-5 indicates the Likert averages for the production/client, project managers
11
and senior/line management respectively. The client places significant focus on
satisfy business goals, customer satisfaction and impact on plant which were rated
significantly higher than time and quality. Other positive criteria were communication,
contractor management and cost. The lowest rating was BEE, which was consistent
with the overall results.
Figure 3: Likert averages - Production/ client
4.38
4.00
4.50
4.38
4.63
4.75
4.13
4.63
3.00
4.13
0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00
Time
Cost
Quality
User Requirements
Customer Satisfaction
Satisfy business goals
Communicaion
Impact on Plant
BEE
Contractor Management
The project managers displayed different opinions and placed high focus on quality.
The other significant criteria which were rated the same were time, cost, user
requirements, and impact on plant. These were consistent with the notion that project
managers concentrate on the immediate outputs of the project which measures direct
progress instead of the long term goals. They awarded a low score for customer
satisfaction, satisfy business goals and communication. Contractor management and
BEE were both rated below 4.00. It was considered unusual that the project
managers did not focus highly on contractor management as it was one of their key
functions.
12
Figure 4: Likert average - Project manager
4.50
4.50
5.00
4.50
4.25
4.00
4.00
4.50
3.00
3.75
0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00
Time
Cost
Quality
User Requirements
Customer Satisfaction
Satisfy business goals
Communicaion
Impact on Plant
BEE
Contractor Management
Senior/Line management rate satisfy business goals and quality as high focus areas;
with time, customer satisfaction, and impact on plant rated equal third highest. User
requirements, communication, contractor management and BEE were all rated below
4.00 indicating that management does not place much focus on these measures.
Figure 5: Likert average - Senior/line management
4.29
4.00
4.57
3.86
4.29
4.57
3.71
4.29
3.29
3.57
0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00
Time
Cost
Quality
User Requirements
Customer Satisfaction
Satisfy business goals
Communicaion
Impact on Plant
BEE
Contractor Management
Figure 6 depicts the response from stakeholders on their perception of time, cost and
quality being the only success criteria for projects. The results indicate that 62% of
13
the respondents think that these are not the only criteria for evaluating project
success. An interesting summary is obtained when these results are broken down
according to the individual stakeholders. It is clearly demonstrated that senior/line
management and system/project engineers are of the opinion that time; cost and
quality are definitely not the only criteria whereas the project manager, finance and
commercial departments are of the contrary opinion. Production/client is indifferent to
these criteria and considers the other highlighted criteria to be just as important as
time, cost and quality.
Figure 6: % Response - "Are Time, Cost, Quality the only success criteria?" Respondents Yes NoSenior/Line manager 0% 100%System/project engineer 23% 77%Production/Client 50% 50%Project manager 75% 25%Finance 100% 0%Commercial 67% 33%
Overall 38% 62%
An interesting point is noted from Figure 7. Respondents who answered “Yes” to the
question “Are time, cost and quality the only success criteria?”, scored higher for all
criteria except for customer satisfaction and satisfy business goals. This gives the
impression that respondents who answered “No” were more interested with the long
term goals of the project and how it satisfied the needs of the stakeholders.
14
Figure 7: Yes/No Likert average scores
4.43
4.79
4.50
4.21
4.21
4.07
4.57
3.14
3.93
4.30
3.87
4.48
4.00
4.30
4.43
4.04
4.22
3.00
3.87
4.29
0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00
Time
Cost
Quality
User Requirements
Customer Satisfaction
Satisfy business goals
Communicaion
Impact on Plant
BEE
Contractor Management
Average Yes Average No
A Pearson moment correlation was performed on the project success criteria to
identify possible correlations and relationships between the criteria. The critical
significance level is taken as <0.05. The results are depicted in Figure 8 and indicate
that time is positively correlated with user requirements, customer satisfaction and
impact on plant.
15
Cost is positively correlated with quality and BEE. Quality demonstrates positive
correlation with satisfy business goals, communication and impact on plant. User
requirement correlates positively with customer satisfaction, impact on plant and BEE.
Communication correlates positively with impact on plant, BEE and contractor
management, with the highest correlation being with contractor management. BEE
and contractor management correlates positively with impact on plant, with BEE and
contractor management also being positively correlated.
Figure 9 summarises the response from stakeholders when asked if they could
highlight alternate criteria not mentioned in the questionnaire. These criteria were
grouped into the relevant categories so as to identify common dimensions of criteria.
Knowledge transfer was highlighted by both project engineers and project managers.
When questioned on this response, stakeholders highlighted that there was a need
for personal development and growth of project team members.
Figure 9: Success Criteria Highlighted by RespondentsOther Criteria Highlighted FrequencyKnowledge Transfer 4Value for Money 4Project Management Style 3Working relationship 2Effective Project Planning 2Long Term Impact 2
Value for money is another criterion that respondents have highlighted as being a
meaningful indicator of project success. This is closely linked to achieving the
business goals at the least cost with the most positive impact on the plant. Project
management style is a further aspect that has been highlighted for consideration as a
project success criterion. Respondents further highlighted that this factor could have
a direct impact on communication, contractor management and customer satisfaction.
16
Other criteria that were mentioned were working relationships, effective project
planning and long term impact of the project.
5 Discussion
From the results obtained in the questionnaire survey, it is clear that time, cost and
quality are not the only criteria for measuring project success. This is consistent with
most studies carried out on project success [3, 5-8]. The questionnaire response
from the different stakeholders re-enforces the choice of criteria selected, as the
averages for all criteria were above the rating of 4.00 for all respondents, except BEE
and contractor management. The BEE criterion is viewed as a short term
performance criterion, which is more of a corporate indicator than one of strategic
business requirements. The low rating was consistent among all respondents.
Although this may be an important business target it has not been a major focus area
on projects, as this was traded off with the other important criteria.
This study also highlights that different stakeholders have different criteria to evaluate
project success. The project manager is concerned more about the short term
requirements and about delivering the project within its constraints. Management is
concerned more about the long term and strategic requirements. The customer
focuses on the impact of the project and how it satisfies his/her needs. Measurement
of their performance can then be based on these criteria. In this way the project team
will be able to adequately focus on the salient measures to ensure project success.
The criteria highlighted in this study would then ensure that both the needs of the
business and customers are met.
17
The five most important project success criteria highlighted for projects within Eskom
Peaking are: quality, impact on plant, satisfies business goals, time and customer
satisfaction. Although cost is not one of the five most important highlighted, it would
form part of the evaluation criteria so that sufficient emphasis is placed on effectively
managing the cost of a project. This is due to the fact that Eskom, like most power
utilities, strives to generate the lowest cost electricity which impacts on the strategic
requirements of the business, industry and the Southern African region. It is felt that
management would gain enormously by using the identified success criteria to
understand and formulate how future projects contribute towards creating value for
the business and improve the processes for managing future projects.
Quality was rated consistently higher by all stakeholders. This is consistent with the
culture portrayed within the Eskom Peaking business unit. The division recently
undertook the ISO9001 rating and the emphasis placed on quality over the past few
years has been significant. Quality has a direct impact on the plant, which could
affect the availability and reliability of the power stations adversely if projects are not
implemented properly or have defects which could result in plant failures. Quality is
also positively correlated with satisfy business goals, communication and impact on
plant. This is consistent with the belief that the better the quality, the more the plant is
positively affected and in this way satisfies the goals of the business. This
relationship also emphasises that the better the communication between all project
stakeholders, the higher the level of quality of the project. It is felt that quality is a
significant success criteria and management must ensure that the level of quality of
projects are always maintained at a high level.
18
The criterion impact on plant plays a major role for the reliable supply of electricity
especially during emergency and peak power loading on the power network. These
are critical demand periods when Peaking plant would be called upon and projects
implemented must ensure that the plant is impacted on positively with respect to
performance and availability. This criterion is deemed relevant and consistent with
the stakeholder ratings. Satisfying the business goals is quite important especially if
it calls for long term sustainability of the plant and a strategy that ensures that Eskom
as a whole has enough capacity to satisfy the rapidly growing demand of electricity.
Projects must be implemented in a manner that effectively supports this need.
Time, which is one of the three traditional measures, is considered relevant especially
when projects are on the critical path of a plant outage (down time). The sooner the
projects are implemented, the sooner the plant is available for production. Time is
directly linked to the plant availability and is the key component and emphasis within
the power utility business, especially with the current growth in electricity demand
within the sub-Saharan region. This ensures that plant capacity is always available
due to power constraints on the power network.
The positive correlation of time to user requirements and customer satisfaction
confirms that projects that are implemented within the allocated time satisfy the
customer as they meet their target of managing the allocated time in which the plant
has to be off-line, which is also one of productions/clients key user requirements. The
other aspect of this correlation is that the more time spent on the plant, the greater
the effect or impact there is on the plant. This could however have a negative impact,
thus the time spent on the plant needs to be effectively controlled. Customer
satisfaction has always been a success criterion within the Eskom Peaking business,
19
especially due to the project team being based centrally and offering a service to six
power stations. Internally, Production is the key customer. They are also directly
accountable for the plant availability and reliability, and the users of the plant and
project outputs. These were consistent with the performance targets that were placed
on senior/line management. They were expected to ensure high availability and
reliability of the plant, that the customer’s electricity requirements were adequately
met with the long term sustainability of the plant and equipment (high quality).
Knowledge transfer was highlighted as a focus area for management owing to most
of the existing project managers and engineers having inadequate experience in
project and contract management. This was due to their being exposed to the project
environment for less than two years.
6 Conclusion
The main conclusion from the results of this study is that there are many more
success criteria that are relevant for evaluating the success of projects and one is not
only limited to the conventional time, cost and quality. The five relevant project
success criteria highlighted in this study are quality, impact on plant, satisfies
business goals, time and customer satisfaction. Stakeholders play a key role in
assessing the level of project success and it is quintessential to first define and agree
on the success criteria at the initiation stage of the project so that all stakeholders are
aware of the focus areas, the purpose and benefits of the proposed project.
Recommendations
The following recommendations are made to Eskom Peaking management:
o The relevant stakeholders for each project must be clearly identified at the start of
the project.
20
o Project success criteria must be discussed and agreed upon with the relevant
stakeholders at the start of the project.
o The most relevant project success criteria for Peaking projects are quality, impact
on plant, satisfy business goals, time and customer satisfaction. Management
should use these measures to glean the project success and the level of
alignment of projects to the business strategy.
o The cost of the project must also form part of the measure to ensure optimum
cost, and that the return on the project is justified.
o Adequate project auditing must be carried out at the completion of the project, and
the success criteria must play a key role in the evaluation of what was initially
agreed with all stakeholders.
21
7 References 1. Maylor H. 2003. Project Management, Prentice Hall, 3rd Edition.
2. Meredith JR. and Mantel, S. J., 2000. Project Management – A managerial
approach, John Wiley & Sons, 4th Edition.
3. de Wit A. Measurement of project management success. International Journal of
Project Management, 1988, 6(3), 164-170.
4. Baccarini D. The logical framework method for defining project success. Project
Management Journal, 1999, 30(4), 25-32.
5. Wateridge J. How can IS/IT projects be measured for success? International
Journal of Project Management, 1998, 16(1), 59-63.
6. McCoy FA. Measuring success: Establishing and maintaining a baseline, Project
Management Institute Seminar/Symposium Montreal Canada, Sep. 1987, 47-52.
7. Pinto JK and Slevin DP. Critical success factors across the project lifecycle.
Project Management Journal, 1988, XIX, 67-75.
8. Saarinen T. Systems development methodology and project success, Information
and Management, 1990, 19, 183-193.
9. Davies TC. The “real” success factors on projects. International Journal of Project
Management, 1999, 17(6), 337-342.
10. Lim CS and Mohamed MZ. Criteria of project success: an exploratory re-
examination. International Journal of Project Management, 1999, 17(4), 243-248.
11. Atkinson R. Project management: cost, time and quality, two best guesses and a
phenomenon, its time to accept other success criteria. International Journal of
Project Management, 1999, 17(6), 337-342.
12. Struckenbruck L. Who determines project success. Project Management Institute
Seminar/Symposium Montreal Canada, September 1987, 13-21.
22
13. Van Aken T De Weg naar project success: Eerder via werkstijl dan instrumenten,
De Tijdstroom, 1996.
14. Gray RJ. Organisational climate and project success. International Journal of
Project Management, 2001, 19, 103-109.
15. Clarke A. The key success factors in project management. Proceedings of a
Teaching Company Seminar, London, December 1995.
16. Morris PWG and Hugh GH. Preconditions of success and failure in major projects,
Templeton College, the Oxford Centre for Management Studies, Kinnington
Oxford, Technical paper No. 3, September 1986.
17. Shenhar AJ, Levy O and Dvir D. Mapping the dimensions of project success.
Project Management Journal, 1997, 28(2), 5-13.