using periodic audits to prevent catastrophic project failure
DESCRIPTION
From May 2008 ICGFM Conference,Paul Doresy, Dulcian on Project ManagementTRANSCRIPT
1 of 30
Dr. Paul Dorsey
Dulcian, Inc.
May 22, 2008
Using Periodic Audits to Prevent Catastrophic Project Failure
2 of 30
The VasaEarly 17th century (1628)“The greatest ship of all time”
3 years to build 2 gun decks with 64 cannons
King Gustavus Adolphus of Sweden set the measurements. 1000 trees were required Triple laminated oak walls (18in/46cm thick) Mast = 190 ft/57m Length = 201 ft/61m
Cost = 5% of Sweden’s GNP
3 of 30
Maiden Voyage
Set sail on a beautiful summer day August 10, 1628 Passed the royal palace of Stockholm Fired proud salutes from cannons Sailed 1400 yards Gust of wind came Ship sank in less than 1 minute
4 of 30
Why is the Vasa relevant?
What sank the Vasa sinks FMIS projects.We are still building Vasas.We can’t stop bad decisions.
We CAN stop ignoring them.If you spend $1M and fail, it is bad.
If you spend $100M and fail, it impacts the whole organization
If you spend $1B and fail, it impacts the country.
If you are going to fail - fail cheap.
5 of 30
6 of 30
The Essence of Project Management
Leading kids on a hikeFrom time to time, climb a treeCheck for obstaclesAdjust directionCall everyone together"LET'S GO!"
7 of 30
Periodic Audits
Critical success factor for failure preventionMust be external
Developers cannot self-assess.Big, substantial effort
Weak audit is worse than useless Provides illusion of safety
8 of 30
Audit Costs
Delay projectExpensive
5%-10% of project costIntrusiveAnnoyingPolitically costly
9 of 30
Team response to audit
Project Manager "Why don't you trust me?" "Waste of time"
Developers "We would rather be coding."
Team may feel threatened… If the team feels threatened, they probably have
reason to be.If the team welcomes the audit….
Sign of professional maturity
10 of 30
Audit Benefits
Early detection of failure If a $300 million project fails after $20 million, that
is a huge savings.Validation of system architectureSecond set of eyesGive team time to take stockMid-project course correction
11 of 30
Auditor Independence
Auditor must be told that there will be no chance for follow-on work. Otherwise the audit is suspect
To enforce independence: No economic attachment to the
results No incentive to skew the results No relationship to the
development team
During the audit: Limit contact with development
team "Stockholm Syndrome" After the audit, auditors can
help with the project plan.
Auditor is the agent of the person who hired him/her (no one else) Only reports to the contract
owner are objective. No professional standard or
certification for IT auditors exists.
Contract creates objectivity.
12 of 30
Audits are never 100% objective
Auditors bring their own biases.There are IT "religions."
.Net vs. Java "Thick database" vs. middle tier logic Service Oriented Architecture (SOA) Repository-based development Business rules Agile
A professional with a different perspective can still detect a "Vasa."
13 of 30
Justifying Audit Cost
An extensive audit requires approximately 10% of the total project cost.
Industry project failure rate is ~ 60%. Example: Audit at the halfway point of a $1 million
project Cost: (50% x 1,000,000) x 10% = $50,000 Benefit: (50%) x 1,000,00) x 60% = $300,000
Given the high failure rate, audits are very cheap insurance.
With other benefits, it is obvious that audits are a good deal.
14 of 30
Finding the right auditor
Not internalNot from the same company as the developersNot dedicated auditors
Must be real developers, DBAs, architects Must have built systems of similar scope and subject
area
15 of 30
The Audit Team
Project Manager Experience in the subject area with projects of similar
scopeDatabase Administrator (DBA)
Experience with same platform and similar database size
Application Architect Skill in the same area (Java, .Net, Oracle, etc.)
Security Specialist Military, financial system or health care experience
16 of 30
Structure of the Audit
Knowledge transfer Deep understanding As if auditor were taking over the project Understand the system before evaluating
Infrastructure audit Evaluate the existing infrastructure to support system Every area needs to be assessed.
Ability to meet current and future user requirements Auditor must understand the requirements
Financial review
17 of 30
Detailed Structure of Audit A. Knowledge Transfer
Allows auditors to understand the entire system architecture as if they were taking over system development.
The following areas should be reviewed for the knowledge transfer portion of the audit:
System Overview Data Model Walkthrough Review/Identify Transaction Use Cases Review User Interface Code
Architecture Review Reporting
Requirements/Architecture Review System Architecture Review System Installation/Upgrade
Mechanism. Internal Control review User Access review Handling system bugs/enhancements Multi-Lingual capabilities Basic System Requirements Process Flows Custom Framework Performance Standards Training
B. Infrastructure Audit Examine from technical soundness perspective. Compare to current industry best practices;
document any discrepancies. System and User Documentation Data Model Audit Database Review User Interface (UI) Architecture Review Distributed System/ETL Audit Security Audit Hardware/Software/Networking Review Backup/Recovery Procedures Appropriateness of system upgrade mechanism
C. Ability to Meet Current/Future Requirements
Examine the current system requirements, identify use cases, and review for suitability, specifically:
Compare documented requirements to the existing use cases and how they are handled.
Assess user satisfaction with the existing system. Are existing backup/recovery procedures sufficient to
meet maximum downtime requirements? Assess system flexibility to meet new requirements.
D. Financial review Review how resources have been spent over the
lifetime of the project including budgeted vs. actual expenditures
18 of 30
Knowledge Transfer
"Seek first to understand." Stephen Covey Learn enough to take over the process:
Architecture Data Model Use Cases Report Audit Configuration Management Internal Controls Documentation Training
System may be bad enough that this is not possible. Do it anyway. Preventing the Vasa from sailing is hard work.
19 of 30
Infrastructure Audit
Compare what was learned in the knowledge transfer with best practices
Each area needs to be assessed: Documentation Data model Database design User interface architecture Security Backup & Recovery Configuration Management Internal Controls
Identify weaknesses in each area: Corrective actions Exposure Controls needed
One mistake can sink the Vasa. System won’t scale Security hole Inflexible design
20 of 30
Ability to Meet Requirements
Requires careful use case documentation Assess user satisfaction
Sit with users Survey Request queue is not a good measure. If system is poor, users
give up. Assess each use case
Functional requirements Performance Downtime
Future requirements Flexibility Deployment
21 of 30
Financial Review
Stewardship Audit If requirements
change, price can balloon.
Does this make sense?
Sunk costs are "somewhat" relevant
Measure decision quality
Financial History
Date Budget $ Spent Mile-stones/
Achieved
Notes
22 of 30
COTS Project Audits
Not very different from custom projectsCOTS projects fail just as often.Review COTS architectureBe careful about how well COTS satisfies
requirements: COTS customizations can be VERY expensive. Customized COTS cannot be upgraded.
23 of 30
Audit Report
2-5 page Executive Summary Report Are we OK?
10-15 page CIO Report Are we OK? Why?
100 page detailed report What we did What we found What needs to be fixed Next steps
24 of 30
Acting on the Audit Report
If report concludes "Vasa…." Get a second opinion Let the development team respond
Sunk costs are sunk costs. The amount of money budgeted for the project is
irrelevant."Upgrade" is a way to change direction without
admitting failure.
25 of 30
Case Study: Ethiopia FMIS
Harvard University ProjectSmall part of major financial reform
$38 M USD over 12 years $3 M USD over 3 years for IT (very frugal)
Harvard was ending the project and wanted to assess quality of system.
Custom development projectDulcian was brought in to do the audit.
26 of 30
Audit Scope
Standard audit As described in this presentation
Total knowledge transfer and team back-up Support for any “what if?” scenarios Total system back-up
27 of 30
Audit Recommendations
Maintain current systemUpgrade system internals
Business rules approach Oracle DBMS Ultra-light Web architecture
28 of 30
Audit Results
Government and Harvard accepted recommendations. Maintain/evolve current system $1.5M USD Redesign architecture $1.5M USD
Dual nature of the audit (audit + handoff) made existing team very uncomfortable. Top three IT people resigned.
29 of 30
Conclusions
Audits don't prevent failure; they just catch failures earlier in the process.
Audits don't replace good design. The audit may only help fail more small projects rather than
one big project. Audits are resource intensive, expensive, annoying,
politically charged. But they are cheaper than not doing them at all in the long
run. Auditors must be kept independent.
No follow-on work Both COTS and custom projects need audits.
30 of 30
Result of Audit
Quite a good design Excellent documentation Very good developer coding Supported current requirements
Some architecture portions needed modification. Database design issues
No Foreign Keys Odd design (inherited from previous team)
Poor performance for parts of system Scalability questions Would not work on Ethiopian internet due to slow/unreliable
connections
31 of 30
Contact Information
Dr. Paul Dorsey – [email protected] Dulcian website - www.dulcian.com
Developer AdvancedForms & ReportsDeveloper AdvancedForms & Reports Designer
HandbookDesignerHandbook
Latest book:Oracle PL/SQL for Dummies
Design Using UMLObject ModelingDesign Using UMLObject Modeling