www.onlyforward.org [email protected] 1 of 37 planning and estimating for projects &...
TRANSCRIPT
www.onlyforward.org
[email protected] 2 of 37
Planning and Estimating
Planning1. Plan Holistically
Efficient process, competent people. Measure, benchmark & improve.2. Plan for the Whole Lifecycle
Understand the business case, stay focussed and deliver for the lowest cost3. Manage Uncertainty and Change
Risk management and change control
Estimating1. Derive the Estimates
Available techniques2. Manage Uncertainty
Estimation uncertainty, 3pt estimates, selective 4pt estimates3. Ensure Delivery
Team buy-in, Scrum & Burn Down charts, co-location
Results5 year case study
www.onlyforward.org
[email protected] 3 of 37
Portfolio Objectives: Holistic, StrategicTeam• Match team to portfolio
• Mix of new, middle & senior PMs• Develop common SMART objectives• Benchmark and align pay & benefits• Learning & Development
• Develop PM Competency Frameworkaligned to APM & business
• Drive PM Qualification & Certification• Run regular PM Forum to share best practice• Embed mentoring & coaching• Develop PM team feedback
• Succession Planning• PM career development• Identify new PMs: internal & external
Results• Develop scalable project governance & review• Analyse and reduce project loss: frequency & scale• Improve project cash-flow
• On-time payment milestone delivery• Payment milestone invoicing & payment
• Deliver better than budget project portfolio result
Customers• Measure and increase repeat business• Improve customer change control• Measure and increase customer satisfaction
• Increase on-time project completion• Seek and act on customer feedback
Processes• Create a unified, efficient process framework
• Co-ordinate best practitioner process development• Actively increase understanding and use
• Rationalise engineering tools & align to process• Benchmark and embed improvements
e.g.• CMMI-Dev: Achieve CMMI Maturity Level 3• EFQM: Achieve >650 score
www.onlyforward.org
[email protected] 4 of 37
Process Framework – Lifecycle
OpportunityAssessment
CONCEPT PHASE
Study Planning
IMPLEMENTATION PHASEDesign I teration A Design I teration B
DesignDesignManu-facture
Site/Final Acceptance Test
Field Trial
Interoperability Test
New Product Introduction (NPI )
Process Design Process Trials Process Proving Ramp-up
Sales & Marketing Definition
Service Definition Service Roll-out
Launch
Volume Production,Sales & Service
End ofLife
G3
ImplementationApproval
G4
ASamples
G5
BSamples
NPI Launch
G7
SalesRelease
G8
ProductSign-off
G9
TerminationDecision
Design/Build/Test
...Increment 1
...
SystemTest
Design/Build/Test
Increment n...
...
SystemTest
Test TestManu-facture
Software &Firmware:
Hardware:
G2
Full Business
Case
G0
Idea
Syn
chro
nis
ati
on
G
ate
sC
o n
c u
r r
e n
t
L i f
e c
y c
l e
P
r o
c e
s s
e s
G1
InitialBusiness
Case
PDRPreliminary Design Review
FDRFinal Design Review
G6
CDRCritical Design Review
PRRProduction Readiness Review
TRRTest Readiness Review
1 2 3 4 5 6 7 8 9 10 11 12
Typical Timeline
www.onlyforward.org
[email protected] 5 of 37
Clear separation of Policy, Process and Guidance
Topic PolicyProcess Work Instruction
Tools Map Template Guide OwnerForm
Lifecycle P01PR01 WI01-00 to WI01-99
TM01 T01-00 to T01-99 G01 Process Owner
F01-00 to F01-99
Bidding P02PR02 WI02-00 to WI02-99
TM02 T02-00 to T02-99 G02 Bid Process Owner
F02-00 to F02-99
Project Management P03PR03 WI03-00 to WI03-99
TM03 T03-00 to T03-99 G03 PM Process Owner
F03-00 to F03-99
20 topics cover whole lifecycle
A little mandatory process
A lot of guidance materialincluding templates and examples from real projects
Project acquisition & delivery focus
1. Lifecycle
2. Bidding
3. Project Management
4. Quality Assurance
5. Procurement
6. Requirements Management
7. Systems Engineering
8. EMC
9. Antenna Development
10. Digital Design
11. Mechanical Engineering
12. Electronics and PCB Design
13. FPGA & Digital ASIC Design
14. RF ASIC Design
15. RF Engineering
16. Software Engineering
17. DSP Engineering
18. Test
19. Manufacturing and Production Engineering
20. Service & Support
Audited Process < > Guidance & Training Material
www.onlyforward.org
[email protected] 6 of 37
ExampleProcess
Concept Planning Phase ProcessCategory F ProjectsCategory A to E Projects
A single concept is detailed sufficiently to quote for a fixed price implementation
RequirementsSystem Design
Electronics Subsystem DesignMechanics Subsystem DesignAntenna Subsystem Design
FPGA Subsystem DesignSoftware Subsystem Design
System Design Review
Implementation Review
Concept StudyPhase
Define System Requirements
Requirements [18]
Analyse & Refine System Design
Subsystem Design
Define Subsystem Requirements
RequirementsSystem Requirements
[18][18.1]
RequirementsSystem Requirements
Subsystem Requirements
[18][18.1][18.2]
Integration Planning
SW Drop PlanFPGA Drop PlanIntegration Plan
System DesignBlock Diagrams
PCB Shapes & Component PlacementBoM & NRE Summary
FPGA Interface SpecificationFPGA Pin Allocation Table
System Design [19]Define Test Concept
System Modelling
Block DiagramsLevel Plans / Link Budget
[19.2][19.6]
[19][19.2][19.4][19.5][19.7][19.8]
System DesignSubsystem Design
Electronics Subsystem DesignMechanics Subsystem Design
3D ModelsMechanics Rapid PrototypesAntenna Subsystem Design
FPGA Subsystem DesignFPGA Behavioural Simulation Scripts
FPGA Complexity AnalysisSoftware Subsystem Design
I terative
Cost ReductionFeature RevisionDesign Selection
Make/Buy Decisions(possible sub-contracts)
Test Planning
Test Concept
Project Planning
Project PlanMonthly Reports
Action ListRisk Register
Change Management ToolSample Production Plan
[21]
Test ConceptTest Plan
Product Integrity TestsSystem Tests
Subsystem TestsIntegration Tests
[18][19][25][26][27][28][29]
[21][22][22.1][22.2][22.3][22.5]
Write Test Reports (no results)
Test Reports 3
System FMEA
ManufacturingKick-off Workshop
Write Production Test Plan
ManufacturingConsensus
System FMEA
Production Test Plan
Implement?
Customer
Y
N
Development Contract
Development Contract Review
Record of Review
ImplementationPhase
Refine PCB Component Placement
PCB Shapes &Component Placement(inc. screening plan)
Schematic Capture
Schematics 1
Refine BoM
BoM & NRE SummaryPreferred Components List
Long Lead Risk Order 2
[19.4]
[25.2]
[20]
[19.5][25.1][34]
[31][30][32]
[24]
[23]
[7][7.6][13][8][14][33]
Product Integrity Plan [6]
1 Unlikely to be complete at Implementation Review, included at this stage to assist BoM & effort estimation.2 May be required – lead time dependent.3 Test Reports included at this stage to assist effort estimation (no tests run or results available).
[19][19.1][25][26][26.1][39.2][27][28][28.1][28.2][29]
G3 I mplementation Approval
CM2 I mplementation Approval
[10]
Project Closure
Closure Review
CM7 Project Closure
www.onlyforward.org
[email protected] 7 of 37
Manage Risk and Control Change
Knowledge
Pre-Bid Bid Concept Implementation
Acquisition Delivery
10 – 25% 75 – 90%Price:
• Run all projects as if they are fixed price
• Early work defines final result
• Aim to get risk down to <20% of contract value before going into implementation
• Need to do work to acquire knowledge – to better estimate the job
• Aim to get under contract in phasesto match the level of risk at each phase
• Bid/ConceptLots of unknowns, lots of riskScoped time & materials – if you canJoint activity with customer
• Gate ReviewsFocus on risk and check for changes
Opportunities
Threats
Time
Volume
www.onlyforward.org
[email protected] 8 of 37
Risk Register: Risk Summary – with Risk History
The saved baselines in the Risk History show the Risk development over time.
www.onlyforward.org
[email protected] 12 of 37
Planning for Projects & ProgrammesStrategy(Need)
Contingency
Opportunities
EnhancementTasks
SecondaryRisks
ProductBreakdownStructure
WorkBreakdownStructure
Work Packages& Tasks
Estimates
Zero Risk(Deterministic)
Cost
INFORM
Inform / Offset
ThreatsMitigationTasks
Programme& Project
Set-up
INFORM
Project Delivery Process,
PDP
Risk Register Tool, RRT
Risk Management Strategy, RMSRisk Management Plan, RMP
Held at Board level: Project, Programme or Business
Held at Project & Programme level
If cost effective
Contingency
ProjectRisk Pot
EstimationUncertainty
www.onlyforward.org
[email protected] 13 of 37
Planning Summary
• Plan holistically and get the right approach• Plan the whole lifecycle, starting at opportunity identification and acquisition• Actively manage Risk – throughout the lifecycle, not just at start
• Planning should not be done by PMs working alone• The quality of early phase work is critical for a successful result
• Understand the business case and the stakeholders• Do enough work to understand and document the requirements• Get the scope and the concept right – cost is largely set by start of implementation
• Ensure PM competence is a match for project value, risk and complexity• In the bid phase use the PM that will run the project
• Planning should ensure the appropriate work is done, at the optimum time
• Planning is not enough if the estimates for the work are wrong…
www.onlyforward.org
[email protected] 14 of 37
Planning and Estimating
Planning1. Plan Holistically
Efficient process, competent people. Measure, benchmark & improve.2. Plan for the Whole Lifecycle
Understand the business case, stay focussed and deliver for the lowest cost3. Manage Uncertainty and Change
Risk management and change control
Estimating1. Derive the Estimates
Available techniques2. Manage Uncertainty
Estimation uncertainty, 3pt estimates, selective 4pt estimates3. Ensure Delivery
Team buy-in, Scrum & Burn Down charts, co-location
Results5 year case study
www.onlyforward.org
[email protected] 15 of 37
Available Estimating Methods1
TransparentBox
Analogical
Group Technologies
Knowledge Based
Case Based Reasoning
Detailed
Product Attributes
Function Costing
FeatureCosting
Accumulationof Parts
Generative
ActivityBased
BlackBox
Statistical
Parametric
Neural Network
Expert Judgement
www.onlyforward.org
[email protected] 16 of 37
Available Estimating Methods I
Method Advantages Limitations
Parametric Clear influences, Objective, Repeatable
Simplistic,Missing parameters
Neural Network Accurate,Updateable
Hidden logic, Complex, Large database required
Expert Judgement Quick,Flexible
Bias, Unstructured, Repeatability
Group Technologies Quick,Intuitive
Case bias,Large case history,Poor for innovations
Knowledge Based Visible logic,Structured
Knowledge obsolescence, Large database required
Case Based Reasoning Quick,Collective memory
Large reliable case base, Poor for innovations
www.onlyforward.org
[email protected] 17 of 37
Available Estimating Methods II
Method Advantages Limitations
Function Costing Integrates requirements with costs
Need to allocate cost to function, Accuracy
Feature Costing Integrates with CAD/CAM, Can be automated
No feature consensus, Large database required
Generative Can be accurate,Detail useful for negotiation
Time consuming,Detailed data may not be available
Activity Based Allocates costs to sources, String indication of potential profitability
Time consuming,Detailed data may not be available,Allocation of overheads.
www.onlyforward.org
[email protected] 18 of 37
Expert Judgement
Expert Judgement most often used for projects with high levels of innovation• No case history or database of past estimates and results to base new estimates on• Often little detail about the task being estimated• Rarely much time to do the estimating
Mitigate estimation method weaknesses• Template for gathering estimates – reduced bias, more consistency between estimators• Use experts – senior / domain experienced – same engineers that will have to deliver• Where possible use a Mini-Delphi2 method to further reduce bias• Include resource grade assumed for the estimate• Don’t include risk in the estimate – estimated separately (don’t build risk on risk)
Estimation SheetProject:<Project Name> Estimator:Bill Gates
Project Manager:<PM> Date:01/10/2013
Task Min ML Max Resources Notes and Assumptions(hours) (hours) (hours)
System Requirements 38 96 192 PM, TA Assumes TSJ for TA roleSystem Design Specifications 96 114 228 HW, SW, DSP TLs Test Plan 38 96 124 V&T TL
www.onlyforward.org
[email protected] 19 of 37
Uncertainty, Optimism Bias, Concurrency & Range Estimates
Project dates and costs are often too optimistic when not considering estimation uncertainty.Assumptions can be too positive, perhaps as a result of making a plan fit fixed targets.This is Optimism Bias.
Concurrent tasks with minimal float and effort estimates using expert judgement given as single point or deterministic estimates is common
The more concurrent tasks, the greater the impact when some tasks finish later than estimated
Deterministic outcomes often have a very low probability
Range estimates are more realistic3 points: minimum, most likely, maximumKey project dates and costs then also become ranges – along with a probability
Typical plan: Yellow line is the probability of achieving the deterministic cost
www.onlyforward.org
[email protected] 20 of 37
Weakness of 3pt Estimates
3 point estimates do have a potential pitfall
Consider estimating the routing of a Printed Circuit Board
Experts use knowledge of board size, number of layers, number and type of components and compare to previous layout routing tasks, resulting in these estimates:
Good progress: 3 days auto-routing, 1 day manual routing so Minimum = 4dNormal progress: 3 days auto-routing , 3 days manual routing so Most Likely = 6d
Problematic progress: 3 days auto-routing, 7 days manual routing so Maximum = 10d
Minimum Estimate Min 4d
Most Likely Estimate ML 6d
Maximum Estimate Max 10d
www.onlyforward.org
[email protected] 21 of 37
Probability of Most Likely Estimate Being Achieved
Most Likely should mean that the task is as likely to be under this duration as overMost Likely should have 50% probability of being achieved
Use the 3 point estimates and apply a triangle distribution:
As the area under this Probability Density Function is normalised to unity, we calculate the probability on the distribution corresponding to the Most Likely point:
PMost Likely = 2/(10-4) = 1/3 or 33% …NOT 50%!
www.onlyforward.org
[email protected] 22 of 37
Survey of Real Project Estimates
Eight projects containing 434 tasks:
ML < mid point between Min & Max = 185
ML = mid point between Min & Max = 220
ML > mid point between Min & Max = 30
A large number of tasks were biased towards the pessimistic resulting in an artificially low probability of completion to Most Likely estimates
Min ML Max
Min ML Max
Min ML Max
www.onlyforward.org
[email protected] 23 of 37
Consider again the Expert Estimate
Quizzing the estimator more closely revealed that for the last 10 similar boards, the duration distribution looked like this:
So similar to the triangle distribution that the 3 point estimate values gave?Consider how many times the task was completed in less than or equal to the Most Likely duration of 6 days…
7 out of 10 – and therefore we would be 70% confident of completing the task within 6 days – but our 3 point estimates predicted we only had a 33% chance!
www.onlyforward.org
[email protected] 24 of 37
Using Range Estimating3 to give a 4th point
Gather 3 point estimate values but also the likelihood that the task will be completed in less than or equal to the most likely duration – in this case, 70%
Then use this information to construct a distribution that reflects the values providedUsing the 70% value we create a double triangle distribution:Calculating points A & B:
0.7 = 0.5*(6-4)*A so A = 0.70.3 = 0.5*(10-6)*B so B = 0.15
Note that the area either side is not equalEach will vary task by task as defined by the 4th pointIn practice this need only be done for the 10 to 20 most critical tasksThese can be found in Oracle’s PRA4 tool using its ‘Tornado view’
Minimum Estimate Min 4d
Most Likely Estimate ML 6d 70%
Maximum Estimate Max 10d
www.onlyforward.org
[email protected] 25 of 37
Monte Carlo Simulation Results
Result for 3 point estimates aloneResult including selected 4 point estimates
Tornado Graph:Sensitivity analysis to select tasks to estimate with 4 points
www.onlyforward.org
[email protected] 26 of 37
Selective 4 Point Estimating: Conclusion
3 Point Estimate 4 Point Estimate
Deterministic(sum of all Most Likely estimates)
£40,477 £40,477
Probability of Achieving Deterministic from Monte Carlo Simulation 1% 22%
Minimum Cost £36,887 £33,989
Maximum Cost £69,165 £61,506
95% Probability Cost £61,602 £54,173
Using 4 point estimation produces a smaller, more realistic risk budget
- producing more competitive pricing
- without decreasing the chances of a successful project result
www.onlyforward.org
[email protected] 27 of 37
Funding Estimation Uncertainty
The business Risk Appetite can inform what probability to use, e.g.:10% Team Target (likely risks do not occur)50% Best Estimate (as many risks occur as not)90% ‘Safe’ Estimate (several unlikely major risks occur)
One strategy:Estimation uncertainty ‘project risk pot’ = deterministic cost – 90% cost
Reward using less of this risk pot, but recognise that a proportion is likely to be required
This encourages behaviour that enhances results whilst recognising uncertainty and setting realistic expectations
PMs use project risk pot to ensure delivery to the deterministic end dateDrives the right behaviour in the team – to deliver on their Most Likely estimates
Selective 4 point estimating maintains competitive pricing
www.onlyforward.org
[email protected] 28 of 37
Ensuring Delivery
Same team to bid, estimate and deliver
Agile Sprints and Burn Down chartsScrum: daily meetings, limited sprints to develop end-to-end testable features, etc.Use of Burn Down charts to display progress for all to seeTeam really buy-in and like to see themselves “below the line”Suitable length expected to be around 4-6 weeks
Burn Down chart example on next slide:Interesting dynamics and effects clearly visibleVHDL Team, 10 week sprint which was too long, too much to verify at the end
Co-locate teams whenever possible and mix the team disciplines and grades
Flexi-time, reward/recognise contribution beyond the norm, exceptional results bonus
www.onlyforward.org
[email protected] 30 of 37
Estimating Summary
• Innovation projects with little re-use from one project to the next limit estimation techniques available
• No database of previous estimates also limits estimation methods available• Build historical data records (estimate & result) where possible
• Quantitative Risk Management3 point estimates and Monte Carlo simulation (double triangle distribution) can give optimum cost and technical risk budget
• 4 point estimates for small number of tasks which most influence key dates can mitigate weakness of 3 point estimation
• Use same people for estimating and delivery• Team buy-in using Scrums, daily meetings, Burn Down charts• Co-locate team wherever possible, mixing expertise
www.onlyforward.org
[email protected] 31 of 37
Planning and Estimating
Planning1. Plan Holistically
Efficient process, competent people. Measure, benchmark & improve.2. Plan for the Whole Lifecycle
Understand the business case, stay focussed and deliver for the lowest cost3. Manage Uncertainty and Change
Risk management and change control
Estimating1. Derive the Estimates
Available techniques2. Manage Uncertainty
Estimation uncertainty, 3pt estimates, selective 4pt estimates3. Ensure Delivery
Team buy-in, Scrum & Burn Down charts, co-location
Results5 year case study
www.onlyforward.org
[email protected] 32 of 37
Case Study: Project Portfolio
The methods described were used in a contract R&D business with this project portfolio
Many complex projects, according to the APM definition5
Projects are categorised to enable PM competence matching.Category A is the highest value, risk and complexity, F the lowest.
41 (40 External & 1 Internal) category A to D projects70% of overall project portfolio by valuePM is always a career PMAverage contract value: £1.3MLargest project: £15M fixed priceAverage duration: 18 months
334 (234 External & 100 Internal) category E & F projectsPM is typically an engineerAverage contract value: £90kAverage duration: 9 months
Red8
3%Amber
145%
Green35392%
ProjectStatus
External27573%
Internal10027%
375Projects
www.onlyforward.org
[email protected] 33 of 37
Case Study: Over-Budget Project Root Causes
Year 1An analysis of project over-spend root causes showed that the largest issue was estimation
‘Project Management’, ‘Incorrectly Bid’, and a large part of ‘Test Failure’ were all indicators that Planning needed improvement
Relocation
Order Reduction
Staff Competence
Grade Mix Change
Customer Dependency
Production Data Pack
Staff Availability
System Design
Integration & Test
Supplied Code Issue
Other
Incorrectly Bid
Project Management
Test Failure
Underestimation
0 10 20
1
1
1
2
2
3
3
4
5
5
9
16
Over Budget - Root Cause Frequency
Year 1
www.onlyforward.org
[email protected] 34 of 37
Case Study: Portfolio Objectives Progress after 5 yearsTeam• Match team to portfolio
• Mix of new, middle & senior PMs• Develop common SMART objectives• Benchmark and align pay & benefits• Learning & Development
• Develop PM Competency Frameworkaligned to APM & business
• Drive PM Qualification & Certification• Run regular PM Forum to share best practice• Embed mentoring & coaching• Develop PM team feedback
• Succession Planning• PM career development• Identify new PMs: internal & external
Results• Develop scalable project governance & review• Analyse and reduce project loss: frequency & scale• Improve project cash-flow
• On-time payment milestone delivery• Payment milestone invoicing & payment
• Deliver better than budget project portfolio result
Customers• Measure and increase repeat business• Improve customer change control• Measure and increase customer satisfaction
• Increase on-time project completion• Seek and act on customer feedback
Processes• Create a unified, efficient process framework
• Co-ordinate best practitioner process development• Actively increase understanding and use
• Rationalise engineering tools & align to process• Benchmark and embed improvements
e.g.• CMMI-Dev: Achieve CMMI Maturity Level 3• EFQM: Achieve >650 score
Continuous improvement
Significant change ongoing
Significant change required
www.onlyforward.org
[email protected] 35 of 37
Case Study: Results after 5 Years
Year 5
Year 4
Year 3
Year 2
Year 1
13%
17%
15%
8%
-52%
Overall Project ResultsResult vs. Budget
• Transformed portfolio results• Happy customers• Happy Project Managers
• Estimation still #1 issue but reduced in frequency• ‘Project Management’ no longer an issue
Project Management
Relocation
Supplied Code Issue
Order Reduction
Incorrectly Bid
System Design
Staff Competence
Production Data Pack
Other
Customer Dependency
Test Failure
Integration & Test
Staff Availability
Grade Mix Change
Underestimation
0 10 20
1
1
1
1
1
2
2
5
5
5
6
8
11
Over Budget - Root Cause Frequency
Year 5
www.onlyforward.org
[email protected] 36 of 37
ReferencesEstimating Methods1. Cost Estimation Method Selection:
Matching User Requirements and Knowledge Availability to MethodsDK Evans, Dr. JD Lanham, Dr. R Marsh, SEEDS (Systems Engineering Estimation and Decision Support)Paper for ICEC (International Cost Engineering Council), 2006
2. Delphi Estimation Method
Range Estimating3. Risk Analysis and Contingency Determinations using Range Estimating
KK Humphries et alAssociation for the Advancement of Cost Engineering, Recommended Practice No. 41R-08, June 2008
Tools4. Oracle Primavera Risk Analysis
‘PRA’
5. APM Complex Project QuestionnaireAssociation for Project Management
www.onlyforward.org
[email protected] 37 of 37
In my board role I led a team of 22 professional Project Managers and 5 Quality Engineers, and ensured Roke’s £79M project portfolio delivered better than budget profit. I set-up and ran a virtual PMO and created the Roke Engineering Process, REP, also managing the engineering tools to support it.
I created a project management competency framework and the PM Excellence Programme, which achieved APM corporate accreditation, scoring 24 out of a possible 25 points in the APM assessment.
I chaired a quarterly PM forum which shared best practice and built a supportive PM community – seven of the project managers I coached have achieved APM RPP, five have PQ, and all gained APMP.Together, these investments in PM professionalism led to a turn-around and annual improvement in project results across a typical portfolio of up to 400 projects a year and delivered an above budget performance in five consecutive years with profits totalling £7.9M above budget.I am a passionate advocate of PM professionalism, a Fellow of the APM and the IET, and author of articles published in Project and PM Today.
After 4 years as an electronics engineer for Siemens, achieving Chartered Engineer, I moved into project management for 14 years, at Siemens and Roke Manor Research. At Roke, my ability to successfully deliver the most challenging whole lifecycle product development projects on time and under budget led to a role as Projects Director and board member for 6 years. In 2013 I went back to hands-on project management, taking a Programme Director role at Cambridge Consultants, in the Cambridge Science Park.
Author Profile