28 aug 2012 prt-148 los angeles washington, d.c. boston chantilly huntsville dayton santa...

28 Aug 2012 PRT-148

Los Angeles Washington, D.C. Boston Chantilly Huntsville Dayton Santa Barbara

Albuquerque Colorado Springs Goddard Space Flight Center Johnson Space Center Ogden Patuxent River Washington Navy Yard

Ft. Meade Ft. Monmouth Dahlgren Quantico Cleveland Montgomery Silver Spring San Diego Tampa Tacoma

Aberdeen Oklahoma City Eglin AFB San Antonio New Orleans Denver Vandenberg AFB

Joint Analysis of Cost and Schedule (JACS) Joint Analysis of Cost and Schedule (JACS)

Australian Department of DefenceAustralian Department of Defence2nd Cost Estimation Conference 2nd Cost Estimation Conference

29 - 30 October 2012 29 - 30 October 2012

Alfred Smith, CCEAAlfred Smith, CCEAJennifer Kirchhoffer, CCEA Jennifer Kirchhoffer, CCEA

Joint Analysis of Cost and Schedule (JACS) Joint Analysis of Cost and Schedule (JACS)

Australian Department of DefenceAustralian Department of Defence2nd Cost Estimation Conference 2nd Cost Estimation Conference

29 - 30 October 2012 29 - 30 October 2012

Alfred Smith, CCEAAlfred Smith, CCEAJennifer Kirchhoffer, CCEA Jennifer Kirchhoffer, CCEA

28 Aug 2012 PRT-148 Approved for Public Release 2 of 35

What is JACS?

Overview of the JACS modeling process

Key reports from a well constructed JACS model

Concluding remarks

AgendaAgenda


Cost, schedule and risk assessments traditionally have been performed by separate teams of professionals

In recent years, it has become more common for the cost analyst to report a “risk adjusted” result as a budget recommendation rather than a point estimate

However, it appears that cost uncertainty models routinely: try to force a 70 or 80% cost result into the point estimate schedule ignore risk management team statements like “High probability this

event will occur and if it does, the consequence will be severe”

What is Joint Analysis of Cost and Schedule (JACS)?

What is Joint Analysis of Cost and Schedule (JACS)?

Joint Analysis of Cost and Schedule is a disciplined, systematic and repeatable process to integrate three critical pieces of information: Cost, Schedule, Risk


Current Approach to Model Cost Estimating UncertaintyCurrent Approach to Model Cost Estimating Uncertainty

Cost

Input, e.g., weight

Cost = a + bxc

Historical data points

Sources of Uncertainty:•Cost estimating method•Cost method inputs

Focus is on estimating total cost uncertainty with limited influence from duration uncertainty or potential events that may influence cost/schedule.


Capture schedule uncertainty on Time-Dependent Costs

Inclusion of Discrete Risks (5x5’s)

Evolving Trends in Uncertainty and Risk Analysis

Evolving Trends in Uncertainty and Risk Analysis

Time-Dependent (TD)[Level of Effort - LOE]

Risk 1

Risk 2

Risk n

.

.

.

Translate 5x5 into probability of occurrence times uncertain consequence which can impact cost and/or duration of one or more tasks

1

n

2


Project Start

Project End

Task Duration

Integrated Risk & Uncertainty Landscape – the JACS Paradigm

Integrated Risk & Uncertainty Landscape – the JACS Paradigm

TD = Time-Dependent Cost, e.g. ‘marching army’ cost

TI = Time-Independent Cost, e.g., Materials

Duration Uncertainty

Burn Rate Uncertainty

TD $ = Segment Duration X Burn Rate

TD $ = Segment Duration X Burn Rate

Probability of Occurrence

Risk Register

TI $

TI $

TI $

TI $

TI $TI $ Uncertainty

TI $


What is JACS?



Concluding remarks

AgendaAgenda


The JACS Process:Develop the Analysis Schedule

The JACS Process:Develop the Analysis Schedule

Risk

Sched

Cost

CollectSched

Data

Create AnalysisSchedule

ValidateBefore

Continuing


Joint analysis of cost and schedule begins with a model of the schedule logic Serves as the backbone for the analysis Cost, risks and uncertainty are mapped into the logic to assess

impacts

Project/program integrated master schedules (IMS) are unsuitable for this role They are generally too big, complex and too detailed Logic common in an IMS can be a problem for a JACS analysis (e.g.,

constraints)

A JACS appropriate schedule must be created from available data (including the program IMS) Typically referred to as an “analysis schedule”

The Need for an Analysis ScheduleThe Need for an Analysis Schedule


√ Captures the major work-flows of the project IMS

√ Provides insight into major cross-dependencies within or across management responsibility boundaries

√ Creates a solid framework to capture cost / schedule uncertainties and discrete risk events

√ Structured around management/ budget responsibility

√ Allows mapping of budgeted work effort to schedule scope

√ Aligns with cost/budget data

√ Identifies key tasks that support major deliverables/ tracking items

√ Detailed IMS step by step work items and task flows are combined while maintaining critical path logic

√ Has traceability and transparency to the more detailed IMS

Attributes of an Analysis ScheduleAttributes of an Analysis Schedule


The JACS Process:Map Costs to Schedule Tasks

The JACS Process:Map Costs to Schedule Tasks

Risk

Sched

Cost

CollectSched Data


UpdateAnalysisSchedule

Collect Cost Data

Identifyas

TD or TI

Map toSched

Activities


Easier to map cost model results to a schedule model rather than replicating schedule network logic in a cost model

Schedule models are generally populated with throughputs, that is they don’t allow equations to estimate the cost of one task based on the cost of another or some technical characteristics

Mapping cost estimate to a schedule model is simplified by: Unifying cost (often product based) and schedule (often task based) work breakdown structures Specifying Time Dependent and Time Independent costs and their uncertainty separately Defining how the TI or TD cost is phased over the task duration

Mapping of Cost to ScheduleMapping of Cost to Schedule

TD Cost

TD Phasing

Total Cost

TI Cost

TI Phasing


The JACS Process:Mapping the Risk Register to Schedule Tasks

The JACS Process:Mapping the Risk Register to Schedule Tasks

Risk

Sched

Cost

CollectSched Data


Collect Risk Data

Assign Likelihood,

EstimateImpact

Map toSched

Activities


Collect Cost Data

Identifyas

TD or TI

Map toSched

Activities


Defined as: If risk event A occurs, there is a cost consequence or opportunity. The probability of A occurring is x% Often modeled as a separate task inserted into the schedule network

If there are only a few such risk events, treat as discrete what-if cases (event cost or schedule impact is either “in” or “out” of the estimate) Point estimate includes the full impact (when there are few)

If there are many such risk events, model using the Yes/No distribution (also known as the Bernoulli distribution) When there are many, there is no standard on how to treat the Point Estimate:

Include none and assess separate from the Point Estimate?

Include all (worst case scenario)?

Include the sum of the expected values? this is a common approach

The risk register should account for: Uncertainty of the cost consequence or opportunity Correlation across duration and cost uncertainties Probabilistic branching rarely attempted

Discrete RiskDiscrete Risk


5 0 0 1 2 1

4 0 0 0 1 0

3 1 2 1 4 0

2 1 3 4 2 4

1 0 1 2 2 4

1 2 3 4 5Likelihood

Total Risks = 36

High = 13

Medium = 13

Low = 10

5x5 Matrix Definitions*5x5 Matrix Definitions*

Risk Management conventions* Consequence

1 Minimal or no impact 2 Additional resources < 5% 3 Additional resources = 5-7% 4 Additional resources = 7-10% 5 Additional resources > 10%

Likelihood of Occurrence 1 Remote (10%) 2 Unlikely (30%) 3 Likely (50%) 4 Highly likely (70%) 5 Near certainty (90%)

Opportunities Should have a separate matrix to

address potential opportunities to save (not addressed in our example)

*Note: Taken from Risk Management Guide for U.S. DoD AcquisitionEvery Agency will set its own standards for these values

Co

nse

qu

en

ce


The JACS Process:Assigning Uncertainties

The JACS Process:Assigning Uncertainties

Risk

Sched

Cost

CollectSched Data


Collect Risk Data

Assign Likelihood,

EstimateImpact

Map toSched

Activities


Collect Cost Data

Identifyas

TD or TI

Map toSched

Activities

AssessDuration

Uncertainty

AssessCost

Uncertainty

Assess Event Cost and Duration

Uncertainty


Most common method to address uncertainty is to assign distributions to uncertain elements and run a Monte Carlo simulation Objective Uncertainty Distributions

Derived from historical data Something you can defend mathematically and historically

Subjective Uncertainty Distributions Based more on expert opinion than statistical analysis Often necessary due to lack of information to characterize it objectively

Every duration, cost and consequence in the model is generally an estimate and therefore uncertain Time Independent Cost Uncertainty Time Dependent Cost (Burn Rate or Resource Utilization) Uncertainty Duration Uncertainty Discrete Risk Uncertainty –Probability of Occurrence

Uncertainty should be applied in a consistent manner across the entire model

The Only Certainty is UncertaintyThe Only Certainty is Uncertainty


Typical Uncertainty DistributionsTypical Uncertainty Distributions

DISTRIBUTION TYPICAL APPLICATIONKNOWLEDGE OF

MOST LIKELY

NUMBER OF PARAMETERS

REQUIREDRECOMMENDED PARAMETERS

Lognormal

Default when no better info.Probability skewed right.

Replicate another model result.Power OLS CER uncertainty.

Mean or median known better than the

mode (most likely)2

50% (median) and high value

(some tools have a 3rd parameter : “Location” . By default, it is zero. Used to “slide” the

lognormal left or right (even into negative region).

Triangular

Expert opinion. Finite min/max. Chance reduces towards

endpoints. Skew possible.Labor rates, labor rate

adjustments, factor methods

Good idea 3 Low, mode, and high

BetaPertLike triangular, but treats mode as 4 times more important than

min or max.Very good idea 3 Low, mode, and high

BetaLike triangular, but min/max

region known better than mode.Not sure

4 Min, low, high, and max

NormalEqual chance low/high.

Unbounded in either directionLinear OLS CER uncertainty.

Good idea, but unbounded in either

direction2

Mean/Median/Mode and high value

UniformEqual chance over uncertainty

range. Finite min/max.No idea 2 Low and High

(some tools require min and max)

Note: Low/high are defined with an associated percentile (by default 15/85). Min/Max are the absolute lower/upper bound (also known as the 0/100). Some policies require truncation at zero.


The Double Counting Time Dependent (TD) Cost Uncertainty Dilemma

The Double Counting Time Dependent (TD) Cost Uncertainty Dilemma

Consider how TD cost is calculated Typically calculated using: Duration * Cost/Day (duration in days) Cost/Day can be derived from similar, completed project totals Cost/Day factor may already capture cost and duration uncertainty Uncertainty on Duration and Cost/Day factor may be double counting

However, basis for Cost/Day must be carefully understood Cost/Day factor may change as the duration changes

Shorter duration achieved by using more resources ($/day larger) Shorter duration achieved by using more expensive resources ($/day larger) Longer duration a consequence of scarce resources ($/day smaller)

In these contexts, uncertainty on Duration and Cost/Day is appropriate Correlation between the two should be considered as well


The JACS Process:Apply Correlation, Validate then Run

The JACS Process:Apply Correlation, Validate then Run

Risk

Sched

Cost

CollectSched

Data


Collect Risk Data

Assign Likelihood,

EstimateImpact

Map toSched

Activities


Collect Cost Data

Identifyas

TD or TI

Map toSched

Activities

AssessDuration

Uncertainty

ValidateFile

RunAnalysis

AssessCost

Uncertainty


Uncertainty


JACS models developed in any tool will have limited “functional” correlation between uncertain elements (correlation due to model mathematics)

Consider applying correlation to ensure elements that should, do move together

Tools should allow application of correlation across any uncertain elements Just because you can apply correlation, does not mean you should! Correlating Dur with TD may be double counting if TD is modeled as a function of Duration

Ideally, measure the correlation present in the model first, then apply as needed

Apply CorrelationApply Correlation


Prior to running an integrated simulation, the user should review the file to ensure that there are no potential issues within the file

There are many commercial tools that will perform a schedule health assessment These tools look for violations of schedule model best practices (e.g., task without predecessor)

A JACS Health check is more comprehensive and should also uncover issues such as: Critical issues: e.g., cost not phased, invalid uncertainty, duration with no cost, invalid correlation Warnings: e.g., uncertainty on zero cost, risk event with zero probability, baseline outside uncertainty Information: e.g., extraordinary float, risk event turned off, duration without uncertainty, no

correlation

Perform a Comprehensive Health Check to Ensure Model Validity

Perform a Comprehensive Health Check to Ensure Model Validity


The JACS ProcessThe JACS Process

Risk

Sched

Cost

CollectSched

Data


Collect Risk Data

Assign Likelihood,

EstimateImpact

Map toSched

Activities


Collect Cost Data

Identifyas

TD or TI

Map toSched

Activities

AssessDuration

Uncertainty

ValidateFile

RunAnalysis

AssessCost

Uncertainty


Uncertainty


What is JACS?



Concluding remarks

AgendaAgenda


Typical Output from a Cost Only Uncertainty Analysis

Typical Output from a Cost Only Uncertainty Analysis

Cost uncertainty is generally performed on total costs

Rarely linked to schedule uncertainties Unable to relate a specific cost result to a specific schedule result

No insight into uncertainty by year, or the impact of schedule slips


A JACS Model Relates Uncertain Cost with Uncertain Duration

A JACS Model Relates Uncertain Cost with Uncertain Duration

New View Shows Cost Aligned with Schedule

70% Cost Confidence Level (CCL) Indicates Reserves Capture Schedule Growth

Lower left identifies joint probability of meeting BOTH

70% cost and schedule (58.4%)

Each dot is the cost and

schedule result from

one trial


Annual Funding ChartsAnnual Funding Charts

Point Estimate vs Annual Uncertainty

The JACS Model provides total and annual uncertainty results Identifies both WHAT the uncertainty is and WHEN it is


Additional Views Are PossibleAdditional Views Are Possible

Point Estimate similar to Mean

Need to use Reserves to move point estimate towards the Mean


Additional Views Are PossibleAdditional Views Are Possible

There are many tools available and they all produce some version of most of these charts


The term “Tornado” is used in a variety of contexts across schedule, cost and general uncertainty analysis; tool documentation and the literature

Various definitions include: Find Cost Drivers: Create a low and high “what-if” case for every “cost driver” based on

their 10/90% values (evaluated one at a time)…find driver that has most impact Problem: ignores correlation effects; does not address drivers influenced by multiple

distributions

Find Uncertainty Drivers: Measure the correlation between each defined input distribution and the output of interest. The highest correlation identifies find distribution that has the most impact on the total uncertainty.

Problem: Element that is highly correlated with output may have nothing to do with that output

Hybrid: Sort the trial results by cost driver (one at a time) to find associated bounds on the output mean or selected percentile

Problem: How you bin the trials has massive effect on results; may require huge number of trials to obtain stable results

Brute Force: Use one of the above methods to find the top 10, then run the simulation 10 times turning one at a time and record the impact on the output of interest.

Problem: Very tedious, time consuming and may be misleading if its loss triggers unexpected conditions in the simulation

Conclusion: Beware of the Tornado chart!

Most Misunderstood Chart:Tornado

Most Misunderstood Chart:Tornado


Total Cost Drivers: (left chart) Measure impact on inflated results, not constant year cost results Run simulation to find combined uncertainty results for 10/90%

Total Uncertainty Drivers: (right chart) Adjust the algorithm to account for applied correlations Identify how many trials used (demonstrate sufficient)

Notice the answers are quite different

Tornado Chart Best PracticesTornado Chart Best Practices

7k Trials Required


Prototype Integrated Time-Based View of Costs, Schedule, Budget, and Risks

Prototype Integrated Time-Based View of Costs, Schedule, Budget, and Risks

Solid line is the point estimate (BCWS) Dots are cost/schedule uncertainty results at various milestones X- Axis identifies when key risk register events occur

Size of symbol indicates impact, color indicates probability (or type)


What is JACS?



Concluding remarks

AgendaAgenda


What Data Do We Need?What Data Do We Need?

Keep it simple and use what you have

Schedule + Risks + Costs = JCL Model Can use any of the below, as long as you have one data source for each

category Schedule

Detailed IMS Simple schedule with just a few moving parts

Costs – preferably time phased Budget data Lower level cost data (LCC databases) / EVM data Parametric costs

“Risks” Risk management system What –if’s Basic uncertainty


Why JACS is Becoming PopularWhy JACS is Becoming Popular

Delivers an integrated view to Project Managers: Schedule probability of success Cost probability of success Impact of discrete program risks Results of any number of what-if scenarios Both total and annual funding reserve requirements

For NASA: Regulatory requirement (7120.5 E) Identify a cost and schedule range by milestone KDP (~milestone) B Baseline program to a specific joint probability level by KDP

(~milestone) C


Questions?Questions?Questions?Questions?

28 aug 2012 prt-148 los angeles washington, d.c. boston chantilly huntsville dayton santa...

Documents