Uncertainty SubgroupARB Expert Workgroup

June 17, 2010

1

• Michael O’Hare, chair• Wes Ingram, ARB staff

• Paul Hodson• Stephen Kaffka• Keith Kline• Michelle Manion• Richard Nelson• Mark Stowers

• With assistance from Richard Plevin, UCB

2

Uncertainty and ILUC

• How has uncertainty about the ‘real value’ of ILUC resulting from substitution of biofuel B for (fossil or other) fuel F been described?

• Given a characterization (PDF function, range, table of estimate values, etc.) ‘how big’ is the uncertainty in a given estimate? Across all estimates?

• How can it be reduced?• How should LCFS implementation accommodate it?

• Cost of error• Objective function

3

Implementation

Estimate physical GWI for MJ/MJ substitution of fuels

GHG emissions

Analyze optimal operationalGWI and choose (implement)

Fuel system responds withblends and driving behavior; nature“responds” with real GWI

Social cost(T etc.) 4

Three distinct variables for a single fuel

• g*ij = estimate of “physical” GWI of fuel i from model j

• i = real “physical” GWI: if fuel i is substituted for fuel k on a MJ/MJ basis, additional GHG release is (i - k). i is a random variable.

• gi = operational GWI of fuel i used in LCFS implementation

5

Sources of uncertainty

• Errors in assumed conceptual frameworks: relationships between biofuel policy and land use change

• Errors and variety in model structure• Errors in model parameters• Intrinsic variation in real-world variablesAre new model results converging on a narrow

range of values?

6

Uncertainty in existing models

• Modelers’ uncertainty reporting

• Variation among model results

• Random error vs. bias

• Can a probability distribution be constructed?

7

Estimates of ILUC emissions from corn ethanol

Estimates of ILUC emissions from corn ethanolStudy Target Year

Shock size(109 L)

ILUC Estimate(g CO2 MJ-1)

ILUC Range(g CO2 MJ-1)

Searchinger et al. 2008

2016 56 104 20 – 200a

Hertel et al. 2010 2001b 50 27 15 – 90c

Dumortier et al. 2009

2018/2019 30 n/a 21 – 118d

USEPA 2010 2012 7.5 81 62 – 104e

USEPA 2010 2017 14 58 43 – 76e

USEPA 2010 2022 10 34 25 – 45e

Al-Riffai et al. 2010 2020f 0.47 36 36 – 53g

CARB 2009 2001b 50 30 18 – 44

Tyner et al. 2010 2015 variable - 14 – 23

Plevin et al. (95% interval)

- - - 20 – 140

Plevin et al. (full range)

- - - 10 – 340

a. Calculated from reported sensitivity results.

b. Analysis was performed using the GTAP-6 database, based on 2001 data, but the results were adjusted post facto to account for the 10% greater average corn yield in 2010.

c. Range is based on a combination of high and low values for various uncertain economic model parameters.

d. Range is based on evaluating alternative model assumptions.

e. Range is 95% CI around mean considering only the uncertainty in satellite data analysis and carbon accounting.

f. Analysis was performed using the GTAP-7 database, based on 2004 data, using the model to project out to 2020.

g. Effect of additional 106 GJ after meeting 5.6% mandate. Higher value is for greater trade liberalization.

Notes about prior table

10

Economic model uncertainty dominates

Source: Plevin, O’Hare et al. 201011

EPA Uncertainty AnalysisSoybean biodiesel

N.B. Does not include uncertainties related to economic modeling or production period.

12

Large, but nebulous uncertainty

“The lesson for policymakers is that results from economic models depend heavily on assumptions, and because we are trying to predict long-run human behavior, there can be legitimate differences in these assumptions.”– Dumortier et al. 2009

“[T]his modeling project has demonstrated how the current limits to data availability create significant uncertainty regarding outcomes predicted by these policy simulations.”– Al-Riffai et al. (IFPRI) 2010

[O]ur experience with modeling, data, and parameter estimation and assumptions leads us to conclude that one cannot escape the conclusion that modeling land use change is quite uncertain. Of course, all economic modeling is uncertain, but it is important to point out that we are dealing with a relatively wide range of estimation differences.– Tyner et al. 2010 13

Uncertainty characterization is inconsistent and incomplete

• Searchinger et al.: Halving and doubling average emissions per hectare, other local SA

• Dumortier et al.: different FAPRI versions; crop yields; GREET vs BESS; w/wo forest conversion

• Hertel et al.: Systematic Sensitivity Analysis on few parameters, with notable limitations.

• USEPA: Uncertainty analysis of remote sensing and emission factors.

• CARB: OAT sensitivity analysis

14

Reducing uncertainty

• Better models• More models• Fewer models• Better data for models• Selection of “best” modelsMost of this work is being done in other

subgroupsUncertainty will not be reduced to zero

15

Policy and implementation challenge

• The LCFS requires publishing a GWI with infinite precision for each fuel, but our knowledge of the ‘real’ GWI is imperfect

• What is the best GWI to use within that framework?

16

Recall: three distinct variables for a single fuel

• g*ij = estimate of “physical” GWI of fuel i from model j

• i = real “physical” GWI: if fuel i is substituted for fuel k on a MJ/MJ basis, additional GHG release is (i - k). i is a random variable.

• gi = operational GWI of fuel i used in LCFS implementation

17

Optimal value for GWIElements of choice

Action: Implement a set of values gi for fuel GWI’sChoose objectiveFactors:

Distribution of i (real values): data is {g*ij}System response R({i}, {gi})Cost of ‘errors’ i – gi , R ≠ Ρ (real response)

The most likely value of i is not necessarily the optimal value of gi (safety factor principle).

18

Optimal gi’s

}{,,, iiihf ggVE R

Classical rule: Max{g} Expectation of value

19

where f and h are distributions of and R

20

What is the goal of the LCFS?• Climate

– Minimize difference i – gi ?

– Minimize T at a date (end of LCFS?)– Minimize additional GHG in atmosphere at a date (end of LCFS?) – Minimize total forcing up to a date?– Minimize probability that a condition will fail (LCFS increases one

of the above)?

• Other– Minimize social cost (including cost of T, food effects,

socioeconomic/welfare effects…?)– As of when (discounted how?)

21

Why might the cost of error be asymmetric?

• Could it be better/worse for i – gi to be +10 than -10? (Using too much/too little biofuel)

• ILUC is irreversible even if biofeedstock production stops.

• Encourage/obstruct advanced biofuels industry.

• Food effects from overuse of biofuel

22

Why might the cost of error be non-linear?

• Catastrophic events (Greenland ice cap, Gulf stream, peat feedback, etc.)

• Socio-economic effects• Other costs may be non-linear even if

symmetric:– Encourage/obstruct advanced biofuels industry.– Food effects from overuse of biofuel

23

Recognition of uncertainty entails attention to cost of error

• Asymmetric distribution of i means E(i) is not necessarily the optimal gi .

• This means the cost of error (functional form and asymmetry) also affects choice of gi

24

25

Uncertainty-explicit policy

• Few examples to learn from in fuel policy

• More theory, few examples in climate policy

• Many examples in health and safety regulation– Safety factor concept– Risk aversity for large costs

26

Policy MetricMetric

UncertaintyProxy

UncertaintyTreatment of Uncert.

Acid rain program

Tons SO2 and NOX

Low Low

Conservative

assumptions

PM2.5 standards

Measured PM2.5 conc.

Low Medium98th

percentile value used

Kyoto protocol

Direct CO2e Low MediumError

propagation recommended

RFS2Life cycle

CO2eHigh High

Partial uncertainty

analysis

LCFSLife cycle

CO2eHigh High None

How much investment in reducing uncertainty?

• --LUC is a complex social and biophysical phenomenon with significant variation across space and time.

• --Validating the role of any one factor/driver of LUC is difficult, at best, and very information-intensive.

• --So, even with substantial additional investments in GTAP, some uncertainties surrounding current iLUC estimates are irreducible.

28

“...in many situations, limitations of data, scientific understanding, and the predictive capacity of models will make (uncertainty) estimates unavailable, with the result that they must be supplemented with other sources of information."

• --Morgan et al. (2010), "Best Practices for Characterizing, Communicating, and Incorporating Scientific Uncertainty in Climate Decisionmaking."

• "Expert elicitation," i.e., a formal assessment by individual experts, based on the full range of scientific evidence, of their best judgement of a subjective probability distribution for the value in question.

• --Expert elicitations have been used to address many questions of climate science (e.g., role of aerosols).

• --While not a substitute for further research, expert elicitation can allow for a formal expression of diversity of opinion not fully reflected in the literature

29

Extending policy scope

• The greatest uncertainty and largest GHG effects are associated with a few landscape types/areas

• A separate policy restricting use of such landscapes for economic purposes may reduce uncertainty and the risk of policy failure more than improved analysis and modeling.

30

Expert elicitation

• --Experts could review full suite of information defining the influence of biofuels on iLUC, not only from GTAP, but including other models (FASOM/FAPRI), databases, and studies from other disciplines

• --In lieu of quantitative values for iLUC, experts could recommend adaptive decision rules/guidance for categorizing feedstocks and fuel pathways according to relative iLUC risk, based on an assessment of all available information

31

Recommendations

• Model estimates should present complete, systematic uncertainty characterization

• Expert elicitation may help infer distribution of i from {g*ij}

• CARB should establish objectives• Cost of error should be examined and estimated• CARB should implement gi with explicit attention to cost

of error and distribution of i

• CARB should explore mechanisms to incorporate estimates from multiple models in determining gi.

32