Mark W. RosegrantSiwa MsangiLiangzhi You

Assessing Climate Forecast Impacts

Advancing Ex Post Methodologies

The Growing Importance of Climate Forecast Information

Increasing frequency of extreme weather events and changing global trends in climate characteristics

The vulnerability of increasingly complex global economic and food systems to environmental factors

In the face of increasing uncertainty, policy makers are demanding better information

How To Measure Forecast Value?

Growing body of literature examining the economic value of forecast information

Theoretical underpinnings are grounded in the theory of decision-making under uncertainty

The majority of this literature employs ex ante methods

Ex Ante or Ex Post ?

A

(prior beliefs)

B(update beliefs)

(model possible response)

C(observe outcome of event) (also observe agent’s actions)

Receive Forecast Signal

Realized Climate Outcome

Ex Ante Ex Post

Modeled behavior

Simulated Benefit

Measured behavior

Realized Benefit

(simulated counter-factual)

Measuring Forecast Value

Within an Ex Ante framework, behavior is modeled and response is simulated to evaluate the net benefits of forecast

Within an Ex Post framework actual behavior is observed and underlying structural relationships driving response must be inferred to estimate the net benefits (comparing to without forecast information)

Overview of the Presentation

Look at Traditional Impact Assessment Discuss the Challenges of Ex Post

Evaluation Look at some promising directions Draw conclusions and

recommendations for advancing research in this area

Ex Post Assessment in Ag Research

Long history of application in empirical literature

Looks mostly at benefits of new technology

Evaluates net benefits with and without innovation

Employs a variety of empirical methods

Two Approaches in Ex Post Assessment

One approach tries to econometrically measure the impact of Ag Research on productivity or production costs with reduced form relationships

Another approach uses consumer welfare theory to relate technology improvements to benefits received by consumers and producers of the agricultural goods within the economy

Econometric Methods

Treating technology as an input, estimate production function, cost function or total factor productivity (TFP)

),,,( ttttt UKZXfQ Conventional Inputs (i.e. land)

Unconventional Inputs (i.e. infrastructure)

Technical knowledge (i.e.R&D investment)

Uncontrollable factors (i.e. weather)

Ag. Output

Econometric Methods

Estimated research coefficients are then used to calculate the value of additional output attributable to the lagged research expenditures ( marginal rate of return to the research investment)

Growth Accounting: contributions by the components in the above equation to the rate of growth of aggregated output

Basic Economic Surplus Model

e

Technology-induced supply shift (S0 to S1)

Total benefits (consumer and producer benefits) are Area of I0I1ab

Basic model can be extended to incorporate multi-markets, to accommodate spillover, to adjust for market distortions etc.

Lessons to Draw from the Literature

Scale (project, program, institution or the whole system)

Attribution (proper accounting for benefits and costs)

Selection bias (random sampling or “cherry-picking”)

Time lags (long lag between R&D investment and final impact)

Lessons to Draw from the Literature

Econometric methods rely on good-quality time series/panel data. More appropriate for entire research system rather than individual projects.

Economic surplus method requires limited data and flexible. It is widely used.

Challenges of Ex Post Assessment

Harder to take a ‘descriptive’ rather than ‘prescriptive’ approach

Non-excludable nature of climate information makes valuation harder

Must infer relevance of forecast from observed behavior which could be driven by a variety of factors

Dis-entangling the underlying structural relationships is non-trivial

Undertaking a Descriptive Analysis

1. Define decision alternatives and determine that the decision is weather-information-sensitive

2. Identify user’s goals

3. Identify all decision-relevant information available to user

4. Develop a model describing the relationship between available information and the decision

5. Evaluate the model. Does it adequately describe the user’s behavior?

6. Use the model to determine the impact of forecast on criteria

Economic Valuation Methods

Observed Behavior(ex post)

Hypothetical Behavior(ex ante)

DirectValuation

Experiments that measure response to better

information in a laboratory

Observed behavior under improved information from

collected data

Questions on willingness-to-pay for better forecasts

Simulated behavior under improved information,

assuming a behavioral model

IndirectValuation

Hedonic values generated by actual behavior within a

related market that can be tied to forecast information

Contingent ranking of attributes that can be indirectly related

to forecast

Direct Valuation Methods

An Ex Ante approach would rely on questioning the information ‘consumer’ directly

An Ex Post approach relies on the revelation of this value through behavior directly tied to climate shocks

Making the structural link between the climate shock and observed action is the challenge

Ex Post Approaches to Direct Valuation

Direct Valuation can be done using reduced-form relationships that derive statistical relationships between forecast information to behavior

Surplus values can also be computed An alternative is to look more closely at

the underlying structural determinants of behavior and estimate models that can link those to climate information

Reduced-Form Methods An econometric Ex Post analysis relies

on the statistical inference of forecast value from observables in both the physical and economic environment (e.g. land value and climate characteristics)

Controlling for non-forecast related factors that affect observed reaction to climate shocks (and information) is the principal challenge

Possible Reduced-Form Model

Relationship tying Farm profits (P) to climate information (K) and other on-farm characteristics

( , , , )t t t t tP f X Z K UConventional Inputs (i.e. land)

Unconventional Inputs (i.e. infrastructure)

Climate Information (knowledge sources)

Uncontrollable factors (i.e. weather)

Controlling for Behavioral Factors

Since the on-farm input levels are endogenous, they are instrumented

The ability of the farmer to adjust also should be accounted for

( , )t t t tX f Y G v

On-farm characteristics

(i.e. credit/labor constraints, farmer experience)

Exogenous factors (environmental, etc. )

Error term

Structural Estimation Approach

Is better able to connect the underlying drivers of response to climate information and environmental shocks than a ‘reduced-form’ approach

Better able to represent the constraints to agent behavior

Comes at a great computational cost as behavioral as well as environmental relationships must be estimated

Promising Trends

Despite relatively thin literature covering ex post methods, recent examples of innovative applications have surfaced

Cover a variety of settings, from pastoral management to crop production

Some methods empirical while others are experimental

A Few Case Studies

Solow et al. (1998) estimate net welfare from the use of ENSO-based climate information to range between $240 and $320 million annually for the U.S. agriculture sector alone

Bradford and Kelajian (1978) looked at the benefit-to-cost ratio of reducing sampling error in government-collected crop and livestock statistics

Livestock Management

Luseno et al. (2003) look at pastoralists in Northern Kenya/Southern Ethiopia

Compare pastoralists own perceptions with climate forecasts and observe how it influences their beliefs

They note the importance of taking flexibility into account, when measuring forecast value, to avoid underestimation

Experimental Approach

Sonka et al. (1988) use agribusiness example to show how decision experiments can be used to test response to varying levels of climate information

By using a controlled setting they are able to measure the impact of information more accurately and directly observe agent behavior

Comments and Critique

Despite the novelty of experimental methods, one may not be entirely sure that ‘real world’ behavior is being observed

Much depends on the design of the experiment and the ‘framing’ of the problem

But much insight can still be gained, and methods are apt to keep improving

Conclusions

Surplus-based methods might be applicable if designed properly

Structural estimation approaches give the most detail of the underlying relationships, but are the most challenging to apply, hard to generalize

Experimental techniques have increasing appeal and potential utility

Conclusions

Econometric methods would be valuable if reliable time series/panel data are available

Need to design panel or cross-sectional data collection efforts for evaluating climate forecast information (i.e. ASTI in agricultural R&D). Needs collaborative effort and long-term commitment.