© 2008 warren b. powell slide 1 the dynamic energy resource model warren powell alan lamont jeffrey...

© 2008 Warren B. Powell Slide 1

The Dynamic Energy Resource Model

Warren PowellAlan Lamont

Jeffrey StewartAbraham George

© 2007 Warren B. Powell, Princeton University


Dynamic energy resource management

Questions:» How will the market evolve in terms of the adoption of

competing energy technologies?• How many windmills, and where?• How much ethanol capacity?• How will the capacity of coal, natural gas and oil evolve?

» What government policies should be implemented?• Carbon tax? Cap and trade?• Tax credits for windmills and solar panels?• Tax credits for ethanol?

» Where should we invest R&D dollars?• Ethanol or hydrogen?• Batteries or windmills?• Hydrogen production, storage or conversion?



Uncertainties:» Technology:

• Carbon sequestration• The cost of batteries, fuel cells, solar panels• The storage of hydrogen, efficiency of solar panels, …

» Climate: • Global and regional temperatures• Changing patterns of snow storage on mountains• Wind patterns

» Markets: • Global supplies of oil and natural gas• International consumption patterns• Domestic purchasing behaviors (SUV’s?)• Tax policies• The price of oil and natural gas



Research challenges:» Making decisions

• Finding the best decisions (capacity decisions, R&D decisions, government policies) requires solving high-dimensional stochastic, dynamic programs.

• How do we obtain practical solutions to stochastic, dynamic programs which exhibit state variables with millions of dimensions?

» Modeling multiple time scales• We have to represent wind, temperature, rain and snow fall, market prices

and government policies.• This requires modeling hourly, daily, seasonal and yearly dynamics.

» Modeling multiple levels of resolution• Spatial: We need to represent the location of windmills at state, regional

and county levels.• Behavioral: We need to capture the differences between travel behavior

patterns (long commutes vs. short trips, commercial fleet vehicles vs. personal use), or the difference between light and heavy industrial power use.



Alternative ways of solving large stochastic optimization problems:» Simulation using myopic policies – Using rules to determine

decisions based on the current state of the system. Rules are hard to design, and decisions now do not consider the impact on the future.

» Deterministic optimization – Ignores uncertainty (and problems are still very large scale).

» Rolling horizon procedures – Uses point estimates of what might happen in the future. Will not produce robust behaviors.

» Stochastic programming – Cannot handle multiple sources of uncertainty over multiple time periods.

» Markov decision processes – Discrete state, discrete action will not scale (“curse of dimensionality”)



Proposed approach: Approximate dynamic programming» Our research combines mathematical programming, simulation

and statistics in a dynamic programming framework.• Math programming handles high-dimensional decisions.• Simulation handles complex dynamics and high-dimensional

information processes.• Statistical learning is used to improve decisions iteratively.• Solution strategy is highly intuitive – tends to mimic human behavior.

» Features:• Scales to very large scale problems.• Easily handles complex dynamics and information processes.• Rigorous theoretical foundation

» Research challenge:• Calibrating the model.• Designing high quality policies using the tools of approximate

dynamic programming.• Evaluating the quality of these policies.


Version 2 of issues:

Next two slides mimic the previous ones, but more compactly.


The dynamic energy resource model

Questions:» How will the market evolve in terms of the adoption of

competing energy technologies?• How many windmills, and where?• How much ethanol capacity?• How will the capacity of fossil fuel generation?

» What government policies should be implemented?• Carbon tax? Cap and trade?• Tax credits for windmills and solar panels?• Tax credits for ethanol?

» Where should we invest R&D dollars?• Ethanol or hydrogen?• Batteries or windmills?• Hydrogen production, storage or conversion?


The dynamic energy resource model

Features we need:» Multiple time scales

• Model will plan decades into the future, but reflect decisions and processes that occur on hourly, daily, seasonal and yearly levels.

» Multiple forms of uncertainty• We will model dynamic information processes that describe

the evolution of technology, climate, weather, prices and wind.

» Multiple levels of spatial granularity• The model will be able to run at different levels of spatial

aggregation, capturing the geographic substitution of different types of energy.

» Multi-attribute representation of markets• We want to be able to distinguish energy demands to capture

usage and lifestyle patterns.


Outline

Traditional models: optimization and simulation


Modeling

The challenge:» We understand the problem of simulating physical

systems• Climate• Hydrology• Technology

» To understand the economics for policy purposes, we need to model decisions.

• How will electricity flow from generating source to market demand on an hourly basis?

• How to operate different energy technologies (daily, weekly, seasonal)?

• How much generation capacity of each type will be added or retired each year?


Deterministic models

Deterministic, linear programming-based models» This is the current class of models used to inform

policy-makers.

» Basic model:

» Features:• Can model flows of energy and substitution of energy

resources over time.• Assumes a deterministic view of the world (everything is

known now).

1 1min subject to , 0t t t t t t t tt

c x A x B x R x



A time-dependent linear program:

TimeSpace



Limitations» Unable to model uncertainty in technology, climate,

prices.» Unable to model activities at a high level of detail.

Large linear program limits the number of rows, which grows rapidly as we use finer representations of resources and markets.

» Traditionally uses a discrete time representation, making it hard to handle fine time scales (e.g. hourly) over long horizons (e.g. 50 years).

» Deterministic models do not adequately represent the decisions policy-makers are faced with when determining climate policy.


Simulation models

Strengths» Very flexible – able to handle a high level of detail.

» Can handle any form of (quantifiable) uncertainty.

Weaknesses» Hard to program rules that mimic the intelligence of

markets, governments and companies.

» Will not recommend the path we should follow to reach a goal.


Modeling alternatives

Simulation» Strengths

• Extremely flexible• High level of detail

» Weaknesses• Low level of “intelligence”• Lower solution quality• May have difficulty

“behaving” properly with new scenarios.

• Difficulty adapting to random outcomes.

Optimization» Strengths

• High level of intelligence• System behaves “optimally”

even with new datasets• Reduces data set preparation.

» Weaknesses• Strict rules on problem structure• Low level of detail• Inflexible!

There is often a competition between deterministic “optimizers” and “simulators.”

. . . Why are we asking this question?


Outline

Elements of a dynamic energy model


Stochastic, dynamic model

Modeling energy resources

Capacity of facilities

Location

Cost

Carbon output

Age

Reserves

ta

Number of resources w/ attrib.

Resource state vectorta

t ta a A

R a

R R



The system state:

, , System state, where:

Resource state (how much capacity, reserves)

Market demands

"system parameters"

State of the technology (costs, pe

t t t t

t

t

t

S R D

R

D

rformance)

Climate, weather (temperature, rainfall, wind)

Government policies (tax rebates on solar panels)

Market prices (oil, coal, ...)



The decision variables:

Hourly dispatch (how much of each type)

Energy flows between regions and technologies

Investment in new capacity for each type of technologytx



Exogenous information:

ˆ ˆ ˆNew information = , ,t t t tW R D

where:

ˆ Exogenous changes in capacity, reserves

ˆ New demands for energy from each source

ˆ Exogenous changes in parameters.

Change in technology

Ch

t

t

t

R

D

ange in climate/weather

Change in prices/market supplies


Stochastic, dynamic model Hourly

» Daily temperature variation» Wind» Equipment failures

Daily» Fluctuation in spot prices» Short term demand» Major weather events» Transportation delays (movement of coal and oil)

Monthly» Seasonal variation (temperature, water flow for hydro, population shifts)» Medium term weather patterns» Significant supply disruptions (major hurricane, wars)

Yearly» Changes in technology» Demand patterns (SUV’s)» Long term climate cycles (including global warming)» Spatial patterns in population growth» New supply discoveries (major oil fields)» Intervention of foreign governments in markets» Long term supply contracts

tW



The transition function

1 1( , , )Mt t t t tS S S x W S

t t+1

Captures the evolution of all aspects of the system over time.



Our strategy:» Basic model:

» Features:• Simulation-based – We simulate forward in time using a very

general-purpose transition model.• Handles virtually any form of uncertainty.• Can use a range of policies for different types of decisions,

from simple dispatch rules to more sophisticated policies that look into the future.

1 1

( )

, ,

t t t

Mt t t t

x X S

S S S x W

Make a decision using policy

Update state using system model


Information and decisionsInformation

T – changes in technologyS – changes in energy suppliesP – changes in energy prices

W – Weather

DecisionsI – Changes in energy capacity infrastructure (new plants, new fields)S – Short term dispatch decisionsR – R&D investmentsM – Market response

T T T T

W WW W WW W WW W WW W WWW WWS S S S S S SSP P PP P P P P P P P P P P P P P P P

I I I IR RR R R R R R R RS S S S S S S S S S S S S S S S S SM M M M M M M M M


Making decisions

Dispatch decisions:» Use the technology with the lowest marginal cost.

» Each hour solve a small linear program to handle substitution of different types of power.

GENERATORS MARKETS

2

1

4

3

A

C

B

I

II

III

ENERGY SOURCES


Making decisions

Hydro power management» Forecast inflow and outflow to reservoirs to determine

amount available for generating electricity

r0l

1l

L

1 2 3 t

r0l

1l

L

1 2 3 t

r

0l

1l

1l

L

1 2 3 t

2lr

0l

1l

1l

L

1 2 3 t

2l r

1l2l

2l

0l

L

1 2 3 t

r

1l2l

2l

0l

L

1 2 3 t


Making decisions

Purchasing new capacity:» A decision to add capacity in year t changes the

capacity available in year t+1.

» We want our capacity acquisition decisions to mimic the intelligence that companies/financial markets make.

» We propose to “simulate Wall St.” by solving the capacity acquisition problem as an optimization problem that balances value now with the expected value of the future.


Outline

General ADP algorithmic strategy


Our general algorithm

Step 1: Start with a post-decision state

Step 2: Obtain Monte Carlo sample of and

compute the next pre-decision state:

Step 3: Solve the deterministic optimization using an

approximate value function:

to obtain .

Step 4: Update the value function approximation

Step 5: Find the next post-decision state:

, 1 ,1 1 1 1 1 1 ˆ( ) (1 ) ( )n x n n x n n

t t n t t n tV S V S v

1 ,ˆ max ( , ) ( ( , ) )n n n M x nt x t t t t t tv C S x V S S x

ntx

, ,1 , ( )n M W x n n

t t tS S S W

,1

x ntS

( )ntW

, , ( , )x n M x n nt t tS S S x

Simulation

Optimization

Statistics


Competing updating methods

Comparison to other methods:» Classical MDP (value iteration)

» Classical ADP (pre-decision state):

» Our method (update around post-decision state):

, 1 ,

, 1 ,1 1 1 1 1 1

ˆ max ( , ) ( ( , ))

ˆ( ) (1 ) ( )

n n x n M x nt x t t t t t t

n x n n x n nt t n t t n t

v C S x V S S x

V S V S v

11( ) max ( , ) ( )n n

x tV S C S x EV S

11

'

11 1

ˆ max ( , ) ( ' | , ) '

ˆ( ) (1 ) ( )

n n n nt x t t t t t t

s

n n n n nt t n t t n t

v C S x p s S x V s

V S V S v

ˆ updates ( )t t tv V S

1 1ˆ updates ( )xt t tv V S

, 1x ntV


Simulating a myopic policy

Approximate dynamic programming

t


Simulating a myopic policy



Using value functions to anticipate the future


t

“Here and now” Downstream impacts


Part VII - CASTLE Lab NewsCASTLE Lab News

New Modeling Language Captures Complexities of Real-World Operations!

75 cents

Spans the gap betweensimulation and optimization.

CASTLE Lab announced the development of a powerful new simulation environment for modeling complex operations in transportation and logistics. The dissertation of Dr. Joel Shapiro, it offers the flexibility of simulation environments, but the intelligence of optimization. The modeling language will allow managers to quickly test continued on page 3

Thursday, March 2, 1999


Outline

Estimating the value functions


Energy resource modeling

oiltx

2008

oiltR ˆ oil

tD ˆ oiltˆ oil

tR

New information 2009

1oiltR 1

oiltx 1

ˆ oiltD 1

ˆ oilt 1

ˆ oiltR

New information

windtxwind

tR ˆ windtD ˆ wind

tˆ windtR 1

windtR 1

windtx 1

ˆ windtD 1

ˆ windt 1

ˆ windtR

coaltxcoal

tR ˆ coaltD ˆ coal

tˆ coaltR 1

coaltR 1

coaltx 1

ˆ coaltD 1

ˆ coalt 1

ˆ coaltR

corntxcorn

tR ˆ corntD ˆ corn

tˆ corntR 1

corntx 1

corntR 1

ˆ corntD 1

ˆ cornt

ˆ corntR


We have to allocate resources before we know the demands for different types of energy in the future:



We use value function approximations of the future to make decisions now:



,,1x ntR

,,2x ntR

,,3x ntR

,,4x ntR

,,5x ntR

This determines how much capacity to provide:



,1ˆ ( )ntv

,2ˆ ( )ntv

,3ˆ ( )ntv

,4ˆ ( )ntv

,5ˆ ( )ntv

Marginal value:

,,1x ntR

,,2x ntR

,,3x ntR

,,4x ntR

,,5x ntR



1, 1,( )xt AB t ABV R

,1,

x nt ABR

Using the marginal values, we iteratively estimate piecewise linear functions.



R1t

ktv

ktv

Right derivativeLeft derivative


,1,

x nt ABR




R1t

( 1)ktv ( 1)k

tv


,1,

x nt ABR




Linear value function approximations:

Linear (in the resource state):

( )t t tl tll

V R v R

L

Two-stage stochastic programming


Piecewise linear, separable value function approximations:

Piecewise linear, separable:

( ) ( )t t tl tll

V R V R

L

Two-stage stochastic programming


Research challenges

Approximate dynamic programming:» At the heart of an ADP algorithm is the challenge of

finding a value function approximation “that works”• Can be used within commercial LP solvers• Can be updated (estimated) easily• Is stable• Provides high quality solutions

» Assessing solution quality• Is it realistic?

– Do we seem to mimic markets and public policy?• Is it robust?

– Do we achieve energy goals under different scenarios?


For the dynamic energy resource model, it is not enough to have a value function that depends purely on the resource vector.» The value of coal plants depends on our ability to sequester carbon.» We need to capture the “state of the world” in our value function approximations.

Strategies:» Let be the full system state vector, capturing the cost of technologies, government policies, etc. etc.» Let be a set of “features” that appear to be important explanatory variables. Identifying features is the “art” of ADP.» We can then fit value functions that depend on the features.

( | ( )) ( | ( ))t t t ta ta ta

V R S V R S

A

Research challenges

tS

( ),f tS f F


Research challenges

Strategies for fitting

» Lookup-table• Very general, but suffers from curse of dimensionality

» Linear regression with low-dimensional polynomials• Can work –depends on the problem.

» Kernel regression• Powerful strategy that combines lookup-table with regression

models.• Use within ADP is surprisingly young.• Variety of issues unique to ADP.

( | ( )) ( | ( ))t t t ta ta ta

V R S V R S

A


Research challenges

Approximate dynamic programming:» How do we establish that we are getting “good”

solutions?• Demonstrate techniques on simpler problems.• Compare against other methods for larger problems.

» We need algorithms that are fast and stable.• Identifying variance reduction methods from the simulation

community that work on this problem class.• Developing kernel regression techniques for improved fitting

of the value function.• Finding the best smoothing techniques for recursive updating.• Parallel processing for accelerating simulations.


Research challenges

System modeling» Modeling the evolution of technology using compact

representations• If we invest in technology, how do we describe the change

process?

» Modeling physical processes at multiple scales• Wind, temperature, rainfall at different levels of discretization.

» We need a software architecture that allows a larger community to participate in the modeling

• We need to tap into various types of domain expertise, such as climate modeling, transportation modeling, …


Notes:» We have developed a classical, deterministic linear

programming model as a basis for comparison and calibration.

» Early results show a close match between ADP and linear programming solution on a deterministic dataset (without storage).

» We intend to continue to use the LP model as a benchmark. But it is limited in the number of time periods it can handle. We cannot model hourly wind fluctuations over 5+ year horizon.


LP vs. ADP comparison

0

5E+12

1E+13

2E+13

2E+13

3E+13

3E+13

0 5 10 15 20 25 30 35 40

Iteration

Cost of ADP solution over LP optimal solution

Iteration

Cos

t ove

r L

P op

tim

al s

olut

ion


Outline

Features and capabilities



Modeling capabilities» Exogenous changes

• Hourly wind fluctuations• Daily price fluctuations• Seasonal rainfall and climate patterns• Yearly changes in energy technology

» Decisions• Adjustments to mix of energy to meet hourly demand• Weekly or monthly adjustments to coal and nuclear output• Seasonal adjustments to hydroelectric power• Yearly changes in energy resource capacity

» Features• Decisions will adapt to uncertainties such as the state of

battery technology or carbon sequestration• Capacity decisions will reflect future uncertainties• Model can be guided by external policies to meet specific

energy goals



Potential reports:» Likelihood of reaching energy goals given a particular

policy

» Usage patterns (hourly, daily, yearly) for different energy sources

» Impact of different energy policies on usage patterns and energy goals

» ???



Planned features:» Spatially distributed energy sources and demands

• Our library already handles this, but we need spatially disaggregate data.

» Hydroelectric storage• Being developed as we speak.

» Modular architecture• Library is very flexible, but we have not made specific efforts

to allow others to link in modules to model climate change, technology, … Plan is to make the overall architecture highly participatory.

© 2008 warren b. powell slide 1 the dynamic energy resource model warren powell alan lamont jeffrey...

Documents

powell slide

dynamic programs

natural gas slide

dynamic programming

princeton university

curse of dimensionality

rd decisions

stochastic programming