energy demand forecasting industry practices and …...© 2010 ibm corporation ibm research -...
TRANSCRIPT
![Page 1: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/1.jpg)
© 2010 IBM Corporation
IBM Research - Ireland
© 2014 IBM Corporation
Energy Demand ForecastingIndustry Practices and ChallengesMathieu Sinn (IBM Research)
12 June 2014ACM e-EnergyCambridge, UK
![Page 2: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/2.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Outline
• Overview: Smarter Energy Research at IBM
• Energy Demand Forecasting– Industry practices & state-of-the-art– Generalized Additive Models (GAMs)– Insights from two real-world projects– Ongoing work and future challenges
• Conclusions
![Page 3: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/3.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
China
WatsonAlmaden
Austin
TokyoHaifa
Zurich
India
Dublin
Melbourne
Brazil
IBM Research Labs
Kenya
![Page 4: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/4.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Smarter Energy Research at IBMOverview
Solar
Wind
Solar
Wind
HydroelectricSolar
NuclearWind
Energy Storage
Energy Storage
Energy Storage
Utility
Plug-in Vehicle
Coal/Natural Gas
c
c c
c
Non-exhaustive project list:
• Pacific Northwest Smart Grid (transactive control, internet-scale control systems)
• Renewable energy forecasts
– Deep Thunder (weather), HyREF (wind power), Watt-Sun (PV)
• IBM Smarter Energy Research Institute (http://www.research.ibm.com/client-programs/seri/)– Outage Prediction and Response Optimization
– Analytics and Optimization Management System (AOMS)
![Page 5: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/5.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Energy Demand ForecastingMotivation
• Market purchases/sales
Tim
e horizo n
Intra-daily
Daily
Weekly
Yearly
Long-term
• Unit commitment, Economic dispatch
• Day-ahead outage planning
• Portfolio structuring
• Power plants maintenance schedule
• Future energy contracts
• Energy storage management
probab
. scenarios
determ
.
scena
riosW
eather
Economy
![Page 6: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/6.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Energy Demand ForecastingMotivation
Beyond forecasting: Load modeling and prediction
?
Source E.Diskin: Can “big data” play a role in the new DSO definition? European Utility Week, Amsterdam October 2013
![Page 7: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/7.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Energy Demand Forecasting
Current practice:
• Forecasting few, highly aggregated series
• Manual monitoring and fine-tuning
Challenges:
• Forecasts at lower aggregation levels → huge amounts of data
• Changes in customer behavior
• Distributed renewable energy sources
Requirements:
} dynamic!
Analytical models
- accurate, flexible, robust
- automated (online learning)
- transparent, understandable
Systems
- scalability, throughput
- data-in-motion and -at-rest
- external interface
![Page 8: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/8.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Energy Demand Forecasting
Objectives:
• Forecasting energy demand at various aggregation levels
– Transmission and Subtransmission networks
– Distribution substations and MV network
– Breakdown by customer groups
Rationales:
• Disaggregate demand for higher forecasting accuracy
– Local effects of weather, socio-economic variables etc.
• More visibility on loads in Subtransmission and Distribution networks– Understanding the effect of exogenous variables– Detecting trends, anomalies, etc.
– Accounting for reconfiguration events
top-down
A B
A1 A2 A3 B1 B2
![Page 9: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/9.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Energy Demand Forecasting
Objectives:
• Forecasting energy demand at various aggregation levels
– Transmission and Subtransmission networks
– Distribution substations and MV network
– Breakdown by customer groups
Rationales:
• Disaggregate demand for higher forecasting accuracy
– Local effects of weather, socio-economic variables etc.
• More visibility on loads in Subtransmission and Distribution networks– Understanding the effect of exogenous variables– Detecting trends, anomalies, etc.
– Accounting for reconfiguration events
top-down
A B
A1 A2 A3 B1 B2
Loadshift
![Page 10: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/10.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
35M Smart Meters
800K Low-Voltage Stations
Substations
Ene
rgy
Con
sum
ptio
n (M
W)
Seasonalities at different times scalesComplex non-stationaritiesTrends and abrupt changes
Energy Demand ForecastingExample
En
erg
y D
ema
nd
(G
Wh
)
![Page 11: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/11.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
35M Smart Meters
800K Low-Voltage Stations
Substations
Ene
rgy
Con
sum
ptio
n (M
W)
Energy Demand ForecastingExample
En
erg
y D
ema
nd
(G
Wh
)
![Page 12: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/12.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
35M Smart Meters
800K Low-Voltage Stations
2,200 ‘Postes Sources’
Ene
rgy
Con
sum
ptio
n (M
W)
Energy Demand ForecastingExample
En
erg
y D
ema
nd
(G
Wh
)
![Page 13: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/13.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Week of July 2, 2007
En
ergy
Con
sum
ptio
n (M
W)
ConsumptionTemperature
Tem
p eratu re (F)
Tem
per ature (F)
Week of July 2, 2007
Ene
rgy
Dem
and
(GW
h) Electrical Load
Energy Demand ForecastingExample
![Page 14: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/14.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Week of July 2, 2007
En
ergy
Con
sum
ptio
n (M
W)
ConsumptionTemperature
Tem
p eratu re (F)
Tem
per ature (F)
Week of February 5, 2007
Ene
rgy
Dem
and
(GW
h) Electrical Load
Energy Demand ForecastingExample
![Page 15: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/15.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Assumption: effect of covariates is additive
Illustrative example:
• xk = (xkTemperature, xk
TimeOfDay) (covariates)
• yk = fTemperature(xk) + fTimeOfDay(xk) (transfer functions)
• Say, xk = (12°C, 08:30 AM)
→ yk = 0.0 GW + 0.02 GW
T (°C)
Nor
m’e
d D
eman
d
Contribution of temperatureon energy consumption
Time of day in 30 min interval
Nor
m’e
d D
eman
d
Contribution of time of dayon energy consumption
Energy Demand ForecastingAdditive Models
![Page 16: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/16.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Energy Demand ForecastingAdditive Models
Demand
Transferfunctions Noise
Basisfunctions
Weights
Categoricalcondition
Formulation:
Transfer functions have the form:
This includes:● constant, indicator, linear functions● cubic B-splines (1- or 2-dimensional)
Covariates:- Calendar variables (time of day, weekday...)- Weather variables (temperature, wind ...)- Derived features (spatial or temporal functionals)- ...
![Page 17: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/17.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Energy Demand ForecastingAdditive Models
Formulation:
Training:
1) Select covariates, design features
2) Select basis functions (= knot points)
3) Solve Penalized Least Squares problem
where λK is determined using Generalized Cross Validation
Linear in basis functions
Penalizer
S. Wood (2006): Generalized Additive Models. An Introduction with R.
![Page 18: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/18.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
En
ergy
Con
sum
ptio
n (M
W)
Tem
p eratu re (F)
Energy Demand ForecastingEDF-IBM NIPS model
Model for 5 years of French national demand (Feb 2006 – April 2011)1
Covariates:
• DayType: 1=Sun, 2=Mon, 3=Tue-Wed-Thu, 4=Fri, 5=Sat, 6=Bank holidays
• TimeOfDay: 0, 1, ..., 47 (half-hourly)
• TimeOfYear: 0=Jan 1st, ..., 1=Dec 31st
• Temperature: spatial average of 63 weather stations
• CloudCover: 0=clear, ..., 8=overcast
• LoadDecrease: activation of load shedding contracts
1A.Ba, M.Sinn, P.Pompey, Y.Goude: Adaptive learning of smoothing splines. Application to electricity load forecasting. Proc. Advances in Neural Information Processing Systems (NIPS), 2012
![Page 19: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/19.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Tem
p eratu re (F)
Energy Demand ForecastingEDF-IBM NIPS modelModel:
Transfer functions: Results:
Trend Lag load Day-type specific daily pattern
Lag temperature (accounting for thermal inertia)
TimeOfDay / Temperature TimeOfYear
1.63% MAPE20% improvement by online learningExplanation: macroeconomic trend effect
![Page 20: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/20.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Energy DemandEDF-IBM Simulation platform Simulation:
• Massive-scale simulation platform for emulating demand in the future electrical grid2
– 1 year half-hourly data, 35M smart meters
– Aggregation by network topology (with dynamic configurations)– Changes in customer portfolio
– Distributed renewables (wind, PV)
– Electric vehicle charging
• Built on IBM InfoSphere Streams
2P.Pompey, A.Bondu, Y.Goude, M.Sinn: Massive-Scale Simulation of Electrial Load in Smart Grids using Generalized Additive Models. Springer Lecture Notes in Statistics (to appear), 2014.
![Page 21: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/21.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Forecasting:
• Statistical approach: Generalized Additive Models (GAMs)
– Accuracy, flexibility, robustness, understandability ...
• Developing GAM operators for IBM InfoSphere Streams
• Online learning:
– Tracking of trends (e.g., in customer portfolio)
– Reducing human intervention– Incorporating new information
Energy Demand Forecasting Online learning
score
learn
score
control
GAMLearner
PMMLPMML = Predictive Models Markup Language
![Page 22: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/22.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Model parameter“Kalman gain”
Forecasting error
Precision matrix
Formulation of GAM learning as Recursive Least Squares:
• Adapt model once actual demand becomes available ( → )
• Implementation:
– Forgetting factor (discounting past observations)
– Complexity: O(p2) (p = number of spline basis functions)
– Sparse matrix algebra → 1000 tuples per second– Adaptive regularization
Energy Demand Forecasting Online learning
![Page 23: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/23.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Stability:
• Incorporate historical sample information in
• Rule of thumb for forgetting factor:
• Hence, for a time window of 1 year = 365*48 data points:
• Another potential issue: divergence of
• “Blowing-up” of Kalman gain
time window size
Don't forgetduring summer whathappened in winter!
Energy Demand Forecasting Online learning
![Page 24: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/24.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Stability:
• is the inverse of the sum of discounted matrix terms
• Divergence can occur, e.g.,
– if subset of basis functions is (almost) collinear
– if subset of basis functions is (almost) always zero
• Solution: Adaptive regularizer– Monitor matrix norm of
– If norm exceeds threshold, then add diagonal matrix to the inverse of
– Complexity: O(p3)
Outer product ofspline basis functions
Energy Demand Forecasting Online learning
![Page 25: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/25.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Learning “from scratch” (initial parameters all equal to zero):
Energy Demand Forecasting Online learning
![Page 26: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/26.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Vermont ProjectScope
Source: wikipedia.org
Weatherdata
Deep Thunder Weather
Analytics Center
Renewable Forecast
Wind
Solar
Demand Forecast
RenewableIntegration
Stochastic Engine
OutcomesMeter data& networks
Power data
![Page 27: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/27.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Vermont ProjectDeep Thunder
Weather variables:- Temperature- Clouds- Humidity- Wind- Solar radiation- ...
![Page 28: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/28.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
Modeling:
• Distributed renewables “behind the meter”
• Forecasting uncertainty
Variable selection & feature extraction:
• Spatial averages of weather variables
• Temporal features (e.g., heat waves)
• Formalization & automatization
Transfer learning:
• How to integrate information from older, lower-resolution data sets?
Transparent analytics:
• GUI which allows users to “drive” analytics withouth in-depth statistical knowledge
Vermont ProjectDemand forecasting challenges
![Page 29: Energy Demand Forecasting Industry Practices and …...© 2010 IBM Corporation IBM Research - Ireland © 2014 IBM Corporation Energy Demand Forecasting Industry Practices and Challenges](https://reader035.vdocuments.site/reader035/viewer/2022071104/5fde6d3d8fce5c46406dcaca/html5/thumbnails/29.jpg)
IBM Research - Ireland
© 2014 IBM Corporation
• Smarter Energy Research at IBM
• Energy demand forecasting
– Current practices & future challenges
– Methodology– Insights from two projects
Thank you!
Conclusions