benchmark of forecasting models - ewea€¦ · benchmark of forecasting models reviewing and...
TRANSCRIPT
Benchmark of forecasting models
Reviewing and improving the state of the art
Daniel Cabezón
Head of Meteorological Models and Special Tasks (Energy Assessment Department)
Santiago Rubín
Energy Forecasting Manager (Energy Assessment Department)
Ignacio Lainez
Director of Energy Assessment
1. Introduction
2. Benchmark of forecasting models
3. Data feed to modelers
4. Error metrics
5. Open benchmark processes
OVERVIEW
2/14
1. Introduction
Objective: To have access on time to the best power forecast available
(expected mean and uncertainty)
1. Day Ahead Market (DAM): Hourly forecast for day D+1 to be delivered before [8-10] am (localtime) at day D
• Symmetric market (deterministic) / Asymmetric market (probabilistic)
• Individual / Aggregated portfolio
2. Intra-Day Market (IDM): refresh of new updated forecasts several times inside a day
Forecast of potential power = wind farm power without energy losses (100% availability and no curtailment)
3/14
4/14
• Best method to scan and identify on real time state-of-the-art forecasting models
• Initial participation necessarily as free trial
• Every participant can get in or quit as desired
• Monthly refresh of wind farm potential power to all modelers
• Milestone every quarter
• EDPR feedback to each individual participant (keeping confidentiality)
• After each Quarter (Jan-Apr-July-Oct)
• Possibility of 1 year contract in case of excellent ratio accuracy-pricing
Static datasupply
Model calibration(M1-M3)
ForecastStart
Real time hourly data feed + Error Monitoring
Q1 Q2 Q3 Q4
QUIT
New forecaster
Previous forecaster
2. Benchmark of forecasting models
• Creation of user in EDPR FTP server: retrieve data + send forecast
• Static Data
• Initial supply of data base for calibration (layout, WT model, historical potential power, etc.)
• Monthly update of wind farm potential power
• Dynamic data
• Real time supply from EDPR Data server to EDPR FTP server
• Submission 1: Active Power + Availability + Curtailment signal
• Submission 2: Preliminary Potential Power
Data server
(SCADA)
Static (Monthly basis)
Dynamic (real time)
FTP server
5/14
3. Data feed to modelers
• Off-line curve characterization: wind farm potential power VS nacelle wind speed
• Real time estimation and monitoring of potential power at all wind farms
• On-line retrieval of:
Nacelle wind speed Potential Power Substation Power
• Applications:
1. Track forecast error in real time
2. Analyze and detect in advance non-planned energy losses (icing, etc.)
0
200
400
600
800
1000
1200
1400
1600
1800
0 5 10 15 20 25 30
Wind speed at nacelle1 [m/s]
6/14
3. Data feed to modelers. Real time potential power
0
5000
10000
15000
20000
25000
30000
1 8
15
22
29
36
43
50
57
64
71
78
85
92
99
10
6
11
3
12
0
12
7
13
4
14
1
14
8
15
5
16
2
16
9
17
6
18
3
19
0
19
7
20
4
21
1
21
8
22
5
23
2
23
9
24
6
25
3
26
0
26
7
27
4
28
1
28
8
29
5
30
2
30
9
31
6
32
3
33
0
33
7
34
4
35
1
35
8
36
5
37
2
37
9
38
6
39
3
40
0
40
7
41
4
42
1
42
8
43
5
44
2
44
9
45
6
46
3
47
0
47
7
Po
ten
tial
Po
we
r [k
Wh
]
Time [h]
Consolidated
Real Time
Real time vs Consolidated potential power
*Failure rate = 3%
7/14
3. Data feed to modelers. Real time potential power
# KPI Parameter (%) Monitored (Weeklyand Monthly)
Description
1 NMAENormalized Mean Absolute ErrorAvg [Absolute Deviated MWh] / Nominal Power [MW]
2 WMAEWeighted Mean Absolute ErrorSum(Absolute Deviated MWh) / Sum(Production [MWh])
3 ImbalancesExcess and Deficit of P50 forecast (normalized to overall production) Excess -> Sum(Deviated MWh > 0) / Sum(Production)Deficit -> Sum(Deviated MWh < 0) / Sum(Production)
4 Percentiles AccuracySignificance of percentiles = % measurements below each percentile% Frequency when Production < P10% Frequency when Production < P90
5 Uncertainty BandDifference between P90 and P10 Avg (P90-P10) / Nominal Power [MW]
6 Time Series Time evolution of P10-P50-P90 forecast against measured potential
power (red line) during the previous week / month
# ForecastDeliverable(Hourly Base)
Description
1 Power P50Expected production(@ 100 % availabilityassumption)
2 Power P10, P90 Uncertainty Range
8/14
4. Error metrics
Targets
Balance
Make accurate
Minimize
Minimize
9/14
4. Error metrics
Month
# KPI Parameter (%) Monitored
Description
1 NMAENormalized Mean Absolute Error
Avg(Absolute Deviated MWh) / Nominal Power [MW]
NMAE (%)
10/14
4. Error metrics
Month
WMAE (%)
KPI Parameter (%) Monitored
Description
2 WMAEWeighted Mean Absolute Error
Sum(Absolute Deviated MWh) / Sum(Production [MWh])
11/14
4. Error metrics
Month
# KPI Parameter (%) Monitored
Description
3 ImbalancesExcess and Deficit of P50 forecast (normalized to overall production)
Excess = Sum(Deviated MWh > 0) / Sum(Production)Deficit = Sum(Deviated MWh < 0) / Sum(Production)
12/14
4. Error metrics
Month
# KPI Parameter (%) Monitored
Description
4 Percentiles AccuracySignificance of percentiles = % measurements below each percentile
% Frequency when Production < P10% Frequency when Production < P90
13/14
4. Error metrics
Month
# KPI Parameter(%) Monitored
Description
5Uncertainty
BandDifference between P90 and P10
Avg (P90-P10) / Nominal Power [MW]
US
Brazil
Canada
Poland
Romania
Italy
Portugal
France
Spain
BelgiumUK
14/14
5. Open benchmark processes
5 Models
12 Models (Wind)
8 Models (PV)
8 Models
45%
26%13%