overview of the current status and future prospects of wind power production forecasting for the...
TRANSCRIPT
Overview of the Current Status and Future Prospects of Wind Power
Production Forecasting for the ERCOT System
John W. Zack
Director of ForecastingAWS Truewind LLC
Albany, New York
ERCOT Workshop
Austin, TX
June 26, 2009
Overview
• How Forecasts are Produced– Overview of forecasting technology, issues and products – ERCOT-specific issues and forecast products
• WGR Data Issues• Forecast Performance
– Recent performance statistics– Comparison to other regions
– Analysis of cases with much worse than average performance • Road to Improved Forecasts
How Forecasts are Produced
Forecasting ToolsInput Data
Configuration for ERCOTERCOT Forecast Products
© 2009 AWS Truewind, LLC
Major Components ofState-of-the-Art Forecast Systems
• Combination of physics-based (NWP) and statistical models
• Diverse set of input data with widely varying characteristics
• Importance of specific models and data types vary with look-ahead period
• Forecast providers vary significantly in how they use forecast models and input data
Input Data, Forecast Model Components and Data Flow for a State-of-the-Art Forecast System
© 2009 AWS Truewind, LLC
Physics-based Models(also known as Numerical Weather Prediction (NWP) Models)
• Differential equations for basic physical principles are solved on a 3-D grid
• Must specify initial values of all variables for each grid cell
• Simulates the evolution of all of the basic atmospheric variables over a 3-D volume
• Some forecast providers rely solely on government-run models; others run their own model(s)
Roles of Provider-run NWP Models• Optimize model configuration for the forecasting of near-surface winds• Use higher vertical or horizontal resolution over area of interest• Execute simulations more frequently• Incorporate data not used by government-run models• Execute ensembles customized for near-surface wind forecasting
© 2009 AWS Truewind, LLC
Statistical Models• Empirical equations are derived from historical predictor and predictand data (“training sample”)
• Current predictor data and empirical equations are then used to make forecasts
• Many different model-generating methods available (linear regression, neural networks etc.)
Predictors Predictand
P1,P2,... F
F = f(P1,P2,...)
TrainingAlgorithm
SMLRANNSVM
Roles of Statistical Models• Account for processes on a scale smaller than NWP grid cells (downscaling) • Correct systematic-errors in the NWP forecasts• Incorporate additional (onsite and offsite) observational data
• received after the initialization of most recent NWP model runs • not effectively included in NWP simulations
• Combine met forecasts and power data into power predictions
MOS
An Important Consideration:Statistical Model Optimization Criterion
• Statistical models are optimized to a specified criterion (either explicitly or by default)
• Result: prediction errors have different statistical properties depending on the criterion
• Sensitivity to forecast error is dependent on the user’s application
• Forecast users and providers should communicate to determine and implement best criterion
• I = input variables• W = weights (from training sample)• O = output variables (the forecast)• T = target (observed variables)• e = forecast error
• Performance criterion (PC) and an optimizing algorithm needed to obtain W’s in the model• A typical PC is mean square error (MSE) - but is it a good one?
© 2009 AWS Truewind, LLC
Plant Output Models
• Relationship of met variables to power production for a specific wind plant
• Many possible formulations• implicit or explicit• statistical or physics-based• single or multi-parameter
Roles of Plant Output Models• Facility-scale variations in wind (among turbine sites)• Turbine layout effects (e.g. wake effects) • Operational factors (availability, turbine performance etc)
© 2009 AWS Truewind, LLC
Forecast Ensembles
• Uncertainty present in any forecast method due to– Input data
– Model type
– Model configuration
• Approach: perturb input data and model parameters within their range of uncertainty and produce a set of forecasts (I.e. an ensemble)
Roles of Ensembles• Composite of ensemble members is often the best performing forecast• Ensemble spread provides case-specific measure of forecast uncertainty • Can be used to estimate structure of forecast probability distribution
Power Forecast Uncertainty
• Two primary sources of uncertainty– Position on power curve– Predictability of weather regime
• Useful to distinguish between sources– Estimate weather-related
uncertainty from spread of forecast ensemble
– Estimate power curve position uncertainty by developing statistics
over all weather regimes
A real facility-scale power curve
How the Forecasting Problem Changes by Time Scale
Minutes Ahead•Large eddys, turbulent mixing transitions•Rapid and erratic evolution; very short lifetimes•Mostly not observed by current sensor network•Forecasting tools: Autoregressive trends•Very difficult to beat a persistence forecast•Need: Very hi-res 3-D data from remote sensing
Hours Ahead• Sea breezes, mountain-valley winds, thunderstorms• Rapidly changing, short lifetimes• Current sensors detect existence but not structure• Tools: Mix of autoregressive with offsite data and
NWP• Outperforms persistence by a modest amount• Need: Hi-res 3-D data from remote sensing
Days Ahead•“Lows and Highs”, frontal systems•Slowly evolving, long lifetimes•Well observed with current sensor network•Tools: NWP with statistical adjustments•Much better than a persistence or climatology forecast•Need: More data from data sparse areas (e.g.
oceans)© 2009 AWS Truewind, LLC
© 2009 AWS Truewind, LLC
Forecast Products
• Deterministic Predictions– MW production for a specific time interval (e.g. hourly)– Typically the value that minimizes a performance metric (not necessarily the
most probable value)
• Probabilistic Predictions– Confidence bands– Probability of Exceedance (POE) values
• Event Forecasts– Probability of defined events in specific time windows – Most likely (?) values of event parameters (amplitude, duration etc.)– Example: large up or down ramps
• Situational Awareness– Identification of significant weather regimes (alerts)– Geographic displays of wind & weather patterns to enable event tracking
ERCOT Forecast System
• Current– Single NWP model run every 6 hrs– No Model Output Statistics (MOS)– WGR output model: mixed approach
• Some statistical using WGR data• Some specified power curve
• Planned– 9 NWP model runs every 6 hrs– 1 NWP model run every hr (RUC)– Model Output Statistics (MOS)– Statistical optimized ensemble– Statistical power curve: all WGRs
ERCOT Forecast Products
• Products– Hourly updated 1 to 48 hr ahead forecasts in hourly increments– STWPF: ~ most likely value– WGRPP: 80% probability of exceedance value– For each WGR and the aggregate of all WGRs
• Delivery Time– By 15 minutes past each hour
• Delivery Vehicles– xml files via web services to ERCOT– csv files via email to each QSE for each WGR for which they schedule– csv files and graphical displays on web page for ERCOT access only
Data Issues
Overview of Data Quality IssuesExamples
Overview of WGR Data Issues
• Uses of WGR Data– Statistically adjust NWP output data– Define wind-power relationship– Identify recent trends for very short-
term (0-4 hrs) forecasts
• Impact of Data Quality Issues– Prevent forecast performance from
reaching its potential– Degrade forecast performance
• Types of Issues– Curtailment– Turbine Availability– Missing data– Erroneous or locked data– Unrepresentative met data
Met Data Status # of WGRs
Useable Data 35
Very high degree of scatter
12
High degree of scatter
11
Data appears not to be in mph
8
No data or < 10 data points
2
Less than 25% of data received
2
Total 70
Data Quality Example #1: Useable Data
• Data available through most of the power curve
• Well-defined wind speed vs power production relationship
• Modest amount of scatter• Only a few outlier points
Hourly average power production vs hourly average wind speed for an ERCOT WGR for non-curtailed hours during the period May 1, 2009 to June 14, 2009
The Good
Data Quality Example #2: Useable But Limited Data
• Generally good data but limited amounts of data in the upper range of the power curve
– Light winds– Curtailment during high wind periods
• Difficult to obtain an accurate wind speed vs power relationship for higher wind speeds Hourly average power production vs hourly average wind
speed for an ERCOT WGR for non-curtailed hours during the period May 1, 2009 to June 14, 2009
The Not as Good
Data Quality Example #3:Questionable scaling of met data
• Wind speeds do not appear to be in mph - perhaps m/s but not clear
• Problem: scaling method is not known and thus data is not useable
• In addition to the scaling issues this example has a moderately high amount of scatter
Hourly average power production vs hourly average wind speed for an ERCOT WGR for non-curtailed hours during the period May 1, 2009 to June 14, 2009
The Bad
Data Quality Example #4:High Amount of Scatter
• A high amount of scatter in the wind speed vs power production relationship is evident
• Data is not useable since its not clear which data has issues
• Possible causes– Turbine availability issues?– Erroneous or unrepresentative
anemometer data?
Hourly average power production vs hourly average wind speed for an ERCOT WGR for non-curtailed hours during the period May 1, 2009 to June 14, 2009
Another Type of Bad
Data Quality Example #5:Extremely High Amount of Scatter
• Enormous amount of scatter in the wind speed vs power production relationship
• Not useable• Difficult to define a WGR-
scale power curve• Possible causes
– Turbine availability issues?– Erroneous or unrepresentative
anemometer data? Hourly average power production vs hourly average wind speed for an ERCOT WGR for non-curtailed hours during the period May 1, 2009 to June 14, 2009
The Ugly
Forecast Performance
IssuesERCOT Results: Wind Speed and Power
Case Examples
Amount and Diversity of Regional Aggregation Impacts Apparent
Forecast Performance• Example: Alberta Wind Forecasting
Pilot Project– 1 year: May 2007-April 2008– 3 forecast providers– 12 wind farms (7 existing + 5 future) divided
into 4 geographic regions of 3 farms each– Hourly 1 to 48 hrs ahead forecasts for farms,
regions and system-wide production
• Regional day-ahead forecast RMSE was 15-20% lower than for the farms
• System-wide day-ahead forecast RMSE was 40-45% lower than for the individual farms
Effect of Aggregation on RMSEAWST 1-48 Hr Ahead Forecasts
Alberta Pilot Project: 1 May 07 - 30 Apr 08
0
5
10
15
20
25
30
35
0 3 6 9
12 15 18 21 24 27 30 33 36 39 42 45 48
Look-ahead Period (hrs)
RM
SE (
% o
f Ca
paci
ty)
Farm Avg (12- 52.8 MW)Regional Avg (4-158.5 MW)System (634 MW)
© 2009 AWS Truewind, LLC
Impact of Aggregation on Performance Comparisons: Misconceptions
• Lack of a consideration of the impact of size and diversity of the generation resource leads to misconceptions about relative forecast performance
• Example: US visitors to the Spanish TSO “RED Electrica” concluded that forecast performance was “phenomenal”
• The size and diversity of this aggregation is so great that there is a huge aggregation effect and when this is considered the performance is typical
From the Visit Report:“SIPREOLICO provides detailed hourly forecasts up to 48 hours updated every 15 minutes. The accuracy of the forecast is phenomenal: The forecast root mean square error for the 48-hour- ahead forecast is below 5.5% of the installed wind generation capacity. “
RED Electrica System Characteristics:• 14,877 MW installed; 575 wind parks • Average of about 30 MW capacity/park • Peak Generation ~ 10,000 MW
© 2009 AWS Truewind, LLC
ERCOTForecast Performance
Wind Speed
Power Production
System-Wide Wind Speed Forecasts
• Parameter – Average wind speed over all
wind generation areas on the ERCOT system
– Provides a perspective on forecast performance that is independent of the power data issues (curtailment, availability etc.)
• Results– Very low bias after January– Low MAE and RMSE
• MAE: ~1.2 m/s (15% of avg)• RMSE: ~ 1.5 m/s (19% of avg)
Monthly System-Wide Average Wind Speed 24-Hr Ahead Forecast Performance Statistics
Jan - May 2009
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Jan Feb Mar Apr May
MonthB
ias,
MA
E o
r R
MS
E (
m/s
)
Bias MAE RMSE
Day-Ahead Wind Speed Forecast MAE:Comparison Between Regions
• Comparison Periods – ERCOT: 1/1/09 - 5/31/09– Alberta: 5/1/07- 4/30/08– NYISO: 1/1/09 - 5/31/09
• Results– ERCOT sites have lower MAE
than Alberta sites but higher than NYISO sites
– ERCOT MAE considerations• For 5 higher MAE months;
annual MAE probably lower• Adversely impacted by wind
speed data issues• Early version of ERCOT
forecast system
Individual Site Day-Ahead Wind Speed Forecast MAE Performance Comparison
0.0
0.5
1.0
1.5
2.0
2.5
3.0
ERCOT Alberta-Fcst A
Alberta-Fcst B
Alberta-Fcst C
NYISO
SystemM
AE
or
RM
SE
(m
/s)
Power Production Forecast Performance Evaluation Issues
• Issue: It is difficult to determine the power production values to use in the forecast evaluation due to curtailment, turbine availability and data quality issues
• AWST’s Performance Evaluation Datasets– QC1: Locked and missing data removed– QC2: QC1 & removal of data during ERCOT-specified curtailed periods– QC3: QC2 & power curve check
• Data rejected if |Prep - Ppc| > 30% of cap• Attempt to eliminate data with significant unreported availability issues
– Synthetic Power: Best guess at power production for all hours• Reported power if deemed representative• Power estimated from power curve if wind speed deemed reliable• Power estimated from capacity factor of highest correlated nearby facility
System-Wide Power Forecasts:Impact of Data on Bias & MAPE
• Lack of consideration of curtailment periods (QC1) leads to much higher Bias & MAPE values in some months (e.g. February)
• Considerable positive bias remains even with QC3 data despite very low wind speed forecast bias
Monthly System-Wide Power Production Forecast Bias: Jan - May 2009
0
5
10
15
20
25
Jan Feb Mar Apr May
Month
Bia
s (
% o
f C
ap
acit
y)
QC1 QC2 QC3 SYNTHETIC
Monthly System-Wide Power Production Forecast MAPE: Jan - May 2009
0
5
10
15
20
25
Jan Feb Mar Apr May
Month
MA
PE (
% o
f C
ap
acit
y)
QC1 QC2 QC3 SYNTHETIC
System-Wide Power Forecasts:Day-Ahead MAPE Comparison
• Comparison Specs:– ERCOT: ~8000 MW; 24 hr ahead– Alberta: 355 MW; 24 hr ahead– CAISO: 815 MW; day-ahead (PIRP)– NYISO: 688 MW; day ahead
• ERCOT MAPE is lower than that achieved by any forecaster in Alberta
• ERCOT MAPE is higher than that for CAISO and NYISO aggregates
Power Production Forecast MAPE Comparison
0
2
4
6
8
10
12
14
16
18
ERCOT Alberta-Fcst A
Alberta -Fcst B
Alberta -Fcst C
California NY
System - Foreacster
MA
PE (
% o
f C
ap
acit
y)
Forecast Performance:Case Examples
Examples of difficult
cases from 2009
April 29, 2009 Case(forecast delivered 3:15 PM CDT April 28)
• Large over estimate of the power production from 1 AM to 2 PM
• Error caused by a poor prediction of the intensification of a storm to the north of Texas and the associated southerly winds
• Error most likely attributable to large scale weather prediction by the National Weather Service data and models
ERCOT Forecast and Observed Generation on 4/29
0
1000
2000
3000
4000
5000
6000
7000
8000
1 3 5 7 9 11 13 15 17 19 21 23
Hour of the Day (CDT)
Gen
erati
on (M
W)
Observed Forecasted
April 29, 2009 Case
• The west to east pressure gradient is too strong and therefore the winds are too strong
• Ensemble with input from non-NWS (e.g. Canadian) NWP data might help in these type of cases
Surface Weather Map: 6 AM
NWP forecast of 70 m wind speed (m/s) for 6 AM (used for 3 PM forecast on 28 April 2009)
May 11, 2009 Case(forecast delivered 3:15 PM CDT May 10)
• Large over estimate of the power production from 8 AM to 2 PM
• Error caused by slight errors in the placement of a slow moving frontal zone across central Texas
• Much of the error associated with higher than forecasted winds in the Sweetwater area
ERCOT Forecast and Observed Generation on 5/11/09
0
1000
2000
3000
4000
5000
6000
7000
8000
1 3 5 7 9 11 13 15 17 19 21 23
Hour of the Day (CDT)
Gen
erati
on (M
W)
Observed Forecast
May 11, 2009 Case
• Frontal zone with large horizontal variation in wind speeds and direction was located near the wind generation areas
• Small errors in frontal placement can lead to large errors in power forecasts
• An example of a typical high uncertainty situation
• Ensemble forecast should help anticipate uncertainty and hedge “most likely” forecast
May 16, 2009 Case(forecast delivered 3:15 PM CDT May 15)
• Large over estimate of the power production from midnight to 5 AM as a cold front approached and passed through the region
• NWP models overestimated the southerly wind speeds ahead of the front
ERCOT Forecast and Observed Generation on 5/16
0
1000
2000
3000
4000
5000
6000
7000
8000
1 3 5 7 9 11 13 15 17 19 21 23
Hour of the Day (CDT)
Gen
erati
on (M
W)
Observed Forecasted
May 16, 2009 Case
• Position and timing of the front is very good
• However, the NWP model greatly overestimated the southerly wind speeds ahead of the front
Surface Weather Map: 3 AM
NWP forecast of 70 m wind speed (m/s) for 3 AM (used for 3 PM forecast on 15 May 2009)
Road to Improved Forecasts
Resolve Data IssuesEnsembles
Regime and Event Specific Methods
Anticipated Steps to Improve Forecast Performance
• Resolve WGR data issues– Obtain maximum information about curtailments and turbine availability– Obtain meteorological data from all WGRs– Resolve issues with meteorological data for many sites
• Implement full statistical ensemble system– Dependent on quality of WGR power and meteorological data– Ensembles will help most in situations with large uncertainty
• Take advantage of regime-based and event-based techniques– Regime-based MOS– Event-based MOS
Relative Role of NWP and MOS in Day-Ahead Forecast Performance: An Example
• The gap between the blue and red lines is a measure of the contribution of WGR-data to forecast performance via MOS
• The raw NWP forecasts have an average MAE improvement over persistence of ~ 40% and the MOS procedure increases that to ~ 60%
• The MOS improvement increment varies from week to week (probably regime effect)
Weekly power production forecast MAEs for an individual WGR in the eastern US for an 8-week period in the fall of 2008. MAEs are scaled by the MAE of a persistence forecast for the entire 8-week period.
© 2007 AWS Truewind, LLC
Impact of Weather RegimesExample: power production forecasts during AESO’s Alberta Pilot Project
• Significant winter wind regimes in Alberta were identified for the 2007-08 season• Forecast performance was analyzed by regime
Power production forecast error was much larger for the SCA regime than the non-SCA cases primarily because the characteristics of NWP forecast errors were quite different in SCA and non-SCA regimes
Shallow cold air (SCA) regime occurs when slowly moving cold air from the N or E undercuts a warmer air mass typically characterized by strong W or SW winds
Event-Specific Forecasting Example
• Large system-wide ramps on multiple time scales occurred in the 10 PM to Midnight period on March 10, 2009
• Caused by a southward propagating cold front
• Investigated as part of the development of ELRAS (ERCOT Large Ramp Alert System)
Wind Speed (m/s) at North Texas Site 1 on March 10-11,
0
5
10
15
20
18 19 20 21 22 23 0 1 2 3 4 5 6 7
Time (CDT)
Sp
eed
(m
/s)
March 10-11Case:Precursor Conditions
• Top: Measured speed at a north Texas WGR
• Right: NWP-produced map of 15-minute wind speed change at 7:15 PM CDT
Use of an Event-Tracking Parameter• Parameters: Distance
to and amplitude of maximum positive wind speed change along a radial path from the forecast site
• Example: parameters indicate a consistent and coherent approach of the feature
• Approach can be used with NWP model output data and/or offsite measured data
Amplitude and Distance of Positive Wind Speed Changes North of Sweetwater on March 11, 2009
0
50
100
150
200
250
300
350
400
450
500
19 9
Dis
tance
(km
)0
5
10
15
20
25
Am
plitu
de (
m/s
)
Distance of Maximum Change
Maximum Wind Speed Change
0
500
1000
1500
2000
2500
3000
3500
19
20
21
22
23 0 1 2 3 4 5 6 7 8 9
Time (UTC)
Pow
er
(MW
)
0
4
8
12
16
Sp
eed
(m
/s)
SWE Estimated Uncurtailed Power(MW)ST2 Wind Speed (m/s)
Causes of the Sixty Largest ERCOT Wind Power Ramps January 1 to April 20, 2009
Coid Front - Up Ramp
Low Level Jet - Up Ramp
Pressure Gradient Change - UpRamp
Daytime Mixing - Up Ramp
Dry Line - Up Ramp
Pressure Gradient Change - BehindCold Front - Down Ramp
Pressure Gradient Change - HighPressure - Down Ramp
Pressure Gradient Change - Other -Down Ramp
Nocturnal Stabilization - DownRamp
Low Level Jet - Down Ramp
Down Ramps Up Ramps
Ramp events have different meteorological causes and thus the optimal parameters for tracking and predicting them are different
Summary• State-of-the-art forecasts are produced from a combination of physics-based and
statistical models– ERCOT forecasts have been generated from an early version of the forecast system that has
minimal reliance on WGR data– The ERCOT forecast system will soon be expanded to include an ensemble of NWP models
and MOS methods which have heavier reliance on WGR data
• Thus far, WGR and system data issues have significantly limited the perceived and actual performance of the ERCOT forecasts
• ERCOT system-wide day-ahead forecasts have lower MAPE than in the Alberta Pilot Project but higher than in California and New York
• Cases with high system-wide error have often been found to be associated with large errors in government-run model predictions or situations of high uncertainty (multi-model ensembles should help in these cases)
• A considerable amount of ERCOT-focused forecasting R&D is in progress to improve the forecasts beyond the expected gains from better WGR data and the expanded forecast system