united states of america before the federal energy ... · federal energy regulatory commission )...

of 18

UNITED STATES OF AMERICA

BEFORE THE

FEDERAL ENERGY REGULATORY COMMISSION

)

PJM Interconnection, LLC ) Docket No. ER14-2940-000

)

RESPONSE OF DR. SAMUEL A. NEWELL AND DR. KATHLEEN SPEES

ON BEHALF OF PJM INTERCONNECTION, LLC

REGARDING VARIABLE RESOURCE REQUIREMENT CURVE

Our names are Dr. Samuel A. Newell and Dr. Kathleen Spees. We are employed by

The Brattle Group as Principal and Senior Associate, respectively. We submit this affidavit

on behalf of PJM Interconnection, LLC (PJM) to respond to the comments and protests

submitted in this docket that relate to our independent assessment of PJM’s Variable

Resource Requirement (VRR) Curve for procuring capacity in its Reliability Pricing Model

(RPM) capacity market. On September 25, 2014, we submitted to the Commission a full

report of our findings from that review, and an affidavit explaining how our analysis

informed PJM’s proposed revisions to the VRR Curve.

We respond here to a subset of the comments and protests that relate to the VRR

Curve shape and to our analyses of the likely reliability and price volatility performance of

that curve. The majority of these comments and protests were submitted by: (a) the PJM

Load Group; (b) Mr. James F. Wilson, on behalf of the PJM Load Group; and (c) the

Maryland Public Service Commission (PSC). We also respond to a subset of the comments

submitted by the North Carolina Electric Membership Commission and the PSEG

Companies.

We first summarize our responses in introduction, and then provide a more detailed

set of responses to the individual points made by the protesters.

of 18

TABLE OF CONTENTS

I. Summary ..........................................................................................................................3

II. Protesters Incorrectly Claim We Evaluated VRR Curves Against a “New”

Reliability Standard ........................................................................................................3

III. Our Probabilistic Simulations Reflect Realistic Variations in Supply, Demand,

and Cleared Quantities ...................................................................................................5

A. Our Updated Simulation Model Provides a Realistic and Empirically-

Grounded Characterization of RPM .......................................................................6

B. Supply, Demand, and Net Supply Variations Are Realistic and Grounded in

Historical Observation ............................................................................................7

C. Modeled Supply Curves are Consistent with Historical Offer Curves .................10

D. Calibrating Entry Until Expected Average Prices Equal Net CONE is

Appropriate ...........................................................................................................10

E. Accounting for the Relationship between Net Energy Revenues and Reserve

Margins Would Have a Negligible Impact on Study Results ...............................11

F. Accounting for Short-Term Supply and Demand Effects Might Increase

Rather than Decrease Estimated Reliability Risks ...............................................12

IV. Protesters’ Critiques Based on Comparisons to Past Reviews and Historical

Data Are Flawed ............................................................................................................13

A. Protesters Misinterpreted Our Findings on the VRR Curve from Prior

Triennial Reviews .................................................................................................13

B. Historical Excess Reserve Margins Cannot be Interpreted to Mean that the

Current VRR Curve Will Maintain Resource Adequacy in the Future ................14

V. The Proposed Convex VRR Curve Shape Reflects a Reasonable Balance among

Competing Objectives ...................................................................................................15

VI. Protesters’ Concerns about Customer Cost Implications Are Overstated ..............17

A. Protestors Substantially Over-Estimate the Customer Cost Implications of

PJM’s Proposed Curve. ........................................................................................17

B. Imposing a “Parent Minimum” Net CONE will Not Result in Excess

Payments to Suppliers and Will Protect against Localized Reliability

Concerns ...............................................................................................................18

of 18

I. SUMMARY

Protesters claim that we evaluated VRR Curves against a “new” reliability standard.

However, the standard we evaluated against was PJM’s long-standing 0.1 Loss of Load

Expectation (LOLE) reliability objective. Our understanding of that standard, which we

confirmed with PJM staff, recognizes that the capacity market will produce reserve margins

that are uncertain and will vary over time. For the distribution of possible reserve margins to

meet the reliability standard in expectation, the average reserve margin will have to be

somewhat above the installed reserve margin at which LOLE equals 0.1.

Other concerns raised by protesters relate to the simulation analyses we conducted to

evaluate VRR Curve performance. As we explain, our model incorporates realistic variations

in supply and demand based on empirical evidence from PJM’s first 10 Base Residual

Auctions (BRAs), and it simulates PJM’s actual auction clearing mechanics. The model does

not account for the relationship between energy margins and reserve margins, but doing so

would have a minor impact on the results. It also does not account for short-term supply and

demand changes between the base auction and the delivery year, which is outside the scope

of the Third Triennial Review. However, contrary to the protesters’ assertions, fully

accounting for short-term market dynamics would not clearly decrease identified reliability

risks, and it could even increase them (assuming PJM’s load forecast is unbiased but subject

to error and that the Short-Term Resource Procurement Target is in place, among other

factors).

Protesters dispute our finding that the current VRR Curve is likely to fall short of

PJM’s reliability objectives, claiming it is inconsistent with past auction results and with our

past reviews of RPM. However, as we explain, their assertions misinterpret historical data

and our past reports. Historical surpluses and low-cost supplies will eventually be exhausted.

Our past reports did in fact identify concerns and uncertainties that we have now been able to

address more fully with the benefit of three additional years of experience with auction

performance.

Protesters also object to the convex VRR Curve shape that we recommended to PJM

and that PJM has proposed to adopt along with a 1% rightward shift. We explain how all

curves involve tradeoffs, and the convex shape reflects a reasonable balance among

competing objectives. It has the advantage over the current VRR Curve in that it sends a

stronger price signal as reserves become short, when reliability consequences rise

increasingly steeply.

Finally, protesters claimed that PJM’s proposed revisions would be very costly. We

explain below why their estimates are substantially overstated, and why imposing a “Parent

Minimum” Net CONE will protect against local reliability shortfalls without resulting in

excess payments.

II. PROTESTERS INCORRECTLY CLAIM WE EVALUATED VRR CURVES

AGAINST A “NEW” RELIABILITY STANDARD

Maryland PSC protests that our analysis assumes a new, enhanced reliability standard

(at page 3). The PJM Load Group similarly claims we applied a new reliability standard

of 18

based on average LOLE instead of average reserve margin. Their witness, Mr. James

Wilson, further notes that our simulation analysis shows that the current VRR Curve would

deliver average reserve margins above the target reserve margin (at ¶12), i.e., meeting the

“old” reserve margin objective even if it does not meet the “new” 0.1 LOLE objective.

Our response is that we did not apply a “new” reliability objective. The objective we

observed was to meet the long-standing 0.1 LOLE standard, and PJM staff confirmed our

understanding of that standard. Our analysis recognizes that 1-in-10 LOLE is a statistical

target that can be realized only on a long-term average basis as weather and generation

availability vary. In the PJM Region, target reserve margins are set to achieve the long-

standing 1-in-10 LOLE, but as PJM has a market-based resource adequacy construct, actual

reserve margins will vary just like the other random variables, so it makes sense to treat that

reserve margin uncertainty in our probabilistic analysis similarly to the other random

variables. Moreover, because (as discussed below) falling below the target reserve margin

has an outsized adverse effect on the average LOLE, meeting the reserve margin on average

will not result in meeting the LOLE standard on average. Instead, the population of reserve

margins that RPM aims to achieve should be high enough so that the expected frequency of

losing load equals the 0.1 LOLE standard, recognizing the effects of all the random variables.

In our Third Triennial Review of PJM’s Variable Resource Requirement Curve

(“Third Triennial Review”), we were able to incorporate the effect of reserve margin

variations on LOLE. Empirical data available from the first 10 auctions informed the

distribution of supply/demand variations, which we implemented in our simulation model to

find a realistic distribution of reserve margin outcomes. We were then able to translate this

distribution into LOLE impacts using the results of PJM’s most recent LOLE study.

Importantly, LOLE is a non-linear function of reserve margins. LOLE improves at a

diminishing rate at reserve margins above the target, and it worsens at accelerating rates at

reserve margins below the target. As a result, a wide distribution of reserve margins has a

higher expected LOLE than a narrow distribution with the same average reserve margin.

Consider one simple example where the target reserve margin is 15.6% and there are only

two years in the sample.1 If both years realize a reserve margin of 15.6%, then the average

reserve margin will equal the target, and the average LOLE will be 0.100, meeting the

desired 1-in-10. But if the two years have a 9.6% reserve margin and a 21.6% reserve

margin, while they will still meet, on average, the target reserve margin of 15.6%, the

expected LOLE will be an unacceptable 0.703 or seven events in ten years. With that wider

distribution, the VRR Curve would have to move further to the right to move the population

of reserve margin outcomes high enough so that the expected LOLE is 0.1.

Indeed, we tuned the Convex VRR Curve so that the average LOLE across all draws

would be 0.1 under our base case modeling assumptions. This means that the long-term

average LOLE would be 0.1, even though the average reserve margin would exceed PJM’s

target Installed Reserve Margin (IRM) by 0.7%. PJM then recommended right-shifting the

curve by 1% to reduce the frequency of low-reliability events (below 0.2 or 1-in-5 LOLE)

that we estimated would occur 13% of the time under base case assumptions, and to protect

1 The analysis in our Third Triennial Review was based on parameters pertaining to the 2016/17

delivery year. For consistency, we use here the 2016/17 IRM of 15.6% as a reference point.

of 18

against supply/demand conditions being even more challenging than our base case

assumptions.

PSEG supports the application of 0.1 LOLE as a reliability standard, but argues that it

is not appropriate to meet that standard on average. Instead, PSEG contends that PJM is

required by NERC reliability standards to meet the 0.1 LOLE every year. Yet PSEG seems

to concede that the model need not meet a 0.1 LOLE every year. They propose that the

LOLE fall below 0.1 LOLE at times, just not as often as with the Brattle Recommended

Curve or PJM Recommended Curve. The specific choice of a reliability standard is a policy

decision for PJM and the Commission, but we offer two observations. First, the VRR Curve

analyses that we have performed for PJM in the past considered how well a given curve

satisfied the target reserve margin on average across all of the simulations; all that has

changed in that regard for this review is that we directly measure the average LOLE in

addition to the average reserve margin. Second, PSEG offers an extreme example of sub-par

reliability which they contend our average LOLE approach would accept, but their example

is incorrect. PSEG suggests that the 1-in-10 average LOLE would be met by five years at 1-

in-1 LOLE (i.e., one event expected every year) offset by one year of 1-in-60 LOLE (i.e., one

event expected in sixty years). That sample, however, would fall far short of the target

LOLE and would be deemed unacceptable under the criteria that governed our analysis. The

correct average of 1/1, 1/1, 1/1, 1/1, 1/1, and 1/60 is 0.84, or over eight events expected in ten

years.

III. OUR PROBABILISTIC SIMULATIONS REFLECT REALISTIC VARIATIONS

IN SUPPLY, DEMAND, AND CLEARED QUANTITIES

Some of the filed comments addressed the simulation modeling that we have

conducted in order to probabilistically characterize the distribution of price, quantity, and

reliability results that might materialize under each demand curve shape. The most extensive

comments relating to this simulation modeling were contained in the Wilson Affidavit

submitted on behalf of the PJM Load Group, and so the majority of our response addresses

their critique of our modeling approach. However, we also respond to the more limited set of

comments submitted by the Maryland PSC.

Mr. Wilson articulates a number of criticisms of the modeling framework that we

have employed to evaluate PJM’s VRR Curve (See Wilson, Section V). While we

acknowledge that our simulation analyses must be interpreted in light of the underlying

assumptions, uncertainties, and limitations, we believe that our approach and our results

provide a realistic characterization of likely VRR Curve performance.

We respond to Mr. Wilson’s primary criticisms by describing how: (a) our model

provides a realistic and empirically-grounded characterization of RPM; (b) the distribution of

supply and demand variations that we model are consistent with historical observation; (c)

the supply curves that we model are consistent with historically observed offers in the

market; (d) our calibration of entry such that expected average prices equal Net CONE is an

appropriate way to model entry; (e) accounting for the interactions with energy and ancillary

service markets would have negligible effect on our results; and (f) accounting for short-term

auctions, while outside the scope of the Third Triennial Review, would have a complex effect

and may increase (rather than decrease) estimated reliability risks.

of 18

A. Our Updated Simulation Model Provides a Realistic and Empirically-Grounded

Characterization of RPM

Mr. Wilson states that the probabilistic simulation model that we have used to estimate

the likely distribution of price, quantity, and reliability outcomes under PJM’s VRR Curve is

“badly flawed” and therefore uninformative. (See Wilson, Section V) Mr. Wilson further

states that the model developed by Dr. Benjamin Hobbs that has previously been used for

evaluating the VRR Curve (while also flawed, in Mr. Wilson’s opinion) does a better job of

representing the market.

We disagree with Mr. Wilson’s critique of our model. We take the opposite view that

our modeling approach provides a realistic description of the distribution of outcomes that

are likely to materialize under different demand curves.

The Hobbs model was developed at the outset of RPM before there was any experience

with forward capacity markets, and so Dr. Hobbs necessarily relied on a relatively stylized

theoretical representation of how the market might function. Although stylized, the Hobbs

model provides directional and conceptual insights into the potential performance of the

VRR Curve, which we and Dr. Hobbs have employed to evaluate various aspects of the VRR

Curve performance in prior Triennial Reviews and for other purposes. However, we, Dr.

Hobbs, and PJM have always interpreted the results of the Hobbs model in the context of

understanding the model’s limitations. In important respects, that model’s results are

sensitive to input parameters that are not directly measurable in the real world. For example,

the Hobbs model incorporates a reasonable expectation that suppliers are risk averse and that

historical years’ revenues influence suppliers’ willingness to enter the market; however, it is

not possible for us to empirically measure these influences. The Hobbs model is also limited

to only one market area and one type of generation technology, and it does not include

supply variations other than those driven by new entry timing, among other simplifications.

The strength of our updated modeling approach is that we have leveraged and

incorporated a decade of experience with BRA auctions to more fully understand how PJM’s

three-year forward auction operates in reality. This depth of information and market

experience was not available at the time that the Hobbs model was developed. We now have

sufficient information on how the market functions to measure and support every input into

the model based on historically observable data, and we have documented how each model

component is supported by market evidence in our Third Triennial Review and briefly

summarize that information in Table 1. We have also adapted the model so that it can now

evaluate locational auction clearing mechanics and calculate LOLE metrics.

of 18

Table 1 Summary of Probabilistic Simulation Model Components and Empirical Basis

Model Component Approach and Empirical Support

Supply Offer Curves Used offer curve shapes from historical Base Residual Auctions

Supply-Demand Fluctuations Characterized variations in supply, demand, transmission parameters, and administrative Net CONE based on actual realized variations in these same parameters from historical data

Accounted for inter-area correlations and time trends

Calibrated net supply (supply minus demand) distributions to be consistent with historical observation

RPM Auction Clearing Mechanics Cleared supply and demand through a locational clearing engine that replicates BRA clearing prices and quantities

Reliability Outcomes Calculated LOLE in each Monte Carlo draw based on cleared quantities and the LOLE curve estimated in PJM’s reliability modeling

Our overall response to Mr. Wilson’s critique is that our modeling approach does in fact

realistically characterize the primary drivers and results that are necessary to evaluate the

reliability and price volatility performance of the VRR Curve, and therefore does provide

substantial information and insight from which to refine the curve’s design. However, we

also acknowledge that alternative modeling approaches could also address the same

questions, and that the results of this model must be interpreted while understanding its

limitations. We provide a more detailed discussion in response to each of Mr. Wilson’s

primary criticisms in the following sections.

B. Supply, Demand, and Net Supply Variations Are Realistic and Grounded in

Historical Observation

Mr. Wilson and the Maryland PSC assert that our estimated variations in supply and

demand are unrealistically large. We disagree with that view, and provide here additional

explanation regarding how we have estimated the likely size of these variations based on

historical data.

We do acknowledge that there is uncertainty in these estimates, and that in the future

these year-to-year variations could be different from the past. In fact, different observers

might anticipate smaller or larger variations in the future depending on their expectations.

For example, Mr. Wilson and the Maryland PSC note several examples of large historical

shocks to supply and demand that are unlikely to occur again in the future, including the

recession, the large coincident retirements occurring under the Mercury and Air Toxics

Standard (MATS), and the RPM design change that incorporated a large quantity of demand

response. (Maryland PSC, p.3-5; Wilson, Section V.C) The protesters argue that by

incorporating these historical events we have overstated our estimates of the likely variations.

However, there are also countervailing arguments that future variations will be higher as a

number of current regulatory and market design uncertainties resolve, and new challenges

of 18

emerge. For example, over the next few years we may see substantial variations in supply

offer quantities associated with changes to demand response participation rules, carbon limits

on existing resources, and PJM’s Capacity Performance proposal. One could argue that these

factors could result in higher variations in the future than we have observed in the past. We

also do not view it as appropriate to exclude the possibility of future recession events from

our analysis. Rather than speculating whether these variations will increase or decrease in

magnitude in future years, we have opted instead to reflect only what is observable from

historical data. However, we have also provided sensitivity analyses to show the

implications if one were to take a different view.

The Maryland PSC expresses concern that we did not account for what they refer to

as a “positive ‘anti-shock’ [in demand] or unexpected load reductions due to load over-

forecast.” (Maryland PSC, p. 5) Their concern is unwarranted. We expressly considered

RPM’s history of changes in the load forecast prior to the Delivery Year, and adopted an

approach that is designed to correct for the effects of any load forecast bias if one exists.2

Mr. Wilson also expressed a concern that modeling supply and demand variations as

functionally independent uncertainties would result in overstating the realized variations in

net demand. (Wilson, ¶ 45-49, 55-61) However, we did consider the possibility that supply

and demand fluctuations are positively correlated as Mr. Wilson suggests, and calibrated our

net supply (supply minus demand) fluctuations such that the net fluctuations in the model

match historically observed net supply fluctuations.

Table 2 (taken from p. 43 of our Third Triennial Review) shows the results of our

calibration on this very issue. We reviewed historical data for the 2009/10 through 2016/17

Delivery Years on fluctuations in supply minus demand in RPM in two ways. The first panel

of Table 2 shows the standard deviation of historic fluctuations in supply minus demand in

RPM. The second panel in Table 2 considers whether there is a trend in changes in net

supply over time, and measures the deviations from that trend. Based on this close

consideration of how supply and demand have changed together in RPM, we incorporated

net supply variations in our modeling (as shown in the third panel of Table 2) that reflect a

standard deviation between the two historic measures of net supply fluctuations. We applied

realistic magnitudes for these variations in net supply modeled in every LDA.

2 In particular, we adjusted for a possible load forecast bias by: (1) calculating the difference between

the four-year ahead and three-year ahead load forecast from the PJM load forecasts over 2007 through

2014 (the average of these deltas would reflect a bias in the load forecast, but this number does not affect our analysis), and (2) calculating the standard deviation in these deltas, which results in the

0.8% standard deviation in load forecasts that we assume in our study. See our Third Triennial

Review, Appendix A2.

of 18

Table 2 Net Supply minus Demand Variations

Sources and Notes: All values calculated over 2009/10 through 2016/17 delivery years, where data were available, see Third Triennial Review,

Section IV.C and Appendix A. [1]: Historical standard deviations calculated from annual BRA Supply Offers. [2]: Historical standard deviations calculated from CETL values in the PJM Planning Parameters. [3]: Historical standard deviations from Reliability Requirement values in the PJM Planning Parameters. [4]: All standard deviations are calculated based on Net Supply, where Net Supply equals [1] + [2] – [3]. [5] – [8]: Equal to columns [1] – [4], divided by the LDA’s 2016/17 Reliability Requirement.

Although the supply-demand variations we modeled are reasonably representative of

historical variations, we also provided sensitivity analyses of greater or lesser variations in

the future. We found that 33% greater variation increased the frequency of load shedding

more than twice as much as 33% lesser variation reduced it. These findings are shown in

Table 15 of our Third Triennial Review.

Standard Deviation Standard Deviation as % of 2016/17 LDA Size

LDA Supply CETL Reliability

Requirement

Net

Supply

Supply CETL Reliability

Requirement

Net

Supply

(MW) (MW) (MW) (MW) (%) (%) (%) (%)

[1] [2] [3] [4] [5] [6] [7] [8]

Historical Absolute Value (2009/10 - 2016/17)

RTO 20,040 n/a 14,783 5,894 12.1% n/a 8.9% 3.5%

MAAC 3,549 811 931 3,480 4.9% 1.1% 1.3% 4.8%

EMAAC 1,900 721 645 2,451 4.8% 1.8% 1.6% 6.2%

SWMAAC 907 910 335 1,652 5.2% 5.3% 1.9% 9.5%

PS 820 352 288 832 6.4% 2.7% 2.2% 6.5%

PS NORTH 534 252 101 585 8.3% 3.9% 1.6% 9.1%

DPL SOUTH 112 206 57 282 3.5% 6.5% 1.8% 8.9%

PEPCO 423 1,060 233 1,673 4.7% 11.8% 2.6% 18.6%

ATSI 717 1,742 38 2,421 4.4% 10.7% 0.2% 14.9%

ATSI-Cleveland n/a n/a n/a n/a n/a n/a n/a n/a

Historical Deviation from Trend (2009/10 - 2016/17)

RTO 4,816 n/a 4,850 2,147 2.9% n/a 2.9% 1.3%

MAAC 1,229 808 792 2,208 1.7% 1.1% 1.1% 3.1%

EMAAC 1,102 717 578 2,091 2.8% 1.8% 1.5% 5.3%

SWMAAC 409 378 283 792 2.4% 2.2% 1.6% 4.6%

PS 657 329 96 759 5.1% 2.6% 0.7% 5.9%

PS NORTH 338 222 84 401 5.3% 3.4% 1.3% 6.2%

DPL SOUTH 70 172 48 193 2.2% 5.4% 1.5% 6.1%

PEPCO 234 236 166 585 2.6% 2.6% 1.8% 6.5%

ATSI 557 n/a n/a n/a 3.4% n/a n/a n/a

ATSI-Cleveland 473 n/a n/a n/a 7.7% n/a n/a n/a

Simulation Analysis

RTO 4,054 n/a 1,499 4,277 2.4% n/a 0.9% 2.6%

MAAC 2,767 794 794 2,984 3.8% 1.1% 1.1% 4.1%

EMAAC 1,591 1,090 492 1,954 4.0% 2.7% 1.2% 4.9%

SWMAAC 644 1,074 279 1,214 3.7% 6.2% 1.6% 7.0%

PS 363 804 215 908 2.8% 6.2% 1.7% 7.1%

PS NORTH 226 359 131 446 3.5% 5.6% 2.0% 6.9%

DPL SOUTH 97 232 76 259 3.1% 7.4% 2.4% 8.2%

PEPCO 328 837 220 935 3.6% 9.3% 2.4% 10.4%

ATSI 663 963 259 1,186 4.1% 5.9% 1.6% 7.3%

ATSI-Cleveland 157 641 164 699 2.5% 10.4% 2.7% 11.3%

of 18

C. Modeled Supply Curves are Consistent with Historical Offer Curves

Mr. Wilson states that the supply curves implemented in our model are too steep,

which results in overstating price volatility and reliability risks. (Wilson, ¶ 13, 62-68) We

respond to three points that Mr. Wilson makes to support this position. First, Mr. Wilson

states that the supply curves we implemented in our modeling are different from the

smoothed shapes that we provided in graphical form in the Third Triennial Review. This is

not the case. The curves that we presented are identical to those that we implemented in

modeling.

Second, Mr. Wilson notes that the PJM supply curves have become more elastic over

time, an observation that we also noted in our Second Performance Assessment of PJM’s

Reliability Pricing Model (Second Triennial Review). While we concur with this

observation to some extent, we disagree that this would suggest any change to the supply

curves we reflect in our analysis because: (a) this increase in supply elasticity is already

reflected in our supply curves, which match the shapes of actual historical BRA supply

curves; (b) we have excluded supply curves from the earliest PJM auctions that were

conducted less than three years forward (with the extension of the forward period to three

years contributing substantially to supply elasticity); (c) a substantial portion of the increased

supply elasticity that we described in the Second Triennial Review was associated with high

offer prices from existing generators that needed to comply with MATS, with the most recent

auction supply curve reverting to a steeper shape in the absence of any similarly high-impact

environmental regulations affecting the auction; and (d) given the recent contraction of

demand response and future uncertainty around demand response participation in future

RPM auctions, one of the drivers of increased supply elasticity has diminished.

Third, Mr. Wilson asserts that it would be more appropriate to assume more supply

elasticity than has been observed historically. Mr. Wilson takes the view that the absence of

offers in the high-price region of the supply curve was a function of market conditions, and

that suppliers who might have offered at higher prices consistent with Net CONE have not

yet entered the market “as there was no chance such offers would clear.” (Wilson, ¶ 63) We

do not agree or disagree with Mr. Wilson on this point, but we do believe that the statement

is speculative and therefore provides an unreliable basis for evaluating PJM’s VRR Curve.

We believe that our approach of incorporating historical evidence about how suppliers have

offered over a decade of experience is more robust than Mr. Wilson’s proposed approach of

incorporating a view of how supplier offer behavior might evolve in the future.

D. Calibrating Entry Until Expected Average Prices Equal Net CONE is

Appropriate

Mr. Wilson asserts that our Monte Carlo simulation model does not properly

represent market entry because the model is “not dynamic,” lacking a chronological

structure. (Wilson, ¶ 49) He is mechanically correct about the fact that our model represents

the distribution of outcomes that can be expected, without expressing the yearly sequence of

outcomes, such as a prolonged period of excess or possible cycles of highs followed by lows

that he believes will occur. However, he is incorrect that our model structure does not

properly model entry or the effects year-to-year dynamics might have on VRR Curve

performance.

of 18

To the extent that year-to-year supply responses might occur and might limit the

magnitude of excursions from the average reserve margin, we already account for such

effects by calibrating our net supply variations to historical market data, as described in

Section III.B above. Apart from that effect, the lack of sequencing is immaterial to properly

estimating the long-term average performance of different VRR Curves.

Contrary to Mr. Wilson’s claim that our model has “no representation of entry

decision-making” and “only the fixed supply curve” (Wilson, ¶ 49), our model does in fact

adjust the quantity of entry based on expected prices, as an investor would. Investors would

analyze supply and demand conditions (including the VRR Curve) and condition their entry

on expecting to recover their capital and fixed costs over the life of their project. We

implement this standard economic assumption in our model by tuning the amount of entry

until the expected average capacity price equals Net CONE. This logic is described further

in Section IV of our Third Triennial Review report and on page 5 of our affidavit

accompanying PJM’s September 25 filing.

E. Accounting for the Relationship between Net Energy Revenues and Reserve

Margins Would Have a Negligible Impact on Study Results

Mr. Wilson correctly points out that we have conducted our simulation models under

the simplifying assumption that suppliers’ Net CONE is independent of the average system

reserve margin. (Wilson, ¶ 13 and Section VIII.C) Mr. Wilson also correctly points out that

Net CONE will actually increase as the average reserve margin increases, because expected

net revenues out of the energy and ancillary services markets will decline.

However, we disagree with Mr. Wilson’s statement that our simplifying assumption

of a constant Net CONE is a “fatal flaw” or even a significant flaw. Adjusting our model to

account for this relationship would have a negligible impact on our study results, because the

effects are small and largely irrelevant.

First, energy prices, net energy revenues, and Net CONE would change by only a

small amount over the 1.3% difference in average reserve margins that PJM’s Recommended

Curve will procure compared to the current VRR Curve.

Second, even if the differences in energy prices, net energy revenues, and Net CONE

were significant, accounting for them would not change our evaluation of the reliability

performance of alternative VRR Curves. In each VRR Curve scenario we analyze, we

assume Net CONE would be set accurately. Our estimated reliability performance and price

volatility of each curve would therefore be largely unaffected by slight differences in the Net

CONE value itself.

Third, even if the differences in energy prices, net energy revenues, and Net CONE

were significant, accounting for them would have a largely offsetting impact on realized

energy plus capacity procurement costs. Customers would pay slightly more (or less) for

capacity and slightly less (or more) for energy, but the all-in price would still be

approximately the same, sufficient to provide new entrants with total net revenues equal to

CONE.

of 18

F. Accounting for Short-Term Supply and Demand Effects Might Increase Rather

than Decrease Estimated Reliability Risks

Mr. Wilson and the Maryland PSC both assert that our simulation modeling

overstates reliability risks because: (1) we assume that the PJM load forecast is unbiased

(although subject to uncertainty); and (2) we model only the three-year forward BRA and do

not model the short-term incremental auctions (IAs).

Regarding load forecast error, both Maryland PSC and Mr. Wilson point to a

“systematic over statement in PJM’s load forecast” to explain why realized reserve margins

would be higher than we have estimated in our simulation modeling. (Maryland PSC, p. 3)

We agree that a systematic over-forecast bias would increase realized reserve margins and

therefore improve reliability relative to what we have estimated. However, we disagree that

we should have incorporated any assumed forecast bias into our study, as it would be

inappropriate to compensate for any known bias in the peak load forecast by adjusting the

parameters, shape, or placement of the VRR Curve. Instead, if such a bias exists, it would be

more appropriate to address that concern by refining the peak load forecast itself (a topic

outside the scope of the Third Triennial Review).

Regarding the impacts of the short-term incremental auctions, Mr. Wilson and the

Maryland PSC both state that by modeling only the three-year forward auctions we have

over-estimated the likely LOLE that would materialize under the current VRR Curve. First,

we agree that design and performance of the short-term incremental auctions are an

important aspect of the overall performance of RPM, but clarify that analyzing the

incremental auctions is outside the scope of this Third Triennial Review.

Moreover, we disagree with the assertions of Mr. Wilson and the Maryland PSC that

accounting for short-term supply and demand effects would necessarily reduce the realized

LOLE. Mr. Wilson states that substantial quantities of resources have been available in the

incremental auctions. Therefore, he posits, if the BRA were to clear short, it would signal

additional supplies to come forward on a short-term basis to fill the need. (Wilson ¶78–80)

He posits that accounting for incremental auctions in our simulation modeling would have

reduced the expected frequency of low reliability events and would have increased the

expected realized LOLE.

However, Mr. Wilson fails to acknowledge a large number of countervailing factors

that would tend to increase LOLE if one were to fully account for all of the uncertainties and

market dynamics that come into play between the three-year-ahead BRA and the delivery

year. These factors that would tend to increase realized LOLE include: (a) three years of

load forecast uncertainty, which is partially but not entirely accounted for in PJM’s LOLE

studies; (b) the 2.5% Short-Term Resource Procurement Target, which assumes that at least

2.5% of needed supplies will be available for procurement on a short-term basis (which has

always been the case to date under excess supply conditions, but which might not be true in a

future scenario when the BRA clears short); and (c) the inability of many of the resources

that can offer into RPM on a three-year forward basis to offer into RPM on a short-term basis

to cover reliability needs, particularly new generation resources that need the full three years

to enter service.

Mr. Wilson’s hypothesis that BRA shortfalls would prompt increased supplies in the

Incremental Auctions is, at this time, speculative. The consistent pattern seen in RPM to date

of 18

is that supply options diminish as the Delivery Year approaches. As shown in Table 3, all

Incremental Auctions conducted for the 2012/13 through 2016/17 Delivery Years exhibit a

pattern of total resource offers steadily decreasing between the three-year forward BRA and

the final Incremental Auction. This can be seen in the far right column, which shows the

total increase (decrease) in supply offers attracted into the Incremental Auctions as a delta

relative to the BRA uncleared supply. The fact that this number is increasingly negative over

the incremental auctions shows a general trend of decreasing supply options as the Delivery

Year approaches. While we cannot say with certainty that Mr. Wilson is incorrect that a

shortage in the BRA would spur additional supply-side activity in the Incremental Auctions,

we can say that Mr. Wilson’s assertion is speculative and that no such activity has been

observed to date.

In sum, we believe accounting for short-term market dynamics would not (as Mr.

Wilson and the Maryland PSC suggest) clearly decrease identified reliability risks, and could

even increase those risks.

Table 3 Incremental Auction Clearing Quantities

Showing a Decline in Supply Offers as the Forward Period Drops from Three to Zero Years Forward

Sources and Notes: PJM Incremental Auction and Base Residual Auction Reports.

IV. PROTESTERS’ CRITIQUES BASED ON COMPARISONS TO PAST

REVIEWS AND HISTORICAL DATA ARE FLAWED

A. Protesters Misinterpreted Our Findings on the VRR Curve from Prior Triennial

Reviews

The PJM Load Group asserts that we concluded in past Triennial Reviews that the

VRR Curve was sufficient (at page 5). Maryland PSC similarly claims that our finding that

the current curve is likely to fall short of the 0.1 LOLE standard contradicts our prior

reviews.

While it is true that our prior reviews found that RPM had performed well to date, it

is also true that we identified a number of concerns that needed to be addressed to maintain

Year Auction Price Participant Offers Participant Cleared Participant Unleared Supply Increase

Sell Offers Buy Bids Sell Offers Buy Bids Net Sell Offers Buy Bids (Decrease) in IAs

($/MW-d) (MW) (MW) (MW) (MW) (MW) (MW) (MW) ($/MW-d)

2012/13 BRA $16 145,373 n/a 136,144 n/a 136,144 9,230 n/a n/a

1st IA $16 7,086 9,339 1,689 1,749 (60) 5,397 7,590 (2,144)

2nd IA $13 6,448 11,560 838 3,215 (2,377) 5,610 8,345 (2,842)

3rd IA $3 5,569 7,459 2,404 4,383 (1,979) 3,166 3,076 (6,098)

2013/14 BRA $28 160,898 n/a 152,743 n/a 152,743 8,155 n/a n/a

1st IA $20 7,471 16,446 2,387 4,882 (2,495) 5,084 11,564 (684)

2nd IA $7 6,073 16,386 1,997 5,599 (3,602) 4,076 10,787 (4,577)

3rd IA $4 5,526 6,372 2,703 3,168 (465) 2,823 3,203 (8,726)

2014/15 BRA $126 160,486 n/a 149,975 n/a 149,975 10,511 n/a n/a

1st IA $6 11,126 13,231 4,240 6,850 (2,610) 6,887 6,381 615

2nd IA $25 6,039 11,133 2,910 4,476 (1,567) 3,129 6,657 (7,082)

3rd IA $26 5,470 6,937 3,978 2,682 (1,296) 1,492 4,255 (9,218)

2015/16 BRA $136 178,588 n/a 164,561 n/a 164,561 14,027 n/a n/a

1st IA $43 6,773 21,305 4,172 5,987 (1,816) 2,602 15,317 (7,253)

2nd IA $136 2,960 10,624 1,780 2,693 (913) 1,180 7,931 (12,883)

2016/17 BRA $59 184,380 n/a 169,160 n/a 169,160 15,220 n/a n/a

1st IA $60 10,917 12,308 4,138 5,557 (1,419) 6,779 6,751 (4,304)

of 18

reliability going forward. We recommended applying a minimum value of at least 0.5 CONE

+ Net CONE to the VRR Curve’s price cap, calibrating E&AS offsets, modeling the

locational deliverability areas (LDAs) more proactively, modifying the 2.5% holdback,

strengthening resource verification, and creating exemptions from the minimum offer price

rule.

PJM has largely implemented most of those recommendations, but that did not

guarantee that RPM would meet its objectives forever. Even with our recommended changes

to the VRR Curve, our Second Triennial Review showed that the revised VRR Curve would

ameliorate most, but not all, of the reliability concerns.3 Reserve margins would vary widely,

averaging 0.5% below IRM, a result that was much better than without reforms but still

below IRM. (See Second Triennial Review, Table 21, Settlement Alternative 3. This

Alternative is the closest to what PJM’s eventually implemented, imposing a minimum value

of CONE to point “a” on the VRR Curve). Expected LOLE, which we did not calculate at

the time, would also be below target. Moreover, we qualified our simulation results, noting

that the model we adapted from Professor Benjamin Hobbs’s original model was “a stylized

representation of RPM and investment behavior and is based on significant simplifications.”

(Second Triennial Review, p. 101). It modeled only one market area and one type of

generation technology, and it did not include supply variations other than those driven by

new entry timing, among other simplifications.

In the Third Triennial Review, we have been able to develop a richer simulation

analysis that is informed by empirical data from the first 10 BRAs. Our model accounts for

realistic distributions of supply and demand fluctuations and realistic supply curve shapes. It

also addresses a number of other identified limitations in the Hobbs modeling, as discussed

in Section III above. Therefore, it is not surprising that our current study might have

identified further concerns that need to be addressed for RPM to meet its reliability

objectives going forward.

B. Historical Excess Reserve Margins Cannot be Interpreted to Mean that the

Current VRR Curve Will Maintain Resource Adequacy in the Future

The PJM Load Group protests revising the VRR Curve, noting that historical cleared

reserve margins were high, showing that the VRR Curve was sufficient. Their witness James

Wilson notes satisfactory performance of the first 11 auctions (Wilson, ¶12). Similarly,

NCEMC notes that RPM has never cleared below IRM.

PJM’s auctions to date have benefited from the market having started in 2007/08 with

a surplus of capacity and plentiful low-cost resources that RPM has been able to attract and

retain. Low-cost entrants included demand response, generation unit uprates, and imports.4

Going forward, there is no reason to expect continued growth of such resources. Imports are

limited by transmission constraints and have already decreased with revised rules that

enforce limits more strictly. Uprates may be limited by opportunities in the existing fleet.

3 “Probabilistic market simulations indicate that increasing the VRR curve cap to 0.5 × CONE above

point “b” would likely offset 80% of the performance deterioration associated with the use of

historical E&AS offsets.” (Second Triennial Review, page ix.) 4 See Second Triennial Review, Figure 10.

of 18

And demand response participation has already contracted and could contract further under

restrictions imposed by the ruling of the federal court of appeals in EPSA v. FERC on the

Commission’s jurisdiction over demand response. In that context, reserve margins can be

expected to tighten as load grows and generation retires, particularly under environmental

compliance pressures on the generation fleet.

Our analysis assumes long-run equilibrium conditions in which there is no longer a

surplus of capacity, but just enough capacity to maintain long-term average prices at Net

CONE. We do not assume prices and reserve margins remain constant; rather, we assume

they vary as supply and demand fluctuate. The supply-demand fluctuations in our

simulations are calibrated to the supply-demand shifts observed in the first 10 auctions—

carrying these distributions forward assumes that the first 10 auctions’ uneven quantities of

supply entry/exit, changes in load forecasts, and modification to transmission limits are

representative of the future. We find that, given such fluctuations and the assumption that

prices equal Net CONE on average in a long-term equilibrium, the current VRR Curve would

not support a LOLE of 0.1.

NCEMC objects to any change in the current VRR Curve because the “current RPM

design, including the Settlement VRR Curve, has encouraged an ‘unprecedented’ level of

investment in new resources despite the small degree of volatility RPM prices have shown

and despite the fact that the unconstrained region in PJM has never cleared at or near Net

CONE.” (NCEMC pp. 9-12) However, the new entry they cite raises questions not about the

shape of the VRR Curve, but rather about the level of the administrative Net CONE. We

have emphasized that estimating Net CONE accurately is very important, and we have aimed

to do so in our 2014 CONE Study, as well as through our evaluation of each component of

the Net CONE estimate in the Third Triennial Review and our recommendation to revise the

indexing method that sets CONE between periodic reviews. However, we caution against

too direct a comparison between individual entry offers and Net CONE estimates for the

same reasons we have cautioned against proposed “empirical Net CONE” mechanisms in the

past: actual offers seem to reflect a wide range of different bidding, hedging, and market-

timing strategies. They may reflect special situations that may not be replicable going

forward. By contrast, our CONE study reflects a generic new combustion turbine’s levelized

cost (which may differ from a bidding strategy but provides the right benchmark for the long-

term average prices needed to sustain entry). Therefore, NCEMC’s observations about new

entry costs do not rebut our showings of the need for changes to the shape of the VRR Curve.

V. THE PROPOSED CONVEX VRR CURVE SHAPE REFLECTS A

REASONABLE BALANCE AMONG COMPETING OBJECTIVES

The PJM Load Group and its witness, James Wilson, argue that the left part of PJM’s

proposed VRR Curve is too steep (raising concerns about volatility and market power) and

the right part is too flat (sometimes procuring significantly more than IRM).

Regarding concerns about volatility, our simulation analysis did in fact show that the

proposed curve exhibits higher volatility than the current VRR Curve, but only slightly. We

found a standard deviation of $107/MW-d between draws under the recommended curve

of 18

versus $95 under the current VRR Curve.5 The overall volatility is similar largely because,

although the left part of the recommended curve is steeper, the right part is less steep. The

slopes of the two sections are almost perfectly interchanged. The convex VRR Curve PJM is

proposing has a slope of -24 $/MW-d per 1% of IRM on the left section of the curve, and -13

$/MW-d on the right section, whereas the current VRR Curve has slopes of -13 $/MW-d on

the left and -20 $/MW-d on the right.6 As we explained in our Third Triennial Review, the

primary reason for making the left part steeper is so that price signals rise faster and

procurement becomes more aggressive as reserve margins get closer to unacceptably low

levels and loss-of-load probabilities climb increasingly rapidly.

As to concerns about market power, Mr. Wilson overlooks that the market monitor

has consistently deemed the market structurally uncompetitive, even at the relatively high

reserve margins experienced in RPM to date, and has consequently mitigated all units. We

therefore reasonably expect that offers would continue to be mitigated, negating much of the

concern about market power.

As to the curve’s procurement of excess capacity when supply is plentiful and

concerns that the marginal reliability value of capacity in the right-most part of the curve is

probably quite low, the curve was not designed to express marginal reliability value, but

rather to meet reliability requirements through a well-functioning market. The purpose of the

gradual foot of the curve is to prevent prices from collapsing so that there is enough money

and stable enough prices in the market to support investment and shift the population of

auction outcomes (as supply and demand fluctuate) far enough to the right to meet reliability

objectives. In that context, a single auction clearing outcome with high reserve margins

cannot be assessed in isolation.

No curve can eliminate volatility and susceptibility to market power, and no curve

can guarantee that sufficient capacity will be procured and no more. Greater price certainty

generally comes at the expense of quantity uncertainty (through flatter curves), just as greater

quantity certainty generally comes at the expense of higher price volatility (through steeper

curves). A demand curve can only aim to strike a reasonable balance among price- and

quantity-related objectives. We have shown that the convex curve shape strikes that

reasonable balance. It performs well in our simulation analyses, and it has the advantage that

it increases the price signal faster as reserve margins become tight in the low reserve margin

range where LOLE deteriorates at an increasing rate.

5 See Third Triennial Review, Table 14.

6 The steeper left section of PJM’s proposed curve is from IRM – 0.2% to IRM + 2.9%, and the flatter

right section is from IRM + 2.9% to IRM + 8.8%. The current VRR Curve’s left section is between

IRM – 3% and IRM + 1%, and the right section is from IRM + 1% to IRM + 5%.

of 18

VI. PROTESTERS’ CONCERNS ABOUT CUSTOMER COST IMPLICATIONS

ARE OVERSTATED

A. Protestors Substantially Over-Estimate the Customer Cost Implications of

PJM’s Proposed Curve.

The Maryland PSC asserts that the recommended curve will impose $1.5 billion

additional costs on consumers each year (Maryland PSC, p. 4). NCEMC refers to the same

alleged $1.5 billion cost increase, based on sensitivity scenarios of recent auctions PJM

provided at stakeholder request, substituting PJM’s recommended VRR Curve in place of the

VRR Curve actually used to clear those auctions. While we do not dispute the mechanics of

PJM’s backward-looking sensitivity analyses, we do not believe this type of analysis is very

informative for evaluating how a revised VRR Curve would perform in future auctions.

One-off sensitivity analyses can be informative to stakeholders, but must be understood as

simplified analyses that do not account for how supply offers might have expanded within a

given auction if suppliers knew the proposed VRR Curve would have been in place. Such

analyses similarly do not account for how supply would expand in the future (which is

exactly what the revised curve aims to achieve). Consequently, such analyses do not account

for the fact that increased supply would substantially moderate the price impacts suggested

by the retrospective, short-run sensitivity analyses that the protesters referenced.

The better approach, as we have attempted through our simulation analyses, is to

consider demand curve performance under a great variety of conditions within a historically

expected range and distribution, and to incorporate economic fundamentals, including the

expectation that prices on a long-term average basis should converge to true Net CONE

under any curve. Such a simulation analysis provides better information about expected

curve performance on reliability and cost metrics. Using this long-run equilibrium approach,

we found that PJM’s recommended curve is superior on reliability metrics to the current

VRR Curve (which falls short of the 0.1 LOLE standard), yet increases long-term average

procurement costs (relative to the current VRR Curve) by only 1.1%, or $216 million per

year.7

NCEMC’s protest also claimed, “PJM provides no information in its filing as to the

extent to which its proposal will increase costs as compared to the Settlement VRR Curve, or

how much the Brattle Recommended Curve would increase costs as compared to the

Settlement VRR Curve.” (NCEMC Protest, p.15). In fact, Table 14 of the 2014 VRR Report

does compare the simulated long-term average costs of all three of these curves. The “Brattle

Recommended Curve” procures a 0.3% higher reserve margin on average than the Settlement

Curve, increasing capacity procurement costs by 0.2% or $43 million per year. Compared to

the “Brattle Recommended Curve,” PJM’s recommended right-shift of the curve procures an

additional 1% higher reserve margin at an incremental cost of 0.9%, or $173 million more

per year.

7 See Third Triennial Review, Table 14.

of 18

B. Imposing a “Parent Minimum” Net CONE will Not Result in Excess Payments

to Suppliers and Will Protect against Localized Reliability Concerns

The PJM Load Group protests PJM’s proposal to require that Net CONE values used

in LDAs be no lower than the values used in the parent LDAs. They assert that this would

provide more than the “missing money” in the LDA, and that any non-price barriers to entry

do not warrant an extra price signal (Load Group Protest, p. 11). They also argue that this

safeguard is unnecessary due to multiple layers of conservatism, including PJM’s use of

level-nominal Net CONE.

However, the PJM Load Group never seems to engage the fundamental reasons we

recommended this change: sub-LDAs are more vulnerable to Net CONE estimation error due

to small sample size and idiosyncratic factors, as we have seen in the Southwest Mid-Atlantic

Area Council (SWMAAC) area, and underestimating Net CONE in an LDA may result in

reliability shortfalls. We recommended setting the LDA Net CONE at the parent level

because doing so would reduce the risk of reliability shortfalls without substantially

increasing costs in the event that true Net CONE is in fact lower in the LDA. If true Net

CONE is lower, developers should favor entering there, and the LDA would not “bind” in the

auctions. If the LDA does not bind, any overstatement of local Net CONE would have no

effect on auction outcomes. (See Third Triennial Review, pp. 24-26)

united states of america before the federal energy ... · federal energy regulatory commission )...

Documents