ma wehinger

8/6/2019 MA Wehinger

1/158

Master thesis

Agent-based modeling in electricity markets:

Introducing a new predictive agent learning

approach

Lukas A. Wehinger

PSL 1021

Department ETH: EEH - Power Systems LaboratoryDepartment CMU: ECE - Electrical and Computer Engineering

Advisors:Prof. Dr. Gabriela Hug (CMU)

Matthias David Galus (ETH)Prof. Dr. Goran Andersson (ETH)

Spring semester 2010November 1, 2010


2/158

i


3/158

Preface

This research project is the final assignment of the Master program Energy Scienceand Technology at the ETH Zurich. The project combines aspects from statistics,

control engineering, machine learning, micro economics and finance. This interdis-ciplinary project allowed me to apply my knowledge gained from my undergraduateand graduate studies at the ETH Zurich.

I owe many thanks to the Power Systems Laboratory at ETH Zurich and theElectrical and Computer Engineering department at Carnegie Mellon University forhosting the master thesis.

I am very grateful to Prof. Gabriela Hug and Matthias Galus for their supervi-sion, support and valuable advices in many regards in elaborating this interestingthesis. A special thanks also goes to Prof. G. Andersson from ETH Zurich, whoenabled to write the master thesis at Carnegie Mellon university in Pittsburgh,Pennsylvania. Besides the research I had the unique opportunity to experience adifferent culture, country and university. I will keep these experiences in good mem-ory. Last but not least I thank the University staff and all the students I met herewho helped me in many different aspects within this project.

Pittsburgh, September 28, 2010 Lukas Wehinger

ii


4/158

iii


5/158

Contents

Abstract vii

Symbols viii

1 Introduction 1

1.1 The European electricity market . . . . . . . . . . . . . . . . . . . . 11.1.1 Goals of the European Union . . . . . . . . . . . . . . . . . . 21.1.2 Implementation of the electricity market liberalization . . . . 2

1.2 Congestion management . . . . . . . . . . . . . . . . . . . . . . . . . 51.2.1 Capacity allocation methods . . . . . . . . . . . . . . . . . . 51.2.2 Implemented congestion management schemes and outlook . 6

1.3 Market-power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.3.1 Market-power in electricity generation . . . . . . . . . . . . . 81.3.2 Market power and congestions . . . . . . . . . . . . . . . . . 8

1.4 Contribution of the thesis . . . . . . . . . . . . . . . . . . . . . . . . 91.5 Structure of the work . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2 Agent-based modeling in electricity markets 11

2.1 Multi-Agent Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2 Benefits of agent-based modeling . . . . . . . . . . . . . . . . . . . . 122.3 When are agent-based models useful? . . . . . . . . . . . . . . . . . 122.4 Using agent-based models for electricity markets . . . . . . . . . . . 132.5 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3 Specification of the model 18

3.1 Market participants and input data . . . . . . . . . . . . . . . . . . . 183.1.1 Market participant definition . . . . . . . . . . . . . . . . . . 183.1.2 Data sources . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.2 Supply curves, market structure and market clearing . . . . . . . . . 213.2.1 Supply curves . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.2.2 The market clearing . . . . . . . . . . . . . . . . . . . . . . . 243.2.3 Time frame of the bidding process . . . . . . . . . . . . . . . 25

3.3 Cost of production and profit calculation . . . . . . . . . . . . . . . . 253.4 Simplifications of the model . . . . . . . . . . . . . . . . . . . . . . . 27

4 Learning-process of the agents: The methodology 28

4.1 Overall implemented structure . . . . . . . . . . . . . . . . . . . . . 304.2 The price predictor . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.2.1 The multi-factor regression model . . . . . . . . . . . . . . . 314.2.2 Exogenous factor predictions . . . . . . . . . . . . . . . . . . 35

4.2.3 Analyzing the price predictor . . . . . . . . . . . . . . . . . . 394.2.4 Using the price predictor for out-of-sample data . . . . . . . . 44

iv


6/158

4.3 The price adjuster . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464.4 The optimization routine . . . . . . . . . . . . . . . . . . . . . . . . 48

4.4.1 Introduction to dynamic programming . . . . . . . . . . . . . 49

4.4.2 The Bellman equation . . . . . . . . . . . . . . . . . . . . . . 494.4.3 Applying the dynamic programming algorithm . . . . . . . . 504.4.4 Discussion of the optimization routine . . . . . . . . . . . . . 62

5 Testing the agent-based model 63

5.1 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635.2 The training round . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

5.2.1 The spot price . . . . . . . . . . . . . . . . . . . . . . . . . . 645.2.2 Power outputs . . . . . . . . . . . . . . . . . . . . . . . . . . 655.2.3 Market-power . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

5.3 Second run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675.3.1 The spot price . . . . . . . . . . . . . . . . . . . . . . . . . . 68

5.3.2 Power outputs and estimation error . . . . . . . . . . . . . . 68

6 Model application: The German electricity market 71

6.1 Simulation settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 716.2 Simulation results reference scenario . . . . . . . . . . . . . . . . . . 72

6.2.1 The training run . . . . . . . . . . . . . . . . . . . . . . . . . 726.2.2 The second run . . . . . . . . . . . . . . . . . . . . . . . . . . 776.2.3 The third run . . . . . . . . . . . . . . . . . . . . . . . . . . . 816.2.4 Conclusion of the simulation . . . . . . . . . . . . . . . . . . 84

6.3 Scenario analysis: Increase in wind generation . . . . . . . . . . . . . 856.3.1 Simulation setup . . . . . . . . . . . . . . . . . . . . . . . . . 866.3.2 The spot price . . . . . . . . . . . . . . . . . . . . . . . . . . 866.3.3 Total cost for consumers . . . . . . . . . . . . . . . . . . . . . 876.3.4 Power outputs and profits . . . . . . . . . . . . . . . . . . . . 886.3.5 Price predictor and market-power . . . . . . . . . . . . . . . 896.3.6 Conclusion of the wind scenarios . . . . . . . . . . . . . . . . 92

6.4 Scenario analysis: Adding storage devices . . . . . . . . . . . . . . . 926.4.1 The spot price . . . . . . . . . . . . . . . . . . . . . . . . . . 936.4.2 Power outputs and profits . . . . . . . . . . . . . . . . . . . . 966.4.3 Price predictor and market-power . . . . . . . . . . . . . . . 996.4.4 Conclusion of the storage scenarios . . . . . . . . . . . . . . . 100

6.5 Scenario analysis: Adding a PHEV cluster . . . . . . . . . . . . . . . 1016.5.1 The PHEV cluster model . . . . . . . . . . . . . . . . . . . . 1026.5.2 The spot price . . . . . . . . . . . . . . . . . . . . . . . . . . 1036.5.3 Power outputs and profits . . . . . . . . . . . . . . . . . . . . 104

7 Model application: The four-country model 106

7.1 Simulation settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1067.2 Cross-border capacity allocation . . . . . . . . . . . . . . . . . . . . 107

7.2.1 Total social welfare vs. total variable cost . . . . . . . . . . . 1077.2.2 Formulating the optimization problem . . . . . . . . . . . . . 1107.2.3 The optimization routine . . . . . . . . . . . . . . . . . . . . 115

7.3 Simulation results reference scenario . . . . . . . . . . . . . . . . . . 1167.3.1 The spot price . . . . . . . . . . . . . . . . . . . . . . . . . . 1167.3.2 Power outputs and profits . . . . . . . . . . . . . . . . . . . . 1177.3.3 Exchanges and congestions . . . . . . . . . . . . . . . . . . . 1227.3.4 Price predictor and market-power . . . . . . . . . . . . . . . 124

7.4 Scenario analysis: Building additional interconnection capacity . . . 1 2 67.4.1 The spot price . . . . . . . . . . . . . . . . . . . . . . . . . . 126


7/158

7.4.2 Exchanges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1287.4.3 Profits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1307.4.4 Price predictor and market-power . . . . . . . . . . . . . . . 131

7.4.5 Conclusion of the line scenarios . . . . . . . . . . . . . . . . . 133

8 Conclusion 134


8/158

Abstract

In this thesis, a new agent learning approach model predictive bidding is intro-duced for modeling electricity markets with multi-agents. In contrast to traditional

agent learning approaches such as Q-learning or learning classifier systems thislearning approach is predictive and model based. The agents create an hourly priceforward curve based on exogenous factor predictions via a multi-factor regressionmodel, whereby the regression coefficients become more accurate as more market-outputs are observed. Additionally, the agents consider their market-power forcreating the hourly price forward curve by measuring a price adjusting value forevery bidding curve with a reinforcement learning algorithm. The price adjustingvalue describes how much the spot market price deviated from the price predictionsbased on exogenous factors in the past. Based on these price forecasts the agents usean optimization routine to find the set of actions which maximizes their expecteddiscounted profit over several time steps. In comparison to Q-learning and learningclassifier systems, this model based learning approach increases the learning rate,uses less memory, optimizes over multi time steps and is able to handle unknownenvironmental inputs.

An agent-based model is set up whereby the agents use the model predictive bid-ding algorithm to model the German electricity wholesale market under referenceconditions as well as a higher wind energy contribution. Every generation technol-ogy is aggregated and modeled by a single agent. Additionally, scenarios with astorage agent which either has a high power output or a high storage capacity arecarried out to assess the impact of the storage agent on spot prices, price volatility,market-power, agent bidding behavior and profitability. The storage agent therebyuses the model predictive bidding algorithm for charging/discharging his storage.In a last simulation scenario, a cluster of PHEVs is simulated which can be chargedor discharged via the electric grid and has a variable charging/discharging power ca-

pacity depending on the amount of PHEVs connected to the grid and their currentstate of charge.

In a next step, the four countries Germany, Switzerland, France and Italy aretaken into account. These countries can exchange electricity, whereby the scarcecross-border capacity between these countries is allocated with an implicit allocationscheme where the total social welfare over the four countries is maximized. Thepower generation technology in each country is modeled by an agent which uses themodel predictive bidding learning algorithm. The simulation is carried out underreference conditions and under conditions where the net transfer capacities (NTC)between certain countries is increased to assess the sensitivity of these transmissioncapacities on spot prices, agent behavior, profitability and market-power. In a last

simulation scenario, the NTC values are increased that no congestions occur andthat the spot price is equal for every hour in all four countries.

vii


9/158

Symbols

Symbols

c C Country c in C, where C = {Germany, Switzerland, France, Italy}

t,c Spot market price at time t in country ct,c Spot market price prediction at time t in country caji,t Agent is bidding curve j at time t

pi,t Agent is power output at time t

V Ctotali,t Agent is total variable cost of production at time t

f c Specific fuel cost

Generator efficiency

j,t Market price of fuel j at day t

Generator ramp cost constant Tons CO2 emissions per M W h fuel burned

t Price of one ton CO2 emission at time t

V Cpi,t Agent is variable production costs at time t

V Cri,t Agent is ramping costs at time t

V Ceai,t Agent is emission allowance costs at time t

tech Generation technology tech {nuclear, hard coal, lignite, gas, oil}

i,t Agent is profit at time t

xt,c Independent exogenous variable x in country c at time t

xc Regression coefficient of exogenous variable x in country c

t,c Disturbance term in country c at time tw Weighting factor of the pav calculation

= {0,...,N1} Policy: an arbitrary set of actions

Optimal policy: set of actions with the highest expected profit

Discount factor

st State s at time t

ut Control input at time t

wt Random disturbance variable at time t

Vi,t Agent is value function at time t

pavi|aji Agent is pav value at time t given action j

pi,t|aji Agent is power output p at time t given action jt,c|aji Spot price prediction in country c given agent i uses action j

viii


10/158

t,c|ai Spot price predictions in country c for all action curves of agent iVi,t|a

ji Agent is value function V at time t given action j

gi,t Agent is profit at time tT Vi Terminal value i

fi,T Agent is filling status at time T

i Agent is maximum energy storage content

i,charge Agent is charging efficiency

i,discharge Agent is discharging efficiency

cumi,t Agent is filling status at time t for the optimal policy

R Return

Rf Risk free return

soc State of charge of the PHEV cluster

PHEV drive constant

P0 Total amount of PHEVs in the PHEV cluster

Pt PHEVs connected to the grid

Acronyms and Abbreviations

OTC Over the counter

ECC European Economic Community

TPA Third-Party-Access

UCTE Union for the Co-ordination of Transmission of Electricity

TSO Transmission system operator

ENTSO-E European Network of Transmission System Operators for Electricity

ATC Available transfer capacity

Belpex Belgian Power Exchange

APX Amsterdam Power Exchange

NordPool Nordic Power Exchange

EU European Union

EEX European Energy Exchange

LPX Leipzig Power Exchange

EET E.ON Energy TradingRWE Rheinisch-Westfalisches Elektrizitatswerk AG

UK United Kingdom

NETA New Electricity Trading Arrangements

GA Genetic Algorithm

LCS Learning Classifier Systems

ETF Exchange Traded Fund

EIA Energy Information Administration

IEA International Energy Agency

EUA Emission Allowance Unit

DWD Deutscher Wetter Dienst

HPFC Hourly price forward curve


11/158

LF Load fraction

MSE Mean squared error

RMSE Root mean squared errorhdd Heating degree day

cdd Cooling degree day

OLS Ordinary least squares

RSS Residual sum of squares

pav Price adjusting value

VAR Value at risk

PAR Profit at risk

std Standard deviation

CMU Carnegie Mellon University

ETH Eidgenossische Technische Hochschule


12/158

Chapter 1

Introduction

Most of the energy of political work is devoted to correcting the effects of misman-agement of governmentMilton Friedman

The European Union tries to set rules for common and well-functioning electricitymarkets and cross-border trade since the electricity market liberalization. In thisintroduction, the regulations No 1228/2003 and its replacement 714/2009 whichspecifies these rules are analyzed. Additionally, market-power often present in elec-tricity generation is discussed.

1.1 The European electricity market

The European electricity industry has undergone substantial changes since the startof the deregulation process in the early 1990s. The electricity markets movedaway from vertically integrated monopolies to a liberalized market. The valuecreation chain was unbundled into generation, transmission and distribution. This

development was driven mostly by economic reasons. It was argued that the marketforces leads to a more efficient and transparent electricity trading. Power exchangeshave been established where electricity is traded in spot and future markets [ 1] [2].

Cross-border acquisitions have surged as national companies try to diversifytheir sources of revenue. Trading companies are operating on various Europeanpower exchanges and are thereby providing liquidity to the various exchanges andOTC markets. Cross-border trading activity has increased since the start of thederegulation process in Europe and is promoted by the European Union to achievethe goal of an internal market for electricity in Europe. These cross-border tradingactivities are introducing new challenges. Some cross-border transmission lines arecongested which makes its access a scarce good. Implementing an efficient and

transparent allocation of cross-border capacity is a major challenge faced by theEuropean Union and subject to current debate [3] [2] [4].

1


13/158

Chapter 1. Introduction

1.1.1 Goals of the European Union

A common energy policy is a key goal since the foundation of the European Eco-nomic Community (ECC). The European Union has set the following objectives fora common electricity policy [2]:

Establish of a competitive electricity market.

Increase the efficiency in generation, transmission and distribution of electric-ity.

Increase the competition by liberalize markets for electricity generation andby building high voltage transmission lines.

Implement the Third-Party-Access (TPA) for industrial- and large-scale con-sumers as well as distribution companies.

1.1.2 Implementation of the electricity market liberalization

The directive 2003/54/EG by the European Parliament and the regulation (EC) No1228/2003 of the European Parliament and the Council was a first approach to setcommon rules for an internal electricity market and cross-border trade in Europe.They were a direct outgrowth of the work of the Florence Regulatory Forum. Thedirective and regulation were in place from July 2004 until they were repealed byregulation No 714/2009 [2] [5].

In the following, some central clauses of the regulation No 1228/2003 are high-lighted. Clause no. 3 states that [6]:

The creation of a real internal electricity market should be promotedthrough an intensification of trade in electricity, which is currently un-derdeveloped compared with other sectors of the economy.

Furthermore, clause no. 4 in regulation No 1228/2003 states that [6]:

Fair, cost-reflective, transparent and directly applicable rules ... shouldbe introduced with regard to cross-border tarification and the allocationof available interconnection capacities, in order to ensure effective accessto transmission systems for the purpose of cross-border transactions.

The regulation No 1228/2003 put a special emphasis on increasing competitionand cross-border trade in achieving the goal of an efficient pan-European electricitymarket. The regulation highlights that although there is electricity trade betweenmember states, there is still no common single European market for electricity. Ina perfect market the prices for the same good should be identical for all consumers,which is not the case for electricity in Europe. Electricity prices vary strongly from

one area to another. These variations are caused by congestions in the transmissionnetwork [7] [1].

2


14/158


It is important to understand the underlying technical characteristics of thetransmission grid. Some facts are highlighted:

The countries in Europe were considered as control areas, the original idea be-hind control areas was the establishment of defined areas, in which generationand consumption of electricity matches more or less at every point in time.Neighboring control areas were connected to help within the UCTE area ifsudden unpredictable shortages in consumption or production occurred. Theoriginal purpose of control areas concerning the balancing of production andconsumption is no longer conformed. With an increase in cross-border tradesince the liberalisation process the transmission lines connecting certain coun-tries in Europe are often critically congested during the daily operation, whichmeans that these transmission lines are driven to their thermal or stabilitylimits, making these capacities scars [2] [1].

The electricity in the network cannot be directed, rather it follows the ohmiclaw. Physical electricity flow between two trading partners does not followthe way of their economic trade. The physical flow affects other lines andareas of the network as well. These so called loop-flows cannot be avoidedand can spread over the whole UCTE network. As a result, network accessrules and the allocation of transmission capacity has to consider the physicalsystem properties [2] [1].

An increase in cross-border transactions and cross-border capacities helps devel-oping a common single electricity market in Europe and promotes that electricityis produced at lowest cost in Europe. The congestions in cross-border lines areaddressed in article no. 6 on General principles of congestion management [6]:

Network congestion problems shall be addressed with non-discriminatorymarket based solutions which give efficient economic signals to mar-ket participants and transmission system operators involved. Networkcongestion problems shall preferentially be solved with non transactionbased methods.

Regulation No 1228/2003 specifies that congestion problems shall be addressedwith non-discriminatory and market-based approaches. A further definition ofnon-discriminatory is not provided. It is also questionable which congestion manage-ment scheme gives efficient economic signals. The regulation addresses the revenueresulting from the allocation of interconnection capacity. The article defines thatthe revenue should be used for one or more of the following purposes [ 6]:

Guaranteeing the actual availability of the allocated capacity;

Network investments maintaining or increasing interconnection capacities;

As an income to be taken into account by regulatory authorities when ap-proving the methodology for calculating network tariffs, and/or in assessingwhether tariffs should be modified.

Since 2009, regulation No 1228/2003 is repealed by the newer regulation No714/2009 of the European Parliament and of the Council. The newer regulation

3


15/158


mentions that regulation No 1228/2003 made significant contributions towards thecreation of such an internal market in electricity by deliver real choice of electricitysupply for all consumers and by encourage cross-border trade. It results in efficiency

gains, more competitive prices and higher standards of service as well as contributeto security of supply and sustainability. The new regulation also highlights someshortcomings in developing an internal electricity market under the old regulation[4]:

However, at the present, there are obstacles to the sale of electricity onequal terms, without discrimination or disadvantage in the Community.In particular, non-discriminatory network access and an equally effectivelevel of regulatory supervision do not yet exist in each Member State,and isolated markets exist. ... Prospectus for the internal gas and elec-

tricity market and Inquiry pursuant to Article 17 of Regulation (EC)No 1/2003 into European gas and electricity sectors (Final Report)demonstrated that the present rules and measures neither provide thenecessary framework nor provide for the creation of interconnection ca-pacities to achieve the objective of a well-functioning, efficient and openinternal market.

This article attributes the source shortcomings in implementing the internalmarket in electricity to the non-discriminatory network access and the creation ofinterconnection capacities. The current congestion scheme does not provide theright economic incentives. The Regulation provides a suggestion to overcome thesedeficits. In clause no. 6 the Regulation notes that [4]:

In particular, increased cooperation and coordination among the trans-mission system operators is required to create network codes for pro-viding and managing effective and transparent access to the transmis-sion network across borders, and to ensure coordinated and sufficientlyforward-looking planning and sound technical evolution of the transmis-sion system in the Community, including the creation of interconnectioncapacities, with due regard to the environment.

The transmission and distribution networks are still a natural monopoly in Eu-rope because of their large economy of scale. They are characterized by high fixedcosts and comparably low variable costs. The European Union requires that the net-works are owned by nationwide companies (TSOs). This federalistic implementationof nationwide TSOs within Europe complicates the cooperation and coordinationamong the TSOs. The European Network of Transmission System Operators forElectricity (ENTSO-E) is an association of Europes TSOs. The association is anattempt to ensure coordination of network operation. But congested power lines area source of revenue for the nationwide TSOs, an increase in cross-border capacitycould decrease the congestion revenues. Additionally, building new interconnection

capacities is oftentimes a difficult task. New power lines would have to be built inpopulated areas where there is a public opposition against them [5].

4


16/158


1.2 Congestion management

In the following different capacity allocation methods are introduced. Examples ofimplemented schemes in Europe are discussed to show how these methods are usedin reality.

1.2.1 Capacity allocation methods

In Europe, different capacity allocation methods are in use. They can be groupedas follows [8]:

First come, first served: Capacity is allocated to the order in which the trans-mission request have been received by the TSO. Starting from the earliestrequest all the requests are fully granted until the available capacity is usedup.

Pro rata: Each applicant is granted a fixed share of his requested capacityamount, the share being equal to the amount of available capacity divided bythe sum of all requested capacity amounts.

Explicit auctions: In explicit auctions the seller determines available trans-fer capacities (ATCs) considering security analysis. After the seller acceptsbids from potential buyers he allocates the capacity to the ones that value itthe most [9]. The following observations and financial implications regardingexplicit auctions can be stated [9]. Explicit auctions are

non-discriminatory

transparent

often a joint co-ordinated mechanism between the concerned TSOs

implemented with different features: uniform clearing price (in mostcases) vs. pay as bid

reflecting the cost of using the cross-border capacity (with perfect marketassumption)

not providing an opportunity for arbitrage. Internal and cross-bordertrade present the same profit opportunity for participants

sending efficient signals to market players for the operation and the valueof the network

Implicit auctions: In implicit auctions an initial energy market clearing isperformed. If ATC are reached, markets split into pre-determined price ar-eas. These areas are then cleared individually. Therefore, implicit auctionsare often called market splitting or market coupling. The following observa-tions and financial implications regarding implicit auctions can be stated [9].Implicit auctions are

non-discriminatory

transparent

a joint co-ordinated mechanism between the concerned TSOs

requiring a centralized power exchange

5


17/158


requiring financial instruments for long term price-hedging and bilateraltrade between price-areas

not providing an opportunity for arbitrage. Internal and cross-bordertrade present the same profit opportunity for participants

sending efficient signals to market players for the operation and the valueof the network

The regulation No 1228/2003 and its repealing regulation No 714/2009 clearlystate that the implementation of market-based congestion management methodsare preferred. These methods should be non-discriminatory, market-based and giveefficient economic signals [4].

A market based allocation would imply that the price for the transmission ca-

pacity is equal to the opportunity cost of the capacity. In other words, the costfor the marginal user of the scarce capacity should be equal to the price differentialbetween the electricity markets at either end of the transmission.

First come first served and pro rata methods of capacity allocation fail to meetthe criteria of being non-discriminatory and market-based. In both of the casesthe capacity allocation is made without considering the users value and there isno guarantee high value users will gain access to the transmission in preference tolow value users. In contrast, the explicit and implicit auctions appear to meet thecriteria of being non-discriminatory and market-based [8]

Article [8] concludes that implicit auctions appear to be the economically mostefficient mechanism for allocating line capacity. Especially taking into account that

there is uncertain (or asymmetric) information and some players exercise market-power (see section 1.3.2).

1.2.2 Implemented congestion management schemes and out-

look

The following table shows implemented congestion management schemes for selectedcross-border capacity allocation. The first table 1.1 represents the allocation inthe day-ahead time frame whereas table 1.2 the allocation for longer time frames(months or years). The selected cross-border transmission lines in the tables are

the ones which are further investigated in this thesis.

In the following, other implemented energy and cross-border allocations schemesin Europe are examined. The NordPool area consisting of Denmark, Finland, Nor-way and Sweden use an implicit cross-border allocation scheme. The exchangecapacity allocation is an integral part of the day-ahead spot price calculation. Con-gestion on interconnections between price areas results in price differences betweenthem. Congestions can occur on lines connecting neighboring countries or on linesconnecting areas within a country. If there are no congestions in the transmissionlines the spot prices in the price areas will be equal. This simultaneous allocationof energy and capacity ensures a maximal utilization of exchange capacity.

The markets Belpex (Belgium), Apx (Netherlands) and Powernext (France)use a trilateral market coupling and an implicit cross-border capacity allocation

6


18/158


Cross-border line Allocation methodItaly-Switzerland Implicit auction (or legal framework different from EU)Switzerland-Italy Implicit auction(or legal framework different from EU)

France-Switzerland Priority list(or legal framework different from EU)Switzerland-France No congestion

Italy-France Implicit auction (or no congestion)France-Italy Explicit and implicit auctions

Germany-France Explicit auctionFrance-Germany Explicit auction

Switzerland-Germany Explicit auctionGermany-Switzerland Explicit auction

Table 1.1: Day-ahead allocation method, source [9]

Cross-border line Allocation method

Italy-Switzerland Auction of financial transmission rightsSwitzerland-Italy Auction of financial transmission rightsFrance-Switzerland No long-term allocationSwitzerland-France No long-term allocation

Italy-France Auction of financial transmission rightsFrance-Italy Auction of financial transmission rights

Germany-France Explicit auctions of physical transmission rightsFrance-Germany Explicit auctions of physical transmission rights

Switzerland-Germany Explicit auctions of physical transmission rightsGermany-Switzerland Explicit auctions of physical transmission rights

Table 1.2: long-term allocation method, source [9]

mechanism. This example shows that a centralized energy market is not a nec-essary requirement for implicit cross-border capacity allocation methods. A jointco-ordinated mechanism between the concerned TSOs is sufficient to implement theimplicit allocation scheme [10].

There is a development toward a centralized European electricity market in Eu-rope. Germanys electricity exchanges, the European Energy Exchange (EEX) andthe Leipzig Power Exchange (LPX) merged in 2002. The new merged electricitybourse is now merging with the French electricity exchange Powernext. E.ON En-ergy Trading (EET), the E.ON groups wholesale trading arm and major Europeanenergy operator said it expects more consolidation and fewer exchanges in the fu-ture. The creation of marketplaces across national borders for different regions

is a natural step on the road to more integrated European power markets (TonyCocker, chief executive of EET). This integrated electricity market will induce moreimplicit capacity allocation method as they are used in the NordPool area.

1.3 Market-power

Market-power is understood as the ability of the market participants to deviateprices from the competitive levels in a profitable way [11]. In electricity markets,market-power describes the ability of market players to influence the spot market

price by their bidding. A higher concentration in electricity generation generallyleads to higher market-power. In contrast, in a competitive market structure, the

7


19/158


decisions on power output are unable to significantly alter the market price and themarket players act as price takers.

1.3.1 Market-power in electricity generation

Throughout the liberalization process in Europe there has been a movement froma more fragmented and competitive electricity wholesale market to an oligopolisticone where a few utilities share a major market stake in electricity production. Inthe German electricity market for instance the overcapacity and the large numberof market players led to a rapid and considerable price drop at the beginning ofthe liberalization process which reduced industry profitability. RWE announced arise in sale of 25% but 15% lower profits in the year 2000 [ 12]. This low profitabil-ity prevented new market-entry and did not set incentives for new investments in

power generation capacity. The result was a consolidation phase where the Ger-man electricity market transformed from one with eight major vertically integratedgenerating companies and many smaller ones in 1999 to one where four dominantfirms have a combined market share of over 90% by the beginning of 2001 [12].

The consolidation in electricity generation in Europe has lead to a high con-centration in power production where markets providesthe opportunity to exercisemarket-power to these dominant firms. An efficient competitive market maximizesthe total social welfare, provides proper investment incentives and enhances the se-curity of supply [13]. The concentration in power production severely impedimentsthis desired outcome and leads to uncompetitive price outcomes. Market-power inthe electricity wholesale industry is therefore a growing concern in the EuropeanUnion.

1.3.2 Market power and congestions

In [1] it is argued that market participants in Spain, California, the UK and Ger-many were suspected to exploit market-power by driving the transmission systemto its limits [1]. By creating these artificially congestions in the network, the marketis split into a high and low priced area. A market participant in the high price areahas an incentive in this market split.

Additionally, under explicit auctions, market participants know the outcome of

the transmission allocation before they decide how to respond in the energy mar-ket. An option is granted to the players which obtained transmission capacity. Aparticipant who can exercise market-power would have a different incentive for theprice differential between the two markets if he gains significant transmission ca-pacity than if he has obtained no or only a small amount of transmission capacity.This implies that a market participant influences the market outcomes in the twoenergy markets depending on his position in the transmission market. This sys-tematic dependency between transmission and energy market outcome may allowthe dominant players to captures rent from others, even under the assumption ofperfect information [8]. The allocation of line capacity is not predetermined in im-plicit allocation schemes, but there might still be ways for market participants withmarket-power to gain advantage of congested lines by observing historical corre-

lations between their bidding behavior and the price outcome under transmissioncongestions. How these market-participants can capture rent is not as straightfor-

8


20/158


ward as it is in explicit auctions [8].

1.4 Contribution of the thesis

To establish a well-functioning market for electricity which sets the right incentivesand leads to an optimal outcome in terms of social welfare is not a trivial task[14]. This thesis aims to simulate the market outcome of an integrated electricitymarket in Central Europe where market-power in electricity generation is present.In particular, the countries Germany, France, Switzerland and Italy are examined.A multi-agent model is carried out for the simulation to incorporate the effect ofmarket-power. The following tasks are performed in the thesis:

Setting up a multi-agent model for the four countries.

Introduce a new agent learning methodology. The new learning process pro-vides a faster learning rate than comparable classifier based learning ap-proaches.

Implement an implicit cross-border allocation scheme and analyze the priceoutcome.

Assess the effect of a higher wind energy contribution in Germany.

Analyze the price outcome in Germany by introducing a higher electricitystorage capacity with an advanced charging/discharging scheme.

Introduce an PHEV (plug-in hybrid electric vehicle) cluster which is charged/dischargedvia the electric grid and assess its effect on the spot market.

Perform a sensitivity analysis on spot prices in these four countries by changingthe cross-border capacities.

1.5 Structure of the work

The thesis is structured as follows: In chapter 2 an introduction into agent-basedmodeling is given including a discussion on the advantage of agent-based models

over other modeling approaches. A special emphasis is put on introducing agent-based models for electricity markets. In chapter 3, the agent-based model used inthis thesis is specified. The market participants and the market clearing process areintroduced and the data sources are defined. Furthermore, the simplifications of themodel are discussed. Chapter 4 presents a new learning algorithm model predictivebidding, which is a enhancement of current learning algorithms and provides a fastand efficient learning for the agents. The agent-based model based on the derivedlearning algorithm is applied to the German electricity wholesale market. The re-sults are discussed in chapter 6. Scenario analysis es are carried out to assess theeffect of a higher wind energy contribution, additional storage devices and a PHEVcluster on electricity spot prices. In chapter 7, the agent-based model is appliedto the four countries Germany, Switzerland, France and Italy. The cross-border

capacity allocation scheme is introduced which maximizes the total social welfare.The simulation results are presented under a reference scenario and scenarios with

9


21/158


increased cross-border capacities. In one simulation scenario the cross-border ca-pacities are set to infinity to assess the outcome under a copper-plate model.

10


22/158


23/158

Chapter 2. Agent-based modeling in electricity markets

complex outcomes on a macro-level. A main feature of the agent-based models isthe repetitive competitive interactions between the agents [17] [18].

According to [19], the most general characterization of agents is given by thefollowing properties:

autonomy: agents operate without the direct intervention of humans or others,and have some kind of control over their actions and internal state.

social ability: agents interact with other agents (and possibly humans) viasome kind of agent-communication language.

reactivity: agents perceive their environment, (which may be the physicalworld, a user via a graphical user interface, a collection of other agents, the

internet, or perhaps all of these combined), and respond to occurring changesin a timely fashion.

pro-activeness: agents do not simply act in response to their environment,they are able to exhibit goal-directed behavior by taking the initiative.

The simulated evolution of the system can be studied both from the perspectiveof the aggregate population as well as the individual agent behavior.

2.2 Benefits of agent-based modeling

Agent-based model offer three main benefits over other modeling techniques [18].Agent-based models:

capture emergent phenomena, these phenomenas result from the interaction ofthe individual entities. The whole is more than the sum of its parts becauseof the interactions between the parts [18]. These emergent phenomenas canhave properties which are decoupled from the properties of the single entities.

provide a natural description of a system. If the system is composed of be-havioral entities, agent-based models are most natural and closer to reality tomodel these systems.

are flexible. The flexibility comes in different dimensions. More agents forinstance can be added, the complexity of the agent, their behavior, degree ofrationality, ability to learn and evolve can be tuned.

2.3 When are agent-based models useful?

In [18] a summary is given about usefulness of agent-based models. The followingsituations are mentioned. Agent-base models are useful when:

the interaction between the agents is complex: For instance nonlinear, discon-tinuous or discrete.

12


24/158


space is crucial and the agents are able to move.

the agents exhibit complex behavior, including learning and adaption.

2.4 Using agent-based models for electricity mar-

kets

As described in 2.1, agent-based models allow to analyze the interdependencies ofthe micro level (participants) and the macro level (the overall market structure).This bottom-up modeling approach appears suitable to assess the evolution of elec-tricity prices. Power generating units can be modeled as agents. They fulfill thebasic requirements for agents given by [19] in section 2.1. The power generating

units are autonomous, they have a social ability in a sense that they interact on amarket place with each other and reactivity because they collect inputs about theirstate and the environment to decide on their actions. Furthermore, these units usea pro-activeness to try different actions to increase their profit.

The interaction between the agents (power generating units) is complex and theagents are able to learn effects related to repetitive behavior and to incorporate theeffect of market-power [20], [21]. An oligopoly market is assumed. Thus, electric-ity producers may bid strategically above their marginal cost as they realize theirpossible influence on market prices [22].

No assumptions concerning the attainment of equilibrium are made in model-

ing an electricity market. There is no guarantee that the aggregated behavior ofthe system is stable if every agent is trying to maximize its profit. Rather, if anequilibrium is obtained or not is assessed from the actual model execution [17].

Most modeling studies assume that there is a bidding process in a central marketplace supervised by a TSO which results in a set of market clearing prices. Moststudies neglect bilateral OTC trading and in terms of network modeling most studiesdisregard transmission constraints entirely [15]. In the model used in this thesisOTC trades are neglected as well, but transmission constraints on cross-bordertransmission lines are taken into account as in the work presented in [ 20].

2.5 Related work

A first group of agent-based models is referred to as simulations applying model-based adaption algorithms according to [23]. These earlier methodologies have naiveor intuitive learning formulations and are tailored for specific designs of the marketsthey simulate. Most of the work in this field has been conducted at the LondonBusiness School.

John Bower, Derek W. Bunn et al. from the Energy Markets Group at theLondon Business School presented agent-based models mainly for the Eng-

land and Wales electricity market in the year 2000. Their simulation resultsin Model-based comparison of pool and bilateral markets for electricity [24]

13


25/158


and Experimental analysis of the efficiency of uniform-price versus discrim-inatory auctions in the England and Wales electricity market [25] compareddifferent market mechanisms and bidding schemes. The generators applied

a simple reinforcement learning algorithm with the goal to maximize profits.The agents adjusted their bidding strategy to their last round success. Trans-mission costs were thereby neglected. The simulation results were validatedagainst classical models of monopoly, duopoly and perfect competition.

In 2001, John Bower, Derek W. bunn and Claus Wattendrup applied thesame model to the German electricity wholesale market in A model-basedanalysis of strategic consolidation in the German electricity industry [26]. Theauthors analyzed the impact of mergers and concentration in the Germanutility industry on spot prices. They concluded that electricity prices wouldrise considerably as an effect of mergers and acquisitions.

In Agent-Based Simulation - An Application to the New Electricity Trading

Arrangements of England and Wales [27], Derek Bunn and Fernando Oliveirapresented an application of the agent-based model to the proposed new elec-tricity trading arrangements (NETA) in the U.K. In contrast to the modelsdescribed above, the authors incorporated an active demand side. A bilat-eral forward market followed by a balancing mechanism was considered andmodeled. This agent-based simulation model provided insights to pricing andstrategic decisions ahead of NETAs actual introduction.

Marija D. Ilic and Poonsaeng Visudhiphan modeled electricity markets byagents, which adjust their bidding strategy by using information about pastmarket prices and their own marginal cost curves. They showed that repeti-tion of bidding plays a significant role in the market dynamics and that tradi-tional static market models are not capable of showing these dynamics. In the

thesis An Agent-based Approach to Modeling Electricity Spot Markets [28] P.Visudhiphan used and Auer et als softmax action selection algorithm in de-termining the generators bid supply functions. Auer et als algorithm assignsa probability to each possible action. The probability of choosing an action isadjusted every round based on the rewards received from the chosen action.

An other group of researcher uses genetic algorithms (GA) for the agent learn-ing process. GA are heuristic methods inspired by biological evolution. In agentlearning methodologies with GAs, the market participants strategies are coded intobitstrings. The fittest strategies are thereby passed from one generation to the nextby the production of offsprings. By using crossover and mutations, the GA are able

to model the genetic dynamics underlying natural evolution. In electricity marketagent-based models, the GAs can give the agents the ability to search for optimalbidding strategies [23].

James Nicolaisen et. al. used GAs for the learning mechanism of the buy-ers and sellers in Concentration and Capacity Effects on Electricity MarketPower [29]. The GA was used to determine the agents bid and ask prices.Different ratios of buyers and sellers as well as buying and selling capacitieswere investigated. The market power was measured in terms of deviation fromcompetitive equilibrium.

In A Co-evolutionary Approach to Modeling the Behavior of Participants in

Competitive Electricity Markets [30] the agents observed a discrete state ofthe environment. Based on this state, they chose an action in the set of

14


26/158


bidding actions. The agents co-evolved their bidding strategy by using agenetic algorithm. The paper concluded that a state-based bidding strategyhelps facilitate the learning process.

Alvin E. Roth and Ido Erev demonstrated that reinforcement learning can makeuseful predictions in experimental games. The reinforcement algorithm presentedin Predicting How People Play Games: Reinforcement Learning in ExperimentalGames with Unique, Mixed Strategy Equilibria [31] is based on psychological find-ings. It incorporates the aspects of experimentation and forgetting. A considerableamount of agent-based electricity models apply this algorithm [23].

A. Weidlich and D. Veit used the Erev and Roth algorithm in Bidding ininterrelated day-ahead electricity markets: Insights from an agent-based simu-lation model. Two markets were cleared sequentially: a day-ahead market anda day-ahead balancing market. The success was evaluated by integrating theopportunity cost of profit that could have been obtained in the other market.The agents learned from past trading results using a modified Erev and RothAlgorithm. The authors showed that the timing of market clearing plays animportant role.

In Market power and efficiency in a computational electricity market withdiscriminatory double-auction pricing J. Nicolaisen used the Erev and Rothlearning algorithm for an agent-based model to study market power in electric-ity markets. The outcomes of the simulation was compared to earlier studiesin which buyers and sellers use genetic algorithms for learning.

Another learning technique often applied is Q-learning. It is a reinforcementtechnique and was first introduced by Watkins [32]. The agents thereby learn anaction-value called Q value that describes the expected reward of taking a givenaction in a given state and following a fixed policy thereafter. The strength ofQ-learning is that it is able to compare the expected utility of available actionswithout using a model of the environment. Q-learning is often applied to Markovdecision processes.

T. Krause and G. Andersson compared different congestion management schemesin Evaluating Congestion Management Schemes in Liberalized Electricity Mar-kets Using an Agent-based Simulator [20]. The market participants used a Q-

learning algorithm in this model. However, the agents were not differentiatingthe state they were in each iteration.

The paper Agent-Based Simulation of Power Markets under Uniform andPay-as-Bid Pricing Rules using Reinforcement Learning [33] studied the priceoutcome under uniform and pay-as-bid (discriminatory) pricing schemes. Theauthors used the SA-Q-learning algorithm, a slightly changed version of Q-learning. Agents usually face a tradeoff between exploration and exploitationin Q-learning. The paper introduced a methodology for a good explorationand exploitation ratio. The authors defined environmental states by the lastrounds market price.

A breakthrough in agent-based models was the introduction of learning classifiersystems (LCS). They combine reinforcement learning to increase the probability

15


27/158


of choosing a successful action and genetic algorithm to reproduce new actionsfrom successful ones. A main feature of LCS is that they are adaptive in a sensethat the ability to choose the best action improves with experience. The source of

improvement is reinforcement. The observed reward after a trial for a given action isused by LCS to alter the likelihood of taking that action in similar circumstances. Incontrast to Q-learning, LCS classifies an environmental input or state. Every agentuses a classifier population, a set of classifier, whereby every classifier proposesa certain action. When a particular environmental input occurs, the LCS formsa match set of classifiers whose conditions satisfy the input. The classificationintroduces generality: A classifier can match two environmental inputs which arenot identical. This allows that the agents are able to classify an input they have notobserved in the past. The classification of the input should thereby be a trade-offbetween accuracy and generality. Accuracy means that the agent should be able toclassify an input correctly and general that unknown environmental inputs (inputswhich have not occurred in the past) should be classified correctly [34].

In the following some papers which use the LCS algorithm for the agent learningprocess are introduced.

In A Multiagent Model of the UK Market in Electricity Generation [35], A.Bagnall and G. Smith presented an agent-based model of the pre NETA UKmarket. The agents used a hierarchical learning classifier system. Three mainresearch questions were addressed by the authors: First, are the agents able tolearn optimal strategies when competing against nonadaptive agents. Second,are the agents able to learn bidding strategies observable in the real world andthird, can cooperation among the agents evolve in certain market situationswithout explicit communication. The authors referred to the work done in [ 25]

and [27] in their paper to highlight the difference in agent learning betweenlearning classifier systems and other methodologies. The authors emphasizedthat the main difference is the complexity of the agent architecture. As [23]correctly states, the authors did not mention why this high complexity ofagent behavior is needed and in which way their results were more valuablethan other simulation models. As a lot of other authors, this paper has notconvincingly made it clear what failures of simpler learning representationsthey avoid with the very complex LCS applied in their model [23].

In What is a Learning Classifier System? some well known researcher inthe field of LCS were asked this question. Some researcher responded areJ. Holland, R. Smith, P. Lanzi and S. Wilson. S. Wilson and P. Lanzi dida considerable work in the development of XCS, which is a subcategory of

LCS. This work did not focus on the mathematical fundamentals of LCS nordid it apply LCS to a certain problem, rather, it described what generally thestrength and justifications of LCS are. P. Lanzi concluded that LCS are betterthan traditional reinforcement learning techniques and are more general thanWatkins Q-learning [32] because Q-learning makes the assumption that theenvironment must b e a Markov Decision Process. Additionally, Q-learningrequires that the agents have a goal which LCS does not require. LCS mightbe very general and not expressible in terms of an optimization problem (e.g.survive).

A main difficulty of LCS is to measure the goodness of a classifier. This

measure is used by the genetic algorithm to select strong classifiers for reproduction,mutation etc. and to delete weak classifiers. This goodness is also used to select a

16


28/158


29/158

Chapter 3

Specification of the model

It is easier to change the specification to fit the program than vice versaAlan Perlis

As described in chapter 2, the electricity generating units are modeled as agents andline flows between neighboring countries are taken into account. The model at handcovers the countries Germany, France, Switzerland and Italy and is referred to asthe four-country model later in the thesis. Every electricity generation technology(e.g. gas, coal, wind) within a country is aggregated in one single decision makingagent.

The agents compete on a countrywide central market place by sending hourlysupply curves to the market. The supply curve specifies the agents electricityproduction level as a function of price. The market collects all the supply curvesand aggregates them to a nationwide market supply curve. It is assumed that theelectricity markets between these four countries are coupled. Taking into accountthe demand curve for every country, the four energy markets are cleared with thepossibility to trade electricity between the countries. An implicit auction methodis assumed for the congestion management scheme. A more detailed description onthe implemented market coupling can be found in chapter 7.

The resulting market prices for the four countries are fed-back to the agents,which compute their profit based on a generator specific cost model. The objective

of the agents is to maximize their profits. The current state of the environment andcurrent state of power output as well as forecasts for the environment are taken intoaccount by the strategic decisions of the agents. The learning process of the agentsis described in chapter 4.

3.1 Market participants and input data

3.1.1 Market participant definition

Each agent represents one of the following electricity generation technologies:

18


30/158

Chapter 3. Specification of the model

Nuclear

Coal

Gas

Oil

Wind

Storage

Hydro

The storage agent represents the pump-hydro storage power plants and the hydroagent the river power plants. The pump-hydro storage power plants can buy andsell electricity and have no natural water inlet whereas the hydro power plants have

a constant water inlet and only a little storage capacity. As a result, the hydropower plants have to sell the constant water inlet in short time frames.

All agents except the wind agent participate actively in the market. A fixed feed-in tariff is paid to the wind producers, which provide the entire available amountof wind energy to the market independently of the current market price.

Table 3.1 shows the installed generation capacity for every agent and the aggre-gated installed capacity per country.

Country Generation technology Installed capacity [MW]Germany Nuclear 20263Germany Coal 37855Germany Gas 16643Germany Wind 21139Germany Total 95900

France Nuclear 63260France Coal 14727France Gas 9818France Hydro 20829France Storage 4303France Total 112937

Switzerland Nuclear 3220Switzerland Hydro 13465Switzerland Storage 1636

Switzerland Total 18321Italy Gas 34382Italy Oil 22921Italy Coal 8022Italy Hydro 13573Italy Storage 7544Italy Total 86442

Table 3.1: Installed capacities for every agent, source EIA

The model is crucial to the input data. Based on observations from electricity

traders, several inputs monitored by the traders are chosen for the model. All agentsreceive the following exogenous input at hour t:

19


31/158


Hour

Current oil price

Wind forecast for Germany

Temperature forecast for all countries

Weekend factor

Load forecast per country

Only a wind forecast for Germany is needed, since the model assumes that onlyGermany has wind power plants. This assumption is reasonable since out of thesecountries only Germany has a significant amount in wind generation. The agentsreceive the temperature forecast for all four countries. The reason is that in Francefor instance the spot price is correlated to the temperature. Agents in countrieswhich are connected to France via transmission lines are able to anticipate thehigher prices in France by knowing the temperature forecast for France. Hard coal,lignite and gas prices tend to be correlated to the oil price, as a result, the agentsreceive the oil price as proxy for their fuel costs. For learning purposes, all agentshave a memory where they store historical exogenous factors, power outputs, thecorresponding market clearing prices k,c and the action curve they used ai,k athours k t 1, t 2,...,t N and country c.

Each agent is parameterized and described by the following parameters:

Generation type (nuclear, coal, gas, oil, wind, storage, hydro) of the agent.

Maximum power output [MW].

Country in which the agent is located (Germany, France, Switzerland or Italy).

Generator efficiency curve as a function of the generation technology andoutput level.

Generator ramp cost constant [e /MW].

CO2 emissions per M W h [ton/MWh] (depending on the output level, theefficiency, the fuel and the specific CO2 emissions of the fuel).

Technical ramp capacity [MW/h].

For the storage device:

Efficiency charge charge (pumping efficiency).

Efficiency discharge discharge (turbine efficiency).

Maximum storage capacity.

3.1.2 Data sources

The data for the parameters as well as for the input vectors are taken from varioussources:

20


32/158


Fuel costs: The coal exchange traded fund (ETF) NYSEArca: KOL, as wellas the FTSE USA - coal index (FTSE: WIUSA1771.L) from Yahoo Financeare used to derive the daily coal price. The Amex Natural Gas Index (XNG)

serves proxy for the daily gas price. The data for the daily oil price is takenfrom EIA. The fuel costs for the nuclear power stations (enriched Uranium)is considered constant.

CO2 costs: For the daily prices of the CO2 emissions , the prices for EmissionAllowance Units (EUA) are used. The EUAs are traded over the EuropeanUnion Emission Trading System. The EEX Carbon Index (Carbix) is takenas reference price.

Wind data: The wind and day-ahead wind forecast data is provided by trans-power stromubertragungs GmbH, a subsidiary of E.ON AG. The feed-in tariff(Einspeisevergutung) is taken from the Bundesverband WindEnergie e.V andamounts to 75 e /MWh.

Historical electricity prices for Germany are provided by the European EnergyExchange (EEX).

The historical load data as well as the values for the net transfer capacities(NTC) are obtained from ENTSO-E.

The hourly temperature data for Germany is provided by the Deutscher Wet-terdienst (DWD). For the European temperature data, the source EuropeanClimate Assessment [38] is used.

3.2 Supply curves, market structure and marketclearing

3.2.1 Supply curves

All agents except the wind agent send an hourly supply curve to the central marketplace. The supply curves represent the agents power output level as a function ofthe spot market price. Three different types of supply curves are differentiated: Theones for the inelastic bidders, for the elastic bidders and for the storage bidders.

Inelastic bidder: It is assumed that the hard coal, lignite and nuclear agenthave ramping constraints which means they can only ramp their productionlevel a certain amount from one hour to the other. As a result, they are sendinga more inelastic supply curve to the market place as the gas fired power plantagent for instance. Figure 3.1 shows a supply curve of an inelastic bidder. Itis important to mention that the supply curve is inelastic, but not perfectlyinelastic; The agents power output is not constant, rather it is still a functionof the spot market price. The supply curve is characterized and parameterizedby two points x and y.

The value x stands for the power output level when the spot price is zero andy for the power output when the spot price is equal to the defined maximum

price which is equal to 200e

/MWh in this example. The power output forevery spot price between these two values is linearly interpolated.

21


33/158


200150100500

40

20

60

80

100

0

x

y

Spot price [e /MWh]

Power

outpu

t[%

]

Figure 3.1: Supply curve for the inelastic bidders

It always holds that x y. If x = y, than the supply curve is perfectlyinelastic and the power output is constant for every spot price.

Elastic bidders: The supply curves for the elastic bidders is characterized bythe points x and y as shown in Figure 3.2. It is assumed that the elasticbidders are able to ramp their production level from full load to zero or fromzero to full load between two consecutive hours. The agents representing gasand oil fired power plants qualify as elastic bidders.

The value x represents the spot price at which the agent starts to increaseits production level and the value y the spot price at which the agent reaches100% power output. If the spot price is smaller than x, the power output is0% and if the spot price is greater than y, the power output is 100%. Allvalues between x and y are linearly interpolated.

It always holds that x y. If x = y, than the supply curve is perfectly elasticand the power output changes from full load to zero at one specific spot price.

The agents with the elastic supply curves have higher marginal costs thanthe inelastic bidders but more flexibility in choosing the power output. Theyusually choose a power output level of zero if the market price is considerablelower then their marginal cost or full load if the spot price is above their

marginal cost.

Storage bidders: The storage bidders have the ability to buy and sell electricitydepending on the spot price. As shown in Figure 3.3, their supply curve isparameterized by the values x and y. A negative power output representsbuying electricity while a positive represents selling electricity. The pump-storage hydro power plants qualify as storage bidders. This type of powerplant pumps water from lower basins to higher ones with cheap electricityand turbines it when the spot price reaches a certain level. It is assumed thatthere is no natural water inflow which means that all the electricity that issold has to be bought at some point in time before. The current reservoirlevel is the integral of all historical buyings and sellings.

Furthermore, it is assumed that the storage agents are symmetric in a sensethat the maximum turbine power level is equal to the maximum pumping

22


34/158


200150100500

40

20

60

80

100

0

x

y

Spot price [e /MWh]

Power

ou

tpu

t[%

]

Figure 3.2: Supply curve for the elastic bidders

power level.

The value x represents the spot price at which the storage agent starts todecrease the pumping from 100% and at value y the storage agent reaches100% turbine level and sells as much electricity as possible. If the price islower then x, the storage agents pumps at full load and if the spot price isgreater then y, the agent turbines at full load. The turbine or pumping power

level between spot values x and y is derived by a linear interpolation. Thepoint of intersection between the x-axis and the power output graph representsthe spot price at which the agent changes from pumping to turbining.

20015010050

-20

-60

20

60

100

80

40

0

-40

-80

x

y

-100

Spot price [e /MWh]

Power

ou

tpu

t[%

]

Figure 3.3: Supply curve for the storage bidders

It always holds that x y. If x = y, than the supply curve is totally elasticand the power output changes from full pumping to full turbining at onespecific spot price.

23


35/158


3.2.2 The market clearing

The market supply curve is derived by aggregating the supply curves for all agentsper country. The storage agents supply curve is decomposed. The positive partof the supply curve is added to the market supply curve while the negative part isadded to the market demand curve.

Starting point for deriving the market demand curve is the load per country cfor a specific hour t. At first, the demand is assumed to be perfectly inelastic atthe country load level. Then, the negative parts of the storage supply curves areadded.

Figure 3.4 shows an example of a market clearing. The market clearing pricet,c is derived by intersecting the aggregated supply curve with the demand curve.After the market clearing, the model calculates the electricity output pi,t of agent

i at time t based on the agents supply curve j (aji,t) and the market clearing pricet,c for every country c.

200150100500

40'000

20'000

60'000

80'000

100'000

0

Spot price [e /MWh]

Power

ou

tpu

t[M

Wh]

Market demand curve

Market supply curve

Storage agent buying

Market clearing price (t,c)

Figure 3.4: Market clearing

The market clearing is not always possible. Two unfavorable outputs can preventthe market supply and demand curve to intersect:

At market price zero, the supply level is higher than the demand level: Thiscase occurs if there are a lot of agents with tight ramping constraints or highramping costs (inelastic bidders) and the demand fluctuates heavily betweenconsecutive timesteps. The agents are not able to decrease the productionlevel enough to encounter the effect of a reduced load for a certain hour. Thesame effect can happen if there is excess wind energy and the remaining agentsare not able to lower their power output. Some electricity spot markets suchas the EEX in Germany allow negative spot prices. In a situation with excesspower generation a negative spot price outcome is likely. In the simulation inthis thesis negative prices are not defined, in this case, the market price (t, c)is set to 0 e /MWh for this country and hour.

At market price equal to the maximum price of simulation, the demand level

24


36/158


is higher than the supply level: The agents are not able to meet the demand.In every country the installed capacity is higher than the maximum load forevery hour t. This case therefore does not occur because there is not enough

generation capacity, but because of ramping constraints or high ramping costs.The agents are not able or willing to ramp their production up to meet thedemand. If this case occurs, the market price (t, c) is equal to the maximumprice of simulation.

The effects described above occur with inexperienced agents but rarely withexperienced agents. The reason is that a price equal to the maximum price ofsimulation is a high profit opportunity and a price equal to zero a loss for everyagent. Since the agents optimize over multiple steps (see chapter 4), the agentsramp their production level over these steps and anticipate very high or low pricesearly. A very high expected price drives the agents to increase the production overseveral price steps in order to exploit the profit opportunity and a very low expected

price induces the agents to start decreasing the production level over several timesteps.

3.2.3 Time frame of the bidding process

At day d, the agents send 24 hourly supply curves for the upcoming day d + 1.The market clearing is then performed for the 24 hours. This implies that theagents know the prices for the upcoming day hours at day d. The implementationof the day-ahead spot market is in accordance with the implemented structure ofthe major electricity spot markets in Europe where the day-ahead auction market

of single hours is the closest to a spot market [39]. In this thesis, there is no future,intra-day or balancing market modeled. The work focuses on the day-ahead market.

3.3 Cost of production and profit calculation

The calculation of the variable cost for the agents is crucial to the model since itis directly related to the calculation of the agents reward. The following table 3.2provides a summary of all the parameters used in the cost model to calculate thetotal variable cost:

Symbol Parameter description

f c Specific fuel cost Generator efficiency Daily market price of fuel Generator ramp cost constant Tons CO2 emissions per MWh fuel burned Daily price of one ton CO2 emission

Table 3.2: Generator specific parameters of the cost model

The quantification of the total variable cost V Ctotali,t for the different generatorsis a modification of the model proposed by [40] and follows:

V Ctotali,t = V Cpi,t + V C

ri,t + V C

eai,t (3.1)

25


37/158


where V Cpi,t represents the variable production costs, V Cri,t the ramping and

V Ceai,t the emission allowance costs at time t for generator i. The productioncosts V Cpi,t comprises specific fuel costs f c (techi, m,t), which depend on agents

generation technology, the daily market price of the specific fuel m (m,t), theagents efficiency (pi,t,techi) and power output pi,t. The technologies tech {nuclear, hard coal, lignite, gas, oil} are differentiated. The efficiency depends onthe generation technology and agent is power output level pi,t at time t. The de-pendency between the efficiency and the power output level is approximated by aquadratic function. The variable cost of production can be calculated according to:

V Cpi,t = pi,t

f c (techi, m,t)

(pi,t,techi)(3.2)

The ramping costs V Cr is calculated by the difference in output level from one

hour to the next multiplied by a generator specific ramp cost constant (techi):

V Cri,t = (pi,t pi,t1) (techi) (3.3)

The emission allowance costs V Ceai,t is determined by a fuel specific constant (fueli), which specifies the number of tons of CO2 emissions per M W h fuelburned and the daily price for the emission allowance t in price per ton CO2produced. Therefore,

V Ceai,t = pi,t (fueli) t 1

(pi,t,techi)(3.4)

The paper [40] suggests using an additional cost term to incorporate the effectsof generator shut-downs. It is argued that if an agent turns off the production, theboiler, pipping etc. cools down. Turning the power plant on later induces additionallosses and therefore costs which depend on the length of the shut-down. These costsare not incorporated into the model used in this thesis. The three main reasons are:

In general, the gas turbines turn off their production because of their highmarginal costs and lax ramping constraints. Since the gas turbines do not usea boiler to heat up pressurized water but burn the gas in the turbine directly

the losses associated to shut-downs are minor.

The data to model the shut-downs is not publicly available. The function tomodel these costs might be reasonable, but the determination of its parametersis difficult.

It is difficult to implement this type of costs because they depend on the lengththe generator has been shut down and are highly nonlinear. They actually donot influence the decision to turn a power plant on again or not because atthe time of the decision to go online these costs can be considered sunk costsand should therefore not affect the decision. As a result, their influence onthe bidding strategy is most likely not significant.

Finally, agent i calculates its profit at time t (i,t) according to

26


38/158


i,t = t,c pi,t V Ctotali,t (3.5)

where t,c represents the spot price in country c of time t.

3.4 Simplifications of the model

The following simplifications are made in this thesis for modeling the central Eu-ropean electricity markets. The simplifications are made to reduce the complexityof the model. It is assumed that the simplifications do not alter the simulationoutcome dramatically.

There are only generators and consumers in the market, no third party tradingintermediaries.

Implicit auctions are assumed for cross-border trades.

The grid within a country is modeled as a copper-plate with no congestionsand losses.

The load part of the aggregated demand function is perfectly inelastic, whichis a feasible assumption since the short run demand for electricity is quiteinelastic in reality. Only the buying of the storage agents leads to an elasticdemand curve.

There is no forward and futures market where electricity can be traded.

The entire volume is traded via the spot market place, i.e. there are nobilateral OTC trades.

27


39/158

Chapter 4

Learning-process of the

agents: The methodology

Good decisions come from experience, and experience comes from bad decisionsUnknown author

The findings from thesis [41] are used to develop a new framework for the agentlearning process. In this thesis the agents should follow a distinct goal: To max-imize their profit. The learning methodology should incorporate basic conceptsof Q-learning, but instead of deriving the Q-values 1 from trying different actions

in different states, the agents calculate the Q-values directly based on a model.The model is a multi-factor regression model and allows calculating the expecteddiscounted rewards (profits) for every combination of action, thereby taking intoaccount exogenous factors and market-power.

Later in the thesis it is shown that the expected discounted profit for a bid-ding strategy (arbitrary set of action curves) can be derived by estimating futurespot prices. The dependency is as follows: With estimated future spot prices, thepower outputs and revenue for every bidding strategy are given because every sup-ply curve defines the agents power output for an estimated future spot price. Withthe estimated power outputs and an appropriate cost model, the agents can alsocalculate the cost for every bidding strategy according to the cost equations in 3.3.By knowing the revenue and costs, the agents derive the profit for these strate-

gies. The price predictions also incorporate the effects of market-power. The pricepredictions become more accurate as more market outcomes are observed.

The algorithm introduced in this thesis is similar to Model Predictive Controlin a way that a model generates predictions and the input (action curve) is set thata given performance criterion (the reward) is maximized. The model predictivebidding learning algorithm is based on and incorporates the following properties:

The learning process is price driven. The agents learn to forecast future spot

1In this context, the Q-values are defined as the discounted expected rewards if an agent usesa certain action in a certain state at time t and follows in an optimal way thereafter. Following in

an optimal way thereafter means that the agent will choose the actions with the highest expectedprofit at t + 1 and thereafter

28


40/158

Chapter 4. Learning-process of the agents: The methodology

prices based on exogenous variables and exogenous variables predictions bymeasuring the sensitivities of these factors on spot prices. The price estimatesthereby incorporate market-power effects.

The agents measure their market-power based on the variations in prices notexplained by the exogenous factors. The measurement of how much a cer-tain action alters the predicted spot price outcome is based on reinforcementlearning.

Based on the spot price predictions, the agents use an optimization routineto choose the supply curves which maximize the expected discounted futureprofits.

The agents optimize over multiple steps in the future. In most classifier basedmodels the agents are only optimizing on a single step basis, which means theymaximize the profit for time t + 1. A multiple step optimization is important

to model the storage agents which have to buy electricity when the expectedspot price is low and sell when it is high. In a single-step optimization, thestorage agents would want to sell as much energy as they can for the next timestep and never buy energy. A multi step optimization is also important forthe inelastic bidders due to their ramping constraints. For them, maximizingonly the front hour can result in unfavorable outcomes.

A simple example explains the reasoning: It is assumed that the expectedspot price will be low at t + 1 where t is the current time step but very high att + 2 and thereafter. If an inelastic bidder is maximizing the reward for t + 1,he would decrease the production level at t + 1. If instead the inelastic bidderuses a multiple period optimization, he might start to increase the productionlevel at t + 1 to be in a high power output level at time t + 2, even if it results

in a loss at t + 1. This ramping scheduling can result in a higher discountedprofit over multiple time steps.

In the following, the advantages of predictive bidding over classifier based learn-ing systems and Q-learning are summarized:

As P. Lanzi highlights in [42], learning classifier systems in general do notrequire that the agents have a distinct goal. LCS might be very general andnot expressible in terms of an optimization problem. In XSC for instance, theagents choose preferably classifiers with a high fitness. The fitness increaseswith a high reward prediction accuracy rather than the absolute value of re-

ward prediction. The agents are therefore not primarily trying to maximizingtheir profits. What they try to maximize is hard to assess. With model pre-dictive bidding, it is clearly defined that the ultimate goal of the agents is tomaximize their profit.

The learning algorithm in LCS is based on trial and error and on evolutionof the classifiers. The outcome of LCS model depends highly on randomnessand is not reconstructible. It is therefore hard to draw conclusions from anLCS model. Furthermore, the outcome from two simulations with identicalenvironments can be very different. In a simulation outcome with an LCSmodel, it is hard to answer why an agent did a certain action in a certainsituation because the agents decision can be based on pure randomness. In

model predictive bidding there is much less randomness; it is therefore easierto draw conclusions about the agents behavior. Additionally, a simulation

29


41/158

Chapter 4. Learning-process of the agents: The methodology

outcome is reconstructible since the agents are always acting the same wayunder the same environmental input.

Not all combinations of supply curves for every state/environmental inputhave to be tried by the agents to experience a relationship between spot priceand state/environmental input as in classifier based systems and Q-learning.The amount of combinations between these factors are manifold with severalexogenous factors. The computation time to discover all these combinationsrequires a high computational effort. In model predictive bidding the agentsonly estimate the regression coefficients of the price predictor model (the linearsensitivities of each exogenous factor on spot prices) and what how a certainsupply curve influences the spot prices (see section 4.3 on the definition ofprice adjusting values) instead of trying all state, environmental input andaction combinations. This enables the agents to learn much faster.

Q-learning and learning classifier systems are very data intense. All the Q-

values for the different combinations of states/environmental inputs have tobe saved in lookup-tables. This very quickly looses viability with increasingenvironmental inputs. For classifier systems, the classifier population becomesvery big if the agents make precise reward predictions. In contrast, only thesensitivities of the exogenous factors on market prices and the price adjustingvalues (pavs) have to be stored in model predictive bidding.

If the agents have no Q-values or matching classifiers for an environmentalinput/state combination, they do not have a reasonable basis to make a de-cision. In these cases, a random action is proposed. Under model predictivelearning the agents are able to make reasonable predictions once they dis-covered the regression coefficients and are able to act on any combination of

environmental input/state. In model predictive learning the simulation breaks the effect of spot price

variations down to the different exogenous factors. Therefore, conclusionscan be drawn on how much an exogenous factor influences the spot price.Furthermore, the model allows indicating the market-power of each agent bythe pav values. This decomposition of spot price variation is not possible withQ-learning or classifier based learning systems.

4.1 Overall implemented structure

Figure 4.1 shows the structure of the implemented learning methodology. Eachagent uses a price predictor t

ma wehinger

Documents