Two time scales stochastic dynamic optimization for microgrids controlPierre Carpentier, Jean-Philippe Chancelier, Michel De Lara & Tristan Rigaut

ENSTA, ENPC, Efficacity

Stochastic optimization for storage management in microgridsEnergy storage is often used to mitigate the uncertainty related to energy demand and production.

Renewable energies are especially inttermittent and some electrical demands particularly uncertain.Stochastic optimization is a tool to ensure supply demand balance without wasting energy or curtail-ing demand.Hierarchical control architectureMicrogrids control architecture is divided into multiple layers to handle multiple time steps. Primarylevel is devoted to ensure the microgrid voltage stability, secondary level is devoted to the energy man-agement while the tertiary level is devoted to economic reliability.

Primary

Secondary

Tertiary

2∆T

2∆T. . .

2∆t∆t

∆T

∆T . . .1 min

∆T

Target

1 s 1 s1 s

∆t ∆t∆t

InfoInfo

Tertiary level

Secondary level

A superior level purpose is to provide a target that the next inferior level has to reach at the end of itshorizon while reaching its own objective.

Example: intraday energy arbitrage and battery long term aging managementWe study hereby a two time scales problem to manage a battery ageing over the long term while usingit everyday for intraday energy arbitrage. The example we use is a battery in a subway station able torecover subways braking energy and to provide power to the station.

SD ⊕ B

Es El

Station node

• D: Demand station

• Es: From grid to station

•: Discharge battery

Subways node

• B: Braking

• El: From grid to battery

•⊕: Charge battery

Medium time scale: energy arbitrage

• Time step ∆t = 1 min• Horizon M = 24h• Decision: battery charge/discharge• Objective: energy cost minimization

d, 0 . . . d,M1 min1 min

For a given charge/discharge decision Ud,m:state of charge S changes

Sd,m+1 = Sd,m −1

ρdU−d,m︸ ︷︷ ︸

+ ρcsat(Sd,m,U+d,m,Bd,m+1)︸ ︷︷ ︸

⊕

state of health H decreases

Hd,m+1 = Hd,m −1

ρdU−d,m − ρcsat(Sd,m,U

+d,m,Bd,m+1)

we save energy and moneyped,m(Es

d,m+1 + Eld,m+1 −Dd,m+1︸ ︷︷ ︸

Saved energy

) with ped,m the cost of

electricity on day d at minute m

Large time scale: long term ageing

• Time step ∆T = 1 day

• Horizon D = 10 years

• Decision: battery replacement

• Objective: long term profitability

2∆T∆T0 . . . D24h24h

For a given replacement decision Rd:battery capacity changes

Cd+1 =

Rd, if Rd > 0Cd, otherwise

we pay a replacement cost P bdRd at uncertain

market prices P bd .

The link between the two scales is the global state at the beginning of each day:

(Sd+1,0,Hd+1,0,Cd+1) = ψd((Sd,M ,Hd,M ,Cd,Rd))

Stating a two time scales optimization problemThe previous example displays 2 state variables that changes every minutes and 1 state variable that

changes every day. We make 1 charge/discharge decision every minute and 1 capacity replacementdecision everyday. We can formulate a two-time scales stochastic optimization problem.

General two time scales stochastic optimization problemWe minimize medium and large scale costs taking into account the medium and large scales dynamics,f index means fast (medium scale), s index means slow (large scale):

minXf ,Xs,Uf ,Us

E[D−1∑d=0

(M−1∑m=0

Lfd,m(Xfd,m,U

fd,m,W

fd,m+1)

)+ Lsd(X

sd,U

sd,W

sd+1,X

fd,0,U

f

d,:,Wf

d,:)]

Xfd,m+1 = F f

d,m(Xfd,m,U

fd,m,W

fd,m+1)

Xfd,0 = ψd(X

sd,X

fd−1,M)

Xsd+1 = F s

d (Xsd,U

sd,W

sd+1,X

fd,0,U

f

d,:,Wf

d,:)

σ(U fd,m) ⊂ Fd,m

σ(U sd) ⊂ Fd,M

we can reformulate the problem with

Xd = (Xfd,0,X

sd), U d = (U f

d,:,Usd), W d = (W f

d−1,:,Wsd)

minX ,U

E[D−1∑d=0

Ld(Xd,U d,W d+1)]

Xd+1 = Fd(Xd,U d,W d+1)

σ(U fd,m) ⊂ Fd,m

σ(U sd) ⊂ Fd,M

The last two constraints, called non anticipativity constraints, states that we make our decisions knowingonly the past realizations of the noises.

Fd,m = σ

Wfd′,m′

, d′<d, m′≤M+1

W sd′, d′≤d

Wfd,m′

, m′≤m

= σ

(previous days fast noisesprevious days slow noises

current day previous minutes fast noises

)

Resolution method: two time scales stochastic dynamic programmingThe resolution method is based on primal decomposition and dynamic programming. We show thatwhen the (W d)d=0,...,D are stage(day)wise independent we can define inductively value functions:

Daily value function

Vd(xd) = minXd+1

[ intraday problem︷ ︸︸ ︷φd(xd, [Xd+1])

+

expected next day value︷ ︸︸ ︷EVd+1(Xd+1)

]s.t Xd+1 σ(Xd,W d+1)

Intraday arbitrage problem with almost surestochastic target

φd(xd, [Xd+1]) = minUd

E Ld(xd,U d,W d+1)

s.t Fd(Xd,U d,W d+1) = Xd+1

U fd,m σ(Xd,W

fd,1:m)

U sd σ(Xd,W

fd,1:M)

U d = (U fd,:,U

sd)

Xd = xd

We use the notation f ([W ]) to emphasize that f ’s domain is L0(Ω,F ,P). Solving these two problemsdirectly is computationally very expensive as the target to reach at the end of the day is measurable w.r.tall the intraday noises. We use 3 tricks to simplify the computation:

• We exploit the problem periodicity (∀d, φd = φ0)

• We exploit value functions monotonicity to relax the target Fd(Xd,Ud,W d+1) ≥Xd+1

• We simplify measurability (Xd+1 σ(Xd)) meaning that we choose the final state target at thebeginning of the day! We loose optimality with this trick.

Numerical applicationWe manage batteries in a subway station over 7 years and want to maximize the Net Present Value(NPV). We have to choose every month a maximum number of daily cycles for the current battery(orange horizontal line). We can decide to buy a new battery every month (green vertical lines are re-placements). We use synthetic scenarios for the price of batteries on the market. We manage chargedischarge of the battery every 15 minutes, giving an effective number of cycles for each day (blue dots).

The algorithm is implemented in Julia with a Core I7, 1.7 Ghz, 8Go ram + 12Go swap SSD. It requires16 min to compute daily value functions and intraday problem value offline. It then takes 20 ms to makea charge/discharge decision online every 15 minutes and to make an ageing daily target decision at thebeginning of every month. We obtain a +128k euros upper bound for the NPV.

ConclusionThis method allows to model two time scales stochastic optimization problems with a proper mod-

elization of the information for the decision maker. The resolution method we present decouples thetwo time scales and could be applied to microgrid hierarchical management, sizing of storage or energystorage investment and long term ageing strategies.