domestic water heaters power optimization using fuzzified rl

8/12/2019 Domestic Water Heaters Power Optimization Using Fuzzified RL

1/24

Domestic Water HeatersPower Optimization Using

Fuzzified RLBy: Khalid Al-jabery


2/24

Outlines

Problem Description Previous Research

Dynamic Programing (DP) and Adaptive DP The Proposed Approach Algorithm Results Discussion


3/24

Problem Description (1)


4/24

Definition How to control the operation of water heaters in

order to minimize peak grid load demand andmaintain worm water with temperature aboveor equal to threshold delivered to the client?

Therefore; There are 3 variables defining thesystem:

1. The temperature of the Water supplied.

2. The current Grid demand3. The instantaneous rate of hot waterconsumption. (User Demand)


5/24

Some Previous Research Demand Side Management, by M.H. Nehrir et. al. 1998.

Applied on a block of dwellings. Need connections among users. Try to reduce the power consumed regardless of the user satisfaction

Control Strategy for Domestic WHs during Peak Periods, by AlanMoreau , 2011.

Based on timing to control the operation of the WHs. Highly depends on the size of the WH tank. Try to reduce the power consumed regardless of the user satisfaction

Demand Side Management using BPSO, by Sepulveda et. al. 2010. Required communications and synchronization among dwellings.

According to the simulation results it doesnt show any improvementin power optimization.


6/24

Dynamic Programing (DP) and Adaptive DP

DP ADP

Adaptive DynamicProgramming ,

, , [ , , + ]| |=

Dynamic Programming , 1 , +

[ , , + ( , )


7/24

The Proposed Approach

1. Define the system states by the linguisticdescription of the control variables. This wasachieved by using Fuzzy Logic.

2. Derive Markov chain process based on the states

and the available actions.3. Using Q-learning with discounted reward to trainthe system in order to reach the near optimalpolicy.

4. The Training algorithm is designed to providebalance between power optimization andcustomer satisfaction.


8/24

Algorithm1. Initialize Q-factors (Ns,Na)=0;2. Initialize state (S) and action (a);3. Repeat (4 to 9) until Stopping condition; 4. Read System Variables ;5. Select new Action =a ;6. Calculate Fuzzy membership ;7. Determine next State =s ;8. Calculate immediate reward ;9. Update Q(s,a) according to Q-learning algorithm;

10. Find Suboptimal policy;


9/24

Sample code O/P


10/24

Result-1Itrs. S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18

10 1 2 2 1 1 2 1 2 2 1 2 2 1 2 2 1 2 2

10 2 2 2 1 2 2 1 2 1 2 2 2 2 2 2 2 2 2

30 2 2 2 1 2 2 1 1 2 2 2 2 2 2 2 1 2 2

30 1 2 2 1 2 1 1 1 2 2 2 2 2 2 2 1 2 2

100 1 2 2 1 1 2 1 1 2 1 2 2 1 2 2 2 2 2

100 1 2 2 1 1 2 1 1 2 1 2 2 1 2 2 1 2 2

States where the grid load is Low States where the grid load is High

Th L M H L M H L M H L M H L M H L M H

W L L L M M M H H H L L L M M M H H H

GL L L L L L L L L L H H H H H H H H H

Actions =0.85


11/24

Results-2Itrs. S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18

10 1 2 2 1 2 2 1 2 2 2 2 2 2 2 2 1 2 2

10 2 2 2 1 1 2 1 2 2 2 2 2 1 2 2 1 2 2

30 2 2 2 1 1 2 1 1 2 1 2 2 2 2 2 1 2 2

30 2 2 2 1 1 2 1 1 1 1 2 2 2 2 2 1 2 2

100 1 2 2 1 1 2 1 1 1 1 2 2 2 2 2 1 2 2

100 1 2 2 1 1 2 1 1 1 1 2 2 2 2 2 1 2 2

States where the grid load is Low States where the grid load is High

Th L M H L M H L M H L M H L M H L M H

W L L L M M M H H H L L L M M M H H H

GL L L L L L L L L L H H H H H H H H H

Actions =0.9


12/24

The Effect of ( )

Itrs.S1

S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18

10 1 2 2 1 1 2 1 2 2 1 2 2 1 2 2 1 2 2 0.85

10 2 2 2 1 1 2 1 2 2 2 2 2 1 2 2 1 2 2 0.9

10 2 2 2 1 2 2 1 2 1 2 2 2 2 2 2 2 2 2 0.85

10 2 2 2 1 1 2 1 2 2 2 2 2 1 2 2 1 2 20.9

30 2 2 2 1 2 2 1 1 2 2 2 2 2 2 2 1 2 2 0.85

30 2 2 2 1 1 2 1 1 2 1 2 2 2 2 2 1 2 2 0.9

30 1 2 2 1 2 1 1 1 2 2 2 2 2 2 2 1 2 2 0.85

30 2 2 2 1 1 2 1 1 1 1 2 2 2 2 2 1 2 2 0.9

100 1 2 2 1 1 2 1 1 2 1 2 2 1 2 2 2 2 2 0.85

100 1 2 2 1 1 2 1 1 1 1 2 2 2 2 2 1 2 2 0.9

100 1 2 2 1 1 2 1 1 2 1 2 2 1 2 2 1 2 2 0.85

100 1 2 2 1 1 2 1 1 1 1 2 2 2 2 2 1 2 2 0.9


13/24

System Output Graph


14/24


15/24


16/24


17/24

System Variables (2)

Hot water demand Energy Distribution

Water Temperature


18/24

Fuzzy Memberships

0

0.20.4

0.6

0.8

1

1.2

90 95 100 105 110 115 120 125 130 135 140 145 150 155 160

Temperature in F

Water Temperature

LOW

Medium

High

0

0.2

0.4

0.6

0.8

1

1.2

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47

Samples Every 30 Mins.

Grid Load

LOW

High

0

0.2

0.4

0.6

0.8

1

1.2

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Gallons /sample

Water Consumption Rate

LOW

Medium

High


19/24

States RepresentationS1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18

L M H L M H L M H L M H L M H L M H

L L L M M M H H H L L L M M M H H H

L L L L L L L L L H H H H H H H H H

L= Low, M=Medium , H= High: e.g: S9 is the state when the water temperature high,the user hot water demand is high and the Grid load is low.

Water Consumption

Temperature

Grid Demand


20/24

Markov Chain for Action1

S1

S2

S3

S4

S5

S6

S7

S8

S9

S10

S11

S12

S13

S14

S15

S16

S17

S18


21/24

Transition Rewards (off action)


22/24

Updating Q The update is according to the eqn.: , 1 , + [ , , +

max , ] Where:S: system old state,a : action selected at S,

: learning rate, here = log(k)/K : is the discount factor,

Q : is the q-factor that associated with each(s,a) pair.

R (s,a,s ) : is the reward for moving from state s to state s using action a.fm : is the fuzzy membership value of the state preferred property *

b: is the action selected at state S and has the max Q-factor.

* The proff ered proper ty means the property that defi ne whether the system is going to be rewar ded orpunished wil l be explain ed in TRM s Der ivation.


23/24

References1. Control Strategy for Domestic Water Heaters during Peak Periods and its

Impact on the Demand for Electricity By A. Moreau, Canada. 2011. 2. Measurement of Domestic Hot Water Consumption in Dwellings, DEFRA

Department for Environment Food and Rural affairs UK.

3. Handbook of Intelligent Control By D. Sofga.

4. Simulation Based Optimization A. Gosavi.

5. Dynamic Programming and Optimal Control, D. Bertskas.

6. A Novel Demand Side Management Program using Water Heaters andParticle Swarm Optimization, A. Sepulveda, 2010.

7. A Reinforcement Learning-Based Architecture for Fuzzy Logic Control,

Hamid R. Berenji.8. A customer-interactive electric water heater demand-side management

strategy using fuzzy logic., M.H. Nehrir, 1998.


24/24

domestic water heaters power optimization using fuzzified rl

Documents