PROSPECTS OF GRADIENT METHODS FOR NONLINEAR CONTROL

Ivo BUKOVSKÝ1

Jiří BÍLA1

Noriasu HOMMA2

Ricardo RODRIGUEZ1

1Czech Technical University in Prague

2Tohoku University, Japan

PROSPECTS OF GRADIENT METHODS FOR NONLINEAR CONTROL

• We consider sample-by-sample adaptation of discrete-time models and controllers by gradient descent

2( )( 1) ( ) ;

... adaptable parameter of a model or controler

kk ki i

i

thi

Qw w i

w

w i

weight update system

• Stability monitoring and maintenance of weight update system of adaptively tuned models and controllers significantly contributes to a stable and convergent control loop

PROSPECTS OF GRADIENT METHODS FOR NONLINEAR CONTROL

PROSPECTS OF GRADIENT METHODS FOR NONLINEAR CONTROL

• In the paper, we introduce derivation of stability condition for gradient-descent tuned models and controllers

• The approach is valid for models and controllers that are nonlinear (incl. linear), but they are linear in parameters– Not suitable for conventional neural networks (MLP,

RBF)– Suitable for Higher-Order Neural Units (HONU, also

known as polynomial neural networks) (not limited to)

PROSPECTS OF GRADIENT METHODS FOR NONLINEAR CONTROL

Further in this presentation

• Fundamental gradient descent schemes for adaptive identification and control

• Static or dynamic Higher Order Neural Units (HONU)

• Stability conditions for static and dynamic HONU and its maintenance at every adaptation step

• Demonstration of achievements with ONU( NOx prediction – EME I, lung motion prediction, nonlinear control loop of a laboratory system)

Plant

Adaptive model-linear

- neural network,

+-

2( )( 1) ( ) ;

kk ki i

i

ew w i

w

( )ku( )krealy

Fundamental gradient descent schemes for adaptive identification and control

Plant Identification by Gradient Descent

( )ke

( )ky

... neural weights

(adaptable parameter)

... control variableu

w

weight update system

(Základní schemata adaptivní identifikace a řízení gradientovými metodami)

Automatické ladění adaptivního stavového regulátoru

Regulovanásoustava

Adaptivní regulátor-lineární

- polynomiální-- klasická neuronová síť

2( )( 1) ( ) ;

kk ki i

i

ew w i

w

( )kv ( )krealy

Referenční model (požadované chování regulované soustavy)

+-

Žádaná hodnota

+-

( )kdesiredy

... adaptovatelný parametr

(váhy u neuronových sítí)

... žádaná hodnota

w

v

Systém adaptovaných

vah

( )ke

Žádaný průběh chování

Fundamental gradient descent schemes for adaptive identification and control (continue)

Tuning of Adaptive Controller in a Feedback Control Loop with Gradient Descent

Plantadaptive controller

- linear PID - neural network,

2( )( 1) ( ) ;

kk ki i

i

ew w i

w

( )kv

( )krealy

Model of desired behavior

+-

( )kdesiredy

... neural weight ,

(adaptable parameter)

... desired value

w

v

+-

2( )( 1) ( ) ;

kk ki i

i

ew w i

w

Plant

Adaptive model-linear

- neural network,

+-

( )ku( )krealy

Fundamental gradient descent schemes for adaptive identification and control (continue)

Updating Control Inputs Directly by Gradient Descent

( )ke

( )ky

2( )( 1) ( ) c

kck ki

eu u

w

+-

( )kdesiredy

eC(k)

The question is:

• How do we assure stability of nonlinear adaptive control loop?• The ways is to assure stability and convergence of adaptive

components in a control loop (plant model + controller)• What nonlinear model to use?

• MLP or RBF networks as models and controllers– Not linear in parameters– Guaranteeing stability is complicated (not

suitable for undergraduate level, difficult for PhD students from non-heavy-math schools)

– Guaranteeing stability is complicated and theoretically heavy for practicioners (thus not attractive for practice)

Static & Dynamic Higher-Order Neural Units

How do we assure stability of the nonlinear adaptive control loop? What model to choose?

Static & Dynamic Higher-Order Neural Units

How do we assure stability of the nonlinear adaptive control loop? What model to choose?

2( )( 1) ( ) ;

kk ki i

i

ew w i

w

Weight-update system:

Example of 2nd-order HONU: 1

( ...)

( ...)

( )

k

k

k

y

y

u

( )kx

( )sk ny

0

r rn n

i j iji j i

x x w

20 0 0 1 0 2 i j ny x x x x x x x x x 0,0 0,1 0,2 i,j n,nw w w w w

“axis of adapted neural weights”

LNU

HONU

convetional NN

2( )k

k

e

0

Approximation strength of neural networks can be improved by adding more neurons or even layers, GA, PSO,…

Static & Dynamic Higher-Order Neural Units (continue)

Sketch of optimization error surfacesLinear x MLP Networks x HONU

Static & Dynamic Higher-Order Neural Units (continue)

Static MLP vs. QNU as MISO models of hot steam turbine averaged data (“steady states”, batch training by Levenberg-Marquardt)

• double hidden layer FFNN

• single hidden layer FFNN

• static QNU• measured data

Static & Dynamic Higher-Order Neural Units (continue)Respiration time series: Training Accuracy for Predicting Exhalation Time -Instances of trained neural architectures trained from different initial conditions by L-M algorithm

2-hidden-layer static MLPs (static feedforward networks)

1-hidden-layer static MLPs (static feedforward networks)

static

QNUs

0 50 100 150

0

20

40

60

80

100

trénovacích epoch

JRNN

JDLNU

JDQNU

trénovacích epoch

Trénování predikce Mackey-

Glass

0 50 100 150

0

20

40

60

80

100

trénovacích epoch

JDQNU

JDLNU

JRNN

Trénování predikce polohy plic0 20 40 60

0

20

40

60

80

trénovacích epoch

JRNN

JRNN

JDQNU

Trénování predikce nelineárního periodického

signálu

0 50 100 150

0

20

40

60

80

100

trénovacích epoch

JRNN

JDLNU

JDQNU

trénovacích epoch

Static & Dynamic Higher-Order Neural Units (continue)

0,0 0 0 0,1 0 1 0,2 0 2

2, ,... i j i j n n n

y w x x w x x w x x

w x x w x

0,0

0,1

0,20 0 0 1 0 2

,

n n

n n

w

w

wy x x x x x x x x

w

rowx colW

1( ...)

( ...)

( )

k

k

k

y

y

u

( )kx

( )sk ny

0

r rn n

i j iji j i

x x w

Static & Dynamic Higher-Order Neural Units (continue)

Stability of weight-update system

• Condition for STATIC HONU

• Condition for DYNAMICAL HONU

( ) ( ) 1k k 1 M colx rowx

( )( ) ( ) ( ) 1

kk n k kse

rowx1 M colx rowx

colW

,

HONU

1( ...)

( ...)

( )

k

k

k

y

y

u

( )kx

0 50 100 150 200 250 300 350 400-2

-1

0

1One Epoch of GD Adaptation of Recurrent QNU to Predict MacKey-Glass Equation (training data vs. neural output)

0 50 100 150 200 250 300 350 400-0.5

0

0.5Prediciton Error during the Epoch of Adaptation

0 50 100 150 200 250 300 350 4000.98

1

1.02

1.04

k

Spectral Radius during the Epoch (stability of weight update system at each adaptation step)

0 100 200 300 400 500 600 700 800

-2

0

2

k

GD Adaptation of Recurrent QNU to Predict MacKey-Glass Equation (training data vs. neural output)

0 100 200 300 400 500 600 700 800-10

0

10

k

Prediciton Error during Adaptation

0 100 200 300 400 500 600 700 800

1

1.5

2

k

Spectral Radius during Adaptation (stability of weight update system at each adaptation step)

600 620 640 660 680 700 720 740 760 780 800

-2

0

2

k

GD Adaptation of Recurrent QNU to Predict MacKey-Glass Equation (training data vs. neural output)

600 620 640 660 680 700 720 740 760 780 800-10

0

10

k

Prediciton Error during Adaptation

600 620 640 660 680 700 720 740 760 780 800

1

1.05

1.1

k

Spectral Radius during Adaptation (stability of weight update system at each adaptation step)

Achievements with QNU

250 300 350 400

-1

-0.5

0

0.5

1

1.5

t [min]

NOx,CO prediction – EME I

trénování testování

Obr. 1: Dobře natrénovaná síť TptRNN pro 3-minutovou predikci klouzavých 3-minutových průměrů NOx, externí měřené vstupy jsou klapky a výkon , (klouzavé průměry se počítají jako průměry předchozích, současných a následujících hodnot, při intervalu predikce 3 minuty to znamená, že externí vstupy jsou již dostupné ale model v principu predikuje 3-minutový průměr který má být za 2 minuty), včase cca 415 ignoruje výpadek měření NOx a výstup modelu dobře nahrazuje měření.

Lung Tumor Motion Prediction

0 500 1000 1500 2000 2500 3000 3500-2

-1

0

1

2

Late

ral a

xis

[mm

]

0 500 1000 1500 2000 2500 3000 3500-10

-5

0

5

Ceph

aloc

auda

l axi

s [m

m]

0 500 1000 1500 2000 2500 3000 3500-2

-1

0

1

2

k

Ante

ropo

ster

ior A

xis

[mm

]

y1

y2

y3

20 40 60 80 100 120

t [sec]

-8

-6

-4

-2

0

2

4

6testing MAE= 0.853120295578 [mm], RMSE= 1.14143756682, treatment time = 86[sec], computing time= 83.385[sec]

20 40 60 80 100 120t [sec]

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0absolute value of prediction error

Lung Tumor Motion Prediction by static QNU

sampling 15 Hz, epochs=100, Ntrain=360, 492 neural weights

Lung Tumor Motion Prediction by static QNU

10^0 10^1 10^2

epochs

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

0.040 Averaged normalized SSE of Retrainings

Nonlinear Control Loop of a Laboratory System

[ ] Ladislav Smetana: Nonlinear Neuro-Controller for Automatic Control,Laboratory System, Master’s Thesis, Czech Tech. Univ. in Prague, 2008.

Nonlinear Control Loop of a Laboratory System

PID Control and Nonlinearity of the Plant

0 100 200 300 400 500 600-30

-25

-20

-15

-10

-5

0

t [s]

y [c

m]

Prubeh PID regulace v zavislosti na hloubce ponoru batyskafu, serizeno na hloubku 20 cm

5 cm

10 cm

15 cm20 cm

25 cm

0 100 200 300 400 500 600-40

-35

-30

-25

-20

-15

-10

-5

0

t [s]

y [c

m]

Prubeh PID regulace v zavislosti na hloubce ponoru batyskafu, serizeno na hloubku 10 cm

5 cm

10 cm

15 cm20 cm

25 cm

Tunned PID controller for 10 cm

30

Tunned PID controller for 20 cm

Nonlinear Control Loop of a Laboratory System

0 100 200 300 400 500 600-30

-25

-20

-15

-10

-5

0

t [s]

y [c

m]

Prubeh PID regulace v zavislosti na hloubce ponoru batyskafu, serizeno na hloubku 20 cm

5 cm

10 cm

15 cm20 cm

25 cm

0 100 200 300 400 500 600-40

-35

-30

-25

-20

-15

-10

-5

0

t [s]

y [c

m]

Prubeh PID regulace v zavislosti na hloubce ponoru batyskafu, serizeno na hloubku 10 cm

5 cm

10 cm

15 cm20 cm

25 cm

310 20 40 60 80 100 120 140

-25

-20

-15

-10

-5

0

t [s]

y [c

m]

Prubeh regulace neuro-regulatoru zavislosti na hloubce ponoru batyskafu

5 cm

10 cm

15 cm20 cm

25 cm

QNU as Adaptive Controller (simplest gradient descent)

Linear PID

Nonlinear Control Loop of a Laboratory System

False Neighbor Analysis is a single-scale analysis

x yyf )(x

( )

( ) ( )

i

j i

x

x x

( )

( ) ( )

i

j i

y

y y

To train neural networks , input (state) vector must be estimated to minimize uncertainty in training data

Děkuji za pozornost

y=f(x)x input data y output data

False Neighbors

1 2 IF AND

THEN and are False Neighbors

=> How much is correct Rx and Ry? - we do not know

=> Let's characterize false neighbors over whole intervals

of Rx and Ry, an

x yR y y R 1 2

1 2

x x

x x

d not just for their single setup

False Neighbor Analysis is a single-scale analysis

Slope of FN in Log-Log plot

FN = 4.2239*log2(id) - 4.5879

-2

0

2

4

6

8

1 1.5 2 2.5 3

log2(id)

log2(FN) Linear (log2(FN))

q(k ) c r (k )H

( )

( )

log

log log

k

k

q

c H r

MULTI-SCALE ANALYSIS approach (MSA)

number of false neighbours on a main diagonal

0

50

100

150

1 2 3 4 5 6

id...index of a diagonal cell

FN

• To characterize a system over the range of setups

• Power law

•What is the fundamental idea?

MULTI-SCALE ANALYSIS approach (MSA)• What is the fundamental idea?

q(k ) c r (k )H

q … quantityH … characterizing exponentr(k) … discretely growing radius

r(k)=2,4,8

•To characterize a system over the range of intervals•The power-law concept

MULTI-SCALE ANALYSIS approach (MSA)• What is the fundamental idea?

q(k ) c r (k )H

q … quantityH … characterizing exponentr(k) … discretely growing radius

r(k)=2,4,8

•To characterize a system over the range of intervals

•The power-law concept

MULTI-SCALE ANALYSIS approach (MSA)• What is the fundamental idea?

k r(k) q A q B

1 2 4 22 4 13 113 8 44 44

r(k)=2,4,8

log2(qB) = 2.2297*log2(r) - 1.1531

log2(qA) = 1.7297*log2(r) + 0.2605

0.9

1.9

2.9

3.9

4.9

1 1.5 2 2.5 3log2(r(k))

q(k ) c r (k )H

MULTI-SCALE ANALYSIS approach (MSA) (cont.)

• How can MSA help to create better neural network models?

j =1 j =2 j =3 j =4 j =5

i=1

max FN (highest chance that y1≠y2

when x1=x2 )

i=2 FN (2,2)

i=3 FN (3,3)

i=4 FN (4,4,)

i=5

min FN (lowest chance that y1≠y2

when x1=x2 )

FN (i ,j ) … count of False Neighbors for Rx (i ) and Ry ( j )

Rx(i

)

Ry ( j )

Smallest Rx - maximum of different states of a system

Largest Rx - minimum of different states of a system

Smallest Ry - maximum of recognized different outputs

Largest Ry - minimum of recognized different outputs

FN decrease

FN decreases

ffecf F

N d

ecre

ase

ffecf F

N d

ecre

ase

( )f yxj

i

False Neighbors Matrix:

Multiscale False Neighbor Approach

MULTI-SCALE ANALYSIS approach (MSA) (cont.)

• What are other potentrials for MSA for signal processing?

• MSA based signal processing

• Variance Fractal Dimension Trajectory (VFDT)

• Mutual Information

– Multiscale approach to calculate mutual information itself

– Mutual information of VFDT processed signals

• Everywhere, where a common analysis is subject to a

single-parameter setup and changing the setup disqualifies

the analysis results.