Parameter Redundancy and Identifiability

Diana Cole and Byron MorganUniversity of Kent

Initial work supported by an EPSRC grant to the National Centre for Statistical Ecology

Herring Gull Wandering Albatross Striped Sea BassGreat Crested Newts

Introduction

• If a model is parameter redundant you cannot estimate all the parameters in the model.

• Parameter redundancy equivalent to non-identifiability of the parameters.

• A model that is not parameter redundant will be identifiable somewhere (could be globally or locally identifiable).

• Parameter redundancy can be detected by symbolic algebra.• Ecological models and models in other areas are getting more

complex – then computers cannot do the symbolic algebra and numerical methods are used instead.

• In this talk we show some of the tools that can be used to overcome this problem.

Example 1- Cormack Jolly Seber (CJS) ModelCapture-Recature

Herring Gulls (Larus argentatus) capture-recapture data for 1983 to 1986 (Lebreton, et al 1995)

Numbers Ringed:

Numbers Recaptured:

111

123

78

R

9100

31030

2467

N

83

84

85Ringing yr

83

84

85

Recapture yr

84 85 86

Example 1- Cormack Jolly Seber (CJS) Model

i – probability a bird survives from occasion i to i+1pi – probability a bird is recaptured on occasion i = [1, 2, 3, p2, p3, p4 ]

recapture probabilities

Can only ever estimate 3 p4 - model is parameter redundant or non-identifiable.

etc1

00

0

22

43

433232

433221322121

pp

p

ppp

pppppp

Q

9100

31030

2467

N

r cn

ij ijicr c

ij

n

i

NRn

ij ij

n

i

n

ij

Nij QQL

11

1

111

123

78

R

Derivative Method (Catchpole and Morgan, 1997)

Calculate the derivative matrix D

rank(D) = 5 rank(D) = 5 Number estimable parameters = rank(D). Deficiency = p – rank(D) no. est. pars = 5, deficiency = 6 – 5 = 1

T

p

pp

p

ppp

pp

p

)ln(

)ln(

)ln(

)ln(

)ln(

)ln(

43

4332

32

433221

3221

21

κ

432321 ppp

j

i

D

T

ppppppR

pppR

ppR

pR

)1( 4332213221211

4332211

32211

211

μ

i

j

D

323211

4232112211

433211321111

432211

4323113211

432321322121

00

0

00

0

ppR

ppRpR

ppRpRR

pppR

pppRppR

pppRppRpR

14

14

14

13

13

13

13

12

12

12

13

13

13

12

12

12

12

11

11

11

000

00

000

000

00

000

ppp

pppp

ppp

Derivative or Jacobian Rank Test

• Jacobian is the transpose of the derivative matrix, so two are interchangeable.

• Uses of rank test:– Catchpole and Morgan (1997) exponential family models,

mostly used in ecological statistics.– Rothenberg (1971) original general use, examples

econometrics.– Goodman (1974) latent class models.– Sharpio (1986) non-linear regression models.– Pohjanpalo (1982) first use for compartment models.

Derivative or Jacobian Rank Test• The key to the symbolic method for detecting parameter

redundancy is to find a derivative matrix and its rank.• Models are getting more complex.• The derivative matrix is therefore structurally more complex.• Maple runs out of memory calculating the rank.• Examples: Hunter and Caswell (2009), Jiang et al (2007)

• How do you proceed?– Numerically – but only valid for specific value of parameters.

But can’t find combinations of parameters you can estimate. Not possible to generalise results.

– Symbolically – involves extending the theory, again it involves a derivative matrix and its rank, but the derivative matrix is structurally simpler.

Wandering AlbatrossMulti-state models for sea birds

Striped Sea BassAge-dependent tag-return

models for fish

Exhaustive Summaries• An exhaustive summary, , is a vector that uniquely defines

the model (Walter and Lecoutier, 1982).

• Derivative matrix

• r = Rank(D) is the number of estimable parameters in a model.• p parameters; d = p – r is the deficiency of the model (how

many parameters you cannot estimate). If d = 0 model is full rank (not parameter redundant , identifiable somewhere) . If d > 0 model is parameter redundant (non-identifiable).

• More than one exhaustive summary exists for a model• CJS Example:

i

j

D

T

p

pp

p

ppp

pp

p

)ln(

)ln(

)ln(

)ln(

)ln(

)ln(

43

4332

32

433221

3221

21

κ

T

ppppppR

pppR

ppR

pR

)1( 4332213221211

4332211

32211

211

μ

Exhaustive Summaries• Choosing a simpler exhaustive summary will simplify the

derivative matrix.• CJS Example:

• Computer packages, such as Maple can find the symbolic rank of the derivative matrix if it is structurally simple.

• Exhaustive summaries can be simplified by any one-one transformation such as multiplying by a constant, taking logs, and removing repeated terms.

• A simpler exhaustive summary can also be found using reparameterisation.

323211

4232112211

433211321111

432211

4323113211

432321322121

00

0

00

0

ppR

ppRpR

ppRpRR

pppR

pppRppR

pppRppRpR

D

14

14

14

13

13

13

13

12

12

12

13

13

13

12

12

12

12

11

11

11

000

00

000

000

00

000

ppp

pppp

ppp

D

Methods For Use With Exhaustive SummariesWhat can you estimate?

(Catchpole and Morgan, 1998, developed separately for compartment models in Chappell and Gunn,1998 and Evans and Chappell , 2000

extended to exhaustive summaries in Cole and Morgan, 2009a)

• A model: p parameters, rank r, deficiency d = p – r• There will be d nonzero solutions to TD = 0. • Zeros in s indicate estimable parameters. • Example: CJS, regardless of which exhaustive summary is used

• Solve PDEs to find full set of estimable pars.

• Example: CJS, PDE:

Can estimate: 1, 2, p2, p3 and 3p4

djfp

i iij ,...,10

1

10000

4

3

pT

α

0434

3

p

ff

p

432321 ppp

Methods For Use With Exhaustive SummariesExtension Theorem

(Catchpole and Morgan, 1997 extended to exhaustive summaries in Cole and Morgan, 2009a)

• Suppose a model has exhaustive summary 1 and parameters 1.

• Now extend that model by adding extra exhaustive summary terms 2, and extra parameters 2. (eg. add more years of ringing/recovery) New model’s exhaustive summary is = [1 2]T and parameters are = [1 2]T.

• If D1 is full rank and D2 is full rank, the extended model will be full rank. The result can be further generalised by induction.

• Method can also be used for parameter redundant models by first rewriting the model in terms of its estimable set of parameters.

i

j

,1

,11

D

2

,1

,21

,2

,2

,1

,2

,1

,1

00 D

DDD

i

j

i

j

i

j

i

j

Methods For Use With Exhaustive Summaries The PLUR decomposition

• Write derivative matrix which is full rank r as D = PLUR (P is a square permutation matrix , L is a lower diagonal square matrix, with 1’s on the diagonal, U is an upper triangular square matrix, R is a matrix in reduced echelon form).

• If Det(U) = 0 at any point, model is parameter redundant at that point (as long as R is defined). The deficiency of U evaluated at that point is the deficiency of that nested model (Cole and Morgan, 2009a).

• Example 2: Ring-recovery model:

Rank(D) = 5

10000

01000

00100

00010

00001

R)1)(1(

)(Det2,11,112,11,1

2,11,1

aa

U

0)(DetIf 2,11,1 U

Therefore nested model is parameter redundant with deficiency 1

4)(Rank2,11,1

U

aa 12,11,1

aa

aaaaa

2,112,1

1,11,111,1

0Q

Finding simpler exhaustive summaries Reparameterisation

1. Choose a reparameterisation, s, that simplifies the model structure.CJS Model (revisited):

2. Reparameterise the exhaustive summary. Rewrite the exhaustive summary, (), in terms of the reparameterisation - (s).

43

32

32

21

21

5

4

3

2

1

p

p

p

p

p

s

s

s

s

s

s

)ln(

)ln(

)ln(

)ln(

)ln(

)ln(

)(

43

4332

32

433221

3221

21

p

pp

p

ppp

pp

p

θ

)ln(

)ln(

)ln(

)ln(

)ln(

)ln(

)(

5

54

3

542

32

1

s

ss

s

sss

ss

s

s

43

433232

433221322121

00

0

p

ppp

pppppp

Q

Reparameterisation3. Calculate the derivative matrix Ds.

4. The no. of estimable parameters = rank(Ds)

rank(Ds) = 5, no. est. pars = 5

5. If Ds is full rank ( Rank(Ds) = Dim(s) ) s = sre is a reduced-form exhaustive summary. If Ds is not full rank solve set of PDE to find a reduced-form exhaustive summary, sre.There are 5 si and the Rank(Ds) = 5, so Ds is full rank. s is a reduced-form exhaustive summary.

15

15

15

14

14

13

13

12

12

11

000

0000

0000

0000

00000

)(

sss

ss

ss

ss

s

s

s

i

js

D

.)(DimRankif

si

js

Reparameterisation

6. Use sre as an exhaustive summary.

A reduced-form exhaustive summary is

Rank(D2) = 5; 5 estimable parameters.Solve PDEs: estimable parameters are 1, 2, p2, p3 and 3p4

43

32

32

21

21

s

p

p

p

p

p

re

3

22

11

4

33

22

2

0000

000

000

0000

000

000

p

pp

pp

s

i

rejD

ReparameterisationExample 2

• Hunter and Caswell (2009) examine parameter redundancy of multi-state mark-recapture models, but cannot evaluate the symbolic rank of the derivative matrix (developed numerical method).

• 4 state breeding success model:

1)...()(

1

loglog

1211

1),(

1

1 1

4

1

4

1

),(),(,

rc

rc

mL

Trrcccc

Trrcr

N

r

N

rc i j

crij

crji

II

)1(0)1(0

0)1(0)1(

)1()1()1()1(

4422

3311

444333222111

444333222111

0000

0000

000

000

2

1

p

psurvival breeding given survival successful breeding recapture

Wandering Albatross

1 3

2 4

1 success

2 = failure

3 post-success

4 = post-failure

Reparameterisation

1. Choose a reparameterisation, s, that simplifies the model structure.

2. Rewrite the exhaustive summary, (), in terms of the reparameterisation - (s).

2

1

333

222

111

14

13

3

2

1

p

p

s

s

s

s

s

s

)1(

)1(

)1(

)(

121

21

211

2222

2221

1112

1111

pp

p

p

p

p

θ

)1(

)(

131321

146

132

145

131

sss

ss

ss

ss

ss

s

Reparameterisation3. Calculate the derivative matrix Ds.

4. The no. of estimable parameters =rank(Ds)

rank(Ds) = 12, no. est. pars = 12, deficiency = 14 – 12 = 2

5. If Ds is full rank s = sre is a reduced-form exhaustive summary. If Ds is not full rank solve set of PDE to find a reduced-form exhaustive summary, sre. Tre sssssssssssssssss 104934837141312116521 //

139

13145513

13131113

0000

)(000

)22(000

)(

ss

sssss

sssss

sD

i

js

s

.)(DimRankif

si

js

Reparameterisation Method

6. Use sre as an exhaustive summary.T

re pp

2244411333

4

4

3

3214433222111222111

s

Breeding Constraint

Survival Constraint

1= 2=

3= 4

1= 3,

2= 4

1= 2,

3= 4

1, 2,

3,4

1= 2= 3= 4 0 (8) 0 (9) 1 (9) 1 (11)

1= 3 ,2= 4 0 (9) 0 (10) 0 (10) 2 (12)

1= 2, 3= 4 0 (9) 0 (10) 1 (10) 1 (12)

1,2,3,4 0 (11) 0 (12) 0 (12) 2 (14)

Reparameterisation MethodFurther Examples

Multi-state models - general exhaustive summary has been developed if there is more than one observable state (Cole, 2009). Maple procedures for finding this exhaustive summary and the derivative matrix. Able to come up with general rules.

Jiang et al (2007) age-dependent fisheries model is more complex, but essentially uses reparameterisation method (Cole and Morgan, 2009b). Able to give general results, whereas Jiang et al (2007) result only applies for 3 years of data.

Wandering Albatross

Striped Sea Bass

Reparameterisation MethodFurther Examples

Multi-state analysis of Great Crested Newts. The parameter redundancy of the more complex models can be examined using the reparameterisation method to find a simpler exhaustive summary. This example consists of 2 states, one observable and one unobservable, so required development of another simpler exhaustive summary (McCrea and Cole work in progress).

Parameter redundancy in Pledger et al (2009)'s stopover models (Matechou and Cole unpublished work).

Clint – a male great crested newt

Sandpiper

Reparameterisation MethodFurther Examples

• Audoly et al (1998) consider a linear compartment model :

Reparameterisation MethodFurther Examples

• In Catchpole and Morgan (2009a) we use the reparameterisation method to show that the model is not redundant.

)(),(),(

),(),(

0100

0010

000/1

0001

)(0

)(0

0)(

0)(

1

0434144341

3403432332

23321221

14124121

ttxtxt

txtV

kkkkk

kkkkk

kkkk

kkkk

T

BuθAθ

θCθyCB

A

TVkkkkkkkkkk 104031441433423321221

Reparameterisation MethodFurther Examples

• We show that an exhaustive summary is:

• By solving si() = bi i = 1,...,11 we find there is a unique solution with k21 = b9, k12 = b5/b9,...,V1 = b11. Hence the model is globally identifiable.

1

3441

21

43134

3223

4114

2112

043414

034323

3212

4121

11

10

9

8

7

6

5

4

3

2

1

)(

)(

)(

)(

V

kk

k

kk

kk

kk

kk

kkk

kkk

kk

kk

s

s

s

s

s

s

s

s

s

s

s

s

Reparameterisation MethodFurther Examples

• Dochain et al (1995) examine the identifiability of models for the activated sludge process using a non-linear compartment model.

• Symbolic method possible for k=1.

• However for k=2 model too complex.

• Using the reparameterisation method with the extension theorem Cole and Morgan (2009a) show that for any k there are 3k estimable parameters (out of 4k+1) of the form

k

ii

i dt

tSYU

1

)()1( ki

tSK

tS

Y

X

t

tS

im

i

i

i ,...,1)(

)()(

1,

1max,

kiYSYSKY

YXiiiiim

i

ii ,...,1)1)(0(),1()0(,)1(

,max,

Conclusion• Exhaustive summaries offer a more general framework for

symbolic detection of parameter redundancy.• Parameter redundancy can be investigated symbolically by

examining a derivative matrix and its rank.• In the symbolic method we can find the estimable parameter

combinations (via PDEs).• The symbolic method can easily be generalised using the

extension theorem.• Parameter redundant nested models can be found using a

PLUR decomposition of any full rank derivative matrix.• The use of reparameterisation allows us to produce structurally

much simpler exhaustive summaries, allowing us to examine parameter redundancy of much more complex models symbolically.

• Methods are general and can in theory be applied to any parametric model.

References– Audoly, S. D’Angio, L., Saccomani, M. P. and Cobelli, C. (1998) IEEE Transactions on

Biomedical Engineering 45, 36-47.– Catchpole, E. A. and Morgan, B. J. T. (1997) Biometrika, 84, 187-196– Catchpole, E. A., Morgan, B. J. T. and Freeman, S. N. (1998) Biometrika, 85, 462-468

– Chappell, M. J. and Gunn, R. N. (1998) Mathematical Biosciences, 148 21-41.– Dochain, D., Vanrolleghem, P. A. and Van Dale, M. (1995) Water Research, 29, 2571-

2578.

– Evans, N. D. and Chappell, M. J. (2000) Mathematical Biosciences 168, 137-159.– Goodman, L. A. (1974) Biometrika, 61, 215-231.– Hunter, C.M. and Caswell, H. (2009). Ecological and Environmental Statistics Volume 3.

797-825– Jiang, H. Pollock, K. H., Brownie, C., et al (2007) JABES, 12, 177-194– Lebreton, J. Morgan, B. J. T., Pradel R. and Freeman, S. N. (1995). Biometrics, 51, 1418-

1428.– Pledger, S., Efford, M. Pollock, K., Collazo, J. and Lyons, J. (2009) Ecological and

Environmental Statistics Series: Volume 3. – Pohjanpalo, H. (1982) Technical Research Centre of Finland Research Report No. 56.– Rothenberg, T. J. (1971) Econometrica, 39, 577-591.– Shapiro, A. (1986) Journal of the American Statistical Association, 81, 142-149.– Walter, E. and Lecoutier, Y (1982) Mathematics and Computers in Simulations, 24, 472-

482

ReferencesRecent Work

– Cole, D. J. (2009) Determining Parameter Redundancy of Multi-state Mark-Recapture Models for Sea Birds. Presented at Eurings 2009 to appear in Journal of Ornithology.

– Cole, D. J. and Morgan, B. J. T (2009a) Determining the Parametric Structure of Non-Linear Models IMSAS, University of Kent Technical report UKC/IMS/09/005

– Cole, D. J. and Morgan, B. J. T. (2009b) A note on determining parameter redundancy in age-dependent tag return models for estimating fishing mortality, natural mortality and selectivity. IMSAS, University of Kent Technical report UKC/IMS/09/003 (To appear in JABES)

– See http://www.kent.ac.uk/ims/personal/djc24/parameterredundancy.htm for papers and Maple code