dart tutorial sec’on 1: filtering for a one variable system · dart tutorial sec’on 1: slide 2...

TheNa'onalCenterforAtmosphericResearchissponsoredbytheNa'onalScienceFounda'on.Anyopinions,findingsandconclusionsorrecommenda'onsexpressedinthispublica'onarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheNa'onalScienceFounda'on.

©UCAR

DARTTutorialSec'on1:FilteringForaOneVariableSystem

DARTTutorialSec'on1:Slide2

Introduc'on

Thisseriesoftutorialpresenta'onsisdesignedtointroducebothbasicEnsembleKalmanfiltertheoryandtheDataAssimila'onResearchTestbedCommunityFacilityforEnsembleDataAssimila'on.ThereissignificantoverlapwiththeDART_LABtutorialthatisalsopartoftheDARTsubversioncheckout.IfyouhavealreadystudiedDART_LAB,feelfreetoskipthroughtheredundanttheoryslides.However,doingtheexercisesinallsec'onsofthistutorialisrecommendedinordertolearnthebestwaystousetheDARTsystem.


−6 −4 −2 0 2 4 60

0.05

0.1

0.15

0.2 Prior PDF

Prob

abilit

y Obs. Likelihood

p A | BC( ) = p(B | AC)p(A |C)p(B |C)


Bayes’Rule



−6 −4 −2 0 2 4 60

0.05

0.1

0.15

0.2 Prior PDF

Prob

abilit

y Obs. Likelihood

Product (Numerator)

p A | BC( ) = p(B | AC)p(A |C)p(B |C)


Bayes’Rule



−6 −4 −2 0 2 4 60

0.05

0.1

0.15

0.2 Prior PDF

Prob

abilit

y Obs. Likelihood

Normalization (Denom.)

p A | BC( ) = p(B | AC)p(A |C)p(B |C)


Bayes’Rule


−6 −4 −2 0 2 4 60

0.05

0.1

0.15

0.2 Prior PDF

Prob

abilit

y Obs. Likelihood

Normalization (Denom.)

Posterior

p A | BC( ) = p(B | AC)p(A |C)p(B |C)



Bayes’Rule


Green==Prior

Red==Observa'on

Blue==Posterior

ThesamecolorschemeisusedthroughoutALLTutorialmaterials.

ColorScheme


ThisproductisclosedforGaussiandistribu'ons.

−4 −2 0 2 40

0.2

0.4

0.6

Prob

abilit

y

Prior PDF

Obs. Likelihood

p A | BC( ) = p(B | AC)p(A |C)p(B |C)


ProductofTwoGaussians



−4 −2 0 2 40

0.2

0.4

0.6

Prob

abilit

y

Prior PDF

Obs. Likelihood

Posterior PDF

p A | BC( ) = p(B | AC)p(A |C)p(B |C)




N (µ1,∑1)N (µ2 ,∑2 ) = cN (µ,∑)

Productofd-dimensionalnormalswithmeansandandcovariancematricesand

isnormal.

µ1 µ2∑1 ∑2




isnormal.

Covariance:

Mean:

N (µ1,∑1)N (µ2 ,∑2 ) = cN (µ,∑)

∑ = (∑1−1+∑2

−1)−1

µ = (∑1−1+∑2

−1)−1(∑1−1µ1 +∑2

−1µ2 )

µ1 µ2∑1 ∑2




isnormal.

Covariance:

Mean:

Weight:

N (µ1,∑1)N (µ2 ,∑2 ) = cN (µ,∑)

∑ = (∑1−1+∑2

−1)−1

µ = (∑1−1+∑2

−1)−1(∑1−1µ1 +∑2

−1µ2 )

µ1 µ2∑1 ∑2

We’llignoretheweightunlessnotedsinceweimmediatelynormalizeproductstobePDFs.


c = 1(2∏)d /2 ∑1 +∑2

1/2 exp − 12

µ2 − µ1( )T (∑1 +∑2 )−1 µ2 − µ1( )⎡

⎣⎤⎦

⎧⎨⎩

⎫⎬⎭



isnormal.

Covariance:

Mean:

Weight:

N (µ1,∑1)N (µ2 ,∑2 ) = cN (µ,∑)

∑ = (∑1−1+∑2

−1)−1

µ = (∑1−1+∑2

−1)−1(∑1−1µ1 +∑2

−1µ2 )

µ1 µ2∑1 ∑2

Easytoderivefor1-DGaussians;justdoproductsofexponen'als.


c = 1(2∏)d /2 ∑1 +∑2

1/2 exp − 12

µ2 − µ1( )T (∑1 +∑2 )−1 µ2 − µ1( )⎡

⎣⎤⎦

⎧⎨⎩

⎫⎬⎭



−4 −2 0 2 40

0.1

0.2

0.3

Prob

abilit

y Prior PDF

Thereareotherfamiliesoffunc'onsforwhichitisclosed…But,forgeneraldistribu'ons,there’snoanaly'calproduct.


p A | BC( ) = p(B | AC)p(A |C)p(B |C)



−4 −2 0 2 40

0.1

0.2

0.3

Prob

abilit

y Prior PDF

Obs. Likelihood


p A | BC( ) = p(B | AC)p(A |C)p(B |C)





−4 −2 0 2 40

0.1

0.2

0.3

Prob

abilit

y Prior PDF

Obs. Likelihood

Posterior PDF


p A | BC( ) = p(B | AC)p(A |C)p(B |C)





Don’tknowmuchaboutproper'esofthissample.Maynaivelyassumeitisrandomdrawfrom‘truth’.

Ensemblefilters:Priorisavailableasfinitesample.

−4 −2 0 2 40

0.2

0.4

0.6

Prob

abilit

y

Prior Ensemble


p A | BC( ) = p(B | AC)p(A |C)p(B |C)



Howcanwetakeproductofsamplewithcon'nuouslikelihood?

Fitacon'nuous(Gaussianfornow)distribu'ontosample.−4 −2 0 2 40

0.2

0.4

0.6

Prob

abilit

y

Prior PDF

Prior Ensemble


p A | BC( ) = p(B | AC)p(A |C)p(B |C)



Observa'onlikelihoodusuallycon'nuous(nearlyalwaysGaussian).

IfObs.Likelihoodisn’tGaussian,cangeneralizemethodsbelow.Forinstance,canfitsetofGaussiankernelstoobs.likelihood.

−4 −2 0 2 40

0.2

0.4

0.6

Prob

abilit

y

Prior PDF

Obs. Likelihood

Prior Ensemble


p A | BC( ) = p(B | AC)p(A |C)p(B |C)



ProductofpriorGaussianfitandObs.likelihoodisGaussian.

Compu'ngcon'nuousposteriorissimple.BUT,needtohaveaSAMPLEofthisPDF.

−4 −2 0 2 40

0.2

0.4

0.6

Prob

abilit

y

Prior PDF

Obs. Likelihood

Posterior PDF

Prior Ensemble


p A | BC( ) = p(B | AC)p(A |C)p(B |C)



Therearemanywaystodothis.

Exactproper'esofdifferentmethodsmaybeunclear.Trialanderrors'llbestwaytoseehowtheyperform.Willinteractwithproper'esofpredic'onmodels,etc.

−2 −1 0 1 2 30

0.2

0.4

0.6Posterior PDF

Prob

abilit

y

SamplingPosteriorPDF


EnsembleAdjustment(Kalman)Filter

−4 −2 0 2 40

0.2

0.4

0.6

Prior Ensemble

Prob

abilit

y




Again,fitaGaussiantosample.

−4 −2 0 2 40

0.2

0.4

0.6

Prior Ensemble

Prob

abilit

y

Prior PDF




ComputeposteriorPDF(sameaspreviousalgorithms).

−4 −2 0 2 40

0.2

0.4

0.6

Prior Ensemble

Prob

abilit

y

Prior PDFObs. Likelihood

Posterior PDF




Usedeterminis'calgorithmto‘adjust’ensemble.1.  ‘Shil’ensembletohaveexactmeanofposterior.2.  Uselinearcontrac'ontohaveexactvarianceofposterior.

−4 −2 0 2 40

0.2

0.4

0.6

Prob

abilit

y

Posterior PDF




−4 −2 0 2 40

0.2

0.4

0.6

Prob

abilit

y

Posterior PDF

Mean Shifted





−4 −2 0 2 40

0.2

0.4

0.6

Prob

abilit

y

Posterior PDF

Mean ShiftedVariance Adjusted




p isprior,u isupdate(posterior),

isstandarddevia'on,overbarisensemblemean.


−4 −2 0 2 40

0.2

0.4

0.6

Prob

abilit

y

Posterior PDF


xiu = xi

p − x p( ) i σ u /σ p( ) + x uσi=1,...,ensemblesize.




−4 −2 0 2 40

0.2

0.4

0.6

Prob

abilit

y

Posterior PDF


Bimodalitymaintained,butnotappropriatelyposi'onedorweighted.Noproblemwithrandomoutliers.




−4 −2 0 2 40

0.2

0.4

0.6

Prob

abilit

y

Posterior PDF


Thereareavarietyofotherwaystodeterminis'callyadjustensemble.Classofalgorithmssome'mescalleddeterminis'csquarerootfilters.



1stlookatDARTDiagnos'cs

cd models/lorenz_63/work inyourDARTsandbox.csh workshop_setup.csh Doesstuffyou’lllearntodolater.matlab -nodesktop

OutputfromaDARTassimila'onin3-variablemodel.20memberensemble.Observa'onsofeachvariableonceevery‘6hours’;errorvariance8.Observa'onONLYimpactsitsownstatevariable.Forassimila'on,lookslike3independentsinglevariableproblems.Modeladvancebetweenassimila'onsisn’tindependent.

Ini'alensemblemembersarerandomselec'onfromlongmodelrun.Ini'alerrorshouldbeanupperbound(randomguess).


1stlookatDARTDiagnos'cs

TrythefollowingMatlabcommands.Eachwillaskyouto:Inputnameofensembletrajectoryfile:<cr>forpreassim.nc

JustselectthedefaultfilebyhipngcarriagereturnforallMatlabexercisesfornow.

'meseriesofdistancebetweenpriorensemblemeanandtruthinblue;spread,averagepriordistancebetweenensemblemembersandmean,inred.(Recordtotalvaluesoftotalerrorandspreadforlater.)'meseriesoftruthinblue;ensemblemeanpriorinred.Figure1isseparatepanelforeachstatevariable.Figure2isthree-dimensionalplot.alsoincludespriorensemblemembersingreenforFigure1.

plot_total_err

plot_ens_mean_-me_series

plot_ens_-me_series


SimpleExample:Lorenz633-variablechao'cmodel

−20

0

20 −20

0

2010203040

Observa'oninred.Priorensembleingreen.Observingallthreestatevariables.Obs.Errorvariance=4.0.Four20-memberensembles.



−20

0

20 −20

0

2010203040

Observa'oninred.Priorensembleingreen.



−20

0

20 −20

0

2010203040




−20

0

20 −20

0

2010203040

Observa'oninred.Priorensembleingreen.Ensembleispassingthroughanunpredictableregion.



−20

0

20 −20

0

2010203040

Observa'oninred.Priorensembleingreen.Partoftheensembleheadsforonelobe,therestfortheother..


SimpleExample:Lorenz633-variablemodel

−20

0

20 −20

0

2010203040



UsingDARTDiagnos'cs

UsingDARTdiagnos'csfromthesimpleLorenz63assimila'on:Canyouseeevidenceofenhanceduncertainty?Wheredoesthisoccur?Doestheensembleappeartobeconsistentwiththetruth?

(Isthetruthnormallyinsidetheensemblerange?)


1.   FilteringForaOneVariableSystem2.   TheDARTDirectoryTree3.   DARTRun>meControlandDocumenta>on4.   Howshouldobserva>onsofastatevariableimpactanunobservedstatevariable?

Mul>variateassimila>on.5.   ComprehensiveFilteringTheory:Non-Iden>tyObserva>onsandtheJointPhaseSpace6.   OtherUpdatesforAnObservedVariable7.   SomeAddi>onalLow-OrderModels8.   DealingwithSamplingError9.   MoreonDealingwithError;Infla>on10.   RegressionandNonlinearEffects11.   Crea>ngDARTExecutables12.   Adap>veInfla>on13.   HierarchicalGroupFiltersandLocaliza>on14.   QualityControl15.   DARTExperiments:ControlandDesign16.   Diagnos>cOutput17.   Crea>ngObserva>onSequences18.   LostinPhaseSpace:TheChallengeofNotKnowingtheTruth19.   DART-CompliantModelsandMakingModelsCompliant20.   ModelParameterEs>ma>on21.   Observa>onTypesandObservingSystemDesign22.   ParallelAlgorithmImplementa>on23.  Loca'onmoduledesign(notavailable)24.  Fixedlagsmoother(notavailable)25.   Asimple1Dadvec>onmodel:TracerDataAssimila>on

DARTTutorialIndextoSec'ons


dart tutorial sec’on 1: filtering for a one variable system · dart tutorial sec’on 1: slide 2...

Documents