dart tutorial section 18:lost in phase space: the challenge of … · 2020-06-04 · dart tutorial...

TheNationalCenterforAtmosphericResearchissponsoredbytheNationalScienceFoundation.Anyopinions,findingsandconclusionsorrecommendationsexpressedinthispublicationarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheNationalScienceFoundation.

©UCAR

DARTTutorialSection18: LostinPhaseSpace:TheChallengeofNotKnowingtheTruth.

RealityStrikes

Inrealapplications,thetruthisunknown.

Allthatwehaveareobservations.

Havingthetruthavailablehasbeenconvenient,butalsomisleading.

Muchlessinformationisavailablefromtheobservations.Theyaregenerallyfunctionsofthestatevariables.Theyarealwayscontaminatedwithobservationalerrors.

DARTTutorialSection18:Slide2

Whattoexpect…

−4 −2 0 2 40

0.2

0.4

0.6

0.8

Prob

abilit

y

Prior PDF Obs. Likelihood

S.D.Inflated S.D. Expected Separation

Actual 3.698 SDs

RecallthatExpected(prior_mean– observation)= σ prior

2 +σ obs2

ErrorisdominatedbyobservationalnoiseifSuppose= 1.0, = 0.1, thenE(RMS) = 1.005.Halvingto0.05 => E(RMS) = 1.001;onlya0.4% reduction!

σ obs2 ≫σ prior

2

σ prior

σ obs σ prior

DARTTutorialSection18:Slide3

&quality_control_nmlinput_qc_threshold = 3.0outlier_threshold = -1.0

/

FirstObservation-spacediagnostics:WhetherornottoassimilateorrejectobservationsbasedontheirExpectedSeparationiscontrolledduringfilter basedonnamelistsettingsininput.nml.

If > outlier_threshold

Theprogramobs_diag post-processesobs_seq.final,calculatesmetricslikeRMSE,bias,ensemblespread,totalspread,#ofobservationsusedorrejected…Startwiththelorenz_96model.

yp − yo σ prior2 +σ obs

2 Observationrejected!(DARTQC==7)

DARTTutorialSection18:Slide4

Observation-spacediagnosticsTheobservationsequencefileisnotinaparticularlyuser-friendlyformat.Toaidintheevaluationandinterpretation,aprogramnamedobs_diagmustberuntoproduceanetCDFfilewithresultsthatcanbeplottedinamannerofyourchoosing.DARThasMatlabfunctions/scriptsthatcreatehigh-qualitygraphics.Forup-to-dateinformationonthelatest,greatestdiagnostics,goto:http://www.image.ucar.edu/DAReS/DART/DART2_Diagnostics.php#obs_diagnostics

&obs_diag_nmlobs_sequence_name = 'obs_seq.final',bin_width_days = -1,bin_width_seconds = -1,init_skip_days = 0,init_skip_seconds = 0,Nregions = 3,trusted_obs = 'null',lonlim1 = 0.00, 0.00, 0.50, -1.0lonlim2 = 1.01, 0.50, 1.01, -1.0reg_names = 'whole’, ‘yin’, ‘yang’, ‘bogus,create_rank_histogram = .true.,outliers_in_histogram = .true.,use_zero_error_obs = .false.,verbose = .false./

DARTTutorialSection18:Slide5

Observation-spacediagnosticsTheobservationsequencefileisnotinaparticularlyuser-friendlyformat.Toaidintheevaluationandinterpretation,aprogramnamedobs_diagmustberuntoproduceanetCDFfilewithresultsthatcanbeplottedinamannerofyourchoosing.DARThasMatlabfunctions/scriptsthatcreatehigh-qualitygraphics.Forup-to-dateinformationonthelatest,greatestdiagnostics,goto:http://www.image.ucar.edu/DAReS/DART/DART2_Diagnostics.php#obs_diagnostics

TheseworkwithANYobs_seq.final fromANYexperimentwithANYmodel!

HereareafewoftheMatlabfunctionsavailableindiagnostics/matlab

• plot_rank_histogram.m• plot_evolution.m• plot_rmse_xxx_evolution.m• two_experiments_evolution.m• plot_profile.m• plot_bias_xxx_profile.m• plot_rmse_xxx_profile.m• two_experiments_profile.m

DARTTutorialSection18:Slide6

Lorenz96observationdiagnosticexample

month/day - Jan.01,1601 01:00:00 start01/01 01/06 01/11 01/16 01/21 01/26 01/31 02/05 02/10 02/15

rmse

0

1

2

3

4

5

6

7

yangRAW_STATE_VARIABLE

forecast: mean=2.9555 analysis: mean=2.8093

forecastanalysis

data file: /Users/thoar/svn/DART/clean_lanai/models/lorenz_96/work/obs_diag_output.nc

# of

obs

: o=

poss

ible

, ∗=a

ssim

ilate

d

23

24

25

26

27

28

29

30

outlier_threshold=-1.0

DARTTutorialSection18:Slide7

&filter_nml…obs_sequence_in_name = "obs_seq.out”obs_sequence_out_name = "obs_seq.final”…/

&quality_control_nmlinput_qc_threshold = 3.0,outlier_threshold = -1.0,/

FirstObservation-spacediagnostics:Trysettingtherejectionthresholdtoasmallpositivenumberandrerunningfilter,andthenrerunningobs_diag onthenewoutputfile.

Don’tforgettorerunfilter!Don’tforgettorerunobs_diag!Don’tforgettousetherightfilenameinobs_diag_nml!

Thisispotentially ,butuseful.Rejecting‘good’observationscanleadtoinflatedestimateofquality.

DARTTutorialSection18:Slide8

FirstObservation-spacediagnostics:

month/day - Jan.01,1601 01:00:00 start01/01 01/06 01/11 01/16 01/21 01/26 01/31 02/05 02/10 02/15

rmse

0

0.5

1

1.5

2

2.5

3

yangRAW_STATE_VARIABLE

forecast: mean=1.6213 analysis: mean=1.5243

forecastanalysis

data file: /Users/thoar/svn/DART/clean_lanai/models/lorenz_96/work/obs_diag_output.nc

# of

obs

: o=

poss

ible

, ∗=a

ssim

ilate

d

0

5

10

15

20

25

30

Observationsbeingrejected!

LowerRMSEthanbefore!$1,000,000question:Why?

outlier_threshold=3.0

DARTTutorialSection18:Slide9

Lorenz_96exercises:

• Pickacasethatworksrelativelywellandlookatobservation-spacediagnostics.

• Pickacasethatissimilar,butclearlydifferent,withphysical-spacediagnostics.

• Seeifyoucandetectthedifferencewithobservation-spacediagnostics.

• Rerunobs_diag withdifferentbin_widths.

DARTTutorialSection18:Slide10

Observation-spacediagnostics:rankhistograms

0 5 10 15 20 25 30 35 40 45 500

10

20

30

40

50

60

70

2297 obs possible, 2269 obs binned

Observation Rank (among ensemble members)co

unt

MPEX

0 5 10 15 20 25 30 35 40 45 500

50

100

150

200

250

300

350

12263 obs possible, 12106 obs binned

Observation Rank (among ensemble members)

coun

t

RADIOSONDE_TEMPERATURE @ 500 hPaFull Domain

May.16,2015 21:00:01 −− May.24,2015 03:00:00data file: /Users/thoar/svn/DART/clean_lanai/models/wrf/work/obs_diag_output.nc

>> fname = ‘obs_diag_output.nc’;>> timeindex = -1;>> varname = ‘RADIOSONDE_TEMPERATURE’;>> plot_rank_histogram(fname, timeindex, varname);

ResultsfromWRFreal-timeforecasting.

DARTTutorialSection18:Slide11

Observation-spacediagnostics:timeevolution(bylevel)

08/01 08/06 08/11 08/160

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

month/day − Aug.01,2005 06:00:00 start

rmse

and

tota

lspr

ead

Northern Hemisphere (20−80)RADIOSONDE_TEMPERATURE @ 500 hPa

rmse pr=1.1971, po=0.98162 totalspread pr=0.91985, po=0.81559

rmsetotalspread

data file: /glade/u/home/thoar/svn/DART/trunk/models/cam/work/obs_diag_output.nc

0

200

400

600

800

1000

1200

1400

1600

1800

2000

# of

obs

: o=

poss

, �=u

sed

Initiallytinyspreadandlargeobservationrejection– systemnotperformingwell– yet!

plot_rmse_xxx_evolution.mplot_evolution.m

Totalspreadisthesqrtofthepooledvarianceoftheobservationerrorandtheensemblevariance.

MuchBetter!Veryfewobservationsbeingrejected.

DARTTutorialSection18:Slide12

Observation-spacediagnostics:time-averagedprofilesplot_profile.m plot_bias_xxx_profile.mplot_rmse_xxx_profile.m

bias (model − observation) and totalspread16−May−2015 21:00:01 through 24−May−2015 03:00:00

hPa

−1 0 1 2 3

100150200250300

400

500

700

850

925

1000

bias pr=0.26525bias po=0.02529totalspread pr=1.563totalspread po=0.99202

0 7 14 21 28# of obs (o=possible, ∗=assimilated) x100

MPEXRADIOSONDE_TEMPERATURE

data file: /Users/thoar/svn/DART/clean_lanai/models/wrf/work/obs_diag_output.nc

bias (model − observation) and totalspread16−May−2015 21:00:01 through 24−May−2015 03:00:00

hPa

−1 0 1 2 3

100150200250300

400

500

700

850

925

1000

bias pr=−0.028017bias po=0.0027291totalspread pr=1.5112totalspread po=1.0003

0 4 8 12 16# of obs (o=possible, ∗=assimilated) x1000

Full DomainRADIOSONDE_TEMPERATURE

data file: /Users/thoar/svn/DART/clean_lanai/models/wrf/work/obs_diag_output.nc

Note:Thesearemuchmoreinformativeformodelswithlevels!(i.e.the1Dmodelsarenotveryinterestingthisway)

DARTTutorialSection18:Slide13

&obs_diag_nmlobs_sequence_name = ‘’obs_sequence_list = ’file_list.txt’first_bin_center = 2005, 8, 1, 6, 0, 0last_bin_center = 2005, 8,26, 0, 0, 0bin_separation = 0, 0, 0, 6, 0, 0bin_width = 0, 0, 0, 6, 0, 0time_to_skip = 0, 0,10, 0, 0, 0max_num_bins = 1000trusted_obs = 'null’…

/

Awordofwarning…

08/01 08/06 08/11 08/16 08/21 08/260

0.5

1

1.5

2

2.5

3

month/day − Aug.01,2005 06:00:00 start

rmse

and

tota

lspre

ad

Northern HemisphereRADIOSONDE_TEMPERATURE @ 500 hPa

rmse pr=1.1176, po=0.91808 totalspread pr=0.91241, po=0.8187

rmsetotalspread

data file: /glade/scratch/raeder/SE30r4_Katrina/Diag_NoSoTrCarib_2005_8_1−23/obs_diag_output.nc

0

400

800

1200

1600

2000

2400

# of

obs

: o=

poss

, +=u

sed

obs_diag ‘time_to_skip’settingwillallowyoutoignorethespinupbeforestartingthetime-averagingfortheverticalprofileswhilestillcalculatingmetricsfortheentireperiodofrecordforthetime-evolutionproducts.

NOTE:The&obs_diag_nml isdifferentforlow-ordermodelsandrealisticmodels.Checkcarefullywhenusing.

DARTTutorialSection18:Slide14

Observation-spacediagnostics:comparingexperiments

08/01 08/06 08/11 08/16 08/21 08/26 08/31−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

31−Jul−2005 18:00:01 through 31−Aug−2005 06:00:00

fore

cast

bia

s (m

odel

− o

bser

vatio

n)

0

30

60

90

120

150

180

210

240

270

# of

obs

(o=p

ossi

ble,

∗=a

ssim

ilate

d)

Southern HemisphereRADIOSONDE_U_WIND_COMPONENT @ 500 hPa

Identical Twin PriorFraternal Twin Prior

data file: /Users/thoar/svn/DART/clean_lanai/models/cam/work/obs_diag_itwin.ncdata file: /Users/thoar/svn/DART/clean_lanai/models/cam/work/obs_diag_ftwin.nc

two_experiments_evolution.mtwo_experiments_profile.m

Thisisusefulforquickcomparisons.Really‘fair’comparisonsrequiremoreprocessingtocomparethesamesetofobservationsacrossexperiments.

assimilation_code/programsobs_common_subset.htmlobs_seq_coverage.htmlobs_selection.htmlobs_seq_verify.html

FYI:‘Identical’meansthemodelthatwasusedtogeneratetheobservationsisalsousedfortheassimilation.‘Fraternal’meanstheobservationscamefromadifferentmodel.

DARTTutorialSection18:Slide15

Observation-spacediagnostics:netCDF

SOMEoftheinformationintheobservationsequencefilescanbeconvertedtonetCDFandeasilyplotted.Aprogramnamedobs_seq_to_netcdfmustberuntoproducethenetCDF.

HereareafewoftheMatlabfunctionsavailableindiagnostics/matlab.

• link_obs.m• plot_obs_netcdf.m• plot_obs_netcdf_diffs.m• plot_coverage.m

DARTTutorialSection18:Slide16

&schedule_nmlcalendar = 'Gregorian’first_bin_start = 2005, 8, 13, 21, 0, 0first_bin_end = 2005, 8, 14, 03, 0, 0last_bin_end = 2005, 8, 14, 03, 0, 0bin_interval_days = 10000bin_interval_seconds = 0max_num_bins = 1000print_table = .true.

/&obs_seq_to_netcdf_nml

obs_sequence_name = ‘cam_obs_seq.2005-08-14-00000.final’obs_sequence_list = ‘’lonlim1 = 160.lonlim2 = 40.latlim1 = 10.latlim2 = 65.

/

Theprogramobs_seq_to_netcdf convertsmuchoftheinformationinanobservationsequencefiletoanetCDFfile.Fornow,we’regoingtoexploreapre-computedfileavailableat:

www.image.ucar.edu/pub/DART/Tutorial_Datasets/obs_epoch_SE30r4_Katrina.nc

Itwasgeneratedwiththefollowinginput:

Complicatedobservation-spacediagnostics.

DARTTutorialSection18:Slide17

MatlabHands-On: link_obsexploringobservations

Thisenablesrotationwiththemouse.

‘paintbrush’allowsyoutoselectobservationsfor‘brushing’

Trydifferentobstypes,Trytolocaterejectedobs,Whyweretheyrejected?Tryplot_obs_netcdf.m

DARTTutorialSection18:Slide18

1. FilteringForaOneVariableSystem2. TheDARTDirectoryTree3. DARTRuntimeControlandDocumentation4. Howshouldobservationsofastatevariableimpactanunobservedstatevariable?

Multivariateassimilation.5. ComprehensiveFilteringTheory:Non-IdentityObservationsandtheJointPhaseSpace6. OtherUpdatesforAnObservedVariable7. SomeAdditionalLow-OrderModels8. DealingwithSamplingError9. MoreonDealingwithError;Inflation10. RegressionandNonlinearEffects11. CreatingDARTExecutables12. AdaptiveInflation13. HierarchicalGroupFiltersandLocalization14. QualityControl15. DARTExperiments:ControlandDesign16. DiagnosticOutput17. CreatingObservationSequences18. LostinPhaseSpace:TheChallengeofNotKnowingtheTruth19. DART-CompliantModelsandMakingModelsCompliant20. ModelParameterEstimation21. ObservationTypesandObservingSystemDesign22. ParallelAlgorithmImplementation23. Locationmoduledesign(notavailable)24. Fixedlagsmoother(notavailable)25. Asimple1Dadvectionmodel:TracerDataAssimilation

DARTTutorialIndextoSections

DARTTutorialSection18:Slide19