****
h
tktk+2
y y
hh
Ensemble Data Assimilation with the Community Land Modeland the US National Water Model
T. Hoar, M. Gharamti, J. McCreight, A. RafieeiNasab, A. Fox,J. Anderson, N. Collins, J. Hendricks, K. Raeder
National Center for Atmospheric ResearchBoulder, Colorado, USA
1. Ensemble Data Assimilation with DART
The Data Assimilation Research Testbed (DART) is an opensource community software facility for ensemble data as-similation developed at the National Center for AtmosphericResearch (NCAR). DART has been free and publically avail-able for more than 10 years. Building an interface betweenDART and a new model does not require an adjoint andgenerally requires no modifications to the model code.DART works with dozens of models of varying complexity,including :
• weather models, e.g. WRF, COAMPS, COSMO, MPASAtmosphere
• climate models components, e.g. CAM, POP, CLM,WACCM, MPAS Ocean, ROMS, FESOM, CICE5, WRF-Hydro, NOAH
• atmospheric chemistry models, e.g. CAM-CHEM, WRF-CHEM
• low-order and simple research models
DART assimilates a wide variety of observations :
• land observations such as snow cover fraction, groundwater depth, tower fluxes, cosmic ray neutron intensity,and microwave brightness temperature observations
• temperature, winds, moisture from NCEP, MADIS, andSSEC
• total precipitable water, radar observations, radio occul-tation observations from GPS satellites
• some observations in the World Ocean Database ...
DART provides both state-of-the-art ensemble data assimi-lation capabilities and an interactive educational platform toresearchers and students.
www.image.ucar.edu/DAReS/DART has infor-mation about how to download DART, the DARTeducational materials, and how to contact us.
2. DART ‘Manhattan’ Release Highlights
Much of the development effort has been to support largermodel states using less memory and improve performance,particularly I/O.
F Ability to handle much larger model states. Distributingthe model state across all tasks means no single taskmust store the entire state at any time.
F One-sided MPI communication allows tasks to requestremote data items from other tasks without interrupt-ing their execution or arranging which data items will beneeded in advance.
F Computing the forward operators for all ensemble mem-bers at the same time leads to better vector code.
F Native NetCDF support eliminates conversion betweenNetCDF model files and DART. This also reduces thehigh-water mark for disk requirements.
F Ensemble data can be read and distributed across alltasks on a variable-by-variable basis, reducing maximummemory requirements.
F Diagnostic files are now written in parallel, resulting infaster I/O and lower memory requirements.
F Supports externally-computed observation operators.
F Supports per-observation-type localization radii.
3. Acknowledgments
The National Center for Atmospheric Research issponsored by the National Science Foundation.Computational resources were provided by NCAR’sComputational and Information Systems Laboratory.
4. Surface Fluxes with CLM4.5
Special Thanks to Andy Fox (University of Arizona)CLM4.5 was used at the Niwot Ridge LTER site to explorethe impact of latent heat, sensible heat, and net ecosystemproduction observations on relevant CLM variables.
• 9.7 km east of the Continental Divide, in a Subalpine Forest
• Spun up for 1500 years with site-specific information.
• 64 ensemble members using an ensemble reanalysis forcing
• Assimilating tower fluxes of latent heat (LE), sensible heat (H), andnet ecosystem production (NEP).
• unobserved variables: LEAFC, LIVEROOTC, LIVESTEMC, DEAD-STEMC, LITR1C, LITR2C, SOIL1C, SOIL2C, SOILLIQ
The model states are being updated at about 8PM local time.
FYI: We are assimilating flux observations …
Sens
ible
hea
tLa
tent
hea
tNe
t Eco
syst
emPr
oduc
tion
Figure 1: The observations of the eddy covariance fluxesare available every 30 minutes. The free run of the model isshown in blue, the assimilation result is shown in red.
Figure 2: The free run is shown in blue. The assimilationresult is shown in red.
Carbon CycleWe have added a number of observations to the CLM-DARTworkflow, such as leaf area index (LAI) and abovegroundbiomass - key components of the terrestrial carbon cycle.DART’s adaptive inflation algorithm allows the ensemble tobecome consistent with the observations and then main-tain a useful ensemble spread. This effort will be moving toCLM5.0 to take advantage of the expanded treatment of thecarbon and nitrogen cycles.
Forecast
Biom
ass
LAI
DA Run
Obs
erva
tions
Free Run
Free Run
Figure 3: This experiment highlights a study in a semi-aridregion in New Mexico. Monthly LAI and annual biomass ob-servations are able to provide a much better representationof the sesaonal cycle and accurate initial conditions for fore-casting. The impact persists for years for different C pools.
5. Soil Moisture with CLM5
The Community Land Model 5.0 was released in February2018 and is the latest in a series of land models devel-oped through the Community Earth System Model (CESM).CLM4.5 biogeophysics and biogeochemistry can be runfrom this release code. A new river model (MOSART)is also included. This release was a land-only release.The Functionally Assembled Terrestrial Ecosystem Simula-tor (FATES) is available as a research option.
Direct solar
Absorbed solar
Diff
use
sola
r
Dow
nwel
ling
long
wav
e
Reflected solar
Emitt
ed
long
wav
e
Sens
ible
hea
t flu
x
Late
nt h
eat
flux
ua0
Momentum fluxWind speed
Ground heat flux
Evaporation
Melt
Sublimation
Throughfall
InfiltrationSurface runoff
Evaporation
Transpiration
Precipitation
Heterotrop.respiration
Photosynthesis
Autotrophicrespiration
Litterfall
Nuptake
Vegetation C/N
SoilC/N N mineralization
Fire
Surface energy fluxes Hydrology Biogeochemical cycles
Aerosol deposition
Soil (sand, clay, organic)Sub-surface
runoff
Phenology
BVOCs
Water tableSaturated Zone
Soil
Dust
Saturatedfraction
N depN fix
DenitrificationN leaching
CH4
Root litter
N2OSCF
Surfacewater
Bedrock
Glacier
Lake
River Routing
Runoff
River discharge
Urban
Land Use Change
Wood harvest
Disturbance
Vegetation Dynamics
Growth
CompetitionWetlandCropsIrrigationFlooding
CLM5.0
Impermeable Bedrock
Figure 4: The representation of the processes in CLM5.0which was released this February.
Gridcell
GlacierLake
Landunit
Column
Patch
UrbanVegetated
Soil
Crop
PFT1 PFT2 PFT3 PFT4 …
Unirrig Irrig Unirrig Irrig
Crop1 Crop1 Crop2 Crop2 …
Roof
Sun Wall
ShadeWall
Pervious
Impervious
TBD
MD
HD
Elevation classes
G
L
CUIrr1
VPFT4
VPFT3
VPFT1
VPFT2
CIrr1
UT,H,MC
Irr2C
UIrr2
Figure 5: The hierarchy of a single CLM5.0 gridcell. Theonly location information is at the gridcell level. We areworking on an approach to more appropriately relate obser-vation locations to specific Landunits, Columns or Patches.
0 45 90 135 180
-30
0
30
60
0
2
4
6
8
10
12
14
16
18
kg/m
2
Figure 6: SMAP locations from 23 July 2015 were the ba-sis for a short proof-of-concept experiment. The soil mois-ture (liquid and ice) from CLM at the SMAP locations wereused in an observing system simulation experiment.
01-Jul 02-Jul 03-Jul 04-Jul 05-Jul 06-Jul 07-JulJul.02,2001 00:00:00 start
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
rmse
and b
ias (m
odel
- obs
erva
tion)
SOIL_MOISTURErmse pr=0.44353, po=0.038084 bias pr=-0.0012196, po=-0.00047593
rmsebias
data file: /Users/thoar/svn/DART/cesm_clm/models/clm/matlab/obs_diag_output.nc
29150
29200
29250
29300
29350
29400
29450
29500
# of o
bs : o
=pos
sible,
$=as
simila
ted
Figure 7: The proof-of-concept result for soil moisture inthe first layer of CLM. This is a 40 member ensemble.
6. U.S. National Water Model
The NCAR-supported community Weather Re-search and Forecasting Hydrologic model(WRF-Hydro) is a modeling system and frame-work for hydrologic modeling and model cou-pling. In 2016 a configuration of WRF-Hydrowas implemented as the National Water Model(NWM) for the continental United States.
Figure 8: Left: The test domain ‘Matthew’ for WRF-Hydroand DART. The domain is approximately 100,000 km2. Thetriangles are at stream gage locations. The LSM runs at ≈1 km resolution. Right: A depiction of the Stream Order.
WRF-Hydro is configured to use the Noah-MP Land SurfaceModel (LSM) to simulate land surface processes. Separatewater routing modules perform diffusive wave surface rout-ing and saturated subsurface flow routing on a 250m grid,and Muskingum-Cunge channel routing down National Hy-drography Dataset (NHDPlusV2) stream reaches.
HydroDART can be run in multiple configura-tions, including a ‘channel-only’ configuration. Asmall case can be run in Docker.
Figure 9: The result from a test experiment on a ‘channel-only’ configuration. This is the verification gage.
HydroDART is currently configured to assimilate streamflowfrom 115 gages in the Matthew domain and update thechannel model state as well as estimate parameters reg-ulating the efficiency of the link response to fluxes. Thereare 50225 links/reaches in the domain with 115 gages thatare currently being assimilated. Research areas include lo-calization, inflation, and parameter estimation.
NWMensemble
DARTNWMmetadata
SetupExperiment
1. SetupExperiment pulls together all the pieces for an experiment and creates ControlDir and RunDir
2. ControlDir is small. Has control scripts Submit, filter, and hydro. RunDir can be quite large.
3. Submit launches (in any order) dependent job filter and dependent job array hydro.
4. Hydro is run after successful completion of filter as determined by the queueing system.
5. Submit can stage multiple cycles of dependent jobs or it can create a ‘cookie’ file and the dependent jobs can submit the next job until the cookie is gone.
RunDir
hydrohydro
SubmitDependent Job Array
Dependent Job
filter
filter operates in RunDir, hydro operates in each underlying directories – one per ensemble member.
Submit, filter and hydro are not the actual script names.
Mem
ber 1
Mem
ber N
Submit
filter hydro
ControlDir
Requires a large filesystem.
Figure 10: Schematic of the workflow of WRF-Hydro andDART. Separate jobs perform the assimilation and a Job Ar-ray is used to run the ensemble of WRF-Hydro instances.Each job can be tuned for an optimal configuration. Sinceeach job runs for a very short time, the jobs may backfill.The queueing system takes care of the job dependenciesand automatically resubmits the next job in the series.
TR32 April 2018
