****

h

tktk+2

y y

hh

Ensemble Data Assimilation with the Community Land Modeland the US National Water Model

T. Hoar, M. Gharamti, J. McCreight, A. RafieeiNasab, A. Fox,J. Anderson, N. Collins, J. Hendricks, K. Raeder

National Center for Atmospheric ResearchBoulder, Colorado, USA

dart@ucar.edu

1. Ensemble Data Assimilation with DART

The Data Assimilation Research Testbed (DART) is an opensource community software facility for ensemble data as-similation developed at the National Center for AtmosphericResearch (NCAR). DART has been free and publically avail-able for more than 10 years. Building an interface betweenDART and a new model does not require an adjoint andgenerally requires no modifications to the model code.DART works with dozens of models of varying complexity,including :

• weather models, e.g. WRF, COAMPS, COSMO, MPASAtmosphere

• climate models components, e.g. CAM, POP, CLM,WACCM, MPAS Ocean, ROMS, FESOM, CICE5, WRF-Hydro, NOAH

• atmospheric chemistry models, e.g. CAM-CHEM, WRF-CHEM

• low-order and simple research models

DART assimilates a wide variety of observations :

• land observations such as snow cover fraction, groundwater depth, tower fluxes, cosmic ray neutron intensity,and microwave brightness temperature observations

• temperature, winds, moisture from NCEP, MADIS, andSSEC

• total precipitable water, radar observations, radio occul-tation observations from GPS satellites

• some observations in the World Ocean Database ...

DART provides both state-of-the-art ensemble data assimi-lation capabilities and an interactive educational platform toresearchers and students.

www.image.ucar.edu/DAReS/DART has infor-mation about how to download DART, the DARTeducational materials, and how to contact us.

2. DART ‘Manhattan’ Release Highlights

Much of the development effort has been to support largermodel states using less memory and improve performance,particularly I/O.

F Ability to handle much larger model states. Distributingthe model state across all tasks means no single taskmust store the entire state at any time.

F One-sided MPI communication allows tasks to requestremote data items from other tasks without interrupt-ing their execution or arranging which data items will beneeded in advance.

F Computing the forward operators for all ensemble mem-bers at the same time leads to better vector code.

F Native NetCDF support eliminates conversion betweenNetCDF model files and DART. This also reduces thehigh-water mark for disk requirements.

F Ensemble data can be read and distributed across alltasks on a variable-by-variable basis, reducing maximummemory requirements.

F Diagnostic files are now written in parallel, resulting infaster I/O and lower memory requirements.

F Supports externally-computed observation operators.

F Supports per-observation-type localization radii.

3. Acknowledgments

The National Center for Atmospheric Research issponsored by the National Science Foundation.Computational resources were provided by NCAR’sComputational and Information Systems Laboratory.

4. Surface Fluxes with CLM4.5

Special Thanks to Andy Fox (University of Arizona)CLM4.5 was used at the Niwot Ridge LTER site to explorethe impact of latent heat, sensible heat, and net ecosystemproduction observations on relevant CLM variables.

• 9.7 km east of the Continental Divide, in a Subalpine Forest

• Spun up for 1500 years with site-specific information.

• 64 ensemble members using an ensemble reanalysis forcing

• Assimilating tower fluxes of latent heat (LE), sensible heat (H), andnet ecosystem production (NEP).

• unobserved variables: LEAFC, LIVEROOTC, LIVESTEMC, DEAD-STEMC, LITR1C, LITR2C, SOIL1C, SOIL2C, SOILLIQ

The model states are being updated at about 8PM local time.

FYI: We are assimilating flux observations …

Sens

ible

hea

tLa

tent

hea

tNe

t Eco

syst

emPr

oduc

tion

Figure 1: The observations of the eddy covariance fluxesare available every 30 minutes. The free run of the model isshown in blue, the assimilation result is shown in red.

Figure 2: The free run is shown in blue. The assimilationresult is shown in red.

Carbon CycleWe have added a number of observations to the CLM-DARTworkflow, such as leaf area index (LAI) and abovegroundbiomass - key components of the terrestrial carbon cycle.DART’s adaptive inflation algorithm allows the ensemble tobecome consistent with the observations and then main-tain a useful ensemble spread. This effort will be moving toCLM5.0 to take advantage of the expanded treatment of thecarbon and nitrogen cycles.

Forecast

Biom

ass

LAI

DA Run

Obs

erva

tions

Free Run

Free Run

Figure 3: This experiment highlights a study in a semi-aridregion in New Mexico. Monthly LAI and annual biomass ob-servations are able to provide a much better representationof the sesaonal cycle and accurate initial conditions for fore-casting. The impact persists for years for different C pools.

5. Soil Moisture with CLM5

The Community Land Model 5.0 was released in February2018 and is the latest in a series of land models devel-oped through the Community Earth System Model (CESM).CLM4.5 biogeophysics and biogeochemistry can be runfrom this release code. A new river model (MOSART)is also included. This release was a land-only release.The Functionally Assembled Terrestrial Ecosystem Simula-tor (FATES) is available as a research option.

Direct solar

Absorbed solar

Diff

use

sola

r

Dow

nwel

ling

long

wav

e

Reflected solar

Emitt

ed

long

wav

e

Sens

ible

hea

t flu

x

Late

nt h

eat

flux

ua0

Momentum fluxWind speed

Ground heat flux

Evaporation

Melt

Sublimation

Throughfall

InfiltrationSurface runoff

Evaporation

Transpiration

Precipitation

Heterotrop.respiration

Photosynthesis

Autotrophicrespiration

Litterfall

Nuptake

Vegetation C/N

SoilC/N N mineralization

Fire

Surface energy fluxes Hydrology Biogeochemical cycles

Aerosol deposition

Soil (sand, clay, organic)Sub-surface

runoff

Phenology

BVOCs

Water tableSaturated Zone

Soil

Dust

Saturatedfraction

N depN fix

DenitrificationN leaching

CH4

Root litter

N2OSCF

Surfacewater

Bedrock

Glacier

Lake

River Routing

Runoff

River discharge

Urban

Land Use Change

Wood harvest

Disturbance

Vegetation Dynamics

Growth

CompetitionWetlandCropsIrrigationFlooding

CLM5.0

Impermeable Bedrock

Figure 4: The representation of the processes in CLM5.0which was released this February.

Gridcell

GlacierLake

Landunit

Column

Patch

UrbanVegetated

Soil

Crop

PFT1 PFT2 PFT3 PFT4 …

Unirrig Irrig Unirrig Irrig

Crop1 Crop1 Crop2 Crop2 …

Roof

Sun Wall

ShadeWall

Pervious

Impervious

TBD

MD

HD

Elevation classes

G

L

CUIrr1

VPFT4

VPFT3

VPFT1

VPFT2

CIrr1

UT,H,MC

Irr2C

UIrr2

Figure 5: The hierarchy of a single CLM5.0 gridcell. Theonly location information is at the gridcell level. We areworking on an approach to more appropriately relate obser-vation locations to specific Landunits, Columns or Patches.

0 45 90 135 180

-30

0

30

60

0

2

4

6

8

10

12

14

16

18

kg/m

2

Figure 6: SMAP locations from 23 July 2015 were the ba-sis for a short proof-of-concept experiment. The soil mois-ture (liquid and ice) from CLM at the SMAP locations wereused in an observing system simulation experiment.

01-Jul 02-Jul 03-Jul 04-Jul 05-Jul 06-Jul 07-JulJul.02,2001 00:00:00 start

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

rmse

and b

ias (m

odel

- obs

erva

tion)

SOIL_MOISTURErmse pr=0.44353, po=0.038084 bias pr=-0.0012196, po=-0.00047593

rmsebias

data file: /Users/thoar/svn/DART/cesm_clm/models/clm/matlab/obs_diag_output.nc

29150

29200

29250

29300

29350

29400

29450

29500

# of o

bs : o

=pos

sible,

$=as

simila

ted

Figure 7: The proof-of-concept result for soil moisture inthe first layer of CLM. This is a 40 member ensemble.

6. U.S. National Water Model

The NCAR-supported community Weather Re-search and Forecasting Hydrologic model(WRF-Hydro) is a modeling system and frame-work for hydrologic modeling and model cou-pling. In 2016 a configuration of WRF-Hydrowas implemented as the National Water Model(NWM) for the continental United States.

Figure 8: Left: The test domain ‘Matthew’ for WRF-Hydroand DART. The domain is approximately 100,000 km2. Thetriangles are at stream gage locations. The LSM runs at ≈1 km resolution. Right: A depiction of the Stream Order.

WRF-Hydro is configured to use the Noah-MP Land SurfaceModel (LSM) to simulate land surface processes. Separatewater routing modules perform diffusive wave surface rout-ing and saturated subsurface flow routing on a 250m grid,and Muskingum-Cunge channel routing down National Hy-drography Dataset (NHDPlusV2) stream reaches.

HydroDART can be run in multiple configura-tions, including a ‘channel-only’ configuration. Asmall case can be run in Docker.

Figure 9: The result from a test experiment on a ‘channel-only’ configuration. This is the verification gage.

HydroDART is currently configured to assimilate streamflowfrom 115 gages in the Matthew domain and update thechannel model state as well as estimate parameters reg-ulating the efficiency of the link response to fluxes. Thereare 50225 links/reaches in the domain with 115 gages thatare currently being assimilated. Research areas include lo-calization, inflation, and parameter estimation.

NWMensemble

DARTNWMmetadata

SetupExperiment

1. SetupExperiment pulls together all the pieces for an experiment and creates ControlDir and RunDir

2. ControlDir is small. Has control scripts Submit, filter, and hydro. RunDir can be quite large.

3. Submit launches (in any order) dependent job filter and dependent job array hydro.

4. Hydro is run after successful completion of filter as determined by the queueing system.

5. Submit can stage multiple cycles of dependent jobs or it can create a ‘cookie’ file and the dependent jobs can submit the next job until the cookie is gone.

RunDir

hydrohydro

SubmitDependent Job Array

Dependent Job

filter

filter operates in RunDir, hydro operates in each underlying directories – one per ensemble member.

Submit, filter and hydro are not the actual script names.

Mem

ber 1

Mem

ber N

Submit

filter hydro

ControlDir

Requires a large filesystem.

Figure 10: Schematic of the workflow of WRF-Hydro andDART. Separate jobs perform the assimilation and a Job Ar-ray is used to run the ensemble of WRF-Hydro instances.Each job can be tuned for an optimal configuration. Sinceeach job runs for a very short time, the jobs may backfill.The queueing system takes care of the job dependenciesand automatically resubmits the next job in the series.

TR32 April 2018