COMPUTATIONAL ASPECTS OF 4D-VAR CHEMICALDATA ASSIMILATION IN ATMOSPHERIC MODELS

A. SANDUa, D.N. DAESCUb, T. CHAIc, G.R. CARMICHAELc, J.H.SEINFELDd, P.G. HESSe, T.L. ANDERSONf

aVirginia Ploytechnic Institute and State University, Blacksburg, VA 24061, USAbPortland State University, City, Portland, OR 97207, USAcThe University of Iowa, Iowa City, IA 52242-1297, USAdCalifornia Institute of Technology, Pasadena, CA 91125, USAeNational Center for Atmospheric Research, Boulder, CO 80307, USAfUniversity of Washington, City, Seattle, WA 98195-1640, USA

Abstract. The task of providing an optimal analysis of the state of the atmosphere requiresthe development of novel computational tools that facilitate an efficient integration ofobservational data into models. In this paper we discuss some of the computational toolsrecently developed for the assimilation of chemical data into atmospheric models. Wereview the discrete and continuous adjoint sensitivity approaches applied to chemicaltransport models. Software tools particularly tailored for direct and adjoint sensitivityanalysis of chemical systems are presented. The adjoint of the state-of-the-art model STEM-III is discussed, together with sensitivity and ozone assimilation results for a realistic testproblem.

Keywords. Data assimilation, Chemical transport models, Adjoint modeling.

1. INTRODUCTIONOur ability to anticipate and manage changes in atmospheric pollutantconcentrations relies on an accurate representation of the chemical state of theatmosphere. As our fundamental understanding of atmospheric chemistry advances,novel computational tools are needed to integrate observational data and modelstogether to provide the best, physically consistent estimate of the evolvingchemical state of the atmosphere. Such an analysis state better defines the spatialand temporal fields of key chemical components in relation to their sources andsinks. This information is critical in designing cost-effective emission controlstrategies for improved air quality, for the interpretation of observational data suchas those obtained during intensive field campaigns, and to the execution of air-quality forecasting. The development of the tools to integrate measurements andmodels is also critical to the challenge of a full utilization of the vast amounts ofsatellite chemical data in the troposphere that are now becoming available, andwhich will become more prevalent in the coming years.

Assimilation of chemical information is only now beginning in airquality/chemistry arenas, but offers the same motivations as those realized in thefield of meteorology. Assimilation techniques can be utilized to produce three-dimensional, time varying optimal representations of the chemical composition ofthe atmosphere, that are consistent with the observed physical and chemical states.These optimal analysis states would be of great value to atmospheric chemistry

1F. Darema (ed.), Dynamic.Data Driven Applications Systems, pp-pp© 2004 Kluwer Academic Publishers. Printed in the Netherlands.

A. SANDU, D.N. DAESCU, T. CHAI, G.R. CARMICHAEL, J.H. SEINFELD, P.G. HESS, T.L.ANDERSONresearch. In the case of intense field experiments, for example, the assimilation ofdata in the forecast operations would improve the predictive capabilities of thechemical transport models (CTMs), while in post-analysis the merging andintegration of the flight observations and modeled fields would provide acomprehensive, analyzed, self-consistent, 3-D chemical and dynamic data set thatall scientists could use to help interpret the measurements. The assimilationtechniques can also be used to systematically improve our ability to refineindividual science components. For example, sensitivity and error analysisembedded in the simulation can be used to provide better estimates of chemicalemissions into the atmosphere (inverse modeling), and to provide methodologies toanalyze the simulation and to systematically design optimal measurement strategies(i.e., what measurements are needed and where should they be conducted toimprove our depiction of the chemical state of the atmosphere).

Figure 1. Overview of our 4D-Var chemical data assimilation in atmospheric transportmodels. Solid lines represent current capabilities. Dotted lines represent new analysiscapabilities that arise through the assimilation of chemical data

A schematic diagram of the uses of 4D-Var data assimilation in atmosphericchemistry is presented in Figure 1. A determination of an optimal analysis staterequires the use of both model and measurement data. The traditional inter-relationships between predicted and measured quantities, where the measurementsare used to provide initial and boundary conditions for the models and to evaluatemodel performance, and the modeled quantities provide context and interpretationof the measured quantities, are included in this framework. But this analysis goeswell beyond these boundaries. The sensitivity and error analysis techniquesrequired to produce the optimal analysis state provide new methodologies foranalyzing model simulations and providing the context for comparisons with

COMPUTATIONAL ASPECTS OF 4D-VAR CHEMICAL DATA ASSIMILATION 3measurements. This envisioned intimate integration of measurements and modelsrepresents formidable challenges. For example, the assimilation of chemicalquantities into atmospheric models greatly increased the computation burden (by atleast an order of magnitude). As described in the following sections, a closerintegration of measurements and models will require significant advances in: (i)data assimilation techniques; (ii) numerical algorithms; (iii) software andapplication-specific data mining strategies; and (iv) interfaces betweenmeasurements data and model data. These four topics, along with scienceapplications, represent the major research elements of this analysis.

In this paper, we present computational tools developed for 4D-Var chemicaldata assimilation into atmospheric transport models. The paper is organized asfollows. In Section 2 we review the mathematical theory of adjoint sensitivityanalysis applied to air quality modeling. Computational tools are presented inSection 3. Numerical results for the simulation of East Asia are shown in Section 4.Conclusions and future research directions are given in Section 5.

2. MATHEMATICAL CONSIDERATIONSVariational methods (3D-Var, 4D-Var) provide an optimal control approach to thedata assimilation problem. Four-dimensional variational (4D-Var) data assimilationallows the optimal combination of three sources of information: an a priori("background") estimate of the state of the atmosphere; knowledge about thephysical and chemical processes that govern the evolution of pollutant fields, ascaptured in the model (CTM); and observations of some of the state variables. Theoptimal analysis state is obtained through a minimization process to provide thebest fit to the background estimate and to all observational data (space and timedistributed) available in the assimilation window. The use of adjoint modeling toevaluate the gradient of the objective functional makes feasible the implementationof the 4D-Var data assimilation for large-scale atmospheric models. The practicalapplicability of the 4D-Var requires an accurate model representation of theatmospheric dynamics (perfect model assumption). In addition, the methodprovides no direct information on the errors in the assimilated fields. The optimalityof 4D-Var and its relationship with the Kalman filter is further discussed by Li andNavon (2001). Computational aspects of 4D-Var are described below.

1. Atmospheric Chemistry and Transport ModelingIn what follows we denote by u the wind field vector, K the turbulent diffusivitytensor, ρ the air density in mole/cm3, and ci the mole-fraction concentration ofchemical species i. The density of this species is ρci mole/cm3. Let Vi

dep be thedeposition velocity of species i, Qi the rate of surface emissions, and Ei the rate ofelevated emissions for this species. The rate of chemical transformations fi dependson absolute concentration values; the rate at which mole-fraction concentrationschange is then fi(ρci)/ρ.

Consider a domain �which covers a region of the atmosphere. Let n

�

be theoutward normal vector on each point of the boundary ∂�.At each time moment theboundary of the domain is partitioned into ∂�= �IN ∩ �OUT ∩�GR where �GR is theground level portion of the boundary; �IN is the set of (lateral or top) boundarypoints where u⋅ n

�

≤0 and �OUT the set where u⋅ n�

>0.

A. SANDU, D.N. DAESCU, T. CHAI, G.R. CARMICHAEL, J.H. SEINFELD, P.G. HESS, T.L.ANDERSON

The evolution of ic in time is described by the material balance equations

( )1 1ñ(ñ)

ññi

i i i i i

cu c K c f c E

t

∂ = − ⋅∇ + ∇ ⋅ ∇ + + ,∂

(1)

0 0( ) ( )i ic t x c x, = , (2)IN IN( ) ( ) forÃi ic t x c t x x, = , ∈ , (3)

OUT0 forÃicK x

n

∂ = ∈ ,∂

(4)

dep GRforÃii i i

cK V c Q x

n

∂ = − ∈ .∂

(5)

We refer to the system (1)–(5) as the forward (direct) model. To simplify thepresentation, in this paper we consider as parameters the initial state c0 of themodel; it is known that this does not restrict the generality of the formulation. Thesolution of the forward model c=c(t,c0) is uniquely determined once the modelparameters c0 are specified.

The direct model (1)–(5) is solved by a sequence of N time steps of length ∆ttaken between t0 and t0=T. At each time step one calculates the numericalapproximation ck(x)≈c(tk, x) at tk=t0+k∆t such that

k k+1 k k+1

N-1k+1 k N 0

[t , t ] [t ,t ]k =0

c = c , c = cÍÍ ⋅ ⋅∏ . (6)

Here the numerical solution operator N is usually based on an operator splittingapproach, where the transport steps along each direction and the chemistry steps aretaken successively. Formally, if we denote by T the numerical solution operator fordirectional transport, and by C the solution operator for chemistry we have

Ä2Ä2Ä2ÄÄ2Ä2Ä2[Ä]

t t t t t t tt t t X Y Z Z Y XN T T T C T T T/ / / / / /, + = ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ (7)

2. Continuous Adjoint AnalysisThe adjoint of the tangent linear model defines the evolution of the adjoint variables

( )ëë(ë)ñ(ñ)ëö

ñTi i

i iiu K F c

t

∂ +∇⋅ = −∇⋅ ∇ − − ∂ , (8)

ë( )ë( ) Fi iT x x, = , (9)

INë( ) 0 forÃi t x x, = ∈ , (10)

OUT(ëñ)ëñ0 forà i

iu K xn

∂ /+ = ∈∂

, (11)

COMPUTATIONAL ASPECTS OF 4D-VAR CHEMICAL DATA ASSIMILATION 5dep GR(ëñ)

ñëforÃii iK V x

n

∂ / = ∈∂

. (12)

To obtain the ground boundary condition we use the fact that u⋅ n�

= 0 at groundlevel. ö i is a forcing function yet to be defined. We refer to (8)–(12) as the(continuous) adjoint system of the tangent linear model. Note that the adjoint initialcondition is posed at the final time T.

The adjoint system (9)–(12) depends on the state of the forward model (i.e. onthe concentration fields c(x,t)) through the nonlinear chemical term F(ρc) andpossibly through the forcing term ö for nonlinear functionals. This means that theforward model must be first solved forward in time, the state c(x,t) saved for all t ,then the adjoint model could be integrated backwards in time from T down to t0.

In practice a hybrid approach is used. The forward model is solved using anumerical method, and the numerical approximation of the state is savedperiodically. These checkpoints are used in the definition of the adjoint equations.The continuous adjoint equation (9)–(12) is a convection-diffusion-reactionequation (with linearized chemistry) and can be solved by any numerical method ofchoice. In particular an operator splitting approach could be employed using thesame numerical methods as for solving the direct model

1 1

11 0

[ ] [ ]0

ëëëë k k N k N k

Nk k N

t t t tk

N N+ − − −

−+

, ,=

= ⋅ , = ⋅∏ . (13)

The forcing function ϕi and the initial values λiF are chosen such that the adjoint

variables are the sensitivities of the cost functional with respect to state variables(concentrations)

ë( )( )i

i

Jx t

c x t

∂, =∂ ,

. (14)

3. Discrete Adjoint AnalysisIn this approach the numerical discretization of the (1)–(5) is considered to be theforward model (6). This is a pragmatic view, as only the numerical model is in factavailable for analysis. For brevity the state of the discretized model will be denotedci

k[j], where i is the species index, j is the space discretization index and k the timediscretization index. ck[j] will refer to the vector of all species at time level k andgrid level j. The cost functional is defined in terms of the discrete model state

0

0

( ) ( [ ])N

k

k j

J c g c j=

= ∑∑ (15)

A. SANDU, D.N. DAESCU, T. CHAI, G.R. CARMICHAEL, J.H. SEINFELD, P.G. HESS, T.L.ANDERSONand one wants the derivatives of this functional with respect to the discrete modelparameters ci

0[j]. A perturbation δc0 in the parameters c0 propagates in timeaccording to the tangent linear discrete equation

1 1

11 0

[ ] [ ]0

ääää k k k k

Nk k N

t t t ti

c c c cN N+ +

−+

, ,=

= ⋅ , = ⋅′ ′∏ , (16)

where N ′ is the tangent linear operator associated with the solution operator N .For an operator splitting approach (7) N ′ is built from the tangent linear transportand chemistry operators

ÄÄ2Ä2Ä2Ä2Ä2Ä2[Ä]

tt t t t t tX Y Z Z Y Xt t tN CT T T T T T/ / / / / /

, + = ⋅ ⋅ ⋅ ⋅ ⋅ ⋅′ ′′ ′ ′ ′ ′ ′ . (17)

To each tangent linear operator corresponds an adjoint operator (denoted here witha star superscript). The adjoint equation of (17) is

ÄÄ2Ä2Ä2Ä2Ä2Ä2[Ä]

tt t t t t tX Y Z Z Y Xt t tN CT T T T T T

/ / / / / /∗ ∗∗ ∗ ∗ ∗ ∗ ∗+ , = ⋅ ⋅ ⋅ ⋅ ⋅ ⋅′ ′′ ′ ′ ′ ′ ′ . (18)

such that the resulting (discrete) adjoint model is

11 1

[ ]ëëöë[ ]ë( )k kk k k N F

jt t j xN ++ +∗

,= ⋅ + , =′ . (19)

This approach was taken to build the adjoint of the 3D chemical transport modelSTEM. The exact formulation of discrete adjoint operators depends on thenumerical methods employed to solve the forward model.

The forcing function ϕ and the initial values λN are chosen such that the adjointvariables are sensitivities of the functional with respect to the state variables

0( )ë[ ]

[ ]ki k

i

J cj

c j

∂=∂

. (20)

4. Data AssimilationIn data assimilation applications, the cost functional measures the distance betweenmodel output and observations, as well as the deviation of the solution from thebackground state.

0 0 b 1 0 b obs 1 obs

0

1 1( )

2 2

NT T

k k k kk

k

J c c c B c c c c R c c− , − ,

=

= − − + − −∑ . (21)

In the above the covariance matrix Raccounts for observation andrepresentativeness errors. cb is the background concentration (the initial guess in theassimilation procedure) and B the covariance matrix of the estimated background

COMPUTATIONAL ASPECTS OF 4D-VAR CHEMICAL DATA ASSIMILATION 7error. The covariance matrices account for error correlations between differentspecies as well as different locations. The discrete adjoint model (19) is thencompletely specified with

1 obs 1 0 bö[ ]äë[ ] 0k T k k T Ni i j k i j k ij e R c c e B c c j− , −

, , = − + − , = (22)

where ei,j is a vector of zeros, with a one in the position corresponding to species iand location j.

In the adjoint data assimilation one distinguishes between the continuous andthe discrete adjoint modeling, see Sirkes and Tziperman (1997). Continuous adjointsensitivity in practice is solved numerically, resulting in a discretization of thecontinuous adjoint equations. On the other hand the discrete adjoints are computedfrom the adjoint of the numerical discretization. The operations of discretizationand adjoint usually do not commute, i.e. the discrete and the continuous adjointapproaches lead to different results. The consistency of discrete adjoints withcontinuous adjoints is a topic of ongoing research. For data assimilation problemsone needs the derivative of the numerical solution, i.e. the discrete adjoints are inprinciple preferred. For sensitivity studies using the adjoint method one wants toapproximate the sensitivities of the continuous model, i.e. in this case a continuousadjoint approach may be preferable.

3. COMPUTATIONAL TOOLSIn this section, we present software developed particularly for data assimilation intoatmospheric models.

1. Stiff Numerical MethodsAtmospheric chemical kinetics result in stiff ODE equations that require specialnumerical integration methods which are stable, preserve linear invariants (a.k.a.mass) and are computationally efficient. Sandu et al. (1997) showed thatRosenbrock methods are well suited for solving atmospheric chemistry problems.The forward discrete chemical model in STEM is given by a Rosenbrockdiscretization of the chemical equations

1

1

i

i n i j jj

Y y a k−

,=

= + ∑ ,

( )1

1

1( ) 1

ã

ii j

n i i jj

cJ y k f Y k i s

h h

−,

=

− = + , = , ,

∑ � ,

11

s

n n j jj

y y m k+=

= + ∑ . (23)

Sandu, Daescu, and Carmichael (2003) showed that the corresponding chemicaldiscrete adjoint reads

A. SANDU, D.N. DAESCU, T. CHAI, G.R. CARMICHAEL, J.H. SEINFELD, P.G. HESS, T.L.ANDERSON

,1 ,

1

1( )

ã

sj iT

n i i n j i j jj i

cJ y k m a v u

h hλ +

= +

− = + +

∑ ,

( ) 1 1Ti i iv J Y u i s s= , = , − , ,� ,

( )11

ëë ( )1

sT

n n in ii

suH y k i

iv+

=

= + ⋅ +×=

∑ ∑ . (24)

Here J denotes the Jacobian and H (a 3-tensor) is the Hessian of the derivativefunction f. Yi are the stage solution vectors computed by the forward method (23).The formulation can be easily extended to non-autonomous systems.

The numerical experiments reported here use the two stage, second orderRosenbrock method Ros-2 (Verwer, Spee, Blom, and Hunsdorfer, 1999) which isdefined by the coefficients ã1 2 2= + / , 1 3 2m = / , 2 1 2m = / , 2 1 1a , = , and 2 1 2c , = − . Other popular numerical integration methods for stiff systems are Runge Kutta andBackward Differentiation Formulas (BDF). The discrete adjoints of BDF schemesare in general (for variable step sizes) inconsistent with the continuous adjointequation. Runge Kutta methods on the other hand are well suited for adjointcomputations.

2. The Kinetic PreProcessor (KPP)The implementation of numerical integrators for chemistry can be doneautomatically using the Kinetic PreProcessor KPP software tools (Damian, Sandu,Potra, and Carmichael, 2002). KPP was recently extended (Daescu, Sandu, andCarmichael, 2000; Sandu, Daescu, and Carmichael, 2002) to produce a quick andefficient implementation of the code for building adjoints and performingsensitivity analysis of chemical kinetic systems. KPP builds Fortran or C simulationcode for chemical systems with chemical concentrations changing in time accordingto the law of mass action kinetics. Tests presented by Daescu et. al. (2000) showeda superior performance over the adjoint code generated with the general purposeadjoint compiler TAMC (Giering, 1997). We present here a tutorial example for theadjoint code implementation in the KPP framework. For simplicity, we consider theproblem ( )y f y′ = .

1. The Discrete Forward ModelThe forward numerical integration, yi→yi+1, is performed using the first orderlinearly implicit Euler method (Hairer, Norsett, and Wanner, 1993)

( ) 11( )i i i i i ihJ t y I y y f t y+

, − − = , , i=0, ∝, N - 1, (25)

with a constant step size h; the final time is reached for tN = Nh = tF. Theimplementation of one forward integration step using KPP generated routines is:

SUBROUTINE LEULER(n,y,h,t)....

C – compute the function, Jacobian CALL FunVar (n,t,y,fval)

COMPUTATIONAL ASPECTS OF 4D-VAR CHEMICAL DATA ASSIMILATION 9CALL JacVar_SP (n,t,y,jac)

C – compute J-(1/h)*I and factorize it jac(lu_diag_v(1:n)) = jac(lu_diag_v(1:n)) -1.0d0/hCALL KppDecomp (n, jac, ier)

C – solve: [y^{i}-y^{i+1}] = ( J - (1/h)*I )**(-1) * fvalCALL KppSolve (jac, fval)

C – update next step solution: y^{i+1} = y^{i} - [y^{i}-y^{i+1}]y(1:n) = y(1:n) - fval(1:n) END

2. Continuous Adjoint Implementation The continuous adjoint system ë( )ë TJ y′ = − is integrated backwards in time. Onestep of the backward integration, λi+1→λi, using the linearly implicit Euler methodis written

( )1 1 11 1( )ëëTi i i i

h hJ t y I+ + +, − = − . (26)The implementation is:

SUBROUTINE CAD_LEULER(n,y,ady,h,t)....

C – compute J-(1/h)*I and factorize itCALL JacVar_SP (n,t,y,jac)jac(lu_diag_v(1:n)) = jac(lu_diag_v(1:n)) -1.0d0/hCALL KppDecomp (n, jac, ier)

C – compute: ( J-(1/h)*I )**(-1) * ( -(1/h)*ady )f(1:n) = (-1.0d0/h)*ady(1:n)CALL KppSolveTR(jac,f,ady)END

We now analyze the cost of the forward and backward integration steps. The sparseLU factorization KppDecomp is the same, and cpu(KppSolve) ≈ cpu(KppSolveTR).An adjoint function evaluation requires the product JTλ which is in general moreexpensive than evaluating f. In addition, forward recomputations may be requiredduring the adjoint integration, therefore in general the adjoint step is moreexpensive than the forward step (this is not the case for the linearly implicit Euler,however).3. Discrete Adjoint ImplementationThe discrete adjoint code is obtained by differentiating (1) with respect to yi. Afterrearranging the terms we obtain

1i ik y y+= − , 11ë

T

i iJ I zh

+ − = , (27)

( )1 1ëë( ) [ ]ë( ) ( )T

i i i T i i i T i Ti

J z J k z J z H z ky

+ + ∂= − − = − − × ∂ , (28)

where Hi=∂Ji/∂yi is the Hessian, a very sparse 3-tensor. The KPP implementation ofthe discrete adjoint step is:

SUBROUTINE DAD_LEULER(n,y,ynext,ady,h,t)....

C – compute J-(1/h)*I and factorize it

A. SANDU, D.N. DAESCU, T. CHAI, G.R. CARMICHAEL, J.H. SEINFELD, P.G. HESS, T.L.ANDERSON

CALL JacVar_SP (n,t,y,jac)jac(lu_diag_v(1:n)) = jac(lu_diag_v(1:n)) -1.0d0/hCALL KppDecomp (n, jac, ier)

C – compute kk(1:n) = ynext(1:n)-y(1:n)

C – compute zCALL KppSolveTR (jac,ady,z)

C – compute Hessian and (hess^T x z)*kCALL HessVar ( y, hess )CALL HessVarTR_Vec ( hess, z, k, ady1 )

C – update adjointsady(1:n) = -z(1:n)/h - ady1(1:n) END

On input ady=λi+1, y=yi, ynext=yi+1, t=ti, on output ady = λi.It can be seen that discrete adjoint model is a more demanding computational

process and its efficient implementation is not a trivial task. The discrete adjoint forlinearly implicit methods needs second order derivatives (the Hessian) as well assparse tensor-vector products. For this reason, the use of discrete adjoints inatmospheric chemistry applications has been previously limited to explicit or loworder linearly implicit numerical methods (Fisher and Lary, 1995; Elbern, Schmidt,and Ebel, 1997, 1999; Elbern and Schmidt, 1999; Daescu, et al., 2000).4. KPP Numerical LibraryThe continuous adjoint model can be easily constructed using KPP-generatedroutines and is integrated with any user selected numerical method. The discreteadjoint models associated with the Ros1, Ros2, and Rodas3 integrators are providedfor variable step size integration. Drivers for adjoint sensitivity and dataassimilation applications are also included. In addition Several Rosenbrock methodsare implemented for direct decoupled sensitivity analysis, namely Ros1, Ros2,Ros3, Rodas3, and Ros4, together with a modified version of the BDF direct-decoupled integrator Odessa (Leis and Kramer, 1988).

The building blocks that KPP generates are listed in Table 1.

TABLE 1. KPP building blocks

KPP building blocks DescriptionFunVar time derivative of concentrationsHessVar Hessian of FunVar (in sparse format)HessVar_Vec Hessian times user vectorsHessVarTR_Vec transposed Hessian transpose times user vectorsJacVar Jacobian of FunVar in full formatJacVar_SP Jacobian of FunVar in sparse formatJacVarReactantProd Jacobian of ReactantProdJacVar_SP_Vec sparse Jacobian times vectorJacVarTR_SP_Vec transposed sparse Jacobian times vectorKppDecomp sparse LU decomposition for the JacobianKppSolve solve sparse system with the Jacobian KppSolveTR solve sparse system with transposed Jacobian ReactantProd vector of reaction ratesSTOICM stoichiometric matrix

COMPUTATIONAL ASPECTS OF 4D-VAR CHEMICAL DATA ASSIMILATION 11dFunVar_dRcoeff derivatives of FunVar with respect to reaction coefficients

(in sparse format)dJacVar_dRcoeff derivatives of JacVar with respect to reaction coefficients

times user vector

3. Parallel Implementation and PAQMSGThe forward and adjoint three-dimensional models are parallelized and were run ona cluster of Linux workstations. Parallelization is based on dimensional splitting assupported by our library PAQMSG (Miehe, Sandu, Carmichael, Tang, and Daescu,2002). The library supports data types for structured grids, and implements routinesfor data decomposition, allocation of local and global entities, data scattering,gathering, and shuffling. We use the horizontal-vertical data decompositionpresented in Figure 2. With data in the horizontal slice format each processor cancompute the horizontal transport; then data is shuffled in vertical column formatand each processor can compute radiation, vertical transport, chemistry and aerosolprocesses in one column. The bulk of the computations is done with data in thecolumn partitioned format; PAQMSG implements a static mapping scheme ofcolumns (tasks) to processors that ensures an excellent load balancing. On a clusterof workstations all input and output is handled by the master process (see Figure 3);and all computations are done by the worker nodes.

Figure 2. The horizontal-vertical data decomposition scheme supported by PAQMSG.

Figure 2. The parallel adjoint STEM implements a distributed checkpointing scheme.

For the adjoint we use a two-level checkpointing scheme. The level-2checkpoints store the concentration fields on the disk every 15 minutes, i.e. at everyoperator split step. Note that the linear transport scheme does not require anyadditional checkpointing storage. The amount of level-2 checkpoint data increasesfivefold if a nonlinear transport scheme (e.g. using flux limiting) is used. The level-1 checkpoints store the concentrations for each process inside the 15 minutes

A. SANDU, D.N. DAESCU, T. CHAI, G.R. CARMICHAEL, J.H. SEINFELD, P.G. HESS, T.L.ANDERSONintervals; level-1 checkpoints use memory buffers. For example, one forwardintegration of each chemical box model for 15 minutes split time interval requires anumber of smaller time steps; these intermediate concentrations are stored in atemporary matrix and used during the backward integration of the adjoint model.Operator splitting and the relative short split time intervals make it feasible to storethe level-1 checkpoints in memory.

The gas phase chemical mechanism is SAPRC-99 (Carter, 2000) whichconsiders the gas-phase atmospheric reactions of volatile organic (VOCs) andnitrogen oxides (NOx) in urban and regional settings. The forward time integrationis done with the Rosenbrock numerical integrator Ros-2; the continuous adjointmodel uses Ros-2 on the same sequence of steps as the forward chemicalintegration. Both the forward and the adjoint models are implemented using KPP.

For our East Asia application discussed in the following section the total level-2checkpoint information stored is ~ 162 MBytes of data for each hour of simulation;or ~ 4 GBytes per 24 hours of simulation. The level-2 checkpoints of the parallelmodel are distributed as shown in Figure 2, where each node stores localinformation on the local disk. This takes full advantage of the total storagecapabilities of the system. It also decreases the communication overhead when theparallel computation runs on a cluster of workstations since the gigabytes of dataare not transmitted over the (relatively slow) connection. The distributedcheckpointing strategy is therefore essential for both efficiency and overall storagecapacity. Note that for the static domain decomposition implemented in PAQMSGthe local entities (i.e. horizontal slices or sets of columns of the concentration field)have the same size throughout the computation, which makes the implementation ofthe distributed checkpointing scheme very efficient. For a dynamic domaindecomposition strategy, on the other hand, the size of local entities change duringthe computation and the implementation of distributed checkpointing becomescomplicated.

The East Asia test case is run on a Beowulf cluster with 20 nodes (Pentium 4,2GHz, 1GB RAM) and Gigabit ethernet connection; the one hour forward andbackward simulation corresponds to 0-1 GMT on March 1st, 2001. On 16 workersthe absolute cpu time for a forward run is about 2 minutes per hour of simulation;and the cpu time for a forward-backward run is about 5 minutes per hour ofsimulation.

4. NUMERICAL RESULTSThe adjoint of the STEM chemical transport model can be used in sensitivityanalysis studies and also for chemical data assimilation. We now present these twoimportant applications of the computational tools developed. The analyzedproblems are in support of the NASA TRAnsport and Chemical Evolution over thePacific (TraceP) field experiment conducted in East Asia. The simulated region isEast Asia, and the simulated interval is one month starting at 0 GMT on March 1st,2001. The meteorological fields are given by a dynamic meteorological model(RAMS) (Pielke et al., 1992), and the initial fields and boundary conditionscorrespond to Trace-P data campaign. The grid has 90×60×18 points and has ahorizontal resolution of 80Km×80Km. Details of the forward model simulation

COMPUTATIONAL ASPECTS OF 4D-VAR CHEMICAL DATA ASSIMILATION 13conditions and comparison with observations are presented in Carmichael et al.(2003).

1. Adjoint Sensitivity AnalysisSensitivities of the response functional g=g(c(tF)) with respect to the state variables(at each time instant) are the adjoint variables λ(t), which can be obtained byintegrating the adjoint model backwards in time. The distributions of the adjointvariables in the three-dimensional computation domain, which are available at anyinstant, provide the essential information for the sensitivity analysis. For instance,isosurfaces of adjoint variables delineate “influence regions”, i.e. areas whereperturbations in some concentrations will produce significant changes in theresponse functional (e.g. ozone at Cheju Island at the final time).

Figure 4 displays the influence areas of ozone at 24 hours before the final timein case 2 (March 4–6) and case 8 (March 22–26), respectively. The influence regionfor case 8 is toward the South and close to the Cheju Island, while that for case 2 istowards the Northwest. This difference reflects different meteorological conditions,as indicated by the wind fields shown in Figure 4.

Figure 4. Influenceareasof ozoneon ozoneat Chejuat 24 hoursbeforethefinal time from(a) March 4–6, and (b) March 22–24.The isosurfacesof λo3=0.001 are shownas darkobjectsfor bothcases.Wind vectorsat 2kmabovethesealevelare also shownover lightlygray-scaledtopography.The differencesin isosurfaceshapesand locations are due todifferences in meteorological fields.

2. Data AssimilationThe preliminary data assimilation tests were conducted using the twin experiments.The descriptions are given in Table 2.

TABLE 2. Descriptions of data assimilation experiments

Item DescriptionReference run all chemical species initial concentrations availableAssimilation window 6 hours starting from t0=0 GMT on March 1stObservations O3 and/or NO2 concentrations at the end of the assimilation

window at all grid points from the reference runControl parameters initial concentrations of O3 or NO2

Initial guess reference initial values increased by 20%

A. SANDU, D.N. DAESCU, T. CHAI, G.R. CARMICHAEL, J.H. SEINFELD, P.G. HESS, T.L.ANDERSON

Cost functional including both observation and background termsOptimization algorithm L-BFGS (Byrd, Lu, and Nocedal, 1995)Termination criteria cost functional reduced by more than 3 orders, or the number

of model integrations exceeds 15

We consider several scenarios, with the control variables being O3 or NO2, andthe observed variables being O3 and/or NO2. The performance of the dataassimilation procedure is measured by the decrease of the cost functional value andthe error of control variables. The decreases of the cost function value and RMSerror versus the number of model runs during the optimization procedure are shownin Figure 5.

With O3 as control variables, the optimization procedure produces a rapiddecrease in the cost function value, and a good decrease in the RMS error. Most ofthe information comes from O3 observations; additional NO2 observations do notseem to bring noticeable benefits. This may be due to the lack of scaling in ourformulation of the cost functional. These results imply that ozone initial conditionsis recoverable through data assimilation. For comparison we include theoptimization of the cost functional without the background term (corresponding toan infinite background covariance). As expected the cost function decreases further.

With NO2 as control variables, the decrease in the cost function, and in the RMSerror, is not as pronounced as that for O3. Again most of the information comesform O3 measurements, with additional NO2 measurements contributing very little tothe optimization process. After about 10 model runs the RMS errors tend tostagnate, even if the cost functional continues to decrease. Perturbing the initial NO2

concentration by 20% results in only a small change in the final (observed) O3

concentration. This may be explained by the fact that NO2 levels are driven mostlyby emissions, and less by the initial conditions, which affects the observability ofthe initial NO2 field through ozone measurements. The results indicate that furtheralgorithmic developments are needed for assimilating NO2. In particular a betterscaling of the cost function, through a rigorous definition of the covariancematrices, is necessary.

COMPUTATIONAL ASPECTS OF 4D-VAR CHEMICAL DATA ASSIMILATION 15

Figure 5. The evolutions of cost function and RMS error of the control variable during theoptimization procedure. The results are normalized by their pre-assimilation values. Severaltests are shown using different control (CTRL) and observed (OBS) variables.

5. CONCLUSIONS AND FUTURE WORKIn this paper we presented the computational tools developed for chemical dataassimilation into atmospheric transport models. The focus is on developing efficientadjoints for the stiff ordinary differential equations arising in the simulation ofchemistry.

An adjoint of the state-of-the-science chemical transport model STEM-III hasbeen developed; it implements an efficient distributed checkpointing scheme usingPAQMSG parallelization library; the chemical subsystem is generated with KPP.Adjoint sensitivity analysis delineate “influence regions” of ozone on ozone atCheju for different cases in TraceP scenario. Data assimilation results are shown fora 6 hours test case, where the perturbed initial concentrations of ozone arerecovered from the known ozone concentrations at the end of the simulationinterval.

Future work will focus on continuing the theoretical analysis of discrete adjointsfor stiff solvers; on using the developed computational infrastructure to run morecomplex tests and to assimilate real measurements data; and on continuing toimprove the computational infrastructure for data assimilation. The analysis is alsobeing extended to include aerosol dynamics and chemistry, and to inverseapplications targeting the recovery of optical emission estimates.

ACKNOWLEDGMENTSThe authors thank the National Science Foundation for supporting this workthrough the award NSF ITR AP&IM 0205198.

REFERENCES1. R. Byrd, P. Lu, and J. Nocedal, A limited memory algorithm for bound constrained optimization,

SIAM J. Sci. Stat. Comput., 16 (1995), 1190–12082. Z. Li and I.M. Navon Optimality of 4D-Var and its relationship with the Kalman filter and Kalman

smoother, Q.J.R. Meteorol. Soc., 127 (2001), 661- 6843. G.R. Carmichael, Y. Tang, G. Karata, I. Uno, D. Streets, N. Thongboonchoo, J.H. Woo, S. Gutikunda,

A. White, T. Wang, D.R. Blake, E. Atlas, A. Fried, B. Potter, M.A. Avery, G.W. Sachse, S.T.Sandholm, Y. Kondo, R.W. Talbot, A. Bandy, D. Thorton, and A.D. Clarke, Regional-Scale Chemical

A. SANDU, D.N. DAESCU, T. CHAI, G.R. CARMICHAEL, J.H. SEINFELD, P.G. HESS, T.L.ANDERSON

Transport Modeling in Support of the Analysis of Observations obtained during the TRACE-PExperiment, J. Geophys. Res., 108D21 (2003)

4. W.P.L. Carter, Implementation of the saprc-99 chemical mechanism into the models-3 framework(Technical report), United States Environmental Protection Agency, January 2000

5. D. Daescu, G.R. Carmichael, and A. Sandu, Adjoint Implementation of Rosenbrock Methods Appliedto Variational Data Assimilation Problems, Journal of Computational Physics, 165 (2000), 496–510

6. D. Daescu, A. Sandu, and G.R. Carmichael, Direct and adjoint sensitivity analysis of chemical kineticsystems with KPP: II – Numerical validation and applications, Atmospheric Environment, 37 (2003),5097-5114

7. V. Damian, A. Sandu, M. Damian, F. Potra, and G.R. Carmichael, The kinetic preprocessor KPP – asoftware environment for solving chemical kinetics, Computers and Chemical Engineering, 26 (2002),1567–1579

8. H. Elbern, H. Schmidt, and A. Ebel, Variational data assimilation for tropospheric chemistrymodeling, Journal of Geophysical Research, 102D12 (1997), 15,967–15,985

9. H. Elbern and H. Schmidt, Eulerian chemistry transport model, Journal of Geophysical Research,104D15 (1999), 18,583–18,598

10. H. Elbern, H. Schmidt, and A. Ebel, Implementation of a parallel 4D-variational chemistry data-assimilation scheme, Environmental management and health, 10 (1999), 236–244

11. M. Fisher and D. J. Lary, Lagrangian four-dimensional variational data assimilation of chemicalspecies, Q.J.R. Meteorol. Soc., 121 (1995), 1681–1704

12. R. Giering, Tangent linear and adjoint model compiler - users manual, Technical report, Max-Planck-Institut fur Meteorologie, 1997

13. E. Hairer, S.P. Norsett, and G. Wanner, Solving Ordinary Differential Equations I. Nonstiff Problems,Springer-Verlag, Berlin, 1993

14. J.R. Leis and M.A. Kramer, ODESSA - An Ordinary Differential Equation Solver with ExplicitSimultaneous Sensitivity Analysis, ACM Transactions on Mathematical Software, 14 (1988), 61–67

15. P. Miehe, A. Sandu, G.R. Carmichael, Y. Tang, and D. Daescu, A communication library for theparallelization of air quality models on structured grids, Atmospheric Environment, 36 (2002),3917–3930

16. R.A. Pielke, W.R. Cotton, R.L. Walko, C.J. Tremback, W.A. Lyons, L.D. Grasso, M.E. Nicholls,M.D. Moran, D.A. Wesley, T.J. Lee, and J.H. Copeland, A comprehensive meteorological modelingsystem - RAMS, Meteorol. Atmos. Phys., 49 (1992), 69–91

17. A. Sandu, J.G. Verwer, J.G. Blom, E.J. Spee, G.R. Carmichael, and F.A. Potra, Benchmarking stiffODE solvers for atmospheric chemistry problems II: Rosenbrock methods, Atmospheric Environment,31 (1997), 3459–3472

18. A. Sandu, D. Daescu, and G.R. Carmichael, Direct and adjoint sensitivity analysis of chemical kineticsystems with KPP: I – Theory and software tools, Atmospheric Environment, 37 (2003), 5083-5096

19. Z. Sirkes and E. Tziperman, Finite difference of adjoint or adjoint of finite difference? Mon. WeatherRev, 49 (1997), 5–40

20. J. Verwer, E.J. Spee, J.G. Blom, and W. Hunsdorfer, A second order Rosenbrock method applied tophotochemical dispersion problems, SIAM Journal on Scientific Computing, 20 (1999), 1456–1480