1

What are retrieval averaging kernels?

Chris Barnet and Eric Maddy

NOAA/NESDIS/STAR

Monday Oct. 19, 2009 University of Maryland

2

Sounding Theory Notes for the discussion today is on-line

voice: (301)-316-5011 email: chris.barnet@noaa.gov ftp site: ftp://ftp.orbit.nesdis.noaa.gov/pub/smcd/spb/cbarnet/ ..or.. ftp ftp.orbit.nesdis.noaa.gov, cd pub/smcd/spb/cbarnet

Sounding NOTES, used in teaching UMBC PHYS-741: Remote Sounding and UMBC PHYS-640: Computational Physics (w/section on Least Square Fitting and Instrument Apodization)

~/reference/rs_notes.pdf ~/reference/phys640_s04.pdf

These are living notes, or maybe a scrapbook – they are not textbooks.

An excellent text book on the topic of remote sounding is:

Rodgers, C.D. 2000. Inverse methods for atmospheric sounding: Theory and practice. World Scientific Publishing 238 pgs

3

For more details on averaging kernels

1)  Rogers 2000, pg. 43-44 & pg. 83-85 2)  Rodgers, C.D. and B.J. Connor 2003. Intercomparison

of remote sounding instruments. J. Geophys. Res. v.108 p.1-14. doi:10.1029/2002JD002299

3)  Maddy, E. S., Barnet, C. D. & Gambacorta, A. (2009), A computationally efficient retrieval algorithm for hyperspectral sounders incorporating a priori information, IEEE Transactions on Geoscience and Remote Sensing Letters v.6 p.802-806, doi: 10.1109/LGRS.2009.2025780

4)  Maddy, E.S. and C.D. Barnet 2008. Vertical resolution estimates in Version 5 of AIRS operational retrievals. IEEE Trans. Geosci. Remote Sens. v.46 p.2375-2384.

5)  My notes – section 8.12.1

4

What are Averaging Kernels

•  Averaging kernels are a linear representation of the weighting of information content of retrieval parameters. –  Related to the partition of information determined from the radiances

and the prior information (usually a first guess). •  Usually employed for assessment of vertical information.

–  Related to the error covariance and vertical resolution of retrievals (i.e., correlation of retrieval parameters)

–  Required by modelers to properly use retrieval products. –  Enables assessment of retrieval skill on a case by case basis.

•  For a perfect instrument (i.e., no noise and 100% information content) and retrieval (i.e., no regularization)

A = I, the identity matrix •  For a single channel retrieval the averaging function is

approximately proportional to the channel kernel function (this is also true for multiple channel retrieval where all kernel kernels are the same, e.g., AIRS CO2).

5

Physical retrieval is a minimization of a constrained cost function

Covariance of observed minus computed radiances: includes instrument noise model and spectral spectroscopic sensitivity to components of the state, X, that are held constant (physics a-priori spectral information).

Derivative of the forward model is required to minimize J.

Covariance of products (e.g., T(p), q(p), CO2(p) ) can be used to optimize minimization of this underdetermined problem. Need to decide how much a-priori statistics is desired in the product. For climate products one can use a minimum variance approach (C = λI) to eliminate inducing correlations. For weather, geophysical correlations (model statistics) are most likely desired.

6

Physics knowledge is what allows interpretation of spectra

•  Given an estimate of the atmospheric state we can compute transmittance.

•  Weighting kernels, dR/dτ, determine where transmittance changes quickly.

•  Channel kernel functions, K, include the effect of temperature lapse rate on a channels sensitivity.

•  If we map measured brightness temperature to altitude of sensitivity we can get a reasonable estimate of the temperature profile directly from the spectrum.

7

Using the inversion equation to derive Vertical Averaging Kernels

•  Our retrieval equation (solution to cost function) can be written as

•  Note that this equation represents a weighted average of the state determined via radiances and the a-priori state. –  The observational covariance, N, can be written as KTNK,

in geophysical units –  The product covariance is given by [KTN-1K + C-1]-1

8

We can derive the averaging function from our minimization equation

•  As we approach a solution, we can linearize the retrieval about a state that approaches the “truth”

•  And we can simplify by replacing the region highlighted in green above with the variable G

zero

9

Computing the averaging function

•  The vertical averaging function is the amount of the derived state that came from the radiances

•  And I-A is the amount that came from the prior

Retrieval covariance Inverse of a-priori covariance

10

Value of the vertical averaging function (slide 1 of 2)

•  Averaging function, A, is the retrieval weighting of the channel kernel functions (think of a retrieval operator as an integrator of channel kernel functions).

•  When using retrieval products the A matrix –  Describes the vertical correlation between parameters –  Tells you how much to believe the product and where to

believe the product. –  A-priori assumptions can be separated from the solution if

we are in a linear domain. •  von Clarmann, T. and U. Grabowski 2007. Elimination of hidden a

priori information from remotely sensed profile data. Atmos. Chem. Phys. v.7 p.397-408.

–  Given the error covariance of the a-priori, Cj,j, the averaging function can be used to derive the propagated error covariance of the retrieval.

11

Value of the vertical averaging function (slide 2 of 2)

•  When comparing correlative measurements (such as high vertical resolution sondes or profiles acquired by aircraft) the validation measurements –  Should have similar vertical smoothing and

•  Otherwise retrieval is penalized for low vertical resolution. •  Sometimes this is done by computing statistics on vertical bins

(e.g., 1K per 1-km) – this is a simple form of vertical averaging. –  Should be “degraded” by the fraction of the prior in the

solution (e.g., in regimes with low information content) •  Useful for retrieval performance assessment (e.g., is it functioning

properly). •  Should NOT be done for product validation, since the prior is part

of the product. –  In essence, the “truth” data is run through the retrieval

filter (averaging function) to produce a profile that is directly comparable to the product derived from the instrument radiances.

12

Example #1: An example of retrieval smoothing and convolution (O3 hole S. Pole)

• Smoothed sonde calculated assuming averaging kernel = identity matrix

–  Ideal case -- what we would do in the absence of damping.

• Convolved sonde using case dependent averaging kernel. Retrieval and convolved sonde compare very well.

13

Aj,j

“truth”

a-priori

retrieval “convolved truth”

Example #2: AIRS CO product

This retrieval is only believed at the 50% level

No information from radiances

14

Comparison of optimal estimate and AIRS science team algorithm’s CO product

•  AIRS science team approach does not minimize a cost function.

•  Instead it uses regularized least squares without a prior constraint.

•  This impacts the averaging kernels in the sense that better information can be acquired if profile shape is more realistic (less errors in Jacobian, K)

Above left: Optimal estimation (red) and AIRS science team (blue) methods produce similar total column amounts. Both profiles have no-skill in lower 3-km, but O-E profile is more realistic statistically.

Above right: O-E averaging kernels (solid) are slightly lower, therefore, O-E allows more lower tropospheric sounding.

15

Example #3: Comparison of NOAA CO2 Product and Kawa 2004 Model for April 2005

•  At first glance, it looks

like the retrieval and model (used as the “truth“ ) do not agree.

•  But if we apply the averaging kernel to the model and we “degrade” it with the retrieval a-priori , they agree quite well.

•  Again, this is because we do not have 100% of information coming from the satellite (this result is within the instrument and propagated error).

ajjjjjjj XAIXAX ⋅−+⋅= )(ˆ

,,

X̂

16

Averaging Kernels Limitations (slide 1 of 2)

•  In a system where a statistical regression is used as a prior (e.g., AIRS science team algorithm), the averaging kernels are a conservative estimate of the vertical correlation of products because the startup regression solution has it’s own averaging kernel that is only known in a statistical sense. –  This becomes important only when our products are over-damped. –  We (NOAA) have the ability to calculate this averaging kernel (via a

brute force method) for case studies if necessary. •  A stepwise retrieval without proper communication of error

covariance can complicate interpretation of averaging kernels –  There is a cross-talk between averaging kernels.

•  The AIRS science team retrieval uses a “background term” and believes a fraction of the radiances derived in the prior. Due to iteration of this term, the averaging kernel for products does not equal the amount of the radiances believed.

•  Separation of signals using propagated noise covariance terms as well as intelligent selection of channels could minimize this effect.

–  Non-linearity is not properly handled by a linear averaging kernel analysis.

17

Simplified Flow Diagram of AIRS Science Team Algorithm

Microwave Physical for T(p), q(p), LIQ(p), ε(f)

Climatological First Guess for all

products

Initial Cloud Clearing, ηj, Rccr

Improved Cloud Clearing, ηj, Rccr

Final Cloud Clearing, ηj, Rccr

IR Regression for Ts, ε(ν), T(p), q(p)

IR Physical Ts, ε(ν), ρ(ν)

IR Physical Ts, ε(ν), ρ(ν)

IR Physical T(p)

IR Physical T(p)

IR Physical Ts, ε(ν), ρ(ν)

IR Physical q(p)

IR Physical O3(p)

IR Physical CO(p)

IR Physical HNO3(p)

IR Physical CH4(p)

IR Physical CO2(p)

IR Physical N2O(p)

Note: Physical retrieval steps that are repeated always use same startup for that product, but it uses retrieval products and error estimates from all other retrievals.

MIT

FG CCR

RET

18

Averaging Kernels Limitations (slide 2 of 2)

•  Vertical weighting is strictly defined on the retrieval parameter grid, not the radiative transmittance algorithm (RTA) grid. –  Any estimate of resolution based on the internal

averaging kernels is limited by the resolution of retrieval parameter functions (Trapezoids in AIRS science team algorithm).

–  Transformations between retrieval parameter functions and radiative transfer grid layers exist; however they assume that we can “upsample” derivatives without loss of accuracy.

•  Not a big problem if we have sampled the atmosphere adequately with respect to channel kernel functions.

•  An unconstrained retrieval (least squares) cannot compute a meaningful averaging kernel.

19

Illustration of retrieval parameter functions AIRS science team trapezoids for T(p)

•  Example of bottom 11 functions used in AIRS T(p) retrieval

•  Functions overlap and are shown in alternating blue, red, black colors.

20

In order to analyze complete system we can compute a “brute force” averaging function

1.  Start with the retrieval state, X0 2.  Perturb X0 in some atmosphere layer by δXk 3.  Compute change in radiance, R(X0+δXk)-

R(X0) 4.  Compute a new retrieval, Xk, using the

perturbed radiance. 5.  Xk-X0 is the jth column of Akj 6.  Goto Step 1 and compute another row of A

This method has the advantage that the entire system, including cloud clearing, regression, and multiple-interacting and non-linear parameters, can be analysed.

21

Inversion in the sonde profile partially captured in the retrieval

Example with an inversion Red: Retrieval, Black: Sonde

22

UfUT = Retrieval averaging kernel

Trapezoids approaching TOA

Trapezoids at Surface

Sum along a row

23

UfUT = Retrieval averaging kernel

24

Averaging kernels enable assessment of where we have increased resolution and information, BUT

•  The increase in information content does not always occur when we would expect it to (e.g. inversions) –  Many cases show oscillations and peaks where we

would not expect them. •  If the retrieval is not properly optimized (e.g. an

incorrect (low) internally propagated error estimate). –  Increased resolution at one layer will result in a loss of

accuracy à This is a “Backus-Gilbert trade off” between retrieval resolution and accuracy

•  Averaging kernels allow evaluation of the cause (or reasonableness) of the vertical oscillations in our biases.

25

Example without an inversion Red: Retrieval, Black: Sonde

No inversion this time, …

26

but there is a still peak and oscillation in the A matrix

Use of water sensitive channels in 2nd T(p) retrieval may be inducing oscillations.

27

Resolution estimates from error covariance matrices and averaging kernels

•  Vertical resolution of any retrieval is related to the width of the channel kernel functions and hence retrieval averaging kernels. –  Backus, G. and Gilbert, F. 1968 The resolving power of gross

Earth data. Geophys. J. R. astr.Soc. v.16 p.169-205. –  Conrath, B.J. 1972 Vertical resolution of temperature profiles

obtained from remote radiation measurements. J. Atmos. Sci. v.29 p.1262-1271.

•  We can also define the vertical resolution in terms of the error correlation between atmospheric layers.

( )iii

ji

jijiρ xxx

xx -ˆ ;

,cov, =Δ

ΔΔ=

σσError correlation matrix

Retrieved value at RTA grid index, i

Truth value at RTA grid index, i

28

Vertical correlation and resolution at ARM-TWP

29

Vertical correlation and resolution at ARM-SGP

30

Vertical correlation and resolution in operational sondes

31

Error estimates and averaging kernels for temperature, a linear analysis

Linear Error is Composed Of: 1.  Instrument Error 2.  Smoothing Term

(Geophysical Functions) 3.  Propagated Error

Covariance Actual retrieval error (blue) lies

within the error estimate (red) for most of the atmosphere.

Predicted error using exact knowledge (magenta) of the

errors of the Initial state error (green) lies on top of actual error (blue). That is, retrieval methodology is linear enough to propagate errors.

32

Error Estimates and Averaging Kernels for Moisture, Linear Analysis

Actual retrieval error (blue) lies within the error bars (red) for most of the atmosphere. Predicted error using exact knowledge (magenta) of the errors of the initial state (green) lies on top of actual error (blue)

33

We Think We Know How To Do This Without Serious Impact to Execution Time!

•  The “trick” is how to propagate the errors through all AIRS science team retrieval steps in a cost effective manner.

•  We can compute the full error covariance and then decompose it and propagate the full covariance with a small number of terms

•  This also could be an approach to communicate retrieval errors to the assimilation community.

34

Summary

•  AIRS averaging kernels and smoothing operators enable “fair” comparison of the physical retrieval to correlative measurements –  smoothing due to trapezoids –  smoothing due to damping (averaging kernel)

•  Averaging kernel derived resolution is similar in vertical shape to resolution derived from error covariance matrices. –  averaging kernels for the physical temperature and moisture

retrievals are good representations of retrieval vertical weighting •  The is a lot to learn about how to communicate retrieval

information content in other applications (e.g., data assimilation). –  Averaging kernels and/or error covariance matrices

35

Backup Slides

Statistical Regression

Least Squares Fitting as a introduction to retrieval

Physical Forms of the retrieval equation

Taylor Expansion of the retrieval equation (AIRS science team methodology)

Effect of AIRS science team trapezoids on averaging kernels.

36

Statistical Regression Retrievals (see Goldberg et al. 2003)

•  Statistical eigenvector regression uses Je observed spectra (on a subset of M “good” channels) to compute eigenvectors. The spectral radiance for scene j, Rn(m),j, can then be represented as principal components, Pk,j

•  The eigenvectors can be determined using a couple of days of satellite (cloudy) radiances by solving

λk = Ek,m·(Δθm,jΔθTj,m)·ET

m,k

37

Statistical Regression Retrievals (continued)

•  A regression, Ai,k, between a “truth” state parameter i, Xi,j, and principal components (centered about mean of ensemble) can be computed.

•  Truth states, as we will discover in lecture #3, are difficult to come by. We can use models (AIRS Science Team Approach uses ECMWF), radiosondes, etc.

•  The equation above is solved by least squares. Since it uses a truncated set of principal components (AIRS Science Team Approach uses 85/1600) the inversion does not need to be regularized.

38

Pro’s and Con’s Of Statistical Regression Retrievals

Pro’s Con’s

Does not require a radiative transfer model for training or application.

Training requires a large number of co-located “truth” scenes.

Application of eigenvector & regression coefficients is VERY fast and for hyper-spectral instruments it is very accurate.

The regression operator does not provide any diagnostics or physical interpretation of the answer it provides.

Since real radiances are used the regression implicitly handles all systematic instrument calibration issues. This is a huge advantage early in a mission.

The regression answer builds in correlations between geophysical parameters. For example, retrieved O3 in biomass regions might really be a measurement of CO with a statistical correlation between CO and O3.

Since clouds are identified as unique eigenvectors, a properly trained regression tends to “see through” clouds.

Very difficult to assess errors in a regression retrieval without the use of a physical interpretation.

39

Review: Traditional Least Squares

•  A linear system of n equations of an observable, yn, and a model, Kn,j, can be expressed as follows

•  An unconstrained least squares fit, when n > j, can be found by inversion of Kn,j

•  Where the inverse of a asymmetric matrix is given by:

40

Example of LSQ #1 Polynomial

•  For example, if the desired fitting equation is a polynomial given by

•  Then Kn,j is given by

41

Example of LSQ #2 Polynomial plus sine function

•  Suppose we wanted to fit an oscillating function (e.g., the Mauna Loa measurement of CO2(t)). The fitting function could be given by

•  And Kn,j is given by

42

Unconstrained LSQ retrieval •  For non-linear LSQ (where Kn,j may be a function of the state

parameters), xj

•  And we may want to impose weighting on the observations

•  The solution can be written in an iterative form

•  The linear algebra solution is identical to minimization of a cost function

43

What we learn from using LSQ analysis of hyper-spectral radiances

•  Linear variables are more stable –  For example, log(q) is better than q

•  Weighting can mitigate geophysical channel interactions

•  Can minimize “null space” error by selecting unique (i.e., non-interacting) geophysical parameters

•  Error in product space can be estimated (and propagated)

44

The Inverse Solution: Low Resolution Instruments

[ ]ajjjnnnTnjjjjnnn

Tnjj XCRNKCKNKX Δ⋅+Δ⋅⋅⋅+⋅⋅=Δ −−−−− 1

,1,,

11,,

1,, ][

Traditional methods (Rodgers, Eyre, etc.) had to rely on the statistics of the a-priori term (models, climatologies, etc) due to lack of information from the measurements (HIRS/MSU had 23 sounding channels). Typically the instrument error is neglected, that is N-1 = I, in this formulation.

Measurement Covariance

Constraint Weighted Average of

Observations & a-priori

45

The Inverse Solution: Hyper-spectral Instruments

•  AIRS: 2378 channels •  IASI: 8461 channels

[ ]ajjjnnnTnjjjjnnn

Tnjj XCRNKCKNKX Δ⋅+Δ⋅⋅⋅+⋅⋅=Δ −−−−− 1

,1,,

11,,

1,, ][

Hyper spectral Instruments measurements have much higher information content: AIRS inverse method exploits the high information content of the instrument & a-priori information in the radiative physics without a penalty in execution time.

46

Iterative Solution to the Cost Function has many forms

•  Optimal estimation can “pivot” off of the a-priori state.

•  Equivalent to “pivoting” from the previous iteration:

•  The background term, modifies obs-calc’s to converge to a regularized solution. Form used in our algorithm:

47

The cost function minimizes differences between observations and computed radiances

•  Linear minimization of cost function is equivalent to expanding Obs-calc’s into a Taylor expansion and minimizing with constrained LSQ fitting.

•  In a linear operator, the different components of geophysical space can be separated.

48

The Problem is Physical and Can be Solved by Parts

•  Careful analysis of the physical spectrum will show that many components are physically separable (spectral derivatives are unique)

•  Select channels within each step with large K and small en

•  This makes solution more stable. •  And has significant implications for

operational execution time.

49

A Note on use of Trapezoidal Vertical Functions (NOTE in these slides we use symbol “A” for the retrieval

parameter and ϕ for Averaging function)

•  In the AIRS science team retrieval algorithm, trapezoidal functions (denoted, ) are used to interpolate the change in retrieval parameters, ΔAj, onto the 100 layer RTA grid:

•  These functions serve three purposes: –  Define a reduced measurement space on which finite difference

derivatives are calculated. –  Ensure a smooth product (interpolation). –  Significantly faster algorithm.

•  Transformation between 100 layer RTA grid and coarse layers, ΔAj, is provided by a least squares estimate:

∑ Δ=Δj

jjLL AFx ,

Coarse layer retrieved quantities Fine level/layer

retrieved quantities interpolated onto RTA grid.

jL,F

)(][ ',0'','1

',,'

'', LLTLjjL

TLj

LLLjj xxFFFxFA −=Δ=Δ −+∑

50

Linear vs. Log derivatives ϕ = averaging kernel in this slide

[Rodgers and Connor 2003] form of the equation assumes linearity in changes in state. For temperature this is true and we have:

For minor constituents (H2O, O3, CO, CH4, etc.) the averaging kernels act in logarithmic or % changes in state:

⎥⎦

⎤⎢⎣

⎡⎟⎟⎠

⎞⎜⎜⎝

⎛ −⋅+=′0

00 x

xxΦ1xx

For small perturbations/low information content we can write in terms of % changes relative to the first guess:

( )00 xxΦxx −⋅+=′

Convolved truth

First Guess

Averaging Kernel

Truth

Unit vector

)/log()log()log( 00 xxΦxx ⋅+=′

51

Retrieval Functions and Convolution Recipe

The retrieval calculates coarse layer derivatives and assigns retrieved changes to fine layers using routine slb2fin (trapezoids denoted ). We can handle the trapezoidal retrieval functions in much the same way that the retrieval handles them by: 1.  Calculating coarse layer delta states. e.g.,

2.  Apply averaging kernel to coarse layer deltas and use the functions to interpolate to the RTA grid.

3.  `Use convolution equation on interpolated convolved delta state:

)(][ ',0'','1

',,'

'', LLTLjjL

TLj

LLLjj xxFFFxFA −=Δ=Δ −+∑

∑∑ Δ⋅Φ⋅=−=Δ'

'',,,0 ][~~~j

jjjj

jLLLL AFxxx

xxx ~' 0 Δ+=

jL,F

Minor gases: Let: x = log(x)

52

Trapezoidal Null Space

•  Projecting the truth-fg onto the trapezoids and interpolating onto the RTA grid.

•  Standard deviation between smoothed truth and truth (note this is dependent on the trapezoid spacing, variability in the truth and variability in the first guess).

'',',,0~~~

LLjjLLLL xFFxxx Δ⋅⋅=−=Δ +

LL

jjLLjjLL

LLjjLL

xxAFFFx

xFFx

Δ=Δ

Δ⋅⋅⋅=Δ

Δ⋅⋅=Δ+

+

~ ~

~

,'','',

'','', Components of the trapezoidal smoothing error are zero if the difference between the first guess and “truth” can be written as a superposition of trapezoidal perturbations!

F+ Slab avg. T(p) 0.25K-0.5K 0.5K-1.0K

H2O(p) 5%-10% 10%-20% O3(p) 5%-10% 10%-20%

53

Retrieval Smoothing Terms

•  Retrieval smoothing is composed two terms: –  Regularization (e.g. a noise threshold value termed Bmax). –  Trapezoidal interpolation rule.

•  Regression can impart high resolution structure, this structure is removed from the comparison by the trapezoidal smoothing terms if it is finer than the width trapezoids.

•  The following slide illustrates each component.

'',1

,,'',, ][ˆ LTLjjL

TLjjjjLL xFFFΦFx Δ⋅⋅=Δ −

Trapezoidal Smoothing

Averaging Kernel Smoothing

54

Examples of Statistics using Averaging Kernels

•  The information content of AIRS spectra is highly scene dependent (e.g. clear vs. cloudy, tropical vs. polar, ocean vs. land, etc. ). –  Therefore, the vertical resolution and accuracy of any given retrieval is a

function of scene. •  In previous slides we have shown that portions of the retrieval error

(e.g. those due to the first guess regression and trapezoidal smoothing) are beyond the physical retrieval capability to characterize.

•  It makes sense to use an estimate of the information content on a case-by-case basis for comparisons of retrievals to correlative measurements. –  Use the averaging kernel and trapezoids to convolve the correlative

measurement such the the correlative profile is more comparable to what the retrieval would “see”.

•  WOUDC Ozone/Radiosondes are weighted toward polar cases –  Water from matched operational radiosonde

•  Comparisons are for temperature and water only.

55

56