robin hogan, julien delanoë, nicola pounder, nicky chalmers, thorwald stein, anthony illingworth...
TRANSCRIPT
Robin Hogan, Julien Delanoë, Nicola Pounder,Robin Hogan, Julien Delanoë, Nicola Pounder,
Nicky Chalmers, Thorwald Stein, Anthony IllingworthNicky Chalmers, Thorwald Stein, Anthony Illingworth
University of ReadingUniversity of Reading
Thanks to Alessandro Battaglia and Richard ForbesThanks to Alessandro Battaglia and Richard Forbes
Towards “unified” retrievals of Towards “unified” retrievals of cloud, precipitation and aerosol from cloud, precipitation and aerosol from
combined radar, lidar and combined radar, lidar and radiometer observationsradiometer observations
Clouds in climate modelsClouds in climate models
14 global models (AMIP)
90N 80 60 40 20 0 -20 -40 -60 -80 90S
0.05
0.10
0.15
0.20
0.25
Latitude
Ver
tical
ly in
tegr
ated
clo
ud w
ater
(kg
m-2) But all
models tuned to give about the same top-of-atmosphere radiation
The properties of ice clouds
are particularly uncertain
• Via their interaction with solar and terrestrial radiation, clouds are one of the greatest sources of uncertainty in climate forecasts
• But cloud water content in models varies by a factor of 10• Need instrument with high vertical resolution…
Stephens et al. (2002)
Vertical structure of liquid water Vertical structure of liquid water contentcontent
• Cloudnet: several years of retrievals from 3 European ground-based sites• Observations in grey (with range indicating uncertainty)
• How do these models perform globally?
– ECMWF has far too great an occurrence of low LWC values
0-3 km
– Supercooled liquid water content from seven forecast models spans a factor of 20
Illingworth, Hogan et al. (2007)
Spaceborne radar, lidar and Spaceborne radar, lidar and radiometersradiometers
The A-Train– NASA– 700-km orbit– CloudSat 94-GHz radar (launch 2006)– Calipso 532/1064-nm depol. lidar– MODIS multi-wavelength radiometer– CERES broad-band radiometer– AMSR-E microwave radiometer
EarthCARE: launch 2012– ESA+JAXA– 400-km orbit: more
sensitive– 94-GHz Doppler radar– 355-nm HSRL/depol. lidar– Multispectral imager– Broad-band radiometer– Heart-warming name
EarthCare
2013201420152016201720182019
OverviewOverview• What do spaceborne radar and lidar see?
– Classification of targets from radar and lidar– Global distribution of supercooled clouds from the LITE lidar
• Towards a “unified” retrieval of cloud, precipitation and aerosol – Variational retrieval framework
• Results from CloudSat-Calipso ice-cloud retrieval– Consistency with top-of-atmosphere radiative fluxes– Evaluation and improvement of models
• Challenges and opportunities from multiple scattering– Fast forward model– Multiple field-of-view lidar retrieval
• EarthCARE• First results from prototype unified retrieval• Outlook for model evaluation and improvement
What do CloudSat and Calipso What do CloudSat and Calipso see?see?
Cloudsat radar
CALIPSO lidar
Target classificationInsectsAerosolRainSupercooled liquid cloudWarm liquid cloudIce and supercooled liquidIceClearNo ice/rain but possibly liquidGround
Delanoe and Hogan (2008, 2010)
• Radar: ~D6, detects whole profile, surface echo provides integral constraint
• Lidar: ~D2, more sensitive to thin cirrus and liquid but attenuated
• Radar-lidar ratio provides size D
CloudSat and Calipso CloudSat and Calipso sensitivitysensitivity
7
• In July 2006, cloud occurrence in the subzero troposphere was 13.3%
• The fraction observed by radar was 65.9%
• The fraction observed by lidar was 65.0%
• The fraction observed by both was 31.0%
Distribution versus temperature & Distribution versus temperature & latitudelatitude
• No supercooled water colder than –40°C (as expected)• Supercooled water more frequent in southern hemisphere storm track
Hogan et al. (2004)
Ingredients of a variational Ingredients of a variational retrievalretrieval
• Aim: to retrieve an optimal estimate of the properties of clouds, aerosols and precipitation from combining these measurements– To make use of integral constraints must retrieve components
together• For each ray of data, define observation vector y:
– Radar reflectivity values– Lidar backscatter values– Infrared radiances– Shortwave radiances (not yet implemented)– Surface radar echo providing two-way attenuation (ditto)
• Define state vector x of properties to be retrieved:– Ice cloud extinction, number concentration and lidar-ratio profile– Liquid water content profile and number concentration– Rain rate profile and number concentration– Aerosol extinction coefficient profile and lidar ratio
• Forward model H(x) to predict the observations from the state vector– Microphysical component: particle scattering properties– Radiative transfer component
The cost functionThe cost function• The essence of the method is to find the state vector x that minimizes
a cost function:
TxxδxBδxδyRδy
x
T1T1T
1
2
211
12
2
12
2
2
1
2
1
2
1
22
1
2
1)(
2
1
n
iiiii
n
i bi
iim
i yi
ii xxxbxHy
J
Each observation yi is weighted by the inverse of
its error variance
The forward model H(x) predicts the observations from the state vector x
Some elements of x are constrained by a
prior estimate
This term can be used to penalize curvature in the retrieved profile
Unified Unified retrievalretrieval
Ingredients developedWork in progress
1. New ray of data: define state vector x
Use classification to specify variables describing each species at each gateIce: extinction coefficient , N0’, lidar extinction-to-backscatter ratio
Liquid: extinction coefficient and number concentrationRain: rain rate, drop diameter and melting iceAerosol: extinction coefficient, particle size and lidar ratio
2a. Radar model
Including surface return and multiple scattering
2b. Lidar model
Including HSRL channels and multiple scattering
2c. Radiance model
Solar and IR channels
3. Compare to observations
Check for convergence
4. Iteration method
Derive a new state vectorAdjoint of full forward modelQuasi-Newton or Gauss-Newton scheme
2. Forward model
Not converged
Converged
Proceed to next ray of data5. Calculate retrieval error
Error covariances and averaging kernel
Unified retrieval: Forward Unified retrieval: Forward modelmodel
• From state vector x to forward modelled observations H(x)...
Ice & snow Liquid cloud Rain Aerosol
Ice/radar
Liquid/radar
Rain/radar
Ice/lidar
Liquid/lidar
Rain/lidar
Aerosol/lidar
Ice/radiometer
Liquid/radiometer
Rain/radiometer
Aerosol/radiometer
Radar scattering profile
Lidar scattering profile
Radiometer scattering profile
Lookup tables to obtain profiles of extinction, scattering & backscatter coefficients, asymmetry factor
Sum the contributions from each constituent
x
Radar forward modelled obs
Lidar forward modelled obs
Radiometer fwd modelled obs
H(x)Radiative transfer models
Adjoint of radar model (vector)
Adjoint of lidar model (vector)
Adjoint of radiometer model
Gradient of cost function (vector)
xJ=HTR-1[y–H(x)]
Vector-matrix multiplications: around the same cost as the original forward
operations
Adjoint of radiative transfer models
yJ=R-1[y–H(x)]
Gauss-Newton method
• Requires the curvature 2J/x2
– A matrix– More expensive to calculate
• Faster convergence– Assume J is quadratic and
jump to the minimum• Limited to smaller retrieval
problems
J
x
x1
J/x2J/x2
Minimization methods - in 1DMinimization methods - in 1DQuasi-Newton method (e.g. L-BFGS)
• Rolling a ball down a hill– Intelligent choice of direction
in multi-dimensions helps convergence
• Requires the gradient J/x– A vector (efficient to store)– Efficient to calculate using
adjoint method• Used in data assimilation
J
x
x2x3x4x5x6x7x8
x1
J/x
x2x3
x4x5
Ice retrievalIce retrieval• First challenge:
– How do we model radar scattering from complex ice particles?
• In Rayleigh regime (»D), backscatter proportional to mass squared– Particle shape irrelevant– Use a mass-D relationship
• But Rayleigh assumption often not valid at 94 GHz (=3 mm)– Most papers assume homogeneous ice-air spheres with a diameter
equal to the maximum dimension D of the particle, and apply Mie theory
• Problems with this approach:– Non-Rayleigh behaviour depends on the vertical dimension of the
particle– Most are irregular aggregates with an average axial ratio of 0.6– They fall with their maximum dimension horizontal
• Alternative approach:– Approximate as horizontally aligned oblate spheroids
Spheres versus spheroidsSpheres versus spheroids
SpheroidSphere
Transmitted wave
Sphere: returns from
opposite sides of particle out
of phase: cancellation
Spheroid: returns from
opposite sides not out of
phase: higherb
Hogan et al. (2011)
– Sphere produces ~5 dB error (factor of 3)– Spheroid approximation matches Rayleigh reflectivity (mass is about right)
and non-Rayleigh reflectivity (shape is about right)
Test with dual-wavelength Test with dual-wavelength aircraft dataaircraft data
Hogan et al. (2011)
Test with 3-GHz differential Test with 3-GHz differential reflectivityreflectivity
Hogan et al. (2011)
• Horizontal polarization Zh
– Differential reflectivity Zdr is larger for more extreme axial ratios
– Agreement with Z and Zdr confirms the Brown & Francis (1995) mass-D relationship and axial ratio = 0.6
• Zdr = 10log10(Zh/Zv)
Ice cloud: non-variational Ice cloud: non-variational retrievalretrieval
• Donovan et al. (2000) algorithm can only be applied where both lidar and radar have signal
Observations
State variables
Derived variables
Retrieval is accurate but not perfectly stable where lidar loses signal
Aircraft-simulated profiles with noise (from Hogan et al. 2006)
Delanoe and Hogan (2008)
Variational radar/lidar Variational radar/lidar retrievalretrieval
• Noise in lidar backscatter feeds through to retrieved extinction
Observations
State variables
Derived variables
Lidar noise matched by retrieval
Noise feeds through to other variables
Delanoe and Hogan (2008)
……add smoothness constraintadd smoothness constraint
• Smoothness constraint: add a term to cost function to penalize curvature in the solution (J’ = id2i/dz2)
Observations
State variables
Derived variables
Retrieval reverts to a-priori N0
Extinction and IWC too low in radar-only region
Delanoe and Hogan (2008)
……add a-priori error add a-priori error correlationcorrelation
• Use B (the a priori error covariance matrix) to smooth the N0 information in the vertical
Observations
State variables
Derived variables
Vertical correlation of error in N0
Extinction and IWC now more accurate
Delanoe and Hogan (2008)
Lidar observations
Radar observations
Visible extinction
Ice water content
Effective radius
Lidar forward model
Radar forward model
Example ice Example ice cloud cloud
retrievalsretrievalsDelanoe and Hogan (2010)
Evaluation using CERES TOA Evaluation using CERES TOA fluxesfluxes
• Radar-lidar retrieved profiles containing only ice used with Edwards-Slingo radiation code to predict CERES fluxes
• Small biases but large random shortwave error: 3D effects?
Chalmers (2011)
ShortwaveBias 4 W m-2, RMSE 71 W m-2
LongwaveBias 0.3 W m-2, RMSE 14 W m-2
CERES versus a radar-only CERES versus a radar-only retrievalretrieval
• How does this compare with radar-only empirical IWC(Z, T) retrieval of Hogan et al. (2006) using effective radius parameterization from Kristjansson et al. (1999)?
Bias 10 W m-2
RMS 47 W m-2
ShortwaveBias 48 W m-2, RMSE 110 W m-2
LongwaveBias –10 W m-2, RMSE 47 W m-2
Chalmers (2011)
How important is lidar?How important is lidar?• Remove lidar-only pixels from radar-lidar retrieval• Change to fluxes is only ~5 W m-2 but lidar still acts to improve
retrieval in radar-lidar region of the cloud
ShortwaveBias –5 W m-2, RMSE 17 W m-2
LongwaveBias 4 W m-2, RMSE 9 W m-2
Chalmers (2011)
A-Train A-Train versus versus
modelsmodels• Ice water
content• 14 July 2006• Half an orbit• 150°
longitude at equator
Delanoe et al. (2011)
• Both models lack high thin cirrus• ECMWF lacks high IWC values; using this work, ECMWF have
developed a new prognostic snow scheme that performs better• Met Office has too narrow a distribution of in-cloud IWC
Evaluation of gridbox-mean ice water Evaluation of gridbox-mean ice water contentcontent
In-cloud mean ice water In-cloud mean ice water contentcontent
Radiative transfer forward Radiative transfer forward modelsmodels
• Infrared radiances– Delanoe and Hogan (2008) model– Currently testing RTTOV (widely used, can do microwave, has
adjoint)• Solar radiances
– Currently testing LIDORT• Radar and lidar
– Simplest model is single scattering with attenuation: ’= exp(-2)– Problem from space is multiple scattering: contains extra
information on cloud properties (particularly optical depth) but no-one has previously been able to rigorously make use of data subject to pulse stretching
– Use combination of fast “Photon Variance-Covariance” method and “Time-Dependent Two-Stream” methods
– Adjoints for these models recently coded– Forward model for lidar depolarization is in progress
Examples of multiple scattering
LITE lidar (<r, footprint~1 km)
CloudSat radar (>r)
StratocumulusStratocumulus
Intense thunderstormIntense thunderstorm
Surface echoSurface echoApparent echo from below the surface
• Regime 0: No attenuation– Optical depth << 1
• Regime 1: Single scattering– Apparent backscatter ’ is easy to
calculate from at range r : ’(r) = (r) exp[-2(r)]
Scattering Scattering regimesregimes
Footprint x
Mean free path l
• Regime 2: Small-angle multiple scattering
– Occurs when l ~ x– Only for wavelength much less than particle size, e.g. lidar & ice clouds
– No pulse stretching
• Regime 3: Wide-angle multiple scattering (pulse stretching)
– Occurs when l ~ x
Time-dependent 2-stream Time-dependent 2-stream approx.approx.• Describe diffuse flux in terms of outgoing stream I+ and incoming stream I–, and
numerically integrate the following coupled PDEs:
• These can be discretized quite simply in time and space (no implicit methods or matrix inversion required)
SII
r
I
t
I
c 211
1
SII
r
I
t
I
c 211
1
Time derivative Remove this and we have the time-independent two-stream approximation
Spatial derivative Transport of radiation from upstream
Loss by absorption or scatteringSome of lost radiation will enter the other stream
Gain by scattering Radiation scattered from the other stream
Source
Scattering from the quasi-direct beam into each of the streams
Hogan and Battaglia (J. Atmos. Sci., 2008)
Fast multiple scattering forward Fast multiple scattering forward modelmodel
CloudSat-like example
• New method uses the time-dependent two-stream approximation
• Agrees with Monte Carlo but ~107 times faster (~3 ms)
Hogan and Battaglia (J. Atmos. Sci. 2008)
CALIPSO-like example
Multiple field-of-view lidar Multiple field-of-view lidar retrievalretrieval
• To test multiple scattering model in a retrieval, and its adjoint, consider a multiple field-of-view lidar observing a liquid cloud
• Wide fields of view provide information deeper into the cloud
• The NASA airborne “THOR” lidar is an example with 8 fields of view
• Simple retrieval implemented with state vector consisting of profile of extinction coefficient
• Different solution methods implemented, e.g. Gauss-Newton, Levenberg-Marquardt and Quasi-Newton (L-BFGS)
lidar
Cloud top
600 m100 m
10 m
Results for a sine profileResults for a sine profile
• Simulated test with 200-m sinusoidal structure in extinction
• With one FOV, only retrieve first 2 optical depths
• With three FOVs, retrieve structure of extinction profile down to 6 optical depths
• Beyond that the information is smeared out
Pounder, Hogan et al. (2011)
Optical depth from multiple Optical depth from multiple FOV lidarFOV lidar• Despite vertical
smearing of information, the total optical depth can be retrieved to ~30 optical depths
• Limit is closer to 3 for one narrow field-of-view lidar
• Useful optical depth information from one 100-m-footprint lidar (e.g. Calipso)!
Pounder, Hogan et al. (2011)
THOTHOR R
lidarlidar
EarthCAREEarthCARE• The ESA/JAXA “EarthCARE”
satellite is designed with synergy in mind
• We are currently developing synergy algorithms for its instrument specification
EarthCARE lidarEarthCARE lidar• High Spectral Resolution
capability enables direct retrieval of extinction profile
EarthCARE radar: Doppler EarthCARE radar: Doppler capabilitycapability
94-GHz radar
10-GHz radar
• Example from NASA airborne cloud radar demonstrates– Can estimate ice fall-speed globally: important for radiation budget– Can identify strong updrafts in convective cores
94-GHz reflectivity in convection disappears very quickly: multiple scattering from CloudSat may be giving us a false impression of how far we are penetrating
Unified algorithm: progressUnified algorithm: progress• Bringing all the aspects of this talk together…• Done:
– Functioning algorithm framework exists– C++: object orientation allows code to be completely flexible:
observations can be added and removed without needing to keep track of indices to matrices, so same code can be applied to different observing systems
– Preliminary retrieval of ice, liquid, rain and aerosol– Adjoint of radar and lidar forward models with multiple scattering
and HSRL/Raman support– Interface to L-BFGS quasi-Newton algorithm in GNU Scientific
Library• In progress / future work:
– Estimate and report error in solution and averaging kernel – Interface to radiance models– Test on a range of ground-based, airborne and spaceborne
instruments
Observations vs forward Observations vs forward modelsmodels
– Radar and lidar backscatter are successfully forward modelled (at final iteration) in most situations
– Can also forward model Doppler velocity (what EarthCARE would see)
• Radar reflectivity factor• Lidar backscatter
Three retrieved componentsThree retrieved components• Liquid water content
• Ice extinction coefficient
• Rain rate
OutlookOutlook• Use of radiances in retrieval should make retrieved profiles consistent with
broadband fluxes (can test this with A-Train and EarthCARE)• EarthCARE will take this a step further
– Use imager to construct 3D cloud field 10-20 km wide beneath satellite – Use 3D radiative transfer to test consistency with broadband radiances
looking at the cloud field in 3 directions (overcome earlier 3D problem)• How can we use these retrievals to improve weather forecasts?
– Assimilate cloud products, or radar and lidar observations directly?– Assimilation experiments being carried out by ECMWF– Still an open problem as to how to ensure clouds are assimilated such
that the dynamics and thermodynamics of the model are modified so as to be consistent with the presence of the cloud
• How can we use these retrievals to improve climate models?– We will have retrieved global cloud fields consistent with radiation– So can diagnose in detail not only what aspects of clouds are wrong in
models, but the radiative error associated with each error in the representation of clouds
• First part of a forward model is the scattering and fall-speed model– Same methods typically used for all radiometer and lidar channels– Radar and Doppler model uses another set of methods
Scattering modelsScattering models
Particle type Radar (3.2 mm) Radar Doppler Thermal IR, Solar, UVAerosol Aerosol not
detected by radarAerosol not detected by radar
Mie theory, Highwood refractive index
Liquid droplets Mie theory Beard (1976) Mie theoryRain drops T-matrix: Brandes
et al. (2002) shapes
Beard (1976) Mie theory
Ice cloud particles
T-matrix (Hogan et al. 2010)
Westbrook & Heymsfield
Baran (2004)
Graupel and hail Mie theory TBD Mie theoryMelting ice Wu & Wang
(1991)TBD Mie theory
Unified algorithm: state Unified algorithm: state variablesvariables
State variable Representation with height / constraint A-priori
Ice clouds and snow
Visible extinction coefficient One variable per pixel with smoothness constraint None
Number conc. parameter Cubic spline basis functions with vertical correlation Temperature dependent
Lidar extinction-to-backscatter ratio Cubic spline basis functions 20 sr
Riming factor Likely a single value per profile 1
Liquid clouds
Liquid water content One variable per pixel but with gradient constraint None
Droplet number concentration One value per liquid layer Temperature dependent
Rain
Rain rate Cubic spline basis functions with flatness constraint None
Normalized number conc. Nw One value per profile Dependent on whether from melting ice or coallescence
Melting-layer thickness scaling factor One value per profile 1
Aerosols
Extinction coefficient One variable per pixel with smoothness constraint None
Lidar extinction-to-backscatter ratio One value per aerosol layer identified Climatological type depending on region
Ice clouds follows Delanoe & Hogan (2008); Snow & riming in convective clouds needs to be added
Liquid clouds currently being tackled
Basic rain to be added shortly; Full representation later
Basic aerosols to be added shortly; Full representation via collaboration?
• Proposed list of retrieved variables held in the state vector x
• Computational cost can scale with number of points describing vertical profile N; we can cope with an N2 dependence but not N3
Radiative transfer forward Radiative transfer forward modelsmodels
Radar/lidar model Applications Speed Jacobian Adjoint
Single scattering: ’= exp(-2) Radar & lidar, no multiple scattering N N2 N
Platt’s approximation ’= exp(-2) Lidar, ice only, crude multiple scattering N N2 N
Photon Variance-Covariance (PVC) method (Hogan 2006, 2008)
Lidar, ice only, small-angle multiple scattering
N or N2 N2 N
Time-Dependent Two-Stream (TDTS) method (Hogan and Battaglia 2008)
Lidar & radar, wide-angle multiple scattering
N2 N3 N2
Depolarization capability for TDTS Lidar & radar depol with multiple scattering N2 N2
Radiometer model Applications Speed Jacobian Adjoint
RTTOV (used at ECMWF & Met Office) Infrared and microwave radiances N N
Two-stream source function technique (e.g. Delanoe & Hogan 2008)
Infrared radiances N N2
LIDORT Solar radiances N N2 N
• Infrared will probably use RTTOV, solar radiances will use LIDORT• Both currently being tested by Julien Delanoe
• Lidar uses PVC+TDTS (N2), radar uses single-scattering+TDTS (N2)• Jacobian of TDTS is too expensive: N3
• We have recently coded adjoint of multiple scattering models• Future work: depolarization forward model with multiple scattering
and 2nd derivative (the Hessian matrix):
Gradient Descent methods
– Fast adjoint method to calculate xJ means don’t need to calculate Jacobian
– Disadvantage: more iterations needed since we don’t know curvature of J(x)
– Quasi-Newton method to get the search direction (e.g. L-BFGS used by ECMWF): builds up an approximate inverse Hessian A for improved convergence
– Scales well for large x– Poorer estimate of the error at the
end
Minimizing the cost functionMinimizing the cost function
Gradient of cost function (a vector)
Gauss-Newton method
– Rapid convergence (instant for linear problems)
– Get solution error covariance “for free” at the end
– Levenberg-Marquardt is a small modification to ensure convergence
– Need the Jacobian matrix H of every forward model: can be expensive for larger problems as forward model may need to be rerun with each element of the state vector perturbed
112 BHRHxTJ
axBaxxyRxy 11
2
1)()(
2
1 TT HHJ
axBxyRHx 11 )(HJ T
JJii xxxx
12
1 Jii xAxx 1
Comparison of convergence Comparison of convergence ratesrates
• Solution is identical• Gauss-Newton method converges in < 10 iterations• L-BFGS Gradient Descent method converges in < 100 iterations• Conjugate Gradient method converges a little slower than L-BFGS• Each L-BFGS iteration >> 10x faster than each Gauss-Newton one!• Gauss-Newton method requires the Jacobian matrix, which must be
calculated by rerunning multiple scattering model multiple times
Unified algorithm: first results for Unified algorithm: first results for ice+liquidice+liquid
Ob
serv
ati
on
s
Retr
ieval
But lidar noise degrades retrieval
TruthRetrieval
First guessIterations
ObservationsForward modelled retrievalForward modelled first guess
Convergence!
Add smoothness constraintAdd smoothness constraintO
bserv
ati
on
s
Retr
ieval
TruthRetrieval
First guessIterations
ObservationsForward modelled retrievalForward modelled first guess
Smoother retrieval but slower convergence