agu 2004 fall meeting, may 19, 2015slide 1 jointly retrieving surface soil moisture from active and...

AGU 2004 Fall Meeting, April 18, 2023 Slide 1

Jointly Retrieving Surface Soil Moisture from Active and Passive Microwave Observations using

Cubist Data-Mining

Xiwu ZhanUMBC-GEST/NASA-GSFC, Code 974.1, Greenbelt, MD

Paul HouserNASA-GSFC HSB, Code 974, Greenbelt, MD

Jeffrey WalkerUniversity of Melbourne, Victoria, Australia


OBJECTIVES

There will be high resolution (up to 1km) radar backscatter observations of land surface soil moisture from NASA ESSP Hydros mission. Radiation transfer models are usually inversed to retrieve soil moisture value. Can data-mining provide an alternative?

There are many global high resolution satellite data sets of land surface parameters that are related to soil moisture. Can we use them to derive soil moisture when radar data are not available?

What are the accuracies of these alternative soil moisture retrieval methods?


METHODOLOGY

Use the 1km geophysical and biophysical data fields and microwave emission and backscatter models (MEBM) from the Observation System Simulation Experiment (OSSE in Crow et al, 2004, Zhan et al, 2005) of NASA ESSP Hydros Mission;

Inverse the simulated radiometer and radar observations with the MEBM for soil moisture retrievals;

Use the Cubist data-mining tool to generate Cubist models and use the models to obtain fine resolution soil moisture retrievals

Use the update equation of the Extended Kalman Filter (EKF) to combine course resolution and fine resolution soil moisture estimations for an optimal soil moisture retrieval data product;

Compute the RMSEs of soil moisture retrievals of the three methods (INV, Cubist, & EKF) against the original soil moisture data fields.


Hydros: Hydrosphere States Mission

Spinning 6m dish

A NASA Earth System Science Pathfinder mission;Surface soil moisture w/ 4%vol. accuracy and Freeze/Thaw state transitions;Revisit time: Global 3 days, boreal area 2 daysL-band (1.41GHz) Radiometer sensing 40km brightness temp. with H & V polarization;L-band (1.26GHz) Radar measuring 1-3km backscatters with hh, vv, hv polarization;Soil moisture products: 3km radar retrievals, 40km radiometer retrievals, 10km radar and radiometer combined retrievals and 5km 4DDA results.


Hydros OSSE: Data Layers

36 km TBh, TBv data from Hydros radiometer

simulator

9 km soil moisture retrieval

product

3 km hh, vv, hv data from Hydros radar simulator

1 km soil moisture data

from nature run

1 36km pixel

9 9km pixels

144 3km pixels

1296 1km pixels


“Truth” Validation

RadiometerInversion

3/9/36km SM

Retrieval Error

1 km Nature Run(Input: LC, ST, NDVI, Rainfall, Met data)

3/9km

Hydros Instrument Simulator

1km SM

36km Tb

Aggregateto

3/9/36km

1km SM

3/9/36kmSM

3/9km SM

EKF Algorithm

36km SM Tb & Errors

Radiometer TbForward Model

Radar Forward Model

EKF Algorithm

Innovations

Calculate optimized SM

Error Models

Based on redAnd white

noise

3/9km

3/9/36km SM

36km Radiometer

Forward Model

Tb Iteration

36km SM

36km Tb

Observations

From Hydrosinstrumentsimulator

Background

From radio-meter

inversion

1km Tsoil 1km Tskin

36km

SM

Data Flow for Using EKF to Retrieve SM from Tb & Observations

RadarInversion

3km Radar

Forward Model

Iteration

1km Radiometer TbForward Model

Aggregateto 36km

White Noise

1km Radar Forward Model

Aggregateto 3/9km

Red Noise

White Noise


“Truth” Validation

RadiometerInversion

3/9/36km SM

Retrieval Error

1 km Nature Run(Input: LC, ST, NDVI, Rainfall, Met data)

3/9km

Hydros Instrument Simulator

1km SM

36km Tb

Aggregateto

3/9/36km

1km SM

3/9/36kmSM

1km SM

EKF Algorithm

36km SM 1km SMErrors

EKF Algorithm

Innovations

Calculate optimized SM

Error Models

Based on redAnd white

noise

1km ndvi, Ts/

1/3/9/36km SM

36km Radiometer

Forward Model

Tb Iteration

36km SM

36km Tb

Observations

From Cubist model

Background

From radio-meter

inversion

1km Tsoil 1km Tskin

36km

SM Cubist

Model

1 km CubistModels

ndvi, Ts/

Observ.

1km SM

1km Radiometer TbForward Model

Aggregateto 36km

White Noise

1km Radar Forward Model

Aggregateto 3/9km

Red Noise

White Noise

Data Flow for Using EKF to Retrieve SM from Tb & other Observations


Radar observational are not handily available for retrieving soil moisture before the launch of Hydros in 2010: spatial coverage, revisit time;

Radar radiation transfer models are not as mature as radiometer models for inversing soil moisture;

Why Alternative for Radar Model?

T*

NDVI*Low Soil Moisturehigh

Soil Moisture

NDVIo

NDVIs

To Ts

Visible/Infrared observations such as NDVI, LST and albedo from MODIS, Landsat and future VIIRS on NPOESS are available everyday at high spatial resolutions;

The “Universal Triangle” relationships between soil moisture and the visible/infrared observations have been documented in literature for many years.


Cubist is used to build regression tree model of the relationships between soil moisture and its related land surface parameters such radar backscatter, or, NDVI, surface temperature and albedo;

Regression tree is similar to the decision tree classifier in that it recursively splits training samples into subsets, two at each split;

Instead of assigning class labels to the subsets, it develops a linear regression model for each of them;

Each splitting is made such that the combined residual error of the models for the two subsets is substantially lower than the residual error of the single best linear model for the samples in the two subsets, and that the combined residual error of the split is the minimum of all possible splits

Cubist: a Data-mining Computer Tool


Noise in data: Sigma - .5dB, Tb – 1K, ndvi – 10%, Ts-.5K, Roughness-5%, VWC-10%

Cubist Model Compared with Radar Model

0.00

0.05

0.10

0.15

0.20

145 150 155 160 165 170 175 180

Day of Year

Cubist Sigmas

Sigma Inversion

Cubist Sigmas vs Sigma Inversion

Tb Inversion

Low noise data

For low noise data, Cubist model of radar backscatters may reduce the RMSEs of radar model inversions by about 1-2 %v/v;


Noise in data: Sigma - 1dB, Tb – 1K, ndvi – 20%, Ts-1K, Roughness-10%, VWC-20%

Cubist Model Compared with Radar Model

0.00

0.05

0.10

0.15

0.20

145 150 155 160 165 170 175 180

Day of Year

Cubist Sigmas

Sigma Inversion

Cubist Sigmas vs Sigma Inversion

Tb Inversion

High noise data

For high noise data, Cubist model of radar backscatters could reduce the RMSEs of radar model inversions by about 3-4 %v/v;


Cubist Model Applicability/Stability

0.000

0.020

0.040

0.060

0.080

0.100

0.120

145 150 155 160 165 170 175 180

Day of Year

Day 166 Model

Daily Cubist Model

Cubist Sigma Model Stability

Day 146 Model

High noise data

Low noise dataDay 146 H Model

Day 166 L Model

Cubist model of radar backscatters using same day or other day training data results very similar accuracy;

Cubist model based on low noise data produces almost the same accuracy as based on high noise data, and the opposite is true too.


Cubist Model Using Visible/IR Data

0.00

0.05

0.10

0.15

0.20

145 150 155 160 165 170 175 180

Day of Year

Cubist ndvi & Ts

Sigma Inversion

Tb Inversion

Low noise dataCubist ndvi & Ts vs Sigma Inversion

If the data noises are low, RMSEs of soil moisture retrievals from Cubist model using only visible/IR observations may be 3-5%v/v higher than radar backscatter model inversions.


Cubist Model Using Visible/IR Data

0.00

0.05

0.10

0.15

0.20

145 150 155 160 165 170 175 180

Day of Year

Cubist ndvi & Ts

Sigma Inversion

Cubist ndvi & Ts vs Sigma Inversion

Tb Inversion

High noise data

RMSEs of soil moisture retrievals from Cubist model using only visible/IR observations may be 1-3%v/v higher than radar backscatter model inversions based on the high noise data. Radar observations are apparently more reliable than visible/IR obs as long as a radar model is known.


][)](([ fbba xXhZKXX

Extended Kalman Filter for SM Retrieval

RHPH

PHK

T

T

x

XhH

b

)(

1434144,1,1,1, ... x

ThvhvvvhhBvBh TTZ

1434)(...)()()()()()( 144,1,1,1, x

TchvchvcvvchhcBhcBh xxxxxTxTXh

Xa – soil moisture retrievalXb – background SMK – Kalman gainZ – observationsh(X) – obs functionH – obs operatorP – bg error covarianceR – obs error covariance

Kalman filter is a statistical data assimilation technique that calculates an optimal observation correction term to the background value based on the relative magnitude of the error covariances of the observations and the background.


0.00

0.05

0.10

0.15

0.20

145 150 155 160 165 170 175 180

Day of Year

Cubist Sigmas

Sigma Inversion

EKF Combining Cubist and Tb Inversion

Tb Inversion

Low noise data

EKF Tb + Cubist

EKF Application Result - 1

EKF retrievals using Cubist model of 1km radar sigmas and 36km Tb inversion are marginally better than Cubist model estimates when the error covariance difference between Tb inversion and Cubist model is large;


0.00

0.05

0.10

0.15

0.20

145 150 155 160 165 170 175 180

Day of Year

Cubist Sigmas

Sigma InversionTb Inversion

High noise data

EKF Tb + Cubist



When the error covariance difference between Tb inversion and Cubist model are smaller, the advantage of the EKF retrievals is larger (2-4% less RMSE);


0.00

0.05

0.10

0.15

0.20

145 150 155 160 165 170 175 180

Day of Year

Cubist ndvi & Ts

Sigma Inversion


Tb Inversion

Low noise data

EKF Tb + Cubist


EKF retrievals using Cubist model of 1km visible/IR obs and 36km Tb inversion are better than both Cubist model estimates and Tb inversion (2-3% less RMSE). But the Cubist model does not produce retrievals as good as using radar backscatter (4-5% larger RMSE).


0.00

0.05

0.10

0.15

0.20

145 150 155 160 165 170 175 180

Day of Year

Cubist ndvi & Ts

Sigma Inversion


Tb Inversion

High noise data

EKF Tb + Cubist


EKF retrievals using Cubist model of 1km visible/IR obs and 36km Tb inversion are marginally better than both Cubist model estimates and Tb inversion for high noise data. Cubist model of ndvi and Ts is not as good as radar backscatter models (1-3% higher RMSE).


Error Distribution of SM Retrievals

36km Tb Inversion 1km Cubist Sigma Model

1km Radar Sigma Inversion 1km EKF Tb Inv + Cubist Sigma

9.1%RMSE in %v/v

3.3%

3.2%5.3%

Low noise data, Day 155


Error Distribution of SM Retrievals

36km Tb Inversion 1km Cubist Sigma Model

1km Radar Sigma Inversion 1km EKF Tb Inv + Cubist Sigma

9.1%RMSE in %v/v

7.1%

5.9%10.9%

High noise data, Day 155


SUMMARYWhen radar backscatter (sigma) observations are available, a Cubist model of sigmas could be used to retrieve soil moisture with better accuracy (1-4% less RMSE) than a radar backscatter model.

The same set of equations of the Cubist model based on one set of training data may be applicable to other sets of data. Thus a Cubist model could be an alternative to a radar backscatter model based on the data we used.

EKF Data Assimilation method can combine high resolution and low resolution soil moisture estimations and improve retrieval accuracy.

Based on the NDVI and Ts used currently, a Cubist model of NDVI and Ts is not as reliable as a Cubist model of radar sigma data. The difference of RMSE could be as high as 1-5%.

However, the radar sigma inversions were obtained with the same radar backscatter model used to generate the radar backscatter data with noises added while the impacts of NDVI and Ts in the radar backscatter model may be as significant as in reality. Thus further investigations using real high resolution soil moisture, and visible/IP observational data are still needed.


Combining Optical/IR RS and MW RS forHigh Resolution Soil Moisture

NDVI,LST,A or Sigmas – SM Relationships

AMSR-E/CMIS SMOS, Hydros,

MW Observations

MODIS/VIIRSTM, SPOT, Hydros

Observations of NDVI, LST, A or Sigmas

Course Rez (20-50km)

Soil Moisture Retrievals

Soil Moisture Truth Data from Airplane/Ground

Observations

High Rez (30m-3km)

Soil Moisture Retrievals

EKF Data Assimilation Algorithms

Cubist: Data Mining Tools

agu 2004 fall meeting, may 19, 2015slide 1 jointly retrieving surface soil moisture from active and...

Documents

soil moisture value

radar data

data mining

hv data

data layers

radar retrievals

radiometer retrievals

hydros radar simulator