spatial forecast verification methods intercomparison project: forecast quality index

Spatial Forecast Verification Methods Intercomparison Project: Forecast Quality Index

Sukanta BasuTexas Tech University

&Efi Foufoula-Georgiou

University of Minnesota

NCAR – Feb 20th, 2007

Forecast Quality Index (FQI)

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

Ingredients:

(a) Partial Hausdorff distance (PHD) → Displacement Error

(b) (Modified) Universal Image Quality Index (UIQI) → Amplitude Error(c) Surrogates

References:

(1)Basu, S., Dodov, B., and Foufoula-Georgiou, E. (2003). A novel measure for QPF verification and its usefulness in multimodel ensemble forecasting, Geophys. Res. Abstracts, 5, 04323, European Geophysical Society.

(2) Venugopal, V., Basu, S., and Foufoula-Georgiou, E. (2005). A new metric for comparing precipitation patterns with an application to ensemble forecasts, J. Geophys. Res., 110, D08111, doi:10.1029/2004JD005395.

Forecast Quality Index: Framework

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

Fcs

Surr # M

Fcs(Fcs>Th)

Fcs>Th

Obs

Obs(Obs>Th)

Surr # 1

Obs>Th

Compute UIQI

Compute PHD

Compute PHD

Compute PHD

Surr # M>ThSurr # 1>Th

1,..., pA a a 1,..., qB b b

, max , , ,H A B h A B h B A

, max minb Ba A

h A B a b

where,

Finite Point Sets: and

then every point A of must be within a distance d of some point of B and vice versa.

,H A B dIf

√ √

√ ,h A B

Hausdorff Distance (for Sets)

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

where,

1 2 1 2 2 1, max , , ,H I I h I I h I I

2 21 1

2 21 1

1 2 1 2,,

,,

, max min , ,

max min

I l m NII i j NI

I l m NII i j NI

h I I I i j I l m

i l j m

2 21 1

2 21 1

1 2 1 2,,

,,

, min , ,

min

th

I l m NII i j NI

th

I l m NII i j NI

h I I k I i j I l m

k i l j m

Effects of Outliers: Partial Hausdorff Distance (PHD)

Hausdorff Distance (for Binary Images)

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

May 13th, 2005 Case Study

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

Surrogates (Stochastic Realizations) of the Observed Field

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

Results (May 13th Case): Forecast Quality Index (FQI)

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

Forecast Quality Index: Sensitivity to Percentile Distance

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

Forecast Quality Index: Sensitivity to # Surrogates

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

Results (May 13th Case): Universal Image Quality Index

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

Results (May 13th Case): Partial Hausdorff Distance (PHD)

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

June 1st, 2005 Case Study

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

Results (June 1st Case): Forecast Quality Index (FQI)

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

Results (June 1st Case): Universal Image Quality Index

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

Results (June 1st Case): Partial Hausdorff Distance (PHD)

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

Results: May 13th vs. June 1st Case

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

← May 13th

June 1st →

What’s Next?

NCAR – Feb 20th, 2007 Basu & Foufoula-Georgiou

Transportation (Kantorovich) Distance-based Metric