correction of daily values for inhomogeneities
DESCRIPTION
Correction of daily values for inhomogeneities. P. Štěpánek. Czech Hydrometeorological Institute, Regional Office Brno, Czech Republic. E-mail: [email protected]. COST-ESO601 meeting, Tarragona, 9-11 March 2009. Using daily data for inhomogeniety detection , is it meaningful ?. - PowerPoint PPT PresentationTRANSCRIPT
Correction of daily valuesCorrection of daily values for for inhomogeneitiesinhomogeneities
P. Štěpánek
Czech Hydrometeorological Institute, Regional Office Brno, Czech Republic
E-mail: [email protected]
COST-ESO601 meeting, Tarragona, 9-11 March 2009
Using daily data for inhomogeniety Using daily data for inhomogeniety detectiondetection, , is it meaningfulis it meaningful??
Homogenization of Homogenization of daily values daily values –– precipitationprecipitation series series
working with individual monthly values (to get rid of annual cycle)
It is still needed to adapt data to approximate to normal distribution
One of the possibilities: consider values above 0.1 mm only
Additional transformation of series of ratios
(e.g. with square root)
Original values - far from normal distribution
(ratios tested/reference series)(ratios tested/reference series) FrequenciesFrequencies
Homogenization of precipitation Homogenization of precipitation – daily values– daily values
Homogenization of precipitation Homogenization of precipitation – daily values– daily values
Limit value 0.1 mm
(ratios tested/reference series)(ratios tested/reference series) FrequenciesFrequencies
Limit value 0.1 mm, square root transformation (of ratios)
(ratios tested/reference series)(ratios tested/reference series) FrequenciesFrequencies
Homogenization of precipitation Homogenization of precipitation – daily values– daily values
Problem of independeProblem of independennce, ce, PrecipitationPrecipitation above 1 mm above 1 mm
August, Autocorrelations
Problem of independece,Problem of independece,
TTemperatureemperature
August, Autocorrelations
Problem of independece,Problem of independece,
TTemperature differences emperature differences (reference – candidate)(reference – candidate)
August, Autocorrelations
HomogenizationHomogenization Detection (preferably on monthly, seasonal and annual values) Correction – for daily values
WP1 WP1 SURVEYSURVEY (Enric Aguilar) (Enric Aguilar) Daily dataDaily data - - CorrectionCorrection (WP4) (WP4)
Very few approaches actually calculate special corrections for daily data.
Most approaches either
– Do nothing (discard data)
– Apply monthly factors
– Interpolate monthly factors
The survey points out several other alternatives that WG5 needs to investigate
0
2
4
6
8
10
12
Ap
ply
mo
nth
lyfa
cto
rs
Ch
an
ge
sN
LR
C N
Dis
card
da
ta
Em
pir
ica
lva
lue
s
Inte
rpo
late
mo
nth
ly
Tra
nsf
er
fun
ctio
ns
CD
FO
verl
ap
pin
gre
cord
s &
LM
Re
fere
nce
s+
mo
de
llin
go
f ho
m.
Lin
ea
ra
dju
stm
en
ts
Trust metadata only
Use a technique to detect breaks
Detect on lower resolution
Daily data correction mDaily data correction methodsethods
„Delta“ methods Variable correction methods – one element Variable correction methods – several
elements
Daily data correction mDaily data correction methodsethods
Interpolation of monthly factors– MASH– Vincent et al (2002) - cublic spline interpolation
Nearest neighbour resampling models, by Brandsma and Können (2006)
Higher Order Moments (HOM), by Della Marta and Wanner (2006) Two phase non-linear regression (O. Mestre) Modified percentiles approach, by Stepanek Using weather types classifications (HOWCLASS), by I. Garcia-
Borés, E. Aguilar, ...
AdjustAdjustinging daily values daily values for inhomogeneitiesfor inhomogeneities, , from from monthlymonthly versus versus dailydaily adjustmentsadjustments(„delta“ method)(„delta“ method)
AdjustingAdjusting from from monthlymonthly data data
monthly adjustments smoothed with Gaussian low pass filter (weights approximately 1:2:1)
smoothed monthly adjustments are then evenly distributed among individual days
-1.0
-0.8
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.1.
1.2.
1.3.
1.4.
1.5.
1.6.
1.7.
1.8.
1.9.
1.10
.
1.11
.
1.12
.
°C
UnSmoothed
B2BPIS01_T_21:00
-1.0
-0.8
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.1.
1.2.
1.3.
1.4.
1.5.
1.6.
1.7.
1.8.
1.9.
1.10
.
1.11
.
1.12
.
°C
ADJ_ORIG ADJ_C_INC
B2BPIS01_T_21:00
AdjustingAdjusting straight from straight from dailydaily data data
Adjustment estimated for each individual day (series of 1st Jan, 2nd Jan etc.)
Daily adjustments smoothed with Gaussian low pass filter for 90 days (annual cycle 3 times to solve margin values)
-5.0
-4.5
-4.0
-3.5
-3.0
-2.5
-2.0
-1.5
-1.0
-0.5
0.0
1.1
.
1.2
.
1.3
.
1.4
.
1.5
.
1.6
.
1.7
.
1.8
.
1.9
.
1.1
0.
1.1
1.
1.1
2.
hP
a
ADJ_ORIG ADJ_SMOOTHB2BPIS01_P_14:00
AdjustmentsAdjustments ((Delta methodDelta method))
-1.5
-1.0
-0.5
0.0
0.5
1.0
01-J
an
01-F
eb
01-M
ar
01-A
pr
01-M
ay
01-J
un
01-J
ul
01-A
ug
01-S
ep
01-O
ct
01-N
ov
01-D
ec
°C
UnSmoothedADJ_C_INCADJ_ORIG
1 3 2
a)
-1.5
-1.0
-0.5
0.0
0.5
1.0
01-J
an
01-F
eb
01-M
ar
01-A
pr
01-M
ay
01-J
un
01-J
ul
01-A
ug
01-S
ep
01-O
ct
01-N
ov
01-D
ec
°C
ADJ_ORIGADJ_SMOOTH30ADJ_SMOOTH60ADJ_SMOOTH90
4 5 6 7
b)
The same final adjustments may be obtained from either monthly averages or through direct use of daily data
(for the daily-values-based approach, it seems reasonable to smooth with a low-pass filter for 60 days. The same results may be derived using a low-pass filter for two months (weights approximately 1:2:1) and
subsequently distributing the smoothed monthly adjustments into daily values)
(1 – raw adjustments, 2 – smoothed adjustments, 3 – smoothed adjustments distributed into individual days), b) daily-based approach (4 – individual calendar day adjustments, 5 – daily adjustments smoothed by low-pass filter for 30 days, 6 – for 60 days, 7 – for 90 days)
Spline through monthly temperature Spline through monthly temperature adjustmentsadjustments („delta“ method)(„delta“ method)
Easy to implement No assumptions about changes in variance Integrated daily adjustments = monthly adjustments But, is it natural?
Variable correction Variable correction
f(C(d)|R), function build with the reference dataset R, d – daily data
cdf, and thus the pdf of the adjusted candidate series C*(d) is exactly the same as the cdf or pdf of the original candidate series C(d)
Trewin & Trevitt (1996) method: Use simultaneous observations of old and new conditions
Variable correctionVariable correction
Variable correctionVariable correction
1996
The HOM method concept: Fitting a modelThe HOM method concept: Fitting a model Locally weighted regression (LOESS)
(Cleveland & Devlin,1998)
HSP2 HSP1
The HOM method concept: Calculating the The HOM method concept: Calculating the binned difference seriesbinned difference series
Decile 1, k=1
Decile 10, k=10
The HOM method concept: The The HOM method concept: The binned differencesbinned differences
DELLA-MARTA AND WANNER,
JOURNAL OF CLIMATE 19 (2006)
4179-4197
SPLIDHOM (SPLIDHOM (SPLIne Daily HOMogenization), Olivier Mestre), Olivier Mestre direct non-linear
spline regression approach (x rather
than a correction based on quantiles),
cubic smoothing splines for estimating regression functions
Variable correctionVariable correction, , q-q functionq-q function
Michel Déqué, Global and Planetary Change 57 (2007) 16–26
Our modified percentiles based Our modified percentiles based approachapproach
0.000
5.000
10.000
15.000
20.000
25.000
30.000
0.00
0
0.04
0
0.10
0
0.16
0
0.22
0
0.28
0
0.34
0
0.40
0
0.46
0
0.52
0
0.58
0
0.64
0
0.70
0
0.76
0
0.82
0
0.88
0
0.94
0
0.99
5
Q_CAND_BE
Q_REF_BE
0.000
5.000
10.000
15.000
20.000
25.000
30.000
0.00
0
0.04
0
0.10
0
0.16
0
0.22
0
0.28
0
0.34
0
0.40
0
0.46
0
0.52
0
0.58
0
0.64
0
0.70
0
0.76
0
0.82
0
0.88
0
0.94
0
0.99
5
Q_CAND_AF
Q_REF_AF
-1.000
-0.500
0.000
0.500
1.000
1.500
0.00
0
0.04
0
0.10
0
0.16
0
0.22
0
0.28
0
0.34
0
0.40
0
0.46
0
0.52
0
0.58
0
0.64
0
0.70
0
0.76
0
0.82
0
0.88
0
0.94
0
0.99
5
Q_DIFF_BE
Q_DIFF_AF
-1.400
-1.200
-1.000
-0.800
-0.600
-0.400
-0.200
0.000
0.200
0.00
0
0.04
0
0.10
0
0.16
0
0.22
0
0.28
0
0.34
0
0.40
0
0.46
0
0.52
0
0.58
0
0.64
0
0.70
0
0.76
0
0.82
0
0.88
0
0.94
0
0.99
5
Q_DIFF
Q_DIFF_SM
-1.400
-1.200
-1.000
-0.800
-0.600
-0.400
-0.200
0.000
0.200
1 8 15 22 29 36 43 50 57 64 71 78 85 92 99
Our percentiles based approachOur percentiles based approach
-1.400
-1.200
-1.000
-0.800
-0.600
-0.400
-0.200
0.000
0.200
0.00
0
0.04
0
0.10
0
0.16
0
0.22
0
0.28
0
0.34
0
0.40
0
0.46
0
0.52
0
0.58
0
0.64
0
0.70
0
0.76
0
0.82
0
0.88
0
0.94
0
0.99
5
Q_DIFF
Q_DIFF_SM
PERC sm25
-0.900
-0.800
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.000 5.000 10.000 15.000 20.000 25.000 30.000
-1.400
-1.200
-1.000
-0.800
-0.600
-0.400
-0.200
0.000
0.200
1 8 15 22 29 36 43 50 57 64 71 78 85 92 99
Q_DIFF_SM
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.000 5.000 10.000 15.000 20.000 25.000 30.000
Variable correction methods – Variable correction methods – complex approach complex approach (several elements)(several elements)
not yet available …
Comparison of the methods, Comparison of the methods, ProClimDB softwareProClimDB software
Correction methods comparisonCorrection methods comparison
PERC
-0.900
-0.800
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.000 5.000 10.000 15.000 20.000 25.000 30.000
EMPIR
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
0 5 10 15 20 25 30
HOM
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
0 5 10 15 20 25 30
SPLIDHOM
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
0 5 10 15 20 25 30
PERC sm50
-0.900
-0.800
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.000 5.000 10.000 15.000 20.000 25.000 30.000
PECR sm75
-0.900
-0.800
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.000 5.000 10.000 15.000 20.000 25.000 30.000
EMPI sm75
-0.900
-0.800
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.000 5.000 10.000 15.000 20.000 25.000 30.000
Correction methods comparison, Correction methods comparison, different parameters settingsdifferent parameters settings
PERC
-0.900
-0.800
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.000 5.000 10.000 15.000 20.000 25.000 30.000
EMPIR
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
0 5 10 15 20 25 30
HOM
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
0 5 10 15 20 25 30
SPLIDHOM
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
0 5 10 15 20 25 30
PERC sm50
-0.900
-0.800
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.000 5.000 10.000 15.000 20.000 25.000 30.000
PECR sm75
-0.900
-0.800
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.000 5.000 10.000 15.000 20.000 25.000 30.000
EMPI sm75
-0.900
-0.800
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.000 5.000 10.000 15.000 20.000 25.000 30.000
PERC
-0.900
-0.800
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.000 5.000 10.000 15.000 20.000 25.000 30.000
EMPIR
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
0 5 10 15 20 25 30
SPLIDHOM
Correction methods comparison, Correction methods comparison, different parameters settingsdifferent parameters settings
PERC
-0.900
-0.800
-0.700
-0.600
-0.500
-0.400
-0.300
-0.200
-0.100
0.000
0.000 5.000 10.000 15.000 20.000 25.000 30.000
EMPIR
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
0 5 10 15 20 25 30
HOM
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
0 5 10 15 20 25 30
SPLIDHOM
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
0 5 10 15 20 25 30
SPLIDHOM
-1.0
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0.0
0.0 5.0 10.0 15.0 20.0 25.0 30.0
HOM
-1.0
-0.9
-0.8
-0.7
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0.0
0.0 5.0 10.0 15.0 20.0 25.0 30.0
Correction of daily valuesCorrection of daily values
We have some methods … - but we have to validate them -> benchmark
dataset on daily data Do we know how inhomogeneites in daily data
behave?we should analyse real datawho and when?, what method for data
comparison?
