PIK, 12 December 2005 The bootstrap approach to analyse trends and extremes in hydrological time series Manfred MudelseeClimate Risk Analysis, Halle (S), Germany Institute of Meteorology, University of Leipzig, Germany Message Provide reliable error bars Use bootstrap Regression Flood risk People G. Tetzlaff, M. Brngen (Meteorology Leipzig) U. Grnewald (Hydrology Cottbus) M. Mudelsee (CRA, Meteorology Leipzig) Challenge:Nonstationarity PastPresentFuture Challenge:Data (1) Right-skewed distributions Lognormal Challenge:Data (2) Uneven time spacing Gap Challenge:Data (3) Noise:measurement proxy Noisy Challenge:Data (3) Noise:measurement proxy persistence* Noisy Challenge:Statistics Statistics (task #1): Estimate true trend in flood risk from data. Challenge:Statistics Because data have errors (measurement, proxy), our estimates have errors. Challenge: Statistics Statistics (task #2): Determine error bars for estimated trend. Challenge: Statistics Statistics (task #2): Determine error bars for estimated trend. Bootstrap can do that also for messy data (distribution, spacing,...). Year Month Season Number of Ice? Stage Magnitude sources Dresden (cm) Feb W 5 I Jan-Feb W 6 I Feb-Mar W 32 I Apr W 16 I W Aug-Sep S Jan-Apr W 4 I Feb-Mar W Aug S Feb W 2 I Feb W 17 I Example: Elbe floods 10212002 Elbe, winter, class 23 Example: Elbe floods 10212002 Elbe, winter, class 23 Steps toward a better methodAdvantage Example: Elbe floods 10212002 Elbe, winter, class 23 Steps toward a better methodAdvantage 1. Continuous shifting (kernel estimation)More estimation points, no ambiguity (bounds) Example: Elbe floods 10212002 Elbe, winter, class 23 Steps toward a better methodAdvantage 1. Continuous shifting (kernel estimation)More estimation points, no ambiguity (bounds) 2. Gaussian (not uniform) kernelSmooth estimate Example: Elbe floods 10212002 Elbe, winter, class 23 Steps toward a better methodAdvantage 1. Continuous shifting (kernel estimation)More estimation points, no ambiguity (bounds) 2. Gaussian (not uniform) kernelSmooth estimate 3. Cross-validated bandwidth* Minimal estimation error Example: Elbe floods 10212002 Elbe, winter, class 23 Example: Elbe floods 10212002 Elbe, winter, class 23 h CV = 35 yr Example: Elbe floods 10212002 Elbe, winter, class 23 h CV = 35 yr * Diggle (1985) Appl. Statist. 34:138. occurrence rate hatestimate ttime hbandwidth KGaussian kernel T(i)flood event dates nnumber of floods Model: Poisson process 1.Prob of event in [t, t + ] = (t) 2.(t):intensity, risk, occurrence rate * Model: Poisson process 3.Independent events (remove persistence, peak-over-threshold data) * Cross-validation Smoothing problem solution Brooks & Marron (1991) Stoch. Proc. Appl. 38:157. * Example: Elbe floods 10212002 Elbe, winter, class 23 h CV = 35 yr OK... but where are the error bars? Example: Elbe floods 10212002 Elbe, winter, class 23 h CV = 35 yr Example: Elbe floods 10212002 Bootstrap resample (with replacement, same size) Elbe, winter, class 23 h CV = 35 yr Example: Elbe floods 10212002 Bootstrap resample (with replacement, same size) Elbe, winter, class 23 h CV = 35 yr Example: Elbe floods 10212002 Bootstrap resample (with replacement, same size) 2nd Bootstrap resample Elbe, winter, class 23 h CV = 35 yr Example: Elbe floods 10212002 Bootstrap resample (with replacement, same size) 2nd Bootstrap resample 2000 Bootstrap resamples Elbe, winter, class 23 h CV = 35 yr Elbe, winter, class 23 h CV = 35 yr Bootstrap resample (with replacement, same size) 2nd Bootstrap resample 2000 Bootstrap resamples Example: Elbe floods 10212002 90% percentile confidence band* Elbe, winter, class 23 h CV = 35 yr Example: Elbe floods 1021 % percentile confidence band* Elbe, winter, class 23 h CV = 35 yr * Bootstrap-t confidence band, other resampling types, Monte-Carlo tests,... Example: Elbe floods 1021 % percentile confidence band* Elbe, winter, class 23 h CV = 35 yr Cowling et al. (1996) J. Am. Statist. Assoc. 91:1516. Mudelsee et al. (2004) J. Geophys. Res. 109:D23101. Document loss !? Deforestation ? Homogeneous from ~1500 CLIMDAT LMM climate: cold & dry Significant ! Less freezing Example: Elbe floods 10212002 Winter floods * LMM climate: not really seen Not significant ! Length reduction Summer floods Example: Elbe floods 10212002 Example: Elbe and Oder floods Mudelsee et al. (2003) Nature 425:166. Conclusions: water management Flood risk... nonstationarity Conclusions: water management Flood risk and... land-use changes: not detectable Conclusions: water management Flood risk and... river engineering: minimal Conclusions: water management Flood risk and... reservoir building:negligible Conclusions: water management Flood risk and... climate: likely Conclusions: water management Flood risk and... climate: likely fewer winter floods (freezing), Conclusions: water management Flood risk and... climate: likely fewer winter floods (freezing), not (yet) more summer floods Conclusions: water management Flood risk and... climate: likely fewer winter floods (freezing), not (yet) more summer floods Even if Elbe and Oder are large rivers, yet monthly interval is not adequate temporal resolution for studying intense precipitation and floods. Kundzewicz et al. (2005) Nat. Hazards 36:165. (p. 179) Conclusions: water management Flood risk and... climate: likely fewer winter floods (freezing), not (yet) more summer floods Even if Elbe and Oder are large rivers, yet monthly interval is not adequate temporal resolution for studying intense precipitation and floods. Kundzewicz et al. (2005) Nat. Hazards 36:165. (p. 179) Elbe, Oder: daily resolution persistence* Conclusions: water management Flood risk and... climate: likely fewer winter floods (freezing), not (yet) more summer floods Conclusions: water management Flood risk and... climate: likely fewer winter floods (freezing), not (yet) more summer floods Middle Elbe and middle Oder only! Conclusions: water management Flood risk and... climate, land-use changes, river engineering, reservoir building Study river by river! More bootstrap... Regression models mean, not extremes fails to quantify trends in flood risk Mudelsee et al. (2004) J. Geophys. Res. 109:D23101. Regression models mean, not extremes fails to quantify trends in flood risk t-test fails also Mudelsee et al. (2004) J. Geophys. Res. 109:D23101. Regression modelx(i) = a + b t(i) + (i) noise(i) assumeE[(i)] = 0 Gaussian shape no persistence minimizeSSQ(a, b) = [x(i) - a - b t(i)] 2 error(b)[SSQ/(n-2)] 1/2 / ([t(i) - t(j)/n] 2 ) 1/2 ^ Regression modelx(i) = a + b t(i) + (i) noise(i) assumeE[(i)] = 0 Gaussian shape no persistence minimizeSSQ(a, b) = [x(i) - a - b t(i)] 2 error(b)[SSQ/(n-2)] 1/2 / ([t(i) - t(j)/n] 2 ) 1/2 ^ Regression modelx(i) = a + b t(i) + (i) noise(i) assumeE[(i)] = 0 Gaussian shape no persistence minimizeSSQ(a, b) = [x(i) - a - b t(i)] 2 error(b)[SSQ/(n-2)] 1/2 / ([t(i) - t(j)/n] 2 ) 1/2 ^ Regression modelx(i) = a + b t(i) + (i) noise(i) assumeE[(i)] = 0 Gaussian shape no persistence minimizeSSQ(a, b) = [x(i) - a - b t(i)] 2 error(b)[SSQ/(n-2)] 1/2 / ([t(i) - t(j)/n] 2 ) 1/2 ^ classical approach fails to quantify error bars Regression modelx(i) = a + b t(i) + (i) noise(i) residualse(i) = x(i) - a - b t(i) ^ ^ bootstrap Regression modelx(i) = a + b t(i) + (i) noise(i) residualse(i) = x(i) - a - b t(i) ^ ^ bootstrap {t(i), x*(i)} {t(i)} a*, b* STD[a*],... Resampled data, x*(i) = a + b t(i) + e*(i) Simulated fit parameters Bootstrap standard errors Repeat 400 times ^ ^ ^ ^ ^ Regression modelx(i) = a + b t(i) + (i) noise(i) residualse(i) = x(i) - a - b t(i) ^ ^ bootstrap {t(i), x*(i)} {t(i)} a*, b* STD[a*],... Resampled data, x*(i) = a + b t(i) + e*(i) Simulated fit parameters Bootstrap standard errors Repeat 400 times ^ ^ ^ ^ ^ draw one by one, with replacement Regression modelx(i) = a + b t(i) + (i) noise(i) residualse(i) = x(i) - a - b t(i) ^ ^ bootstrap nopersistence preserved! Resampled data, x*(i) = a + b t(i) + e*(i) ^ ^ draw one by one, with replacement Regression modelx(i) = a + b t(i) + (i) noise(i) residualse(i) = x(i) - a - b t(i) ^ ^ bootstrap persistence preserved! Resampled data, x*(i) = a + b t(i) + e*(i) ^ ^ draw data blocks with replacement block length, l Regression modelx(i) = a + b t(i) + (i) noise(i) residualse(i) = x(i) - a - b t(i) ^ ^ block bootstrap Resampled data, x*(i) = a + b t(i) + e*(i) ^ ^ draw data blocks with replacement Knsch (1989) Ann. Statist. 17:1217. block length, l Regression modelx(i) = a + b t(i) + (i) noise(i) residualse(i) = x(i) - a - b t(i) ^ ^ block length: o via AR(1) process Resampled data, x*(i) = a + b t(i) + e*(i) ^ ^ draw data blocks with replacement block length, l Regression modelx(i) = a + b t(i) + (i) noise(i) residualse(i) = x(i) - a - b t(i) ^ ^ block length: o via AR(1) process o more sophisticated... Resampled data, x*(i) = a + b t(i) + e*(i) ^ ^ draw data blocks with replacement block length, l Regression modelx(i) = a + b t(i) + (i) noise(i) residualse(i) = x(i) - a - b t(i) ^ ^ block length: o via AR(1) process o more sophisticated... Resampled data, x*(i) = a + b t(i) + e*(i) ^ ^ draw data blocks with replacement Lahiri (2003) Resampling methods for dependent data. Springer. block length, l Message Regression Flood risk Message Provide reliable error bars Regression Flood risk Message Provide reliable error bars Use bootstrap Regression Flood risk Message Provide reliable error bars Use bootstrap Regression Flood risk Talk downloadable from GOODIES Oder Elbe Dresden Eisenhttenstadt Erzgebirge Sudeten Mountains Elbe and Oder u :test statistic,t :flood event dates [t1; t2] : observation intervaln : data size Under H0, statistic u is standard normally distributed. Example: u = 3.0 means a highly significant downward trend. Cox & Lewis (1966) The Statistical Analysis of Series of Events. Methuen, London. Test of H0: "Constant flood risk" Homogeneous from ~1500 LMM climate: not seen Significant ! Less freezing Reduced data quality in 18501920 CAUTION ! Example: Oder floods 12692002 Winter floods LMM climate: not really seen Not significant ! Length reduction Reduced data quality in 18501920 CAUTION ! Example: Oder floods 12692002 Summer floods Conclusions "Although extreme floods with return periods of 100 yr and more occurred in central Europe in July 1997 (Oder) and August 2002 (Elbe), there is no evidence from the observations for recent upward trends in their occurrence rate. Global climate changes affect many and various processes in regional hydrology, such as river and soil freezing in the case of winter floods under continental climate." Mudelsee et al. (2003) Reservoir-size correction Reservoir size, present:237 million m 3 Reservoir size, January 1920:12 million m 3 Correction (assuming 100% utilization):class 1 instead of 2 Corrected records (heavy floods, 23):same trends Corrected records (all floods):fewer downward trends Oder:same results as for Elbe Time resolution of flood records Even if Elbe and Oder are large rivers, yet monthly interval is not adequate temporal resolution for studying intense precipitation and floods. However, it is important to note that climate is just one of several important factors controlling the process of river flow. Figure 8 illustrates changes in annual maximum river flow of the river Warta (large right-hand tributary to the Odra) at the Poznan gauge. There has been a clearly decreasing long-term tendency in annual maximum flow, which is difficult to explain by climatic impacts. The tendency can be explained by direct human interference (changes in storage, land use, and melioration). Kundzewicz et al. (2005) Nat. Hazards 36:165. (p. 179) Elbe, Oder: daily resolution persistence* Correlation: flood occurrence vs atmospheric pressure Flood occurrence, y :0:no flood 1:flood Atmospheric pressure variable, z :SLP (5 x 5 grid) or z 500 (2.5 x 2.5 grid), 1658 to 1999,monthly resolution, reconstructed from measured and documentary data (Luterbacher et al Climate Dynamics 18:545561) Biserial correlation coefficient, r yz :Students t distributed Correlation: flood occurrence vs atmospheric pressure Winter: Zonal air-flowSummer: Meridional air-flow Mudelsee et al. (2004) J. Geophys. Res. 109:D23101. Correlation: flood occurrence vs atmospheric pressure r(Elbe/Oder floods, NAO) insignificant Analyze Weser, Rhine, Main Persistence: AR(1) vs ARFIMA Mudelsee (in prep.) Persistence: AR(1), even spacing Positive autocorrelation, memory, red noise e(i)= e(i1) a + N(0;1) (i) [1a 2 ] 1/2 Persistence: AR(1), uneven spacing e(i)= e(i1) exp[[t(i)t(i1)]/ ] + N(0;1) (i) [1exp[2[t(i)t(i1)]/ ]] 1/2 Robinson PM (1977) Stochastic Processes and their Applications 6:924. Mudelsee M (2002) Computers and Geosciences 28:6972.