Journal of Hydrology 529 (2015) 302–315

Contents lists available at ScienceDirect

Journal of Hydrology

journal homepage: www.elsevier .com/locate / jhydrol

Comparing various artificial neural network types for water temperatureprediction in rivers

http://dx.doi.org/10.1016/j.jhydrol.2015.07.0440022-1694/� 2015 Elsevier B.V. All rights reserved.

⇑ Corresponding author. Tel.: +48 22 6915 858; fax: +48 22 6915 915.E-mail address: adampp@igf.edu.pl (A.P. Piotrowski).

Adam P. Piotrowski a,⇑, Maciej J. Napiorkowski b, Jaroslaw J. Napiorkowski a, Marzena Osuch a

a Institute of Geophysics, Polish Academy of Sciences, Ks. Janusza 64, 01-452 Warsaw, Polandb Environmental Engineering Faculty, Warsaw University of Technology, Poland

a r t i c l e i n f o s u m m a r y

Article history:Received 17 November 2014Received in revised form 22 June 2015Accepted 27 July 2015Available online 1 August 2015This manuscript was handled by AndrasBardossy, Editor-in-Chief, with theassistance of Fi-John Chang, Associate Editor

Keywords:Streamwater temperature forecastingArtificial neural networksTemperate climate zonesNearest neighbour approachANFIS

A number of methods have been proposed for the prediction of streamwater temperature based on var-ious meteorological and hydrological variables. The present study shows a comparison of few types ofdata-driven neural networks (multi-layer perceptron, product-units, adaptive-network-based fuzzyinference systems and wavelet neural networks) and nearest neighbour approach for short timestreamwater temperature predictions in two natural catchments (mountainous and lowland) locatedin temperate climate zone, with snowy winters and hot summers. To allow wide applicability of suchmodels, autoregressive inputs are not used and only easily available measurements are considered.Each neural network type is calibrated independently 100 times and the mean, median and standarddeviation of the results are used for the comparison. Finally, the ensemble aggregation approach is tested.

The results show that simple and popular multi-layer perceptron neural networks are in most cases notoutperformed by more complex and advanced models. The choice of neural network is dependent on theway the models are compared. This may be a warning for anyone who wish to promote own models, thattheir superiority should be verified in different ways.

The best results are obtained when mean, maximum and minimum daily air temperatures from theprevious days are used as inputs, together with the current runoff and declination of the Sun from tworecent days. The ensemble aggregation approach allows reducing the mean square error up to severalpercent, depending on the case, and noticeably diminishes differences in modelling performanceobtained by various neural network types.

� 2015 Elsevier B.V. All rights reserved.

1. Introduction

Streamwater temperature is an important factor of water qual-ity that has significant impact on the distribution of biotic organ-isms in the aquatic ecosystem (see e.g. Sharma et al., 2012; Shenet al., 2015). It is governed by a number of environmental factors,including air temperature, net radiation, river runoff, groundwaterinputs, river width and depth, shading and many others. Probablybecause of global warming the temperature increase in many nat-ural rivers has already been observed (Webb et al., 2008; Orr et al.,2015) and is expected to continue (Kurylyk et al., 2014), what mayespecially impact the ecosystem of rivers located in cold and tem-perate climate zones (Blaen et al., 2013; Ficklin et al., 2013), due tothe shortening of freezing periods (e.g. Takacs and Kern, in press)and the increasing possibility of exceedance of the extreme maxi-mum air temperatures. However, the relations between global rise

of air temperatures and streamflow temperatures are not trivialand proving a link still poses a significant challenge (Mohseniand Stefan, 1999; van Vliet et al., 2011; Kurylyk et al., 2014).

Apart from the global warming, various local factors induced bynatural, semi-natural or anthropogenic processes may severelyimpact the streamwater temperatures (Poole and Berman, 2001;Caissie, 2006; Meier, 2012). However, the importance of such fac-tors like discharge variations (Pekarova et al., 2008), flow regula-tions (Dickson et al., 2012), groundwater runoff contribution(Loinaz et al., 2013; MacDonald et al., 2014), mesoscale habitattypes (Long and Jackson, 2014), urbanization (Anderson et al.,2010; Xin and Kinouchi, 2013), forest burning (Wagner et al.,2014), ice cover (Caissie et al., 2014) or spillage of thermal pollu-tions (Vega et al., 1998; Kalinowska et al., 2012) highly dependon the site-specific conditions and cannot be easily generalized.This poses additional challenges for models describing relationshipbetween meteorological conditions and streamwatertemperatures.

A.P. Piotrowski et al. / Journal of Hydrology 529 (2015) 302–315 303

Streamwater temperature prediction approaches proposed inthe past mainly included physically-based, temperature equilib-rium concept-based or simple statistical models (Webb et al.,2008; Wehrly et al., 2009; Bustillo et al., 2014). In recent years var-ious kinds of deterministic models (Caissie et al., 2007),data-driven approaches (St-Hilaire et al., 2012; Grbic et al., 2013;Cole et al., 2014) or artificial neural networks (ANNs) (Sahooet al., 2006; Sivri et al., 2007; Chenard and Caissie, 2008; Sahooet al., 2009; Daigle et al., 2009; Faruk, 2010; McKenna et al.,2010; Jeong et al., 2013; Napiorkowski et al., 2014; Piotrowskiet al., 2014; Hadzima-Nyarko et al., 2014; Rabi et al., in press) havebeen applied to this task. In some studies (Sahoo et al., 2009;Bustillo et al., 2014) regression and ANN models are claimed toperform not worse than the more sophisticated empirical or heatbudget-based models.

In recent years a few comparative studies have been publishedin the field. For example Sahoo et al. (2006) compared regression,chaotic and multi-layer perceptron ANN models, Wehrly et al.(2009) compared various statistical models, Cole et al. (2014) com-pared three data-driven approaches with heat flux model andBustillo et al. (2014) verified the performance of various regressionand temperature equilibrium-based models in the context ofstreamwater temperature prediction. However, although a largenumber of different types of neural networks have been developedso far, for the prediction of streamwater temperatures almostalways the ‘‘classical’’ multi-layer perceptron ANNs (MLP) havebeen used (Sahoo et al., 2006; Sivri et al., 2007; Chenard andCaissie, 2008; Daigle et al., 2009; McKenna et al., 2010; Jeonget al., 2013; Piotrowski et al., 2014; Hadzima-Nyarko et al., 2014;Cole et al., 2014; Rabi et al., in press). Similar MLP networks werealso applied for somehow related problem, the prediction of tem-peratures of stormwater runoff in urban watershed (He et al.,2011; Sabouri et al., 2013). In a number of studies it is claimed thatMLP outperforms various non-ANN-based regression approaches(Chenard and Caissie, 2008; Daigle et al., 2009; Faruk, 2010;Jeong et al., 2013; Rabi et al., in press); however, according toour knowledge, among other ANN types for streamwater tempera-ture prediction only radial-basis function ANNs (RBF) were com-pared with MLP ones in just two papers (Sahoo et al., 2009;Hadzima-Nyarko et al., 2014). In both papers it was found thatMLPs trained either by a kind of Genetic Algorithm (Sahoo et al.,2009) or Levenberg–Marquardt algorithm (Hadzima-Nyarkoet al., 2014) outperform other modelling methods, including RBFneural networks.

This lack of interest in using various ANN types may be a sur-prise, as in other hydrological applications wide number of versa-tile kinds of ANNs were successfully tested, including wavelet ANN(WNN) (Nourani et al., 2014), adaptive-network-based fuzzy infer-ence systems (ANFIS) (Nayak et al., 2004), Bayesian neural net-works (Kingston et al., 2005), recurrent neural networks(Coulibaly et al., 2000), modular neural networks (Zhang andGovindaraju, 2000), product-units neural networks (PUNN)(Piotrowski and Napiorkowski, 2012) and many others (Maieret al., 2010; Abrahart et al., 2012; Kisi et al., 2012). The presentpaper aims at filling this gap by verifying the applicability of fewANN types to streamwater temperature predictions in two catch-ments, one mountainous and one lowland, located in temperateclimate zone of eastern Poland. Obviously, comparing all kinds ofexisting ANN models, or even all that were tested for hydrologicalapplications so far, would be an almost intractable task. One mustalso remember that some ANN types are designed for specific kindsof problems, and are not necessary easily ‘‘comparable’’ with otherANNs for the streamwater temperature prediction. Taking this intoaccount, four ANN types were selected, including (1) MLP that iswidely applied in the field, (2) ANFIS and (3) WNN that are popularin other hydrological applications, and (4) PUNN that has been

relatively recently introduced to water-related studies. As severalstudies mentioned above showed the superiority of MLP over var-ious regression methods in streamwater temperature prediction, inthis paper instead of classical regression, the four ANN types arecompared with the nearest neighbour approach that was recentlyintroduced to the field (St-Hilaire et al., 2012).

2. Study sites and data

In this study streamwater temperature prediction models aretested at daily time scale. Although water temperature in riversdepends on numerous natural and anthropogenic factors, the dataon such important features like solar radiation, cloudiness, ground-water inflow, snow melting, etc. are frequently not available. Toallow wide applicability of the performed comparison, only themost important and easily accessible data are used in this study(consult the critical comments on using unneeded variables inwater quality modelling by ANNs in Chau, 2006). The water tem-perature in the following day TW(t + 1) is predicted based on thedaily mean (T2), maximum (T3) and minimum (T1) air tempera-tures, the river runoff (Q) and the declination of the Sun (S) only,measured during the recent days (up to day t). The past water tem-peratures are not used to predict the future ones (in other wordsno autoregressive inputs are used) as the water temperature pre-diction models are expected to be also used for the future climateconditions.

Data from two rivers located in the eastern part of Poland areconsidered in this study, namely Biala Tarnowska and Suprasl.Both catchments are located in the Humid Continental Zoneaccording to the Köppen Climate Classification. Of major interestis the Biala Tarnowska River, a right-bank tributary of theDunajec River that flows north from the Carpathian Mountains(see left part of Fig. 1). Biala Tarnowska catchment’s area up tothe Koszyce Wielkie gauging station (the village located on theoutskirts of Tarnow city), at which both runoff and streamwatertemperature are measured, equals 956.9 km2. Air temperature ismeasured at a meteorological station in Tarnow. The highest peakswithin the catchment reach almost 1000 m a.s.l. and are located inthe far south; in the north, close to the city of Tarnow, hills arelower, reaching up to 550 m a.s.l. Biala Tarnowska is a typicalmountainous river with slopes up to 8.6‰ in the upper part andabout 0.9‰ in the lower part. During winter it frequently freezes,in summer water temperature may occasionally exceed 22 �C.Note that if the river is frozen, for the purpose of this study it isassumed that its water temperature equals 0 �C. Fluctuations ofwater temperature are frequent and highly correlated with theweather patterns. The river is especially interesting as it is locatedin the mountains with frequent snowfalls during half of the yearand very dynamic weather conditions. For this catchment 17 yearsof daily data are available (01.11.1983–31.10.2000) (see left part ofFig. 2) and both gauging and meteorological stations are locatedclose to each other.

The second considered river is the lowland Suprasl that flowsthrough the flat part of eastern Poland. The data from SupraslRiver are used in this study to test if the performance ranking ofmodels achieved for Biala Tarnowska would be representative alsofor another river located in similar climate zone, even if environ-mental conditions and data availability would differ. Main differ-ences between experiment performed on Suprasl River and thatperformed on Biala Tarnowska are as follows: (1) Suprasl River islocated in different orographic conditions; (2) its slopes and watervelocities are low; (3) it is mainly supplied by groundwater andsnow melting; (4) available data cover only ten years period (01.11.1990–31.10.2000); (5) the distance between the gauging station(located in the village of Zaluki) and the station from which

Fig. 1. Location of Biala Tarnowska (left map) and Suprasl (right map) catchments.

304 A.P. Piotrowski et al. / Journal of Hydrology 529 (2015) 302–315

meteorological data are collected (located in the city of Bialystok)is large (over 25 km, see right part of Fig. 1).

The last two factors obviously affect the prediction perfor-mance, but the question is if this would also be reflected in thechange of the ranking of the models. The catchment size ofSuprasl up to Zaluki village gauging station equals 344 km2. Thehydro-meteorological regimes of air temperature, runoff and watertemperature from Suprasl catchment are illustrated in Fig. 2.

To diminish the possibility of overfitting when ANNs are used(see the discussion in Section 3.5), the daily data from both riversare divided into three sets: training (used during calibration)[Biala Tarnowska: 01.11.1983–31.10.1990; Suprasl: 01.11.1990–31.10.1994], validation (used to decide when to stop ANN training)[Biala Tarnowska: 01.11.1990–31.10.1994; Suprasl: 01.11.1994–31.10.1997] and testing (calibration-independent) [BialaTarnowska: 01.11.1994–31.10.2000; Suprasl: 01.11.1997–31.10.2000]. The mean water temperatures measured for BialaTarnowska in periods included into training, validation and testingsets are equal to 8.83 �C, 9.05 �C and 8.73 �C respectively, henceshow no sign of warming. However, respective measurements forthree data sets from Suprasl River show positive trend in watertemperature (7.78 �C, 7.88 �C and 8.44 �C).

The initial selection of input variables used in this study isbased on the results published in Piotrowski et al. (2014) thatwas restricted to MLP models only. However, the ANN input vari-ables (and hence ANN architectures) are tested again, as (1) dailyminimum air temperature was not used in the previous studyand, what may be even more important, various input variablesmay give best results for different ANN types; and (2) to achieveconvincing results, each ANN is optimized 100 times in this studyand the mean and the median of the results are compared. Theselection of the proper set of input variables is performed forBiala Tarnowska River only, and the same inputs are used forSuprasl River; this saves space and facilitates the comparison ofthe performance rankings of different ANN models on both rivers.

3. Models

With exception of statistical tests and wavelets, codes of allother methods used in this study have been implemented inMATLAB or FORTRAN. For statistical tests and wavelets respectiveMATLAB functions were used.

3.1. Multi-layer Perceptron ANN (MLP)

MLP is probably the most popular among ANN types. It consistsof nodes grouped into input, hidden and output layers. Singlehidden layer is considered sufficient to approximate continuousfunctions, but there is no widely accepted rule regarding the num-ber of hidden nodes (Haykin, 1999). MLP with a single hidden layerin which logistic (sigmoidal) activation function is used may bewritten as:

yP ¼ v0 þXJ

j¼1

v jf wj0 þXK

i¼1

wjixi

!

where f ðaÞ ¼ 11þ e�a

ð1Þ

Such MLP version is used in this study. In Eq. (1) yP is a pre-dicted value of the output variable, xi, i = 1, . . . K represent inputvariables, w and v are MLP weights and J is the number of hiddenunits. Although various activation functions may be used in MLPnetworks, the difference in the performance between them is usu-ally limited and in this study the reliable (Shamseldin et al., 2002)logistic transfer function defined in Eq. (1) is used. The number ofhidden nodes (J) is often determined empirically (Zhang et al.,1998; Maier et al., 2010). As a common practice facilitated training,input and output variables are linearly normalized to [0, 1] intervalbefore being used by MLP (Zhang et al., 1998).

Biala Tarnowska River Suprasl River

Fig. 2. The average monthly values of basic hydro-meteorological data for Biala Tarnowska and Suprasl catchments.

A.P. Piotrowski et al. / Journal of Hydrology 529 (2015) 302–315 305

3.2. Product-Unit ANNs (PUNN)

In Rumelhart et al. (1986) a type of ANN with both multiplica-tive and additive nodes, called Sigma-Pi units, has been suggested.However, as in Sigma-Pi units the number of multiplicative termsquickly increases with the number of model inputs, Durbin andRumelhart (1989) proposed a modified version of such networks,called Product-Unit ANNs (PUNN). The simplest version of PUNNthat is used in this study may be defined as

yP ¼ f v0 þXJ

j¼1

v j

YK

i¼1

xwji

i

!ð2Þ

Durbin and Rumelhart (1989) suggested using the logistic func-tion as f, but in practice the PUNN proposed in Eq. (2) with f beingsimply an identity function (f(a) = a) gained more attention(Martinez-Estudillo et al., 2006; Piotrowski and Napiorkowski2012) and is applied in this study.

306 A.P. Piotrowski et al. / Journal of Hydrology 529 (2015) 302–315

The interesting feature of PUNN is that, contrary to the majorityof ANN types, it has fewer parameters to be optimized than MLP(when the same number of input, output and hidden nodes is con-sidered). Although PUNNs are claimed to be difficult to train(Janson and Frenzel, 1993), the convergence of thegradient-based algorithm for PUNN has been proved (Zhanget al., 2008). To facilitate training in Piotrowski andNapiorkowski (2012) an approach of limiting the allowable sizeof PUNN weights was proposed for rainfall–runoff modelling, butthis turned out not necessary for the application considered inthe present study.

Another feature of PUNN is that the possible negative inputsraised to the non-integer exponents produce complex numbers.To avoid inputs lower than 0, or equal 0 or 1, that could enterthe multiplicative units, PUNN inputs and outputs are linearly nor-malized to the [0.1, 0.9] interval (following Martinez-Estudilloet al., 2006). However, when data are extrapolated there is stillthe possibility that the negative inputs may be encountered. Formore discussion on PUNN the reader is referred to Durbin andRumelhart (1989), Schmitt (2001), Zhang et al. (2008) andPiotrowski and Napiorkowski (2012).

3.3. Adaptive-network-based fuzzy inference systems (ANFIS)

Among a number of fuzzy ANN types used in various fields ofscience probably the most popular one, called ANFIS, was devel-oped by Jang (1993). Classical ANFIS uses the Takagi–Sugeno infer-ence rules, in which the consequent part is defined by a non-fuzzyfunction. An example of such rule may look as follows: if x1 is LOWand x2 is HIGH. . . and xK is LOW, than y = f(x, b), where LOW andHIGH are linguistic variables and b is a vector of parameters. Infuzzy rules each input variable (xi) may be, for example, partly con-sidered as, HIGH, partly MEDIUM or LOW, depending on the defi-nition of so-called membership functions. Frequently theGaussian membership functions are used in the following form

ljiðxiÞ ¼ expxi � a1ji

2a2ji

� �2 !

ð3Þ

where ljiðxiÞ is the membership function of j-th linguistic variable,xi is the i-th input to a particular first-layer node and a1ji and a2ji arethe centre and spread parameters, respectively.

The simplest variant of ANFIS, tested in this study, has five lay-ers and is defined as follows. When the number of rules NR (set to 4in this study) and input variables K (note that various combina-tions of inputs, hence various K values, are tested in this paper)are set, in layer 1 a fuzzification takes place, following Eq. (3). Inlayer 2 for each j-th rule (j = 1, . . . , NR) its firing strength is com-puted as

wj ¼YK

i¼1

lji j ¼ 1; . . . ;NR ð4Þ

In layer 3 the ratio of each rule’s firing strength is computed as

v j ¼wjPNR

k¼1wk

j ¼ 1; . . . ;NR ð5Þ

Then, in layer 4, the Takagi–Sugeno inference is performed bymeans of simple linear relation

TSj ¼ v j bj0 þXK

i¼1

bjixi

!j ¼ 1; . . . ;NR ð6Þ

and the final output is obtained by means of the summationprocedure

yP ¼XNR

j¼1

TSj ð7Þ

The main disadvantage of ANFIS is large number of parameters,which is equal to the number of elements in vectors a and b – inthe discussed version one obtains 2�NR�K + (K + 1)�NR parameters.Note that in this paper we assume that the number of rules NR isset by the user (to 4 in this paper), instead of the number of linguis-tic labels. Using all possible combinations of a few linguistic labelsleads to heavily over-parameterized networks when the number ofinput variables is relatively high, as in this study (depending on thecase, K varies around 10 in this paper, what even for only two lin-guistic variables leads to 2K possible rules). In this paper input andoutput variables are linearly normalized to [�1, 1] interval beforebeing used by ANFIS.

3.4. Wavelet ANN (WNN)

In a number of studies it was shown that various datapre-processing techniques (Wu et al., 2009), especially the wavelettransform (Alexandridis and Zapranis, 2013), may improve the per-formance of hydrological modelling (Wang and Ding, 2003;Nourani et al., 2009; Tiwari and Chatterjee, 2010; Adamowskiand Chan, 2011; Kisi and Cimen, 2011; Maheswaran and Khosa,2014; Adamowski and Prokoph, 2014). The wavelet transform isan effective decomposition method that provides a way of analyz-ing signal in both time and frequency domains, contrary to the con-ventional Fourier transforms that do provide time–frequencyanalysis for the variables with stationary signals.

The time-scale wavelet transform of a continuous time signal,x(s), is defined as

Wðs; dÞ ¼ jsj�1=2Z 1

�1w�

s� ds

� �xðsÞds;

d; s; s 2 R; s–0ð8Þ

where w (s) is mother wavelet (‘*’ denotes the complex conjugate); sis the scale, d is the time factor, R is the set of real numbers. Theabove equation describes, that wavelet transform represents thedecomposition of x(s) under different resolution scales(Daubechies, 1990).

The successive wavelet is often discrete in real application. For adiscrete time series xt, the discrete wavelet transform DWT can bedefined as

Wðj; kÞ ¼ 2�j=2XN�1

t¼0

wð2�jt � kÞxt ð9Þ

where t is integer time step, j and k are integers that control thescale and time; W(j, k) is the wavelet coefficient for the scale factor,s = 2j, and the time factor, s = 2jk.

Data pre-processing by means of Mallat algorithm (Mallat,1989) is used for the calculation of the discrete wavelet transform(Eq. (9)) according to the following steps. In the first step, forselected wavelet, four filters are computed, namely the decompo-sition low-pass filter, the decomposition high-pass filter, thereconstruction low-pass filter and the reconstruction high-pass fil-ter. In the second step, two sets of coefficients, approximation coef-ficients cA1, and detail coefficients cD1 are calculated. Thesevectors are obtained by convolving x with the low-pass filter forapproximation, and with the high-pass filter for detail, followedby dyadic decimation (downsampling). In the next step approxi-mation coefficients cA1 are split in two parts using the samescheme, producing cA2 and cD2, and so on until the assumeddecomposition level K is reached.

The next step is so-called reconstruction. Based on the approx-imation coefficients at level K (cAK), and the details coefficientsfrom level K to 1 (cDK, . . . , cD2, cD1) the level K approximationand the level 1, 2, . . ., K details are reconstructed with the help ofthe reconstruction low-pass filter and the reconstruction

A.P. Piotrowski et al. / Journal of Hydrology 529 (2015) 302–315 307

high-pass filter. After this step any hydrological time series can berepresented (decomposed) as a sum of one approximation seriesAK and K detail series D1, . . . , DK.

In the present study, several wavelet types are coupled withMLP architecture. Daubechies (db) wavelets (suggested to performbest in a detailed study by Shoaib et al. (2014) on WNN applicationto river runoff forecasting), Symmlet (sym), also known asDaubechies’ least-asymmetric wavelets, and Coiflet (coif) of order1–6, are considered as the mother wavelets. These wavelets offeran appropriate trade off between wave-length (for evaluation oflocal behaviour of signal) and smoothness, resulting in an appro-priate behaviour for short term forecasting.

The hydrological time series used in WNN training are decom-posed into approximation part and detail parts for differentdecomposition levels, from 2 to 8. These data are then linearly nor-malized to [�1, 1] interval.

3.5. ANN training, initialization, objective function, approach toprevent overfitting and split of data

Gradient-based algorithms are the most popular ANN trainingmethods (Haykin, 1999). Although they do not have generalizationcapabilities and hence tend to stick in the closest local optima, inmany papers (Schaffer, 1994; Mandischer, 2002; Ilonen et al.,2003; Socha and Blum, 2007; Almeida and Ludermir, 2010;Piotrowski et al., 2012; Piotrowski, 2014; Bullinaria and AlYahya,2014) it was shown that their performance is frequently similarto the performance of more sophisticated global search methods,and the speed is much slower. A similar conclusion was found inour previous study on streamwater temperature modelling bymeans of MLP neural networks (Piotrowski et al., 2014). Hence,in this study Levenberg–Marquardt (LM) algorithm (Hagan andMenhaj, 1994), one of the most efficient algorithms for ANN train-ing that use information on objective function derivatives(Adamowski and Karapataki, 2010), is applied.

The application of LM training method requires Mean SquareError (MSE) to be used as objective function:

MSE ¼ 1N

XN

n¼1

yPn � yn

� �2 ð10Þ

where yn is the measured value of the output variable and N is thenumber of observations. The initial weights of each ANN type arerandomly generated within [�1, 1] (apart from the spread parame-ter in ANFIS, that allows positive values only); starting from lowvalues of weights is important for good generalization propertiesof trained ANNs (Nowlan and Hinton, 1992; Haykin, 1999).

Because ANNs are non-parametric regression models (Gemanet al., 1992) that are universal approximators (Hornik et al.,1989), they may suffer from overfitting to the training data(Holmstrom and Koistinen, 1992; Prechlet, 1998; Haykin, 1999).Overfitting is understood as fitting the ANN parameters not onlyto the signal, but also to a noise that is usually present in the train-ing sample. A number of techniques to prevent ANN overfittingwere recently compared on hydrological data in Piotrowski andNapiorkowski (2013); so-called early stopping, based onPrechlet’s (1998) Generalization Loss (GL) class turned out themost reliable approach. To use this method in the present study,the data sets from Biala Tarnowska and Suprasl catchments aredivided into three parts: training (TR, composed of 41% and 40%of the data for Biala Tarnowska and Suprasl catchments), validation(V, 24% and 30%, respectively) and independent testing (TE, 35%and 30%, respectively), hence three different MSE values (see Eq.(10)) are computed in each iteration (MSETR, MSEV and MSETE).During calibration of ANN models the derivatives and the step size

are determined according to MSETR only. The training is stopped atiteration g at which (see Prechlet, 1998)

GLðgÞ ¼ MSEVðgÞMSEVðcÞ

� 1� �

> a ð11Þ

where a is set to 0.2 in this study and c (c 6 g) is the number of theiteration at which the lowest value of MSEV was obtained. The train-ing may also be stopped after pre-defined number of function callsthat is limited to 300 in this study. After termination, the best solu-tion returned by the algorithm is chosen according to the perfor-mance for the validation data (i.e. the solution with the lowestMSEV(c)).

As LM is a local search method, it is prone to be trapped in localoptima that occasionally may represent a very poor solution (inother words, MSE may be very high in some local optima). Suchvery poor solution could significantly affect the 100-run mean per-formance of the model. As after a few trials one may easily learnwhat the performance of the model should be, and as LM is a quickoptimization method, it is easy to eliminate such useless calibra-tions by setting a maximum allowed value on MSEV (we may dothis as the validation set is know during optimization and theselection of the best solution is based on this data). Then, if localoptimum found by LM at particular run has higher MSEV than suchpre-defined maximum allowed value, the calibration is simplyrepeated for that run, starting from another initial solution. In thispaper the maximum allowed values of MSEV are set to 2 in case of1 lead day prediction for Biala Tarnowska, 3 – for 1 lead day predic-tion for Suprasl; 4 – for 3-days ahead and 5 – for five days aheadpredictions on both rivers. Numbers of ANNs that fall into suchdefined very poor local optima vary, depending on ANN type andsize, but rarely exceed several percent. By this approach the poorlocal optima are removed from the 100 runs considered for com-parison. However, there is still a possibility that the performanceof some calibrated ANNs could be truly poor for the testing data(MSETE values), that are treated as unknown during the wholetraining process to be fully calibration-independent.

3.6. Nearest neighbour approach

The k-nearest neighbour approach (kNN) has been applied tostreamwater temperature prediction by St-Hilaire et al. (2012). Inthe present study the kNN variant that requires neither calibrationnor the use of validation set is applied. In the simplest version eachinput is considered of similar importance, and all input and outputvariables are standardized to [0, 1] interval prior to computations.First, based on the training and validation data (used together),according to the least squares the coefficients w of linear relation

yPlin ¼

PINi¼1wixi between input and output variables are found.

Then k representing the number of nearest neighbours must bespecified (tests with k set between 1 and 100 neighbours are per-formed in this study) and for each input its k-nearest neighbours(from training and validation sets) are considered to make the finalprediction of the water temperature. For each among k-nearestneighbours the prediction of water temperature is performedaccording to the linear model with already evaluated weights w.The final prediction for each input data is done by weighted aver-age of the predictions obtained for all k-nearest neighbours, whenthe significance of each among k-nearest neighbours depends lin-early on its Euclidean distance in the input space from the currentobservation.

3.7. Statistical tests

Although very different opinions on the necessity of using sta-tistical tests to validate results may be found in the literature

308 A.P. Piotrowski et al. / Journal of Hydrology 529 (2015) 302–315

(Schmidt, 1996; Clarke, 2008; Vecek et al., 2014), at leastnon-parametric tests, which application is not restricted byassumptions that are rarely fulfilled, may facilitate the evaluationof the research. Wilcoxon rank-sum and Kolmogorov–Smirnovtests (Corder and Foreman, 2014) are chosen in this study. The sig-nificances of pair-wise differences between the results achievedfrom the selected best architectures of each ANN type are verifiedat significance level a = 5%. As 6 pair-wise comparisons among 4models are tested for each prediction horizon in this study, onemust note that the actual probability of making an error amongall considered comparisons is in fact higher than a (and reachesabout 26%, see Clarke, 2010), hence for clarity the detailed(un-corrected) p-values are also provided.

3.8. ANN ensemble aggregation

When ANNs are trained a number of times, for every timeinstance (in this paper – day) each trained model would usuallypredict different values of output variable. Hansen and Salamon(1990) found that the modelling performance may be improvedwhen such individual predictions are somehow aggregated.Plenty of aggregation methods were developed since 1990s (asan example we refer the reader to Islam et al., 2003; Windeattand Zor, 2013 and a review by Mendes-Moreira et al., 2012), butthe simplest and most widely used is computing the mean,weighted mean or median of predictions obtained from varioustrained ANNs at each time step (See and Abrahart, 2001; Zhouet al., 2002; Granitto et al., 2005; Zheng, 2009; Piotrowski et al.,2014).

In many papers the problem of constructing and selectingensemble members is addressed. Various ANN ensemble membersmay have different architecture, be trained by different optimiza-tion methods or their training may be based on various data sam-ples selected according to so-called boosting or bagging methods(see Granitto et al., 2005; Pasti and de Castro, 2009). It is alsoknown that the performance of aggregation approach may dependon the ensemble size (see Zheng, 2009; Boucher et al., 2010;Windeatt and Zor, 2013). In this study we use a very simpleapproach and for every n-th time instance the finalensemble-aggregated prediction (yn

P,agg) is evaluated by taking themedian prediction of all 100 neural networks of particular typeand architecture that were previously trained. Then the MeanSquare Error of aggregated predictions is computed (MSEagg)

MSEagg ¼ 1N

XN

n¼1

yP;aggn � yn

� �2

yP;aggn ¼ median yP;i

n ; i ¼ 1; . . . ;100� � ð12Þ

This approach is similar to that applied recently by DeWeberand Wagner (2014) for water temperature predictions in differentcatchments. However, the median is chosen instead of the mean, asit rules out occasional poor predictions (that, as will be shownlater, do happen, especially when PUNN is used). The ensembleaggregation is tested in this study only on the best architectureof each ANN type.

4. Results and discussion

4.1. Choice of model architectures

The choice of ANN architectures requires determination ofinput variables, the number of hidden nodes (in case of MLP,PUNN and WNN) or the number of rules (in case of ANFIS). Thisis a difficult task, frequently done by trial and error (Zhang et al.,1998; Haykin, 1999), also in case of stream temperature modelling(Chenard and Caissie, 2008; Daigle et al., 2009). In case of kNN,

input variables and number of nearest neighbours must be deter-mined. The choice of ANN and kNN input variables is based on datafrom Biala Tarnowska catchment in this study. The same inputvariables are than used for Suprasl catchment. However, full archi-tectures of each ANN type, including the number of hidden nodes,rules and wavelet types are determined separately for both catch-ments, based on the performance for daily river temperature pre-diction; in case of 3- and 5-days ahead predictions the samearchitectures are used.

4.1.1. Biala Tarnowska catchmentAt first we consider the MLP architecture from our previous

studies that also aimed at streamwater temperature prediction atBiala Tarnowska catchment (Piotrowski et al., 2014;Napiorkowski et al., 2014). This architecture include 5 hiddennodes and 7 inputs (such architecture we call 7-5-1). The inputsare: declination of the Sun (S) at days t and t � 1, river runoff (Q)measured at day t, daily maximum air temperature (T3) measuredat day t and daily mean air temperatures (T2) measured at days tand t � 1 as well as a sum of daily mean air temperatures mea-sured at 5 consecutive days t � 2 to t � 6 (see Table 1). The lastinput provides aggregated information about the air temperaturesthat occur during the last week before the prediction, which areimportant for daily streamwater temperature prediction(Mohseni et al., 1998; Meier, 2012; DeWeber and Wagner (2014).Including the mean air temperatures from each of seven recentdays would lead to an overparameterized ANN model (see discus-sion by Gaume and Gosset (2003) on ANN overparameterization inhydrology), but number of input variables may become a problemnot only when ANNs are used, see Sahoo et al. (2009). Table 1shows that mean air temperatures from the last two days and asum of mean air temperatures from another previous 5 days arebetter choices than other variants of mean air temperature inputs,based on MLP neural networks with number of hidden nodes fixedto 5. Table 1 contains the information on 100-run mean and med-ian MSE achieved for training (MSETR), validation (MSEV) and test-ing sets (MSETE). However, other input variables, like themaximum (T3) and minimum (T1) air temperature or river runoff(Q), measured at one or more recent days also may, or may not beimportant, depending on particular type of ANN. Hence, for BialaTarnowska catchment various architectures of each ANN kind aretested.

The discussion on ANN architectures is divided into three cases,based on information on air temperatures used. The detailedresults achieved by each considered ANN architecture for dailystreamwater temperature prediction are to be found inSupplementary Tables 1–12 (Appendix A); the results from thebest architectures of each ANN type are compared in Table 2.

In the first case (Case 1), which is the same as in Piotrowskiet al. (2014), the mean and maximum daily air temperatures areused – the results are given in Suppl. Tables 1–4. Each consideredANN architecture uses at least 7 inputs that were discussed abovein this section, for larger architectures T3(t � 1) is used as input 8,and Q(t � 1) as input 9. Number of tested hidden nodes varybetween 3 and 8 (in case of MLP and PUNN), number of rulesbetween 3 and 6 (for ANFIS). In the case of WNNs 2–4 hiddennodes and three different wavelets (Daubechies, Symmlet andCoiflet) with proper order and the decomposition level were tested.

In Case 2 among air temperatures only the mean ones are con-sidered (minimum and maximum ones are skipped, see Suppl.Tables 5–8), what is also done in some studies (e.g.Hadzima-Nyarko et al., 2014). Hence the number of input nodesequals only 6 – in case of MLP, PUNN and ANFIS, or 10 – in caseof WNN (consult Suppl. Table 8).

Finally in Case 3 mean, minimum and maximum daily air tem-peratures are used as ANN inputs (see Suppl. Tables 9–12). In the

Table 1Biala Tarnowska River. The results obtained by MLP architectures with 5 hidden nodes and mean daily air temperatures (T2) measured at various recent days (t). Other inputvariables, namely S(t, t � 1); Q(t); T3(t), are kept fixed. sum(t � 2, t � 6) and sum(t � 3, t � 6) is a sum of the daily averaged air temperatures measured 2–6 (first case) or 3–6(second case) days before t. The lowest MSE values are shaded, the architecture chosen as the best is bolded.

Table 2Biala Tarnowska River. Results achieved by various ANN types. The detailed results showing architecture selection for each ANN type is given in the appendix. S – declination ofthe Sun, Q – river runoff; T1 – minimum daily air temperature; T2 – mean daily air temperature; T3 – maximum daily air temperature; sum(t � 2, t � 6) is a sum of the dailyaveraged air temperatures measured 2–6 days before day t. The lowest MSE values are shaded.

A.P. Piotrowski et al. / Journal of Hydrology 529 (2015) 302–315 309

case of MLP, PUNN and ANFIS (see Suppl. Tables 9–11) the testswith 8, 9 and 10 inputs are considered. WNN is tested with 18input variables (see Suppl. Table 12 for details).

Fig. 3 shows the performances of kNN approaches with 6–10inputs and numbers of nearest neighbours varying between 1and 100. The best results from kNN with each input variant (basedon the performance on testing data) are given in Table 3.

The overall best architecture of each ANN type is bolded inSuppl. Tables 8–11. Although in Case 3 neural networks are the lar-gest (hence more difficult in training and more prone to overfit-ting), this case turns out advantageous for all approaches(including kNN) but WNN (which performs best using only meantemperatures). Note that Chenard and Caissie (2008) also foundthat mean, minimum and maximum air temperatures are togetherneeded for streamwater temperature prediction at CatamaranBrook, New Brunswick, Canada. Including this variables adds infor-mation on the scale of cooling the stream surface during night andheating during the day, especially at clear air weather in higherlatitudes.

The best MLP architecture is that with 10 inputs; PUNN requireonly 8 inputs, skipping maximum and minimum air temperaturesnoted two days ago. Both ANN types perform best with 6 hiddennodes. However, the best ANFIS architecture is the one with 10inputs but only 3 rules. As far as the WNNs are concerned, verysimple architecture with 10 inputs (that includes only mean tem-perature T2, runoff Q and declination of the Sun S), Coiflet of order3 with decomposition level 3 and 4 hidden nodes outperform theothers (see Suppl. Table 8). The superiority of simple WNN andANFIS architectures over more complex ones may indicate the dif-ficulty in proper calibration of larger neural networks. Moreover, itseems that in case of WNN there are some problems at the upperboundary observed for signals with the finite length. This is espe-cially true for filters with long support that extends beyond theobserved signal length during the filtering. Note, that in time seriesforecasting boundary values are important in the prediction offuture values. It may explain why in case of WNN good perfor-mance on training and validation data that are decomposedtogether do not results in similar performance on testing data,where the new observations are used to compute the wavelettransform.

4.1.2. Suprasl catchmentIn case of Suprasl catchment only the ANN variants that per-

formed best for Biala Tarnowska catchment are considered. Withexceptions of WNNs they include mean, minimum and maximumdaily air temperatures as inputs (discussed as Case 3 for BialaTarnowska River). In case of WNN the variant that among dailyair temperatures uses only the mean is applied. The detailedresults of using various number of hidden nodes, rules and wavelettypes are given in Suppl. Tables 13–16. The simplest tested archi-tectures turns out the best for the Suprasl River when PUNN,ANFIS and WNN are used; only in case of MLP the simplest archi-tecture (8-3-1) performs slightly worse than the best (10-3-1).

Fig. 4 shows the respective predictions obtained from kNNapproach with number of nearest neighbours varying from 1 to100. The best results are obtained for k = 26, but are again muchpoorer (MESTE = 2.63) than the ones obtained from any ANN type.

4.2. Model performance comparison

The comparison of the performance of different models will bemainly based on the results achieved for calibration-independenttesting data.

4.2.1. Biala Tarnowska catchment4.2.1.1. One day ahead predictions. A comparison of result from dif-ferent ANN types (see Table 2) shows that, although the differencesin MSETE are small, MLP and PUNN perform slightly better thanANFIS and WNN for testing data. All ANN types perform much bet-ter than kNN approaches (see Fig. 5). Overall the best predictionsare achieved: (1) according to the median MSETE when MLP isused; (2) according to the mean MSETE when PUNN is applied.The best results achieved by ANFIS are about 5% poorer. WNNand especially kNN are outperformed by other tested models. AsANNs do not use past water temperatures as model inputs, noinformation is given to the model that it over- or underestimatesthe stream temperature, and hence the autocorrelation betweenresiduals is, as could be expected, high and vary between 0.33and 0.77.

Both statistical tests show that almost all pair-wise differencesbetween the results obtained from considered ANN types are

310 A.P. Piotrowski et al. / Journal of Hydrology 529 (2015) 302–315

statistically significant at a = 5% level, the exception is the differ-ence between MLP and PUNN. The respective p-valus, given inSuppl. Table 17, are very small in most cases, hence the differences

Fig. 3. One-lead day forecast for Biala Tarnowska catchment. The performances ofthe k-nearest neighbour approach with 6–10 inputs and numbers of nearestneighbours varying between 1 and 100, achieved for testing data.

Table 3Biala Tarnowska River. Best kNN results achieved for different input variables.

Nr. of inputs Nr. of neighbours MSETR MSEV MSETE Input variables

6 20 2.9323 3.0847 2.6708 S(t, t � 1); Q(t);T2(t, t � 1, sum(t � 2, t � 6))

7 38 2.6691 2.9298 2.4221 S(t, t � 1); Q(t);T2(t, t � 1, sum(t � 2, t � 6)); T3(t)

8 47 2.6042 2.9242 2.3847 S(t, t � 1); Q(t);T2(t, t � 1, sum(t � 2, t � 6));T3(t, t � 1)

9 44 2.6172 2.9091 2.3672 S(t, t � 1); Q(t, t � 1);T2(t, t � 1, sum(t � 2, t � 6));T3(t, t � 1)

10 42 2.4428 2.6893 2.1510 S(t, t � 1); Q(t); T1(t, t � 1);T2(t, t � 1, sum(t � 2, t � 6));T3(t, t � 1)

Fig. 4. One-day forecast for Suprasl catchment. The performances of the k-nearestneighbour approach with 10 inputs and numbers of nearest neighbours varyingbetween 1 and 100.

Fig. 5. The performance of one, three and five lead day forecasts for BialaTarnowska (BT) and Suprasl (S) catchments for testing data. The k-nearestneighbour approach with 10 inputs and numbers of nearest neighbours varyingbetween 1 and 100.

A.P. Piotrowski et al. / Journal of Hydrology 529 (2015) 302–315 311

between the performances of various ANN types may be consid-ered significant even though a number of pair-wise comparisonsis made.

The average annual predicted water temperature (averagedover 100 calibrations of particular ANN type) is for independenttesting data only up to 0.1 �C warmer than the average annualmeasured temperature for the respective period (the biggest differ-ence is noted for WNN). For other data sets the differences aresmaller, and reach up to +0.06 �C for validation and no more than�0.02 �C for training set (interestingly, for training the largest neg-ative bias is noted for PUNN).

Finally, it must be noted that the performance of kNN, althoughclearly inferior to the performance of ANN models, noticeablyimproves when more information on air temperatures is included.

4.2.1.2. Three and five days ahead prediction. To verify if the rankingof ANN types would be similar irrespective of the prediction hori-zon, the 3- and 5-days ahead predictions are performed. To savespace only the architectures chosen as the best ones for eachANN type on 1-day ahead tests (included in Table 2) are comparedand the results are given in Table 4. Like in case of 1-day ahead pre-diction, kNN approach performs poorer than ANN models as maybe noted from Fig. 5. Due to the same reasons as discussed for1-day predictions, autocorrelation coefficients between ANNmodel residuals from consecutive days are often high, and depend-ing on ANN type and particular run, vary from just below 0 to over0.8. One may find that again PUNN and MLP outperform ANFIS andWNN for independent data. The only difference is that PUNN isgenerally slightly better than MLP for 3- and 5-days ahead predic-tions. The large mean MSETE noted for PUNN in case of 3-daysahead predictions is due to the very poor performance of onePUNN model (out of 100) for testing data that affects the mean,but not the median results. Such unfortunate predictions may hap-pen for the specific data when ANNs are used, especially PUNN thathas exponential weights. This should be a warning that using a sin-gle data-based ANN model may occasionally lead to erroneous

Table 4Biala Tarnowska River. The performance of 3- and 5-days ahead streamwater temperatureprediction are considered. The lowest MSE values are shaded.

Table 5Suprasl River. Results achieved by various ANN types. The detailed results showing architec– river runoff; T1 – minimum daily air temperature; T2 – mean daily air temperature; T3 –temperatures measured 2–6 days before day t. The lowest MSE values are shaded.

results, and some kind of ensemble aggregation approach isneeded.

According to both statistical tests performed (see Suppl.Table 17) all pair-wise comparisons show statistically significantdifferences between the performance of ANN types with the excep-tion of MLP and ANFIS pair for 5-days predictions.

The differences between average annual predicted and mea-sured water temperatures are not much larger than the ones notedfor 1-day predictions varying, depending on the data set and ANNtype, from �0.07 �C to +0.15 �C.

4.2.2. Suprasl catchment4.2.2.1. One day ahead prediction. One may find that 1-day aheadstreamwater temperature prediction performances by ANN typesare poorer for Suprasl River than for Biala Tarnowska River (consultTable 5). This was expected due to two main factors: much shorterdata (only 10 years of daily data were available for Suprasl, insteadof 16; testing data cover only 3 years, instead of 6) and much largerdistance between hydrological and meteorological stations. Thisprobably explains, why in most cases the simplest tested architec-tures (see Suppl. Tables 13–16) are chosen for Suprasl River.Although the information on minimum and maximum air temper-atures noted during previous days seems less important than incase of Biala Tarnowska River, MLP (see Table 5) is again the overallbest prediction method.

The statistical significance of almost all differences betweenresults achieved from almost all ANN types is confirmed bypair-wise comparisons (see Suppl. Table 18). Only differencebetween results obtained from PUNN and ANFIS is not significant.

The ranking of models is similar to the one observed for BialaTarnowska River (MLP is followed by PUNN, ANFIS, WNN andkNN); just the difference in the performance of MLP and PUNN islarger.

4.2.2.2. Three to five lead days prediction. For 3- and 5-days aheadstreamwater temperature predictions only the architectures of

prediction. Only the architectures of each ANN type that perform best for 1-day ahead

ture selection for each ANN type is given in the appendix. S – declination of the Sun, Qmaximum daily air temperature; sum(t � 2, t � 6) is a sum of the daily averaged air

Table 6Suprasl River. The performance of 3- and 5-days ahead streamwater temperature prediction. Only the architectures of each ANN type that perform best for 1-day ahead predictionare considered. The lowest MSE values are shaded.

Table 7MSE of the predictions obtained after ANN ensemble aggregation by means ofmedian. Members of each 100-run ensemble are from the same ANN type andarchitecture. Only best architectures of each ANN type are considered (the onesbolded in Supplementary Tables 8–11 and 13–16).

ANN type Training Validation Testing

1-day ahead

Biala TarnowskaWNN 1.2267 1.2100 0.9028MLP 0.9551 1.0961 0.7909ANFIS 0.9551 1.0668 0.8089PUNN 0.9949 1.1156 0.8249

SupraslWNN 1.2447 2.4359 1.8165MLP 1.1465 2.2994 1.5035ANFIS 1.0207 2.0824 1.5563PUNN 1.1940 2.1289 1.5384

3-day ahead

Biala TarnowskaWNN 1.5539 1.8658 3.0282

312 A.P. Piotrowski et al. / Journal of Hydrology 529 (2015) 302–315

each ANN type that were considered the best for 1-lead day predic-tions are tested. The results achieved by kNN approach aredepicted in Fig. 5. From Table 6 one may note the superiority ofMLP over PUNN for testing data. The overall ranking of ANN typesis, contrary to what was observed for the Biala Tarnowska River,exactly the same as in case of 1-lead day predictions. The differ-ences between the results achieved with different ANN types arestatistically significant (see Suppl. Table 18). One may also notethat even if 1- and 3-days ahead streamwater temperature predic-tions achieved for Suprasl River are poorer than the ones noted forBiala Tarnowska River, this is not the case for 5-lead days predic-tions. This is probably an effect of larger variability of streamwatertemperature in the mountainous, than in the lowland river.However, due to the positive trend observed in measured watertemperatures at this catchment, for testing data the annual averagepredicted stream temperature is up to (depending on ANN typeand prediction horizon) 0.55 �C lower than the measured one;the respective differences for validation and training sets are muchlower (up to +0.22 �C and �0.05 �C).

MLP 2.0607 2.5987 2.0826ANFIS 2.0505 2.5637 2.0838PUNN 2.1063 2.5764 2.0823

SupraslWNN 1.5860 2.6685 2.5973MLP 1.9561 3.1551 2.2751ANFIS 1.8585 2.8936 2.2577PUNN 1.9911 2.8570 2.2998

5-day ahead

Biala TarnowskaWNN 2.2904 2.6714 4.2869MLP 3.1790 3.7377 3.2271ANFIS 3.1149 3.7104 3.1852PUNN 3.1851 3.6654 3.1922

SupraslWNN 1.9100 3.3406 3.5772MLP 2.4629 3.7414 2.9133ANFIS 2.4365 3.5232 2.8188PUNN 2.5321 3.5574 2.9581

4.3. Improving results by means of a simple ANN ensemble aggregationapproach

The MSEagg values obtained for 1-, 3- and 5- days aheadstreamwater temperature predictions for Biala Tarnowska andSuprasl Rivers when ensemble aggregation approach is used aresummarized in Table 7. Ensemble aggregation improves theprediction performance for each ANN type, each river and everyprediction horizon considered. In most cases the improvement var-ies between 5% and 10%, sometimes is even higher (e.g. one leadday prediction for Biala Tarnowska River by means of MLP neuralnetworks). One should also note that when ensemble aggregationis used the performances of MLP, PUNN and ANFIS become verysimilar, in some cases (e.g. 3-days ahead predictions for BialaTarnowska River) almost indistinguishable. Finally, with surpriseANFIS, that do not performed so well in comparisons discussedin earlier sections, turns out the best ANN type for 5-days aheadprediction when ensemble aggregation approach is used.

The above discussion shows that various ANN types may per-form equally well for streamwater temperature predictions,depending on the way the comparison is made. Is this result sur-prising? As mentioned in the Introduction, no similar researchhave been done for streamwater temperature modelling so far,hence there is no way to relate such findings to the literature inthe topic area. We may only note that very different opinions onsuperiority of some ANN types over others may be found in the lit-erature aiming at other hydrological problems, however, applica-tions of some ANN types, like PUNN, even if numerous in various

fields of science, are still hard to find in hydrology, hence ourresults have to remain incomparable. For example Wang et al.(2009) suggested that ANFIS outperform MLPs for forecastingmonthly discharge; the superiority of ANFIS over MLP was alsofound for suspended sediment modelling (Kisi et al., 2012). Onthe other hand, Ay and Kisi (2014) showed the superiority ofMLPs over ANFIS in modelling chemical oxygen demand in riversand He et al. (2014), who studied the possibility of river flowforecasting in semi-arid mountainous areas, reached the akin con-clusion to ours, namely that the performance of both models is

A.P. Piotrowski et al. / Journal of Hydrology 529 (2015) 302–315 313

very similar and superiority of one over another may depend onthe number of input variables used. Finally, Elshorbagy et al.(2010) showed that very different artificial intelligence modelsmay perform better or worse depending on to which hydrologicalproblem they are applied, and when some problems may be veryselective, for others little difference in modelling performance bymeans of various techniques may be found. This could be the rea-son why WNNs, that are widely praised in hydrological literature(Nourani et al., 2014), but were according to our best knowledgenever tested on streamwater temperature modelling, fail to showits superiority in this study.

5. Conclusions

In the present paper the performance of four ANN types havebeen compared on streamwater temperature prediction at twocatchments, one mountainous and one lowland, both located inmoderately cold climatic conditions of eastern Poland. It wasshown that, when the mean or median performances of singlemodels are compared, the multi-layer perceptron ANNs are aslightly better choice than product-unit ANNs in most cases;adaptive-network-based fuzzy inference systems and waveletANNs perform poorer. However, any ANN type turns out much bet-ter than the k-nearest neighbour approach.

To properly compare different neural network models, thearchitecture of each of them must be selected separately. Muchattention to the selection of ANN input variables is needed, espe-cially those describing the daily air temperatures. It was found thatthe information on mean, maximum and minimum air tempera-tures from the last 1–2 days is important, and the information onmean air temperature measured during few more recent daysshould also be provided to the model.

The performance of various ANN types for prediction horizonsvarying from 1 to 5 days was verified. Obviously the longer leadtimes the poorer results, but when mean or median performancesof separately applied models are compared, the ranking of ANNtypes do not vary significantly with the prediction horizon.However, in some cases product-units turned out the best ANNtype. Although the differences between the prediction perfor-mances of different ANN types are visually small, they are oftenstatistically significant.

Using a simple ensemble aggregation approach allows reduc-tion of the mean square error by at least few percent, in some casesover 10%. Of similar importance is that even simple ensembleaggregation makes the difference in modelling performance bythree ANN types (except wavelet ANNs) barely recognizable. Forlonger prediction horizons the application of ensemble aggregationmay change the ranking of ANN models, allowingadaptive-network-based fuzzy inference systems to slightly out-perform the others. This shows that the ranking of ANN types forthe streamwater temperature predictions heavily depend on theway the comparison is performed. Such finding may be a warningfor anyone that introduces a novel kind of models.

Acknowledgment

This paper has been financed from the Polish public budget forscience (2013–2015) by MNiSW, grant nr. IP2012 040672.

Appendix A. Supplementary material

Supplementary data associated with this article can be found, inthe online version, at http://dx.doi.org/10.1016/j.jhydrol.2015.07.044.

References

Abrahart, R.J., Anctil, F., Coulibaly, P., Dawson, C.W., Mount, N.J., See, L.M.,Shamseldin, A.Y., Solomatine, D.P., Toth, E., Wilby, R.L., 2012. Two decades ofanarchy? Emerging themes and outstanding challenges for neural network riverforecasting. Prog. Phys. Geogr. 36, 480–513.

Adamowski, J., Karapataki, C., 2010. Comparison of multivariate regression andartificial neural networks for peak urban water-demand forecasting: evaluationof different ANN learning algorithms. J. Hydrol. Eng. 15, 729–743.

Adamowski, J., Chan, H.F., 2011. A wavelet neural network conjunction model forgroundwater level forecasting. J. Hydrol. 407, 28–40.

Adamowski, J., Prokoph, A., 2014. Determining the amplitude and timing of streamflow discontinuities: a cross wavelet analysis approach. Hydrol. Process. 28,2782–2793.

Alexandridis, A.K., Zapranis, A.D., 2013. Wavelet neural networks: a practical guide.Neural Networks 42, 1–27.

Almeida, L.M., Ludermir, T.B., 2010. A multi-objective memetic and hybridmethodology for optimizing the parameters and performance of artificialneural networks. Neurocomputing 73, 1438–1450.

Anderson, W.P., Anderson, J.L., Thaxton, C.S., Babyak, C.M., 2010. Changes in streamtemperatures in response to restoration of groundwater discharge and solarheating in a culverted, urban stream. J. Hydrol. 393, 309–320.

Ay, M., Kisi, O., 2014. Modelling of chemical oxygen demand by using ANNs, ANFISand k-means clustering techniques. J. Hydrol. 511, 279–289.

Blaen, P.J., Hannah, D.M., Brown, L.E., Milner, A.M., 2013. Water temperaturedynamics in High Arctic river basins. Hydrol. Process. 27, 2958–2972.

Boucher, M.A., Laliberte, J.P., Anctil, F., 2010. An experiment on the evolution of anensemble of neural networks for streamflow forecasting. Hydrol. Earth Syst. Sci.14, 603–612.

Bullinaria, J.A., AlYahya, K., 2014. Artificial Bee Colony training of neural networks:comparison with back-propagation. Memetic Comput. 6, 171–182.

Bustillo, V., Moatar, F., Ducharne, A., Thiery, D., Poirel, A., 2014. A multimodelcomparison for assessing water temperatures under changing climateconditions via the equilibrium temperature concept: case study of the MiddleLoire River, France. Hydrol. Process. 28, 1507–1524.

Caissie, D., 2006. The thermal regime of rivers: a review. Freshw. Biol. 51, 1389–1406.

Caissie, D., Satish, M.G., El-Jabi, N., 2007. Predicting water temperatures using adeterministic model: application on Miramichi River catchments (NewBrunswick, Canada). J. Hydrol. 336, 303–315.

Caissie, D., Kurylyk, B.L., ST-Hilarie, A., El-Jabi, N., MacQuarrie, K.T.B., 2014.Streambed temperature dynamics and corresponding heat fluxes in smallstreams experiencing seasonal ice cover. J. Hydrol. 519, 1441–1452.

Chau, K.W., 2006. A review on integration of artificial intelligence into water qualitymodeling. Mar. Pollut. Bull. 52, 726–733.

Chenard, J.F., Caissie, D., 2008. Stream temperature modeling using artificial neuralnetworks: application on Catamaran Brook, New Brunswick, Canada. Hydrol.Process. 22, 3361–3372.

Clarke, R.T., 2008. A critique of present procedures used to compare performance ofrainfall–runoff models. J. Hydrol. 352, 379–387.

Clarke, R.T., 2010. On the (mis)use of statistical methods in hydro-climatologicalresearch. Hydrol. Sci. J. 55 (2), 139–144.

Cole, J.C., Maloney, K.O., Schmid, M., McKenna Jr., J.E., 2014. Developing and testingtemperature models for regulated systems: a case study on the Upper DelawareRiver. J. Hydrol. 519, 588–598.

Corder, G.W., Foreman, D.I., 2014. Nonparametric Statistics: A Step-by-StepApproach. John Wiley & Sons. http://dx.doi.org/10.1002/9781118165881, ISBN978-1118840313.

Coulibaly, P., Anctil, F., Rasmussen, P., Bobee, B., 2000. A recurrent neural networksapproach using indices of low-frequency climatic variability to forecast regionalannual runoff. Hydrol. Process. 14, 2755–2777.

Daigle, A., St-Hilaire, A., Ouellet, V., Corriveau, J., Ouarda, T.B.M.J., Bilodeau, L., 2009.Diagnostic study and modeling of the annual positive water temperature onset.J. Hydrol. 370, 29–38.

Daubechies, I., 1990. The wavelet transform, time-frequency localization and signalanalysis. IEEE Trans. Inf. Theory 36, 961–1005.

DeWeber, J.T., Wagner, T., 2014. A regional neural network ensemble for predictingmean daily river water temperature. J. Hydrol. 517, 187–200.

Dickson, N.E., Carrivick, J.L., Brown, L.E., 2012. Flow regulation alters alpine riverthermal regimes. J. Hydrol. 464, 505–516.

Durbin, R., Rumelhart, D.E., 1989. Product units: a computationally powerful andbiologically plausible extension to backpropagation networks. Neural Comput.1, 133–142.

Elshorbagy, A., Corzo, G., Srinivasulu, S., Solomatine, D.P., 2010. Experimentalinvestigation of the predictive capabilities of data driven modeling techniquesin hydrology – Part 2: application. Hydrol. Earth Syst. Sci. 14, 1943–1961.

Faruk, D.O., 2010. A hybrid neural network and ARIMA model for water quality timeseries prediction. Eng. Appl. Artif. Intell. 23, 586–594.

Ficklin, D.L., Stewart, I.T., Maurer, E.P., 2013. Effects of climate change on streamtemperature, dissolved oxygen, and sediment concentration in the SierraNevada in California. Water Resour. Res. 49, 2765–2782.

Gaume, E., Gosset, R., 2003. Over-parameterisation, a major obstacle to the use ofartificial neural networks in hydrology? Hydrol. Earth Syst. Sci. 7, 693–706.

Geman, G., Bienenstock, E., Doursat, R., 1992. Neural networks and the bias/variancedilemma. Neural Comput. 4, 1–58.

314 A.P. Piotrowski et al. / Journal of Hydrology 529 (2015) 302–315

Granitto, P.M., Verdes, P.F., Ceccatto, H.A., 2005. Neural network ensembles:evaluation of aggregation algorithms. Artif. Intell. 163, 139–162.

Grbic, R., Kurtagic, D., Sliskovic, D., 2013. Stream water temperature predictionbased on Gaussian process regression. Expert Syst. Appl. 40, 7407–7414.

Hadzima-Nyarko, M., Rabi, A., Sperac, M., 2014. Implementation of Artificial NeuralNetworks in modeling the water-air temperature relationship of the RiverDrava. Water Resour. Manage 28, 1379–1394.

Hagan, M.T., Menhaj, M.B., 1994. Training feedforward networks with theMarquardt algorithm. IEEE Trans. Neural Networks 5, 989–993.

Hansen, L.K., Salamon, P., 1990. Neural network ensembles. IEEE Trans. PatternAnal. Mach. Intell. 12, 993–1001.

Haykin, S., 1999. Neural Networks, a Comprehensive Foundation. Macmillan CollegePublishing Co., New York, USA.

He, J.X., Valeo, C., Chu, A., Neumann, N.F., 2011. Prediction of event-basedstormwater runoff quantity and quality by ANNs developed using PMI-basedinput selection. J. Hydrol. 400, 10–23.

He, Z.B., Wen, X.H., Liu, H., Du, J., 2014. A comparative study of artificial neuralnetwork, adaptive neuro fuzzy inference system and support vector machinefor forecasting river flow in the semiarid mountain region. J. Hydrol. 509, 379–386.

Holmstrom, L., Koistinen, P., 1992. Using additive noise in back-propagationtraining. IEEE Trans. Neural Networks 3, 24–38.

Hornik, K., Stinchcombe, M., White, H., 1989. Multilayer feed forward networks areuniversal approximators. Neural Networks 2, 359–366.

Ilonen, J., Kamarainen, J.K., Lampinen, J., 2003. Differential Evolution trainingalgorithm for feed-forward neural networks. Neural Process. Lett. 17, 93–105.

Islam, M.M., Yao, X., Murase, K., 2003. A constructive algorithm for trainingcooperative neural network ensembles. IEEE Trans. Neural Networks 14, 820–834.

Jang, J.S.R., 1993. ANFIS: adaptive-network-based fuzzy inference systems. IEEETrans. Syst. Man Cybern. 23, 665–685.

Janson, D.J., Frenzel, J.F., 1993. Training product unit neural networks with geneticalgorithms. IEEE Expert – Intell. Syst. Appl. 8 (5), 26–33.

Jeong, D.I., Daigle, D., St-Hilaire, A., 2013. Development of a stochastic watertemperature model and projection of future water temperature and extremeevents in the Ouelle river basin in Québec, Canada. River Res. Appl. 29, 805–821.

Kalinowska, M.B., Rowinski, P.M., Kubrak, J., Mirosław-Swiatek, D., 2012. Scenariosof the spread of a waste heat discharge in a river – Vistula River case study. ActaGeophys. 60, 214–231.

Kingston, G.B., Lambert, M.F., Maier, H.R., 2005. Bayesian training of artificial neuralnetworks used for water resources modeling. Water Reour. Res. 41, art. no.W12409.

Kisi, O., Cimen, M., 2011. A wavelet-support vector machine conjunction model formonthly streamflow forecasting. J. Hydrol. 399, 132–140.

Kisi, O., Dailr, A.H., Cimen, M., Shiri, J., 2012. Suspended sediment modeling usinggenetic programming and soft computing techniques. J. Hydrol. 450, 48–58.

Kurylyk, B.L., MacQuarrie, K.T.B., Voss, C.I., 2014. Climate change impacts on thetemperature and magnitude of groundwater discharge from shallow,unconfined aquifers. Water Resour. Res. 50, 3253–3274.

Loinaz, M.C., Davidsen, H.K., Butts, M., Bauer-Gottwein, P., 2013. Integrated flow andtemperature modeling at the catchment scale. J. Hydrol. 495, 238–251.

Long, S.L., Jackson, C.R., 2014. Variation of stream temperature among mesoscalehabitats within stream reaches: southern Appalachians. Hydrol. Process. 28,3041–3052.

MacDonald, R.B., Boon, S., Byrne, J.M., 2014. A process-based stream temperaturemodelling approach for mountain regions. J. Hydrol. 511, 920–931.

Maheswaran, R., Khosa, R., 2014. Wavelet–Volterra coupled model for monthlystream flow forecasting. J. Hydrol. 450, 320–335.

Maier, H.R., Jain, A., Dandy, G.C., Sudheer, K.P., 2010. Methods used for thedevelopment of neural networks for the prediction of water resource variablesin river systems: current status and future directions. Environ. Modell. Softw.25, 891–909.

Mallat, S.G., 1989. A theory for multi resolution signal decomposition: the waveletrepresentation. IEEE Trans. Pattern Anal. Mach. Intell. 11, 674–693.

Mandischer, M., 2002. A comparison of evolution strategies and backpropagationfor neural network training. Neurocomputing 42, 87–117.

Martinez-Estudillo, A., Martinez-Estudillo, F., Hervias-Martinez, C., Garcia-Pedrajas,N., 2006. Evolutionary product unit based neural networks for regression.Neural Networks 19, 477–486.

McKenna, J.E., Butryn, R.S., McDonald, R.P., 2010. Summer stream watertemperature models for Great Lakes streams: New York. Trans. Am. Fish. Soc.139, 1399–1414.

Meier, T.D., 2012. Controls of summer stream temperature in the Pacific Northwest.J. Hydrol. 475, 323–335.

Mendes-Moreira, J., Soares, C., Jorge, A.M., De Sousa, J.F., 2012. Ensemble approachesfor regression: a survey. ACM Comput. Surv. 45 (1). http://dx.doi.org/10.1145/2379776.2379786. Art. No. 10.

Mohseni, O., Stefan, H.G., Erickson, T.R., 1998. A nonlinear regression model forweakly stream temperatures. Water Resour. Res. 34 (10), 2685–2692.

Mohseni, O., Stefan, H.G., 1999. Stream temperature air temperature relationship: aphysical interpretation. J. Hydrol. 218 (3–4), 128–141.

Napiorkowski, M.J., Piotrowski, A.P., Napiorkowski, J.J., 2014. Stream temperatureforecasting by means of ensemble of neural networks: importance of inputvariables and ensemble size. In: Schleiss et al. (Eds.), River Flow 2014. Springer,London, pp. 2017–2025, � 2014, ISBN 978-1-138-02674-2.

Nayak, P.C., Sudheer, K.P., Rangan, D.M., Ramasastri, K.S., 2004. A neuro-fuzzycomputing technique for modeling hydrological time series. J. Hydrol. 291, 52–66.

Nourani, V., Alami, M.T., Aminfar, M.H., 2009. A combined neural-wavelet model forprediction of watershed precipitation, Lighvanchai, Iran. Eng. Appl. Artif. Intell.16, 1–12.

Nourani, V., Baghanam, A.H., Adamowski, J., Kisi, O., 2014. Applications of hybridwavelet–artificial intelligence models in hydrology: a review. J. Hydrol. 514,358–377.

Nowlan, S.J., Hinton, G.E., 1992. Simplifying neural networks by soft weight-sharing.Neural Comput. 4, 473–493.

Orr, H.R., Simpson, G.L., des Clers, S., Watts, G., Hughes, M., Hannaford, J., Dunbar,M.J., Laize, C.L.R., Wilby, R.L., Battarbee, R.W., Evans, R., 2015. Detectingchanging river temperatures in England and Wales. Hydrol. Process. 29 (5),752–766.

Pasti, R., de Castro, L.N., 2009. Bio-inspired and gradient-based algorithms to trainMLPs: the influence of diversity. Inf. Sci. 179, 1441–1453.

Pekarova, P., Halmova, D., Miklanek, P., Onderka, M., 2008. Is the water temperatureof the Danube River at Bratislava, Slovakia, rising? J. Hydrometeorol. 9, 1115–1122.

Piotrowski, A.P., Rowinski, P.M., Napiorkowski, J.J., 2012. Comparison ofevolutionary computation techniques for noise injected neural networktraining to estimate longitudinal dispersion coefficients in rivers. Expert Syst.Appl. 39, 1354–1361.

Piotrowski, A.P., Napiorkowski, J.J., 2012. Product-units neural networks forcatchment runoff forecasting. Adv. Water Resour. 49, 97–113.

Piotrowski, A.P., Napiorkowski, J.J., 2013. A comparison of methods to avoidoverfitting in neural networks training in the case of catchment runoffmodeling. J. Hydrol. 476, 97–111.

Piotrowski, A.P., Osuch, M., Napiorkowski, M.J., Rowinski, P.M., Napiorkowski, J.J.,2014. Comparing large number of metaheuristics for artificial neural networkstraining to predict water temperature in a natural river. Comput. Geosci. 64,136–151.

Piotrowski, A.P., 2014. Differential evolution algorithms applied to neural networktraining suffer from stagnation. Appl. Soft Comput. 21, 382–406.

Poole, G.C., Berman, C.H., 2001. An ecological perspective on in-streamtemperature: natural heat dynamics and mechanisms of human-causedthermal degradation. Environ. Manage. 27, 787–802.

Prechlet, L., 1998. Automatic early stopping using cross-validation: quantifying thecriteria. Neural Networks 11, 761–777.

Rabi, A., Hadzima-Nyarko, M., Sperac, M., in press. Modelling river temperaturefrom air temperature in the River Drava (Croatia). Hydrol. Sci. J. http://dx.doi.org/10.1080/02626667.2014.914215.

Rumelhart, D.E., Hinton, G.E., McClelland, J.L., 1986. A general framework forparallel distributed processing. In: Rumelhart, D.E., McClelland, J.L. (Eds.),Parallel Distributed Processing. MIT Press, Cambridge, MA USA.

Sabouri, F., Gharabaghi, B., Mahboubi, A.A., McBean, E.A., 2013. Impervious surfacesand sewer pipe effects on stormwater runoff temperature. J. Hydrol. 502, 10–17.

Sahoo, G.B., Ray, C., De Carlo, E.H., 2006. Use of neural network to predict flash floodand attendant water qualities of a mountainous stream on Oahu, Hawaii. J.Hydrol. 327, 525–538.

Sahoo, G.B., Schladow, S.G., Reuter, J.E., 2009. Forecasting stream water temperatureusing regression analysis, artificial neural network, and chaotic non-lineardynamic models. J. Hydrol. 378, 325–342.

Schaffer, J.D., 1994. Combinations of genetic algorithms with neural networks orfuzzy systems. In: Zurada, J.M., Marks, R.J., Robinson, C.J. (Eds.), ComputationalIntelligence. Imitating Life. IEEE Press, New York, pp. 371–382.

Schmidt, F.L., 1996. Statistical significance testing and cumulative knowledge inpsychology: implications for training of researchers. Psychol. Methods 1 (2),115–129.

Schmitt, M., 2001. On the complexity of computing and learning with multiplicativeneural networks. Neural Comput. 14, 241–301.

See, L., Abrahart, R.J., 2001. Multi-model data fusion for hydrological forecasting.Comput. Geosci. 27, 987–994.

Shamseldin, A.Y., Nasr, A.E., O’Connor, K.M., 2002. Comparison of different forms ofthe multi-layer feed-forward neural network method used for river flowforecasting. Hydrol. Earth Syst. Sci. 6, 671–684.

Sharma, L., Greskowiak, J., Ray, C., Eckert, P., Prommer, H., 2012. Elucidatingtemperature effects on seasonal variations of biogeochemical turnover ratesduring riverbank filtration. J. Hydrol. 428, 104–115.

Shen, X.M., Sun, T., Liu, F.F., Xu, J., Pang, A., 2015. Aquatic metabolism response tothe hydrologic alteration in the Yellow River estuary, China. J. Hydrol. 525, 42–54.

Shoaib, M., Shamseldin, A.Y., Melville, B.W., 2014. Comparative study of differentwavelet based neural network models for rainfall–runoff modeling. J. Hydrol.515, 47–58.

Sivri, N., Kilic, N., Ucan, O.N., 2007. Estimation of stream temperature in FirtinaCreek (Rize-Turkiye) using artificial neural network model. J. Environ. Biol. 28,67–72.

Socha, K., Blum, C., 2007. An ant colony optimization algorithm for continuousoptimization: application to feedforward neural network training. NeuralComput. Appl. 16, 235–247.

St-Hilaire, A., Ouarda, T.B.M.J., Bargaoui, Z., Daigle, A., Bilodeau, L., 2012. Daily riverwater temperature forecast model with a k-nearest neighbour approach.Hydrol. Process. 26, 1302–1310.

A.P. Piotrowski et al. / Journal of Hydrology 529 (2015) 302–315 315

Takacs K., Kern, Z., in press. Multidecadal changes in the river ice regime of thelower course of the River Drava since AD 1875. J. Hydrol. http://dx.doi.org/10.1016/j.jhydrol.2015.01.040.

Tiwari, M.K., Chatterjee, C., 2010. Development of an accurate and reliable hourlyflood forecasting model using wavelet-bootstrap-ANN (WBANN) hybridapproach. J. Hydrol. 394, 458–470.

van Vliet, M.T.H., Ludwig, F., Zwolsman, J.J.G., Weedon, G.P., Kabat, P., 2011. Globalriver temperatures and sensitivity to atmospheric warming and changes in riverflow. Water Resour. Res. 47, art. no. W02544.

Vecek, N., Mernik, M., Crepinsek, M., 2014. A chess rating system for evolutionaryalgorithms: a new method for the comparison and ranking of evolutionaryalgorithms. Inf. Sci. 277, 656–679.

Vega, M., Pardo, R., Barrado, E., Deban, L., 1998. Assessment of seasonal andpolluting effects on the quality of river water by exploratory data analysis.Water Res. 32, 3581–3592.

Wagner, M.J., Bladon, K.D., Silins, U., Williams, C.H.S., Martens, A.M., Boon, S.,MacDonald, R.J., Stone, M., Emelko, M.B., Anderson, A., 2014. Catchment-scalestream temperature response to land disturbance by wildfire governed bysurface-subsurface energy exchange and atmospheric controls. J. Hydrol. 517,328–338.

Wang, W., Ding, J., 2003. Wavelet network model and its application to theprediction of hydrology. Nat. Sci. 1, 67–71.

Wang, W.C., Chau, K.W., Cheng, C.T., Qiu, L., 2009. A comparison of performance ofseveral artificial intelligence methods for forecasting monthly discharge timeseries. J. Hydrol. 374, 294–306.

Webb, B.W., Hannah, D.M., Moore, R.D., Brown, L.E., Nobilis, F., 2008. Recentadvances in stream and river temperature research. Hydrol. Process. 22, 902–918.

Wehrly, K.E., Brenden, T.O., Wang, L., 2009. A Comparison of statistical approachesfor predicting stream temperatures across heterogeneous landscapes. J. Am.Water Resour. Assoc. 45, 986–997.

Windeatt, T., Zor, C., 2013. Ensemble pruning spectral coefficients. IEEE Trans.Neural Networks Learn. Syst. 24, 673–678.

Wu, C.L., Chau, K.W., Li, Y.S., 2009. Predicting monthly streamflow using data-drivenmodels coupled with data-preprocessing techniques. Water Resour. Res. 45, art.no. W08432.

Xin, Z.H., Kinouchi, T., 2013. Analysis of stream temperature and heat budget in anurban river under strong anthropogenic influences. J. Hydrol. 489, 16–25.

Zhang, G., Patuwo, B.E., Hu, M.Y., 1998. Forecasting with artificial neural networks:the state of the art. Int. J. Forecast. 14, 35–62.

Zhang, B., Govindaraju, R.S., 2000. Prediction of watershed runoff using Bayesianconcepts and modular neural networks. Water Resour. Res. 36, 753–762.

Zhang, C., Wu, W., Chen, X.H., Xiong, Y., 2008. Convergence of BP algorithm forproduct unit neural networks with exponential weights. Neurocomputing 72,513–520.

Zheng, J., 2009. Predicting software reliability with neural network ensembles.Expert Syst. Appl. 36, 2116–2122.

Zhou, Z.H., Jiang, Y., Yang, Y.B., Chen, S.F., 2002. Lung cancer cell identification basedon artificial neural network ensembles. Artif. Intell. Med. 24, 25–36.