review of surrogate modeling in water resourceshomepage.usask.ca/~ser134/publications/razavi et al....
TRANSCRIPT
Review of surrogate modeling in water resources
Saman Razavi,1 Bryan A. Tolson,1 and Donald H. Burn1
Received 19 October 2011; revised 4 June 2012; accepted 12 June 2012; published 31 July 2012.
[1] Surrogate modeling, also called metamodeling, has evolved and been extensively usedover the past decades. A wide variety of methods and tools have been introduced forsurrogate modeling aiming to develop and utilize computationally more efficient surrogatesof high-fidelity models mostly in optimization frameworks. This paper reviews, analyzes,and categorizes research efforts on surrogate modeling and applications with an emphasison the research accomplished in the water resources field. The review analyzes 48references on surrogate modeling arising from water resources and also screens out morethan 100 references from the broader research community. Two broad families of surrogatesnamely response surface surrogates, which are statistical or empirical data-driven modelsemulating the high-fidelity model responses, and lower-fidelity physically based surrogates,which are simplified models of the original system, are detailed in this paper. Taxonomieson surrogate modeling frameworks, practical details, advances, challenges, and limitationsare outlined. Important observations and some guidance for surrogate modeling decisionsare provided along with a list of important future research directions that would benefit thecommon sampling and search (optimization) analyses found in water resources.
Citation: Razavi, S., B. A. Tolson, and D. H. Burn (2012), Review of surrogate modeling in water resources, Water Resour. Res., 48,
W07401, doi:10.1029/2011WR011527.
1. Introduction[2] Computer simulation models, which simulate abstract
representations of physically based systems using mathemat-ical concepts and language, are playing a key role in engi-neering tasks and decision making processes. There arevarious types of problems utilizing computer simulationmodels (for simplicity referred to as simulation models here-after) including prediction, optimization, operational man-agement, design space exploration, sensitivity analysis, anduncertainty analysis. There are also problems such as modelcalibration and model parameter sensitivity analysis dealingwith simulation models to enhance their fidelity to the real-world system. Fidelity in the modeling context refers to thedegree of the realism of a simulation model. Modern simula-tion models tend to be computationally intensive as they rig-orously represent detailed scientific knowledge about thereal-world systems [Keating et al., 2010; Mugunthan et al.,2005; Zhang et al., 2009]. Many model-based engineeringanalyses require running these simulation models thousandsof times and as such demand prohibitively large computa-tional budgets.
[3] Surrogate modeling, which is a second level ofabstraction, is concerned with developing and utilizingcheaper-to-run ‘‘surrogates’’ of the ‘‘original’’ simulation
models. Throughout this paper, the terms ‘‘original func-tions,’’ ‘‘original simulation models,’’ and ‘‘simulationmodels’’ are used interchangeably. A wide variety of surro-gate models have been developed to be intelligently appliedin lieu of simulation models. There are two broad familiesunder the large umbrella of surrogate modeling, responsesurface modeling and lower-fidelity modeling. Responsesurface surrogates employ data-driven function approxima-tion techniques to empirically approximate the modelresponse. Response surface surrogates may also be referredto as ‘‘metamodels’’ [Blanning, 1975; Kleijnen, 2009] as aresponse surface surrogate is a ‘‘model of a model.’’‘‘Model emulation’’ is another term referring to responsesurface surrogate modeling [O’Hagan, 2006]. The term‘‘Proxy models’’ has also been used in the literature to referto response surface surrogates [Bieker et al., 2007]. Unlikeresponse surface surrogates, lower-fidelity surrogates arephysically based simulation models but less-detailed com-pared to original simulation models, which are typicallydeemed to be high-fidelity models; they are simplified sim-ulation models preserving the main body of processes mod-eled in the original simulation model [Forrester et al.,2007; Kennedy and O’Hagan, 2000].
[4] In a surrogate modeling practice (response surfacesurrogates or lower-fidelity physically based surrogates), thegoal is to approximate the response(s) of an original simula-tion model, which is typically computationally intensive,for various values of explanatory variables of interest. Thesurface representing the model response with respect to thevariables of interest (which is typically a nonlinear hyper-plane) is called ‘‘response surface’’ or ‘‘response landscape’’throughout this paper. For the majority of response surfacesurrogate modeling techniques, different response surfacesmust be fit to each model response of interest (or each
1Department of Civil and Environmental Engineering, University ofWaterloo, Waterloo, Ontario, Canada.
Corresponding author: S. Razavi, Department of Civil and Environmen-tal Engineering, University of Waterloo, 200 University Ave. West, Water-loo, ON N2L 3G1, Canada. ([email protected])
©2012. American Geophysical Union. All Rights Reserved.0043-1397/12/2011WR011527
W07401 1 of 32
WATER RESOURCES RESEARCH, VOL. 48, W07401, doi:10.1029/2011WR011527, 2012
function aggregating multiple model responses). The neuralnetwork technique is one exception capable of fitting multi-ple model responses. In contrast, since lower-fidelity surro-gates retain some physically based characteristics of theoriginal model, one lower-fidelity surrogate model is typi-cally capable of approximating multiple model responses ofinterest.
[5] The main motivation of developing surrogate model-ing strategies is to make better use of the available, typi-cally limited, computational budget. Simpson et al. [2008]report that the common theme in six highly cited metamod-eling (or design and analysis of computer experiments)review papers is indeed the high cost of computer simula-tions. Global optimization algorithms based on responsesurface surrogates such as EGO [Jones et al., 1998],GMSRBF and MLMSRBF [Regis and Shoemaker, 2007a],and Gutmann’s method [Gutmann, 2001] and also uncer-tainty analysis algorithms such as ACUARS [Mugunthanand Shoemaker, 2006] and RBF-enabled MCMC [Bliznyuket al., 2008] all have been developed to circumvent thecomputational budget limitations associated with computa-tionally intensive simulation models. In this regard, surro-gate modeling may only be beneficial when the simulationmodel is computationally intensive, justifying the expenseof moving to a second level of abstraction (reduced model fi-delity) which typically leads to reducing the accuracy ofanalyses. Therefore, even though Jones [2001] and Simpsonet al. [2008] both point out that surrogate modeling is morethan simply reducing computation time, reviewing otherpossible motivations for surrogate modeling is beyond thescope of this paper.
[6] Many iterative water resources modeling analysespotentially stand to benefit from surrogate modeling. Bene-fits are only potential because any surrogate-enabled mod-eling analysis provides an approximation to the analysiswith the original model and the error of the analysis resultseems difficult or impossible to assess without repeatingthe exact analysis with the original simulation model. Forexample, there is no guarantee that a model parameterdeemed insensitive on the basis of surrogate modeling analy-sis is truly insensitive in the original simulation model. Anincomplete list of classic or popular iterative modeling anal-yses in water resources, which are candidates for efficiencyenhancement with surrogate modeling, include deterministicmodel parameter optimization (calibration) studies with evo-lutionary algorithms [e.g., Duan et al., 1992; Wang, 1991],uncertainty-based or Bayesian model calibration studies[e.g., Beven and Freer, 2001; Kavetski et al., 2006; Vrugtet al., 2009], management or design optimization with evo-lutionary algorithms [e.g., McKinney and Lin, 1994; Savicand Walters, 1997], multiobjective optimization algorithms[e.g., Cieniawski et al., 1995; Reed et al., 2003], global sen-sitivity analysis methods [e.g., Hornberger and Spear, 1981;Saltelli et al., 2000], and any traditional Monte Carlo–basedreliability or uncertainty analysis [e.g., Melching et al.,1990; Skaggs and Barry, 1997].
[7] This paper aims to review, analyze, and classify theresearch on surrogate modeling with an emphasis on sur-rogate modeling efforts arising from the water resourcesmodeling field. Simpson et al. [2001] and Wang and Shan[2007] also review the literature on response surfacesurrogates for engineering design optimization problems.
Simpson et al. [2004] summarize a discussion panel onresponse surface surrogate modeling held at the 9th AIAA/ISSMO Symposium on Multidisciplinary Analysis andOptimization. Simpson et al. [2008] review the literature onresponse surface modeling and motivations from a histori-cal perspective and also emphasize the appeal of lower-fidelity physically based surrogate modeling. Forrester andKeane [2009] review recent advances in surrogate modelingincluding advances in lower-fidelity physically based surro-gates in the field of optimization. Special journal issues onsurrogate modeling summarize the first and second Interna-tional Workshops on Surrogate Modeling and Space Map-ping for Engineering Optimization (see Bandler andMadsen [2001] and Bandler et al. [2008]). Another specialissue publication on surrogate modeling is a recent thematicjournal issue on surrogate modeling for the reduction andsensitivity analysis of complex environmental models (seeRatto et al. [2012]). In addition, there are more specificreview papers focusing on specific tools/strategies involvedin surrogate modeling. Kleijnen [2009] reviews kriging andits applications for response surface surrogate modeling. Jinet al. [2001] and Chen et al. [2006] review and comparemultiple function approximation models acting as responsesurface surrogates. Jin [2005] focuses on response surfacesurrogate modeling when used with evolutionary optimiza-tion algorithms.
[8] Surrogate modeling has been increasingly more pop-ular over the last decade within the water resources com-munity and this is consistent with the increasing utilizationof metamodels in the scientific literature since 1990 asdocumented by Viana and Haftka [2008]. A research data-base search of formal surrogate modeling terminology inmid-2011 (search of Thomson Reuters (ISI) Web of Knowl-edge) in 50 journals related to surface water and groundwaterhydrology, hydraulic, environmental science and engineering,and water resources planning and management returned110 articles on surrogate modeling in water resources. Webelieve that the actual number of articles on surrogate model-ing in water resources is higher as there are articles not usingthe formal terminology of surrogate modeling and/or not pub-lished in water resources related journals. Forty-eight of theavailable surrogate modeling references published until mid-2011 dealing with water or environmental resources problemswere selected for detailed review based on their relevance,our judgment of their quality and clarity of reporting and thesignificance of surrogate modeling to the contribution of thepublication. The phrase ‘‘water resources literature’’ usedthroughout this paper refers to the 50 water resources journalsand these 48 surrogate modeling references.
1.1. Goals and Outline of Review
[9] Our primary objective (1) is to provide water resourcesmodelers considering surrogate modeling with a more com-plete description of the various surrogate modeling techni-ques found in the water resources literature along with someguidance for the required subjective decisions when utilizingsurrogate models. The depth of the review of the topics cov-ered here generally varies with the popularity of the topic inthe water resources literature and as such, discussion largelyrevolves around optimization applications. Additional morespecific objectives are as follows: (2) describe each of thecomponents involved in surrogate modeling practice as
W07401 RAZAVI ET AL.: REVIEW W07401
2 of 32
depicted in Figure 1; (3) provide a categorization of the dif-ferent surrogate-enabled analysis frameworks (i.e., the differ-ent ways the components in Figure 1 can interact); (4) relateexisting surrogate modeling efforts in the water resources lit-erature with similar efforts in the broader research commu-nity; and (5) identify relevant underutilized ideas forconsideration in future water resources studies.
[10] Figure 1 presents a diagram that shows all the com-ponents involved in the surrogate modeling analysis frame-work and the sections in the paper that are directly related toeach component. Conventional frameworks not involvingsurrogate models, such as different simulation-optimizationframeworks, consist of only the original model and thesearch or sampling algorithm components being directlylinked together. In surrogate-enabled frameworks, however,three new components, design of experiments, response sur-face surrogate, and/or lower-fidelity surrogate, may also beinvolved. These three components, and the frameworkthrough which all the components interact, are of particularinterest in this paper. Such frameworks generally begin witha design of experiments to generate a sample with which totrain or fit a response surface or lower-fidelity surrogatemodel; then the sampler/search algorithm repeatedly runsthe original computationally expensive model and/or the sur-rogate and collects their response. During this metamodel-enabled analysis, the surrogate model can be static ordynamically updated. Any original model evaluation whichis utilized to fit the surrogate model is referred to as a designsite. Section 2 details the elements associated with theresponse surface surrogates and presents their advances, con-siderations, and limitations. Section 3 presents the motiva-tion and different types and frameworks for lower-fidelitysurrogate modeling. Section 4 discusses how the perform-ance of a surrogate-enabled framework should be evaluatedand benchmarked against other alternatives. The paper endswith summary and concluding remarks in section 5.
1.2. Case Study or Problem CharacteristicsInfluencing Surrogate Model Design
[11] The most critical problem characteristics that shouldinfluence surrogate model/technique selection are asfollows:
[12] 1. Model analysis type to be augmented by the sur-rogate model—search or sampling. For the remainder ofthis paper, search analysis is meant to refer to optimization(management, calibration, single-objective or multiobjec-tive) or uncertainty-based/Bayesian model calibration pro-cedure while all other modeling analyses are referred to assampling analyses.
[13] 2. Computational budget constraints. This refers tohow many original model evaluations can be utilized tobuild the surrogate model and ultimately perform the modelanalysis of interest. In applications where a surrogatemodel is to be repeatedly utilized after it is initially con-structed (i.e., optimize real-time operational decisions), thetime available for each utilization can be critical.
[14] 3. Dimensionality of the problem. In general, as thenumber of explanatory variables increases, surrogate mod-eling becomes less advantageous and even infeasible.
[15] 4. Single-output versus multioutput surrogates. Thisis a key distinction in the context of environmental simula-tion modeling, where model outputs are typically variablein both time and space. Single-output surrogates are com-mon where the original model output of interest is a func-tion calculated from a large number of model outputs (i.e.,calibration error metric).
[16] 5. Exact emulation versus inexact emulation. Inother words, should the surrogate model exactly predict theoriginal model result at all design sites?
[17] 6. Availability of original model developers/experts.Some surrogate modeling techniques require these experts(lower fidelity modeling), and Gorissen [2007] notes thatthey can provide valuable insight into the significance ofsurrogate modeling errors relative to original model errors.
[18] Although not an original problem characteristic, theavailability of surrogate modeling software and expertsalso has an impact on surrogate model design. The afore-mentioned surrogate modeling reviews and the literature ingeneral do not precisely map all problem characteristics tospecific or appropriate types of surrogate models. We alsodo not attempt this and instead only make periodic observa-tions and judgments as to when certain types of surrogatemodeling techniques might be more or less appropriate
Figure 1. Diagram of the elements involved in surrogate (metamodel) enabled analysis framework.
W07401 RAZAVI ET AL.: REVIEW W07401
3 of 32
than others. Properly considering the case study specificfactors above is the first key to avoid implementing a poorsurrogate modeling technique.
2. Response Surface Surrogates[19] Response surface surrogate modeling as a research
field arising from various disciplines has been in existencefor more than six decades and has become very active sincethe beginning of 1990s [Simpson et al., 2008]. The firstgeneration of response surface surrogates, initiated by Boxand Wilson [1951], relied heavily on polynomials (typicallysecond-order) and have been the basis of the so-calledresponse surface methodology (RSM). Response surfacesurrogates do not emulate any internal component of originalsimulation models; instead they approximate the relation-ships between several explanatory variables, typically thesimulation model parameters and/or variables affectingmodel inputs, and one or more model response variables. Inother words, a response surface surrogate is an approxima-tion or a model of the ‘‘original’’ response surface defined ina problem domain—response surface surrogates are meta-models of original models. The terms ‘‘metamodel’’ and‘‘response surface surrogate’’ are used interchangeablythroughout this paper. Notably, the existence of the responsesurface concept dates back to well before formalizing mod-ern response surface approaches as, for example, the tradi-tional nonlinear local optimization techniques based on theTaylor series expansion (i.e., different variations of New-ton’s method) use simple approximations. In such methods,a response surface surrogate, typically a variation of polyno-mials, is locally fitted on the (single) current best solutionthrough the use of first- and/or second-order derivative infor-mation of the original function, unlike the formalizedresponse surface approaches in which surrogates are fittedon multiple design sites usually regardless of derivatives.Trust-region methods are another example family of tradi-tional local optimization strategies based on the responsesurface concept, as they iteratively approximate a certainregion (the so-called trust region) of the original functiontypically using polynomials.
[20] Research and advances in response surface surro-gate modeling can be classified into three main categories :(1) identification and development of experimental designsfor effective approximation, (2) developing and applyingfunction approximation techniques as surrogates, and (3)framework development utilizing surrogates. The researchefforts in the water resources community tend to focus onthe second and third categories. In the following, the litera-ture on response surface surrogate modeling for waterresources applications is reviewed in section 2.1. Then sec-tions 2.2–2.6 further detail this review in relation with theabove three categories and outline the advances, considera-tions, and limitations.
2.1. Response Surface Surrogates in Water ResourcesLiterature
[21] Response surface surrogate modeling has beenwidely applied in various water and environmental model-ing problems for decades. Table 1 summarizes 32 studiesutilizing response surface surrogate models in water resour-ces problems published since 2000. Although this table
does not cover all surrogate modeling studies over that pe-riod, we believe that it provides readers with an adequatecoverage of the subject area. Note that not all these studieshave been published in water resources related journals.According to Table 1, about 45% of the surrogate modelingstudies focus on automatic model calibration. Most surrogate-enabled auto-calibration studies involve surrogates inconjunction with optimization algorithms; in four studies,surrogates have been used with uncertainty analysis algo-rithms for the purpose of model calibration (i.e., GLUE inthe work of Khu and Werner [2003] and Zhang et al. [2009],ACUARS in the work of Mugunthan and Shoemaker [2006],and Markov Chain Monte Carlo in the work of Bliznyuket al. [2008]). Schultz et al. [2004, 2006] and Borgonovoet al. [2012] also use surrogates to speed up Monte Carlosampling studies. Ten studies of the surrogate model applica-tions listed in Table 1 are groundwater optimization prob-lems, mostly groundwater remediation. Four surrogatemodeling studies are in the context of water distribution sys-tem design and optimization. Borgonovo et al. [2012] is theonly study in Table 1 using surrogate modeling for sensitivityanalysis and interested readers are thus referred to Blatmanand Sudret [2010], Ratto et al. [2007], and Storlie et al.[2009] for metamodel-enabled sensitivity analysis examplesfrom the broader research community. Five studies, Lionget al. [2001], Bau and Mayer [2006], Behzadian et al.[2009], di Pierro et al. [2009], and Castelletti et al. [2010],use response surface surrogates in multiobjective optimiza-tion settings. In addition, most studies fit the response surfacesurrogates on continuous explanatory variables. Five studies[i.e., Bau and Mayer, 2006; Behzadian et al., 2009; Broadet al., 2005; Broad et al., 2010; Castelletti et al., 2010] applysurrogates to integer optimization problems (response surfacesurrogates are fitted on discrete variables), and Yan andMinsker [2006, 2011] and Hemker et al. [2008] apply surro-gate modeling in mixed-integer optimization settings. Notethat problems with discrete or mixed variables require specialconsiderations and have been solved largely on an ad hoc ba-sis [Simpson et al., 2004]. For mixed-integer optimizationproblems, Hemker et al. [2008] utilize response surface sur-rogates within a branch-and-bound optimization frameworksuch that the surrogate model is employed when solving therelaxed optimization subproblems (integer variables allowedto assume noninteger values). Shrestha et al. [2009] use neu-ral networks as a surrogate to completely replace computa-tionally intensive Monte Carlo sampling experiments. Intheir approach, neural networks are used to emulate the pre-dictive uncertainty (i.e., 90% prediction intervals) of a hydro-logic model. The surrogate in this study does not follow thegeneral strategy of response surface modeling where surro-gates map model parameters or other problem variables tothe system response, and instead their neural network modelmaps observed rainfall and runoff in the preceding time inter-vals to the model predictive uncertainty in the next time inter-val. In sections 2.2–2.6, and also section 4, we refer back tothe studies listed in Table 1 to elaborate the details related tothe subject of each section.
2.2. Design of Experiments
[22] Deterministic simulation systems like computersimulations can be very complex involving many variableswith complicated interrelationships. Design of Experiments
W07401 RAZAVI ET AL.: REVIEW W07401
4 of 32
Tab
le1.
Sum
mar
yof
Clo
sely
Rev
iew
edM
etam
odel
ing
App
lica
tion
sin
the
Wat
erR
esou
rces
Lit
erat
ure
Wor
kby
Typ
eof
Pro
blem
Typ
eof
Met
amod
elT
ype
ofS
earc
hor
Sam
plin
gA
lgor
ithm
Typ
eof
Fra
mew
ork
Num
ber
and
Typ
eof
Exp
lana
tory
Var
iabl
esC
ompu
tati
onal
Sav
ing
John
son
and
Rog
ers
[200
0]G
roun
dwat
erre
med
iati
onL
inea
rR
egre
ssio
nan
dN
eura
lne
twor
k(s
ingl
ehi
dden
laye
r)
Sim
ulat
edA
nnea
ling
Bas
icse
quen
tial
fram
ewor
k30
cont
inuo
usN
otre
port
ed
Lio
nget
al.[
2001
]A
utom
atic
cali
brat
ion
ofth
eH
ydro
Wor
ksw
ater
-sh
edm
odel
(Bi-
obje
ctiv
e)
Neu
ral
netw
ork
(thr
eehi
d-de
nla
yers
)A
ccel
erat
edco
nver
genc
eG
A(A
CG
A)
Bas
icse
quen
tial
fram
ewor
k8
cont
inuo
usN
otre
port
ed
Khu
and
Wer
ner
[200
3]U
ncer
tain
tyba
sed
Aut
o-m
atic
cali
brat
ion
ofth
eS
WM
Mra
infa
ll-r
unof
fm
odel
Neu
ral
netw
ork
(sin
gle
hidd
enla
yer)
GL
UE
Bas
icse
quen
tial
fram
ewor
k(I
nste
adof
form
alD
oE,
GA
wit
hni
chin
gw
asus
edto
gene
rate
desi
gnsi
tes)
8co
ntin
uous
80%
innu
mbe
rof
full
eval
uati
ons
Schu
ltz
etal
.[20
04,
2006
]U
ncer
tain
tyan
alys
isof
aw
ater
qual
ity
mod
elfo
rre
gula
tory
deci
sion
supp
ort
Lin
ear
Reg
ress
ion,
poly
no-
mia
l,an
dpr
oces
s-in
spir
edsi
mpl
efu
ncti
ons
Mon
teC
arlo
Sim
ulat
ion
Bas
icse
quen
tial
fram
ewor
k9
cont
inuo
usN
otre
port
ed
Khu
etal
.[20
04]
Aut
omat
icca
libr
atio
nof
MIK
E11
/NA
MR
adia
lba
sis
func
tion
sG
enet
ical
gori
thm
sM
etam
odel
-em
bedd
edev
olut
ion
fram
ewor
k,N
ofo
rmal
DoE
9co
ntin
uous
60%
innu
mbe
rof
full
eval
uati
ons
Reg
isan
dSh
oem
aker
[200
4]G
roun
dwat
erbi
orem
edia
-ti
onop
tim
izat
ion
and
17te
stfu
ncti
ons
Rad
ial
basi
sfu
ncti
ons
and
seco
nd-o
rder
poly
nom
ial
(bot
hlo
cal)
(�,�
)-e
volu
tion
stra
tegy
Met
amod
el-e
mbe
dded
evol
utio
nfr
amew
ork
12co
ntin
uous
Not
repo
rted
Ost
feld
and
Salo
mon
s[2
005]
Aut
omat
icca
libr
atio
nof
CE
-QU
AL
-W2
wat
erqu
alit
ym
odel
k-ne
ares
tne
ighb
ors
(kN
N)
Gen
etic
algo
rith
ms
Met
amod
el-e
mbe
dded
evol
utio
nfr
amew
ork,
No
form
alD
oE
3co
ntin
uous
Not
repo
rted
Mug
unth
anet
al.
[200
5]A
utom
atic
cali
brat
ion
ofa
grou
ndw
ater
bior
emed
ia-
tion
mod
el
Rad
ial
basi
sfu
ncti
ons
Gau
ssia
nra
ndom
sam
pler
App
roxi
mat
ion
unce
rtai
nty
base
dfr
amew
ork
8co
ntin
uous
Not
repo
rted
Bro
adel
al.[
2005
]W
ater
dist
ribu
tion
syst
emde
sign
opti
miz
atio
nN
eura
lne
twor
k(s
ingl
ehi
dden
laye
r)G
enet
ical
gori
thm
sB
asic
sequ
enti
alfr
amew
ork
22di
scre
te21
%of
CP
Uti
me
Mug
unth
anan
dSh
oem
aker
[200
6]A
utom
atic
cali
brat
ion
and
para
met
erun
cert
aint
yan
alys
isof
grou
ndw
ater
mod
els
Rad
ial
basi
sfu
ncti
ons
Gau
ssia
nra
ndom
sam
pler
App
roxi
mat
ion
unce
rtai
nty
base
dfr
amew
ork
3an
d7
cont
inuo
usM
ore
than
87%
innu
mbe
rof
full
eval
uati
onsa
Bau
and
May
er[2
006]
Pum
p-an
d-tr
eat
opti
miz
a-ti
on(B
i-ob
ject
ive)
Kri
ging
Com
plet
een
umer
atio
nA
ppro
xim
atio
nun
cert
aint
yba
sed
fram
ewor
k4
disc
rete
Not
repo
rted
Yan
and
Min
sker
[200
6]G
roun
dwat
erre
med
iati
onde
sign
–M
OD
FL
OW
and
RT
3Dm
odel
sus
edfo
rfl
owfi
eld
and
cont
ami-
nant
conc
entr
atio
n
Neu
ral
netw
ork
(sin
gle
hidd
enla
yer)
Gen
etic
algo
rith
ms
Met
amod
el-e
mbe
dded
evol
utio
nfr
amew
ork,
No
form
alD
oE
9an
d28
mix
edin
tege
r85
-90%
innu
mbe
rof
full
eval
uati
ons
Zou
etal
.[20
07]
Aut
omat
icca
libr
atio
nof
the
WA
SP
wat
erqu
alit
ym
odel
Neu
ral
netw
ork
(sin
gle
hidd
enla
yer)
Gen
etic
algo
rith
ms
Bas
icse
quen
tial
and
adap
t-iv
e-re
curs
ive
fram
ewor
ks19
cont
inuo
us97
%of
CP
Uti
me
Reg
isan
dSh
oem
aker
[200
7a]
Gro
undw
ater
bior
emed
ia-
tion
opti
miz
atio
nan
d17
test
func
tion
s
Rad
ial
basi
sfu
ncti
ons
Gau
ssia
nan
dun
ifor
mra
ndom
sam
pler
sA
ppro
xim
atio
nun
cert
aint
yba
sed
fram
ewor
kT
est
func
tion
s:2-
14G
roun
dwat
erpr
oble
m:
12co
ntin
uous
Not
repo
rted
Shoe
mak
eret
al.
[200
7]A
utom
atic
cali
brat
ion
oftw
oS
WA
Tw
ater
shed
mod
els
Rad
ial
basi
sfu
ncti
ons
(�,�
)-ev
olut
ion
stra
tegy
Met
amod
el-e
mbe
dded
evol
utio
nfr
amew
ork,
No
form
alD
oE
8an
d14
cont
inuo
usN
otre
port
ed
W07401 RAZAVI ET AL.: REVIEW W07401
5 of 32
Tab
le1.
(con
tinu
ed)
Wor
kby
Typ
eof
Pro
blem
Typ
eof
Met
amod
elT
ype
ofS
earc
hor
Sam
plin
gA
lgor
ithm
Typ
eof
Fra
mew
ork
Num
ber
and
Typ
eof
Exp
lana
tory
Var
iabl
esC
ompu
tati
onal
Sav
ing
Hem
ker
etal
.[20
08]
Wel
lfi
eld
desi
gnop
tim
iza-
tion
and
hydr
auli
cca
ptur
epr
oble
m
Kri
ging
Bra
nch-
and-
boun
don
mai
npr
oble
m�
Seq
uent
ial
quad
rati
cpr
ogra
mm
ing
onsu
bpro
blem
s
Ada
ptiv
e-re
curs
ive
fram
ewor
k5þ
inte
ger
wit
h15þ
cont
inuo
usN
otre
port
ed
Bli
znyu
ket
al.[
2008
]A
utom
atic
cali
brat
ion
and
Bay
esia
nun
cert
aint
yan
alys
isof
anen
viro
n-m
enta
lm
odel
Rad
ial
basi
sfu
ncti
ons
Mar
kov-
Cha
inM
onte
Car
loS
ampl
erB
asic
sequ
enti
alfr
amew
ork
4co
ntin
uous
Mor
eth
an90
%in
num
ber
offu
llev
alua
tion
s
Kou
rako
san
dM
anto
-gl
ou[2
009]
Coa
stal
aqui
fer
pum
ping
man
agem
ent
Neu
ral
netw
ork
(mod
ular
)si
ngle
hidd
enla
yer
Evo
luti
onar
yA
nnea
ling
Sim
plex
Sch
eme
opti
miz
atio
n
Met
amod
el-e
mbe
dded
evol
utio
nfr
amew
ork,
No
form
alD
oE
34co
ntin
uous
(For
met
a-m
odel
ing,
disa
ggre
gate
din
tosm
alle
rsu
bset
sha
ving
mem
bers
wit
hne
glig
ible
corr
elat
ions
wit
hm
embe
rsof
othe
rsu
bset
s)
95%
ofC
PU
tim
e
Fen
etal
.[20
09]
Cos
tan
dco
ntam
inan
tre
-m
oval
opti
miz
atio
nfo
rso
ilva
por
extr
acti
onsy
stem
desi
gn
Sec
ond-
orde
rpo
lyno
mia
lsan
dex
pone
ntia
lfu
ncti
ons
Gen
etic
algo
rith
ms
Ada
ptiv
e-re
curs
ive
fram
ewor
k6
and
9co
ntin
uous
40-9
4%of
tota
lC
PU
tim
eac
ross
4ca
sest
udie
sa
Beh
zadi
anet
al.
[200
9]W
ater
dist
ribu
tion
syst
emm
onit
orin
glo
cati
ons
opti
-m
izat
ion
(Bi-
obje
ctiv
e)
Neu
ral
netw
ork
(sin
gle
hidd
enla
yer)
NS
GA
-II
Met
amod
el-e
mbe
dded
evol
utio
nfr
amew
ork,
No
DoE
15an
d50
disc
rete
87%
and
96%
ofC
PU
tim
e(�
34%
ofto
tal
savi
ngis
due
toca
chin
g)Z
hang
etal
.[20
09]
Unc
erta
inty
-bas
edau
to-
mat
icca
libr
atio
nof
SW
AT
mod
els
Sup
port
vect
orm
achi
nes
(SV
Ms)
and
Neu
ral
net-
wor
k(s
ingl
ehi
dden
laye
r)
GL
UE
(was
used
only
wit
hS
VM
s,A
NN
was
used
for
bech
mar
ikin
gS
VM
s)
Bas
icse
quen
tial
fram
ewor
k6,
9,12
,and
16co
ntin
uous
20-4
2%of
tota
lC
PU
acro
ss4
case
stud
ies
Zou
etal
.[20
09]
Aut
omat
icca
libr
atio
nof
the
WA
SP
wat
erqu
alit
ym
odel
Neu
ral
netw
ork
(sin
gle
hidd
enla
yer)
Gen
etic
algo
rith
ms
Ada
ptiv
e-re
curs
ive
fram
ewor
k19
cont
inuo
usN
otre
port
ed
Reg
isan
dSh
oem
aker
[200
9]G
roun
dwat
erbi
orem
edia
-ti
onop
tim
izat
ion
prob
-le
ms,
auto
mat
icca
libr
atio
nof
agr
ound
-w
ater
bior
emed
iati
onm
odel
,and
20te
stfu
ncti
ons
Rad
ial
basi
sfu
ncti
ons
Gau
ssia
nan
dun
ifor
mra
ndom
sam
pler
sA
ppro
xim
atio
nun
cert
aint
yba
sed
fram
ewor
kT
estf
unct
ions
:2-
6G
roun
d-w
ater
opti
miz
atio
npr
ob-
lem
s:12
Aut
omat
icca
libr
atio
n:
6co
ntin
uous
Rou
ghly
50-9
7%in
num
ber
offu
llev
alua
tion
sa
Shre
stha
etal
.[20
09]
Pre
dict
ive
unce
rtai
nty
esti
mat
ion
ofa
hydr
olog
icm
odel
(HB
V)
Neu
ral
netw
ork
(sin
gle
hidd
enla
yer)
N/A
(see
sect
ion
2.1)
N/A
(see
sect
ion
2.1)
N/A
(see
sect
ion
2.1)
Not
repo
rted
diP
ierr
oet
al.[
2009
]W
ater
dist
ribu
tion
syst
emde
sign
opti
miz
atio
n(b
i-ob
ject
ive)
Kri
ging
Gen
etic
algo
rith
ms
App
roxi
mat
ion
unce
rtai
nty
base
d34
and
632
(con
tinu
ous)
N/A
Infe
rior
perf
orm
ance
Bro
adet
al.[
2010
]W
ater
dist
ribu
tion
syst
emde
sign
opti
miz
atio
nN
eura
lne
twor
k(s
ingl
ehi
dden
laye
r)G
enet
ical
gori
thm
sB
asic
sequ
enti
alfr
amew
ork
49di
scre
te98
%of
CP
Uti
me
Cas
tell
etti
etal
.[2
010]
Wat
erqu
alit
yre
habi
lita
tion
inre
serv
oirs
(mul
tiob
jec-
tive
opti
miz
atio
n)
Rad
ial
basi
sfu
ncti
ons,
n-di
men
sion
alli
near
inte
r-po
lato
r,an
din
vers
edi
s-ta
nce
wei
ghte
d
Com
plet
een
umer
atio
nA
dapt
ive-
recu
rsiv
efr
amew
ork
3di
scre
teN
otre
port
ed
W07401 RAZAVI ET AL.: REVIEW W07401
6 of 32
(also referred to as DoEs) employ different space fillingstrategies to empirically capture the behavior of the under-lying system over limited ranges of the variables. As a pri-ori knowledge about the underlying (original) responsesurface is usually unavailable, DoEs tend to assume uni-formity in distributing the commonly called ‘‘design sites,’’which are the points in the explanatory (input) variablespace evaluated through the original simulation model.Most metamodel-enabled optimizers start with DoEs as in23 out of 32 studies listed in Table 1. There are a wide vari-ety of DoE methods available in the literature; however,full factorial design (e.g., used for metamodeling in thework of Gutmann [2001]), fractional factorial design, cen-tral composite design [Montgomery, 2008], Latin hyper-cube sampling (LHS) [McKay et al., 1979], and symmetricLatin hypercube sampling (SLHS) [Ye et al., 2000] appearto be the most commonly used DoE methods. Full and frac-tional factorial designs and central composite design aredeterministic and typically more applicable when the num-ber of design variables is not large. For example, the size ofthe initial DoE sampled by the full factorial design in a 10dimensional space with only two levels would be 1024(¼210), which may be deemed extremely large and beyondthe computational budget. Latin hypercube sampling andsymmetric Latin hypercube sampling both involve randomprocedures and can easily scale to different numbers ofdesign variables. Research efforts on the metamodeling-inspired DoEs mostly focus on determining the optimaltype [e.g., Alam et al., 2004] and size of existing DoEs(i.e., number of initial design sites, e.g., Sobester et al.[2005]) for a specific problem or to develop new and effec-tive DoEs [e.g., Ye et al., 2000].
2.3. Function Approximation Models
[23] Response surface surrogates approximate theresponse surface of computationally intensive models (i.e.,original models) by fitting over a set of previously eval-uated design sites. A variety of approximation techniqueshave been developed and applied as surrogates. Examplesof such techniques include: polynomials [Fen et al., 2009;Hussain et al., 2002; Myers and Montgomery, 2002;Wang, 2003], kriging, which is also sometimes referred toas design and analysis of computer experiment (DACE) inthe surrogate modeling context [Sacks et al., 1989; Sakataet al., 2003; Simpson and Mistree, 2001], k nearest neigh-bors (kNN) [Ostfeld and Salomons, 2005], artificial neuralnetworks (ANNs) [Behzadian et al., 2009; Khu andWerner, 2003; Papadrakakis et al., 1998], radial basisfunctions (RBFs) [Hussain et al., 2002; Mugunthan et al.,2005; Mullur and Messac, 2006; Nakayama et al., 2002;Regis and Shoemaker, 2007a], support vector machines(SVMs) [Zhang et al., 2009], multivariate adaptive regres-sion splines (MARS) [Friedman, 1991; Jin et al., 2001],high-dimensional model representation [Rabitz et al.,1999; Ratto et al., 2007; Sobol, 2003], treed Gaussianprocesses [Gramacy and Lee, 2008], Gaussian emulatormachines [Kennedy and O’Hagan, 2001; O’Hagan, 2006],smoothing splines ANOVA models [Gu, 2002; Ratto andPagano, 2010; Storlie et al., 2011; Wahba, 1990], andproper orthogonal decomposition [Audouze et al., 2009]. Inthe RBF references above, the RBFs are not associatedwith ANNs, whereas in some publications [e.g., Jin, 2005;T
able
1.(c
onti
nued
)
Wor
kby
Typ
eof
Pro
blem
Typ
eof
Met
amod
elT
ype
ofS
earc
hor
Sam
plin
gA
lgor
ithm
Typ
eof
Fra
mew
ork
Num
ber
and
Typ
eof
Exp
lana
tory
Var
iabl
esC
ompu
tati
onal
Sav
ing
Yan
and
Min
sker
[201
1]R
elia
bili
ty-b
ased
grou
nd-
wat
erre
med
iati
onde
sign
–M
OD
FL
OW
and
RT
3Dm
odel
sus
edfo
rfl
owfi
eld
and
cont
amin
ant
conc
entr
atio
n
Thr
eeN
eura
lne
twor
ks(s
in-
gle
hidd
enla
yer)
Noi
syG
enet
ical
gori
thm
Met
amod
el-e
mbe
dded
evo-
luti
onfr
amew
ork,
No
for-
mal
DoE
28m
ixed
inte
ger
86-9
0%in
num
ber
offu
llev
alua
tion
s(t
his
figu
rein
clud
essa
ving
due
toca
chin
g)
Sree
kant
han
dD
atta
[201
1]C
oast
alaq
uife
rpu
mpi
ngm
anag
emen
tE
nsem
ble
ofge
neti
cpr
o-gr
amm
ing-
base
dm
odel
sN
SG
A-I
IB
asic
sequ
enti
alfr
amew
ork
33co
ntin
uous
Not
repo
rted
Raz
avi
etal
.[20
12]
Aut
omat
icca
libr
atio
nof
aS
WA
Tm
odel
and
agr
ound
wat
erm
odel
and
4te
stfu
ncti
ons
Kri
ging
,rad
ial
basi
sfu
nc-
tion
,and
Neu
ral
netw
orks
(sin
gle
hidd
enla
yer)
Gen
etic
algo
rith
ms
and
Gau
ssia
nra
ndom
sam
pler
Ada
ptiv
e-re
curs
ive
and
appr
oxim
atio
nun
cert
aint
yba
sed
fram
ewor
ks
7,10
,14,
and
15co
ntin
uous
Not
repo
rted
Bor
gono
voet
al.
[201
2]S
ensi
tivi
tyan
alys
isof
anen
viro
nmen
tal
nucl
ear
was
tem
odel
and
thre
ete
stfu
ncti
ons
Sm
ooth
ing
spli
neA
NO
VA
and
krig
ing
Mon
teC
arlo
Sim
ulat
ion
Bas
icse
quen
tial
fram
ewor
kT
est
func
tion
s:2,
3,an
d3
Env
iron
men
tal
prob
lem
:12
cont
inuo
us
96%
ofC
PU
tim
e
a Com
puta
tion
alsa
ving
isno
tex
plic
itly
men
tion
edin
the
pape
r,an
dth
isva
lue
isin
terp
rete
dba
sed
onth
eav
aila
ble
info
rmat
ion.
W07401 RAZAVI ET AL.: REVIEW W07401
7 of 32
Khu et al., 2004], as well as in the MATLAB neural net-work toolbox [Beale et al., 2010], RBFs are considered as atype of feed-forward artificial neural networks.
[24] In the recent water resources related response sur-face surrogate modeling literature, ANNs and RBFs are themost commonly used function approximation techniquesas, among all surrogate modeling studies listed in Table 1,14 and 10 of the 32 studies have applied ANNs and RBFs,respectively. In addition, according to Table 1, polynomialshave been used in five studies, and in two of these [i.e.,Johnson and Rogers, 2000; Regis and Shoemaker, 2004],polynomials were used to highlight their weakness in com-parison with other more promising alternatives. Five stud-ies in Table 1 have employed kriging. Each of SVMs, kNN,and smoothing spline ANOVA has been used in only onestudy. Moreover, the approximation models mostly act asglobal surrogates of underlying functions, which representthe original models over the entire input range. However,in some studies [e.g., Regis and Shoemaker, 2004; Wanget al., 2004], the approximation model is fitted locally onlyover a specified (limited) number of design sites (a subsetof all available design sites), which are in close vicinity ofthe point of interest in the explanatory variable space.
[25] The information used to fit the response surface sur-rogates are typically the response values of the originalfunction (i.e., original simulation model) at the designsites; however, there are studies aiming to include the sen-sitivity (gradient information) of the original function withrespect to the explanatory variables (i.e., derivative values)to enhance the approximation accuracy and form the so-called ‘‘gradient-enhanced response surface surrogates.’’Examples include Kim et al. [2005] and van Keulen andVervenne [2004] for gradient-enhanced polynomials andLiu [2003] for gradient-enhanced kriging and gradient-enhanced neural networks. In practice, such methods haveserious limitations as in most of the problems thatresponse surface surrogates are applied, the derivatives arenot readily available and have to be calculated using nu-merical methods requiring the evaluation of the originalfunction at extra points. The extra computational burdenimposed can become prohibitively large when the numberof dimensions in the explanatory variable space is morethan a few, whereas this extra computation could be savedto evaluate the original function at new more intelligentlysampled points.
[26] Selection of an appropriate function approximationtechnique for a given surrogate-enabled analysis requirescareful consideration. There are significant differences inthe logic inspiring the development of different techniquesand also in their level of practicality for a particular prob-lem. In the following, some practical and technical detailsare presented on five common function approximation tech-niques used as response surface surrogates of computersimulation models (polynomials, RBFs, kriging, SVMs,and ANNs), all of which have been used for surrogate mod-eling in the water resources literature. SVMs were selectedfor detailed review in addition to the four most commontechniques in Table 1 because Viana and Haftka [2008]include them as one of the four main classes of techniquesin their surrogate modeling review paper. The basic infor-mation and formulations of these techniques were notincluded as they are available elsewhere.
2.3.1. Polynomials and Simple Functions[27] Polynomials have the simplest type of parameters
(i.e., coefficients in a polynomial regression), which areobjectively determined usually through the least squareregression method. Second-order polynomial functions,which are the most popular polynomials used as responsesurface surrogates, have (D þ 1)(D þ 2)/2 parameterswhere D is the number of explanatory variables (dimensionof the input space). First- and second-order polynomialshave had very successful applications in local nonlinear pro-gramming optimization algorithms (e.g., different variationsof gradient-descent and Newton/quasi-Newton methods)where, for example, a second-order polynomial is used toemulate a local mode in the original response landscape.However, the use of polynomials as global surrogates maybe only plausible when the original response landscape is,or is reasonably close to, unimodal, which is not often thecase in many water resources related problems.
[28] Application of higher-order polynomials (third-order or more, which are common in curve-fitting) is typi-cally infeasible when D is greater than only a few variables.This is largely because specifying a proper polynomialform for a particular problem may become very challeng-ing. Second, the number of polynomial parameters to betuned (and therefore the minimum number of design sitesrequired, see also section 2.6.3) becomes excessively large.The inferior performance of polynomials compared to otherfunction approximation techniques mostly due to havingfairly inflexible prespecified forms and being inexact emu-lators (see section 2.6.2) has been acknowledged in severalstudies [e.g., Hussain et al., 2002; Regis and Shoemaker,2004; Simpson and Mistree, 2001]. Note that when theform of the underlying function is similar to a polynomialand when this form is known a priori (e.g., in the work ofFen et al. [2009] polynomials had been reportedly success-ful for the problem of interest) polynomials can be one ofthe best options to choose for surrogate modeling.
[29] Other simple functional forms such as exponentialfunctions [Aly and Peralta, 1999; Fen et al., 2009] fitted byleast square methods may also be applied for response sur-face modeling. Schultz et al. [2004] and Schultz et al. [2006]develop ‘‘reduced-form models’’ based on components of theactual process equations in an original model and fit them todesign sites sampled from the original model. They point outthat when the specified functional forms ‘‘are informed bythe mechanics’’ of the original model, the reduced-formmodels demonstrate better predictive (generalization) ability.Notably, what is called a reduced-form model in these publi-cations is different from the lower-fidelity physically basedsurrogates outlined in section 3.
2.3.2. Radial Basis Functions[30] Radial basis function (RBF) models consist of a
weighted summation of typically n (sometimes fewer) ra-dial basis functions (also called correlation functions) and apolynomial (usually zero- or first-order) where n is thenumber of design sites. There are different forms of basisfunctions including Gaussian, thinplate spline, and multi-quadric. Some forms (e.g., Gaussian) have parametersspecifying the sensitivity/spread of the basis function overthe input domain, while some (e.g., thinplate spline) havefixed sensitivity (no parameter in the basis function)
W07401 RAZAVI ET AL.: REVIEW W07401
8 of 32
regardless of the scale (unit) and importance or sensitivityof each input variable. To address the scale problem, all thedata are typically normalized to the unit interval. Moreover,in RBF models, the sensitivity of the basis functions in allD directions is typically assumed identical (only one singleparameter, if utilized, in all dimensions for all basis func-tions) treating all variables as equally important, althoughsuch an assumption is often not true. An RBF model isdefined by the weights of the basis functions, the coeffi-cients of the polynomial used, and the basis function pa-rameter if it exists. Weights of the basis functions as wellas the polynomial coefficients can be objectively deter-mined by efficient least squares techniques; however, thebasis function parameter is usually specified arbitrarily orby trial and error. The minimum number of design sitesrequired to fit an RBF model is the number of coefficientsof the polynomial used to augment the RBF approximation.
2.3.3. Kriging[31] Application of kriging in the context of design and
analysis of computer experiments (DACE) was first formal-ized by Sacks et al. [1989] and since then has been fre-quently called DACE in some publications [Hemker et al.,2008; Ryu et al., 2002; Simpson et al., 2001]. More recentliterature uses DACE to refer to the suite of all metamodel/emulation techniques [e.g., Ratto et al., 2012; Simpsonet al., 2008]. Similar to RBF models, the kriging model isalso a combination of a polynomial model, which is aglobal function over the entire input space, and a localizeddeviation model (correlation model consisting of basisfunctions) based on spatial correlation of samples. The spe-cial feature of kriging (main difference from RBF models)is that kriging treats the deterministic response of a com-puter model as a realization of a stochastic process, therebyproviding a statistical basis for fitting. This capability ena-bles kriging to provide the user with an approximation ofuncertainty associated with the expected value predicted bykriging at any given point. Approximation of uncertainty isthe basis of the so-called ‘‘approximation uncertainty basedframework’’ for surrogate-enabled analyses, described insection 2.4.4. Note that such approximation of uncertaintymay be available (heuristically or directly) for other func-tion approximation models (see section 2.4.4).
[32] As opposed to RBF models, the correlation parame-ters (sensitivities) in kriging are typically different alongdifferent directions in the input space (D different valuesfor correlation functions in a D-dimensional space), result-ing in higher model flexibility. All the kriging parameters,including the correlation parameters, can be determinedobjectively using the maximum likelihood estimation meth-odology. Like RBF models, the minimum number of designsites needed to fit a kriging model is the number of coeffi-cients in the polynomial augmenting the approximation.Kriging users only need to specify the lower and upperbounds on the correlation parameters, although the appro-priate bounds are sometimes hard to specify [Kleijnen,2009]. The kriging correlation parameters can be interpretedto some extent in that large values for a dimension indicatea highly nonlinear function in that dimension, while smallvalues indicate a smooth function with limited variation.Moreover, Jones et al. [1998] pointed out that the correla-tion matrix in kriging may become ill-conditioned (nearly
singular) toward the end of an optimization run as then theoptimization algorithm tends to sample points near the pre-viously evaluated points (design sites); this ill-conditioningis usually manageable.
2.3.4. Support Vector Machines[33] Support vector machines (SVMs) are a relatively
new set of learning methods designed for both regressionand classification. Although SVMs to some extent rely onutilizing the concept of basis (correlation) functions as usedin RBFs and kriging (especially when using Gaussian ker-nel function), unlike RBFs and kriging, they only involve asubset of design sites lying outside an "-insensitive tubearound the regression model response, referred to as sup-port vectors, to form an approximation. When fitting theSVMs on data, the "-insensitive tube is formed to ignoreerrors that are within a certain distance of the true values.This capability enables SVMs to directly control andreduce the sensitivities to noise (very suitable for inexactemulation, see section 2.6.2). The other special feature ofSVMs is that in the SVM formulation, there is a termdirectly emphasizing the regularization (smoothness) of thefitted model. There are two specific SVM parameters whichare associated with the radius of the "-insensitive tube andthe weight of the regularization term. The Kernel functionused within SVM (e.g., Gaussian function) may also have aparameter to be determined; this parameter acts like thecorrelation parameters in RBF and kriging models andadjusts the sensitivity/spread of the Kernel function withrespect to explanatory variables. Users only require dealingwith these two or three SVM parameters, and once theirvalues are available, the dual form of the SVM formulationcan be efficiently solved using quadratic programming todetermine all the other SVM formulation parameters. SVMscan better handle larger numbers of design sites as the opera-tor associated with design site vectors in the SVM formula-tion is dot product [Yu et al., 2006]. The two aforementionedspecific parameters are mutually dependent (changing onemay influence the effect of the other) and usually determinedthrough a trial-and-error process or optimization. Cherkasskyand Ma [2004] present some practical guidelines to deter-mine these parameters. Optimal SVM parameter values aredifficult to interpret and relate to characteristics of theresponse surface.
2.3.5. Neural Networks[34] Feedforward artificial neural networks (ANNs) are
highly flexible tools commonly used for function approxi-mation. ANNs in this paper refer to multilayer perceptrons(MLPs), which are by far the most popular type of neuralnetworks [Maier et al., 2010]. Development and applica-tion of ANNs involve multiple subjective decisions to bemade by the user. Determination of the optimal structure ofANNs for a particular problem is probably the most impor-tant step in the design of ANN-based surrogates. ANNstructural parameters/decisions include number of hiddenlayers, number of neurons in each hidden layer, and thetype of transfer functions. Various methodologies havebeen developed to determine appropriate ANN structuresfor a given problem, including the methods based on thegrowing or pruning strategies [Reed, 1993] such as themethods presented in the work of Teoh et al. [2006] and Xuet al. [2006], methods based on the network geometrical
W07401 RAZAVI ET AL.: REVIEW W07401
9 of 32
interpretation such as Xiang et al.’s [2005] method, andmethods based on the Bayesian approaches such as themethods in the work of Kingston et al. [2008] and Vilaet al. [2000]. However, these methods, each of which mayresult in an appropriate but different structure for a givenproblem, typically require extensive numerical analyses onthe training data as they generally attempt to test differentnetwork structures in systematic ways. Despite these meth-odologies, trial-and-error is the approach to determine thenumber of hidden neurons in all ANN-based surrogatemodeling studies listed in Table 1. Considering alternativetypes of neural network architectures beyond MLP (such asgeneralized regression neural network (GRNN) [Maieret al., 2010]) may provide another way to reduce or elimi-nate subjective ANN building decisions.
[35] ANNs with one sigmoidal hidden layer and the lin-ear output layer have been proven capable of approximat-ing any function with any desired accuracy provided thatassociated conditions are satisfied [Hornik et al., 1989;Leshno et al., 1993]. Although one hidden layer is adequateto enable neural networks to approximate any given func-tion, some researchers argue that neural networks withmore than one hidden layer may require fewer hidden neu-rons to approximate the same function. It is theoreticallyshown in the work of Tamura and Tateishi [1997] that tobe an exact emulator (interpolating approximator, see alsosection 2.6.2), neural networks with two hidden layersrequire considerably fewer hidden neurons compared toneural networks with one hidden layer. However, develop-ing an exact emulator is not usually the objective of neuralnetwork practitioners (except when used as response sur-face surrogates of deterministic computer simulation mod-els) as the data used are usually noise-prone and thenumber of input-target sets is typically large. The need forexact emulation is still a case study specific determination.In addition, to be an exact emulator, excessively large ANNstructures are required, whereas such networks would morelikely fail in terms of generalization (perform poorly atunsampled regions of input space). Section 2.6.2 deals withexact emulation versus inexact emulation. From a morepractical perspective, it is shown in the work of de Villiersand Barnard [1993] through extensive numerical experi-ments that single-hidden-layer neural networks are superiorto networks with more than one hidden layer with the samelevel of complexity mainly due to the fact that the latter aremore prone to fall into poor local minima in training.
[36] Neural network practitioners tend to use single-hidden-layer neural networks as, for example, 13 of 14 neu-ral network applications in response surface surrogatemodeling listed in Table 1 have used feed-forward neuralnetworks with only one hidden layer; Liong et al. [2001] isthe only study in Table 1 using more than one hidden layer.Single-hidden-layer neural networks form the approximationby combining m sigmoidal units (i.e., sigmoidal lines,planes, or hyperplanes in the 1-, 2-, or 3-and-more-dimen-sional problem space) where m is the number of hidden neu-rons. The number of parameters (weights and biases) of thesingle-hidden-layer neural networks is m � (2 þ D) þ 1where D is the dimension of input space (i.e., number ofinput variables of the response surface surrogate). The opti-mal number of hidden neurons, m, is a function of shape andcomplexity of the underlying function [Xiang et al., 2005] as
well as the training data availability [Razavi et al., 2012]. Inthe response surface surrogate modeling context, the form ofthe original function is often unclear, therefore, the numberof data points available (i.e., design sites) for training, p, isthe main factor involved in determining m. It is usually pre-ferred that the number of ANN parameters be less (or muchless) than p as discussed in the work of Maier and Dandy[2000], although mathematically, there is no limitation whenthe number of parameters is higher than p. A possible strat-egy is to enlarge m dynamically as more design sites becomeavailable. Generally, for a specific problem, there are multi-ple appropriate ANN structures, and for each structure, thereare many appropriate sets of network weights and biases.The term ‘‘appropriate’’ here refers to the network structuresand weight and bias values that can satisfactorily representthe training data. Neural network training (adjusting the net-work parameters) can be a time-consuming optimizationprocess depending on the ANN structure and trainingmethod selected (see for example discussion on the work ofKourakos and Mantoglou [2009] in section 2.6.5). Second-order variations of backpropagation algorithms (i.e., quasi-Newton algorithm such as Levenberg-Marquardt) are themost computationally efficient ANN training methods [Hammet al., 2007] the role of each weight and bias in forming thenetwork response is typically unclear.
2.4. Metamodel-Enabled Analysis Frameworks
[37] Research efforts on metamodeling arising from waterresources modeling are mainly focused on framework devel-opment for utilizing metamodels. Metamodel-enabled analy-sis frameworks in the literature can be categorized underfour main general frameworks. The basic sequential frame-work and adaptive-recursive framework outlined in sections2.4.1 and 2.4.2 are multipurpose frameworks (conceivablyapplicable in all sampling or search analyses), while themetamodel-embedded evolution framework in sections 2.4.3is clearly limited to search analyses dependent on evolution-ary optimization algorithms. The approximation uncertaintybased framework outlined in section 2.4.4 is primarily usedfor search analyses but may also be applicable in some sam-pling studies.
2.4.1. Basic Sequential Framework[38] Basic sequential framework (also called off-line) is
the simplest metamodel-enabled analysis framework con-sisting of three main steps. It starts (Step 1) with design ofexperiment (DoE) through which a prespecified number ofdesign sites over the feasible space are sampled and theircorresponding objective function values are evaluatedthrough the original function. In Step 2, a metamodel isglobally fitted on the design set. Then in Step 3, the metamo-del is fully substituted for the original model in performingthe analysis of interest. In this step, a search or samplingalgorithm is typically conducted on the metamodel. Theresult obtained from the metamodel is assumed to be theresult of the same analysis with the original model; for exam-ple, in metamodel-enabled optimizers with the basic sequen-tial framework, the optimal point found on the metamodel istypically evaluated by the original function and is consideredas the optimal (or near optimal) solution to the original func-tion. In some studies on optimization with metamodelingsuch as Broad et al. [2005] and Broad et al. [2010], at Step 3,
W07401 RAZAVI ET AL.: REVIEW W07401
10 of 32
extra promising points on the metamodel found on the con-vergence trajectory of the optimization algorithm may also beevaluated by the original function. The size of DoE in Step 1of this ‘‘off-line’’ framework is large compared to the size ofinitial DoEs in more advanced ‘‘online’’ metamodeling frame-works since almost the entire computational budget allocatedto solve the problem is spent on Step 1 in the basic sequentialframework. In this regard, the other frameworks (see sections2.4.2–2.4.4) may be called ‘‘online’’ as they frequently updatethe metamodel when new data become available.
[39] According to Table 1, eleven out of 32 recent meta-modeling studies use the basic sequential framework. Asthe sampler in Step 3, Khu and Werner [2003] and Zhanget al. [2009] use the GLUE uncertainty-based calibrationalgorithm, Schultz et al. [2004, 2006] use traditional MonteCarlo simulation for uncertainty analysis, Borgonovo et al.[2012] use Monte Carlo simulation for sensitivity analysis,and Bliznyuk et al. [2008] use a Markov Chain Monte Carlo(MCMC) sampler for uncertainty-based calibration. In allthe other studies with the basic sequential framework, dif-ferent optimization algorithms are used in Step 3. In thework of Liong et al. [2001] and Khu and Werner [2003],instead of fitting the metamodel over the design sitesobtained through a formal DoE, which tends to assume uni-formity, an optimization trial is conducted on the originalfunction and the set of points evaluated over the conver-gence trajectory is used for metamodel fitting (see section2.5 for discussion on whether an initial DoE is required).Instead of having a conventional DoE at Step 1, Bliznyuket al. [2008] first locate a high posterior density region ofthe explanatory variable space by direct optimization onthe original function and then fit a metamodel on the ap-proximate high posterior region (local) rather than theentire space (global). Zou et al. [2007] contrast the basic se-quential framework with the adaptive-recursive frameworkdescribed in section 2.4.2.
[40] Although widely used, the basic sequential frame-work has potential failure modes arising from the fact thatthe metamodel, especially when developed off-line, is notnecessarily a reasonably accurate representation of theoriginal model in the regions of interest in the explanatoryvariable space. For example, in the optimization context,the global optimum of the metamodel found in Step 3 (i.e.,the final solution returned by the framework) is veryunlikely to be a local optimum of the original function. Inother words, there is no guarantee that the returned solutionis even located close to a stationary point of the originalfunction. Figure 2 demonstrates a simple case in which thebasic sequential framework fails and returns a solutionclose to a local maximum of the original function in a mini-mization problem. The metamodel in Figure 2 is a tunedkriging model rather than a conceptual example. The origi-nal function used in Figure 2 (and also used in Figures 3–7)was generated in this review to demonstrate the behavior ofdifferent surrogate modeling strategies. When the originalfunction is simple (e.g., unimodal functions), the probabil-ity of such failures is minimal.
2.4.2. Adaptive-Recursive Framework[41] Like the basic sequential framework, in Step 1, the
adaptive-recursive framework starts with a DoE to design theinitial set of design sites. In Step 2, a global/local metamodel
is fitted on the set of design sites. In Step 3, a search or sam-pling algorithm is employed on the metamodel to identifythe regions of interest in the explanatory variable space andscreen out one or multiple points. When used for optimiza-tion, an optimization algorithm is typically used to find thenear-optimal point (or multiple high-quality points) on themetamodel. The point(s) in the explanatory variable spaceobtained in Step 3, are evaluated by the original function andadded to the set of design sites to update the metamodel.Step 2 and Step 3 are subsequently repeated many times toadaptively evolve the metamodel until convergence or stop-ping criteria are met. When the adaptive-recursive frame-work is used for optimization, the best point the frameworkfinds (available in the final set of design sites) is consideredas the optimal (or near-optimal) solution to the originalfunction.
[42] Six out of the 32 studies listed in Table 1 apply pro-cedures lying under the adaptive-recursive framework. Fivestudies utilize formal optimization algorithms in Step 3, butCastelletti et al. [2010] conduct a complete enumeration atthis step since their combinatorial optimization problem hasonly three decision variables. Johnson and Rogers [2000]point out that the quality of solutions obtained through ametamodel-enabled optimizer is mostly controlled by meta-modeling performance, not the search technique applied onthe metamodel.
[43] The adaptive-recursive framework attempts toaddress the drawbacks of the basic sequential framework[Zou et al., 2007]. When used for optimization purposes,however, the adaptive-recursive framework is helpful atbest for local optimization as reported by Jones [2001] whoidentified some possible cases where the adaptive-recursiveframework may even fail to return a stationary point on theoriginal function. Figure 3 which depicts possible behav-iors of the adaptive-recursive framework, was partiallyinspired by the discussion in the work of Jones [2001]. Asshown in Figure 3a, if the optimal point on the metamodel(found in Step 3) is located at a previously evaluated pointon the original function (i.e., a point already existing in theset of design sites used in Step 2), the algorithm would stallas reevaluating and adding this point to the set of design
Figure 2. A failure mode of basic sequential frameworkfor response surface surrogate modeling where the mini-mizer of surrogate function very poorly approximates thelocal or global minimum of the original function.
W07401 RAZAVI ET AL.: REVIEW W07401
11 of 32
sites would not change the metamodel (may also causemathematical problems in some function approximationmethods such as kriging). Another similar failure mode thatis less likely to occur is when the optimal point on the meta-model (found in Step 3) has the same objective functionvalue in both the metamodel and the original function
(see Figure 3a but assume there was no design site at thesurrogate minimizer, which is on a nonstationary point ofthe original function); evaluating and adding this new pointto the set of design sites would not have any effect on theupdated metamodel. Although it seems very unlikely thatStep 3 returns a point exactly the same as the pointsdescribed in Figure 3a, returning new points lying in theirclose vicinity may also result in very little change in theupdated metamodel and lead to the same problems. Someprocedure is required in this framework to address thesealgorithm stalls and revive the search (one example proce-dure is in the work of Hemker et al. [2008]). Jones [2001]notes that one way to mitigate these algorithm stalls is tomatch the gradient of the response surface surrogate withthe gradient of the original function at the sampled points(at least on the sample point at which the search stalls).However, as explained in section 2.3, the application of the‘‘gradient-enhanced response surface surrogates’’ is nontri-vial with practical limitations.
[44] The adaptive-recursive framework when used foroptimization can easily miss the global optimum of theoriginal function [Gutmann, 2001; Jones, 2001; Schonlau,1997; Sobester et al., 2005]. Although significance ofmissing the global optimum depends on a number of fac-tors, Figure 3b depicts a case where the framework hasfound a local optimum, but as the metamodel is mislead-ingly inaccurate in the neighborhood of the global opti-mum, Step 3 would never return a point in the globalregion of attraction to be evaluated by the original model.The adaptive-recursive framework with gradient-enhancedresponse surface surrogates (see section 2.3) may onlyguarantee convergence to a stationary point on the originalfunction, which may be a global/local optimum, plateau, orsaddle point [Jones, 2001]. Figure 3c shows a case wherethe framework has converged to a point on a plateau on theoriginal function at which the gradient of the metamodelmatches the gradient of the original function. Notably, thesurrogate functions in Figures 3a–3c represent the actualresponse of a kriging model developed for the purpose ofthis review and tuned for the given sets of design sites.
2.4.3. Metamodel-Embedded Evolution Framework[45] The metamodel-embedded evolution framework is a
popular framework for optimization. Among the 32 studieson metamodeling listed in Table 1, eight studies utilizealgorithms lying under this framework. The metamodel-embedded evolution framework shares some characteristicswith the adaptive-recursive framework but with significantdifferences including: the metamodel-embedded evolutionframework is inherently designed to be used with evolu-tionary (i.e., population-based) optimization algorithms, noformal DoE is typically involved at the beginning (exceptvery few studies [e.g., see Regis and Shoemaker, 2004]),and decision criteria through which candidate solutions areselected for evaluation by the original function are differ-ent. In the metamodel-embedded evolution framework, apopulation-based optimization algorithm is employed andrun initially on the original function for a few generations.All the individuals evaluated by the original function in thecourse of the first generations are then used as design sitesfor metamodel fitting. In the following generations, individ-uals are selectively evaluated by either the metamodel or
Figure 3. Failure modes of adaptive-recursive frameworkfor response surface surrogate modeling: (a) minimizer ofsurrogate function is located at a previously evaluatedpoint, (b) minimizer of surrogate function is a local opti-mum of original function far from the global optimum assurrogate function is misleadingly inaccurate in the vicinityof global region of attraction, and (c) minimizer of surro-gate function is located at a plateau of original function.
W07401 RAZAVI ET AL.: REVIEW W07401
12 of 32
the original model. The metamodel is usually updated(refitted) a couple of times in the course of optimization asmore design sites become available. Jin et al. [2002] adoptthe metamodel-embedded evolution framework and formal-ize the concept of evolution control with two approaches:controlled individuals through which a subset of theindividuals in the population at each generation (i.e., thebest � individuals or � randomly selected individuals)are evaluated by the original function and controlled gener-ations through which all the individuals in the population,but at selected generations, are evaluated by the originalfunction. The parameters of metamodel-embedded evolu-tion framework, such as � and the frequency of metamodelupdating, can be adaptively changed in the course of optimi-zation depending on the accuracy of the metamodel.
[46] When an evolutionary algorithm is enabled withmetamodels through the metamodel-embedded evolutionframework, it typically exhibits less consistent behavior com-pared to when the same evolutionary algorithm is used with-out metamodeling [Jin et al., 2002]. This degradation instability is expected as such an algorithm switches betweentwo different response landscapes (the original function andthe metamodel) while searching. A metamodel-enabledoptimizer with this framework should satisfy at least two nec-essary conditions to become a global optimizer: (1) the evo-lutionary algorithm used should be a global optimizer, and(2) any solution (i.e., any individual in any generation)regardless of the approximate function value obtained by themetamodel should have the chance to be selected for evalua-tion through the original function. Thus, like the optimizerswith the adaptive-recursive framework, metamodel-enabledoptimizers with evolution control strategies that only evaluatebest individuals (best in terms of approximate function val-ues) through the original function are at best local optimizers.In such cases, the failure modes explained in Figure 3 for theadaptive-recursive framework (see section 2.4.2) may alsoapply to the metamodel-embedded evolution framework.Convergence properties of metamodel-embedded evolutionframework with other evolution control strategies couldlikely overcome these failure modes.
[47] Some variations of the Learnable Evolution Model(LEM) [Michalski, 2000] might also fall under the metamo-del-embedded evolution framework. LEM is an attempt toimprove upon the basic and inefficient Darwinian evolutionoperators by using machine learning tools. Unlike the con-ventional metamodeling practice which typically producesa continuous approximate response surface, LEM employsclassifying techniques (e.g., decision tree learners) to dis-criminate promising/nonpromising solutions based on thesearch history. The classifier in LEM can involve domain-specific knowledge for rule induction. Jourdan et al.[2006] develop a multiobjective version of LEM, calledLEMMO (LEM for multiobjective), for water distributionnetwork design problems. LEMMO embeds a decision treeclassifier within the NSGAII multiobjective optimizationalgorithm; the solutions generated by the evolution opera-tors in NSGAII are first evaluated by the classifier and thenmodified if needed. di Pierro et al. [2009] compareLEMMO with ParEGO which is a multiobjective metamo-del-enabled optimization algorithm (see section 2.5.6) onwater distribution network design problems.
2.4.4. Approximation Uncertainty Based Framework[48] The approximation uncertainty based framework
may be deemed as an extension to the adaptive-recursiveframework designed for optimization. As explained in sec-tion 2.4.2, the adaptive-recursive framework relies solelyon the approximate values from the response surface surro-gate in the course of optimization. As the adaptive-recursive framework assumes these approximate values astrue, it may easily miss the main region of attraction wherethe global optimum lies. This behavior is illustrated insection 2.4.2 and also reported in the work of Gutmann[2001], Jones [2001], Schonlau [1997], and Sobester et al.[2005]. Addressing this shortcoming, the approximationuncertainty based framework considers the uncertaintiesassociated with the approximation. In this framework, theapproximation value resulting from the response surface sur-rogate is deemed as approximation expected value and thena measure is utilized to quantify the associated approxima-tion uncertainty. Such a measure is explicitly available insome function approximation techniques, e.g., in polyno-mials, kriging, Gaussian radial basis function models[Sobester et al., 2005], and smoothing spline ANOVA [Gu,2002]. To provide the approximation uncertainty, these tech-niques assume that the deterministic response of a simulationmodel is a realization of a stochastic process. Unlike thesetechniques, which provide some statistical basis for approxi-mation uncertainty, Regis and Shoemaker [2007a] propose adistance-based metric (i.e., the minimum distance from pre-viously evaluated design sites) as a heuristic measure of theapproximation uncertainty applicable to any given responsesurface surrogate. Other deterministic function approxima-tion techniques may also provide measures of uncertaintywhen trained with Bayesian approaches; for example,Bayesian neural networks [Kingston et al., 2005] providethe variance of prediction. Bayesian learning typicallyinvolves a Markov Chain Monte Carlo procedure, which ismore computationally demanding than conventional learn-ing methods.
[49] The approximation uncertainty based frameworkconsists of the three steps outlined in section 2.4.2 with amajor difference in Step 3 being, instead of optimizing theapproximate response surface (i.e., the surface formed byapproximation expected values) as in the adaptive-recur-sive framework, it optimizes a new surface function that isdefined to emphasize the existence of approximation uncer-tainty. Different ways have been proposed in the literatureto define such a surface function, each of which emphasizesthe approximation uncertainty to a different extent. In thisregard, the adaptive-recursive framework, in which the sur-face function totally ignores the approximation uncertainty,can be deemed an extreme case that typically yields a localoptimum. The other extreme is to build and maximize asurface function solely representing a measure of approxi-mation uncertainty (e.g., approximation variance availablein kriging) over the explanatory variable space. In theapproximation uncertainty based framework with such asurface function, Step 3 returns the point where the approxi-mation uncertainty is the highest, and as such, subsequentlyrepeating Step 2 and Step 3 in the framework would evolvea globally more accurate metamodel on the basis of a well-distributed set of design sites. Although globally convergent
W07401 RAZAVI ET AL.: REVIEW W07401
13 of 32
under some mild assumptions [Sobester et al., 2005], theframework solely using the approximation uncertaintywould require impractically large number of original func-tion evaluations, especially when the number of decisionvariables is more than a few.
[50] An effective uncertainty based surface function to beoptimized in Step 3 combines the approximation expectedvalue and the associated approximation uncertainty in a waythat balances exploration (i.e., searching unexplored areas ofhigh approximation uncertainty) and exploitation (i.e., fine-tuning a good quality solution by reducing the attention toapproximation uncertainty) during the search. Most commonmethods to define the uncertainty based surface functionsassume that the hypothetical stochastic process is normalwith the expected value y and standard deviation � generatedby the surrogate model for any given point in the explana-tory variable space, x. Figure 4a depicts the information typi-cally available from a response surface surrogate that iscapable of producing approximation uncertainty—this plot
is the outcome of an actual experiment with kriging. Thereare two popular approaches to make use of such informationin Step 3 of the approximation uncertainty based frame-work: (1) maximizing a new surface function representingthe probability of improvement and (2) maximizing theso-called ‘‘expected improvement’’ surface function [Schonlau,1997].
[51] Figures 4a–4c illustrate the concept of probability ofimprovement during optimization based on a real experi-ment with a tuned kriging model. As can be seen, fmin is thecurrent best solution found so far (the design site with theminimum original function value), and as such, any func-tion value which lies below fmin is an improvement. Thus,at any given point x in the explanatory variable space, apossible improvement, I, and the probability of improve-ment, PI, over the current best are given by:
I ¼fmin � Y ðxÞ if Y < fmin
0 otherwise;
((1)
PI ¼ �fmin � yðxÞ�ðxÞ
� �; (2)
where Y(x) is a possible function value at the point xbeing a random number following NðyðxÞ; �ðxÞ2Þ; and �is the normal standard cumulative distribution function.Notably, if the probability of improvement over the cur-rent best is used as the surface function in the framework,the search will be highly local around the current best so-lution unless there is a point on the response surface surro-gate having an estimated expected value, y, less than fmin.To address this drawback, a desired target improvement,T, which is smaller than fmin, is assumed and then theprobability that the original function value is equal orsmaller than T would be:
PI ¼ �T � yðxÞ�ðxÞ
� �: (3)
[52] The higher the desired improvement is (the smallerthe value of T is), the more global the search is. As such,when desired improvement is assumed very small, thesearch would be very local typically around the currentbest until the standard error of the surrogate in that localarea becomes very small. Very large desired improvements(T values much smaller than fmin) may force the algorithmto search excessively global resulting in very slow conver-gence. As Jones [2001] also points out, the performance ofthe algorithm based on the probability of improvement ishighly sensitive to the choice of desired target, T, and deter-mining the appropriate value for a given problem is not triv-ial. To diminish the effect of this sensitivity, Jones [2001]presents a heuristic way to implement the probability ofimprovement approach based on multiple desired target val-ues. Further details on the probability of improvementapproach are available in the work of Jones [2001], Sasenaet al. [2002], and Watson and Barnes [1995].
[53] The so-called expected improvement approach mightbe considered as a more advanced extension to the probabil-ity of improvement approach. Expected improvement is ameasure that statistically quantifies how much improvement
Figure 4. (a) Probabilistic output of a response surfacesurrogate capable of producing approximation uncertaintyassuming the deterministic response of simulation modelfollows a normal stochastic process, (b) approximationuncertainty function (AU or �), (c) function representingprobability of improvement (PI), and (d) function repre-senting expected improvement (EI).
W07401 RAZAVI ET AL.: REVIEW W07401
14 of 32
is expected if a given point is sampled to evaluate throughthe original function. Expected improvement at a givenpoint x, EI(x), is the expectation of improvement, I(x),(defined in equation (1)) for all possible Y(x) function val-ues, which follow NðyðxÞ; �ðxÞ2Þ; and calculated by:
EI ¼�
fmin � yðxÞ�
�fmin � yðxÞ�ðxÞ
� �þ �ðxÞ� fmin � yðxÞ
�ðxÞ
� �; (4)
where � is standard normal cumulative distribution func-tion, and � is standard normal probability density function.Interested readers are referred to Schonlau [1997] for deri-vation of equation 4. Figure 4d is a real example of the newsurface function formed by the expected improvementapproach. The EGO (Efficient Global Optimization) algorithmdeveloped by Jones et al. [1998] is the most commonly usedmetamodel-enabled optimizer with the approximation uncer-tainty based framework that utilizes the expected improvementsurface function.
[54] The approximation uncertainty based framework mayutilize a different statistical approach than the approachesexplained above to build a new surface function to use inStep 3. This approach, introduced by Gutmann [2001] andimplemented first using Gaussian radial basis functions,hypothesizes about the location and the objective functionvalue of the global optimum, and then evaluates the ‘‘credi-bility’’ of the hypothesis by calculating the likelihood of theresponse surface surrogate passing through the design sitesconditioned to also passing through the hypothetical globaloptimum. As such, for any given hypothetical objective func-tion value, a surface function can be formed over the entirevariable space representing the credibility of having the hypo-thetical objective function value at different points. If krigingis used as the response surface surrogate, its parameters canalso be optimized for any given point in the variable space tomaximize the conditional likelihood [Jones, 2001]—differentkriging parameter values are generated for different points inthe variable space. Jones [2001] points out that the key bene-fit of this approach over the expected improvement approachis that the next candidate solution in this approach is notselected solely based on the parameters of the surrogate,which may be in substantial error when the initial design sitesare sparse and/or deceptive. However, as even an estimate ofthe optimal objective function value is not known a priori inmany real-world problems, hypothesizing about the optimalobjective function value to be used in the framework may benontrivial. Therefore in practice, the hypothetical optimalfunction value(s) should be heuristically defined and changedas the framework proceeds [Gutmann, 2001; Jones, 2001]. Inaddition, this approach is typically more computationallydemanding than the expected improvement approach, espe-cially when the surrogate parameters (e.g., kriging parame-ters) are also to be optimized to calculate the conditionallikelihood for any given point. Regis and Shoemaker [2007b]propose a strategy to improve the method by Gutmann[2001] by controlling and increasing its local search ability.
[55] Among the studies listed in Table 1, three studiesutilize the statistics-based approaches for the approximationuncertainty based framework: di Pierro et al. [2009] usethe expected improvement approach through the ParEGOalgorithm, Mugunthan et al. [2005] use the conditionallikelihood approach developed by Gutmann [2001], and
Bau and Mayer [2006] make use of the kriging error esti-mates to make termination decisions in their surrogate-enabled optimization framework. Regis and Shoemaker[2007a] also emphasize the need of considering the approx-imation uncertainty from a nonstatistical point of view anddevelop a metamodel-enabled optimizer involving a heuris-tic distance-based measure indirectly representing the uncer-tainty in approximation. Among the studies listed in Table 1,Mugunthan et al. [2005], Razavi et al. [2012], and Regis andShoemaker [2007a, 2009] use the Regis and Shoemaker[2007a] metamodel-enabled optimizer.
[56] The metamodel-enabled optimizers with the approx-imation uncertainty based framework such as EGO areessentially for global optimization. They work very wellwhen the general shape of the original function and itsdegree of smoothness are properly approximated typicallyas a result of a good initial DoE [Schonlau, 1997]. How-ever, there are two possible drawbacks associated with thisframework. Any new surface function based on the approx-imation uncertainty is highly multimodal, and as such find-ing its global optimum in Step 3 is not easy especiallywhen the number of decision variables is large. To searchfor the global optimum in Step 3 of the framework, theoriginal EGO algorithm [Jones et al., 1998] uses thebranch-and-bound algorithm, the EGO-based metamodel-enabled optimizer developed in the work of Sobester et al.[2005] uses a multistart BFGS algorithm, and the algorithmby Regis and Shoemaker [2007a] uses a brute-force randomsampler.
[57] The other possible drawback of this frameworkarises from the fact that the approximation uncertaintybased framework relies extensively on a measure ofapproximation uncertainty to select the next candidate solu-tion as if it is correct ; whereas such a measure is only anestimation of the approximation uncertainty suggesting thatif poorly estimated, the approximation uncertainty wouldbe very deceptive in guiding the search. The approximationuncertainty at a given point is mainly a function of thedegree of nonsmoothness of the underlying (original) func-tion being approximated through the design sites and thenthe distance from surrounding previously evaluated points(i.e., design sites) (note that the measure of uncertainty pro-posed in Regis and Shoemaker [2007a] is solely based ondistance). As such, if the degree of nonsmoothness of theoriginal function is poorly captured by poorly distributedinitial design sites, the framework would result in a veryslow convergence or even, in extreme cases, prematurestalls. Figure 5 demonstrates one of our experiments withkriging where the degree of nonsmoothness of the originalfunction is underestimated by the response surface surrogatedue to a poor distribution of the initial design sites. In sucha case, the framework tends to search locally around theminimum of the surrogate (similar to the behavior of theadaptive-recursive framework—see section 2.4.2) until apoint better representing the degree of nonsmoothness ofthe original function is evaluated or until an exhaustivesearch is completed in this local region of attraction, whichresults in very small approximation uncertainty values atthis region guiding the search to other regions. Jones [2001]identifies an extreme case where the framework stalls.There are different modifications in the literature to improveupon the approximation uncertainty based framework when
W07401 RAZAVI ET AL.: REVIEW W07401
15 of 32
enabled with the expected improvement concept. Thesemodifications, include the generalized expected improvement[Schonlau, 1997] and the weighted expected improvement[Sobester et al., 2005], dynamically changing the emphasison the global search capability of the approximation uncer-tainty based framework.
2.5. Design Considerations of Metamodel-EnabledSearch (Optimization) Frameworks
2.5.1. Local Optimization Versus Global Optimization[58] As illustrated in section 2.4.4, the metamodel-
enabled optimizers under the approximation uncertaintybased framework aim to improve upon other frameworksby recognizing the importance of approximation uncer-tainty for global optimization. In this framework, thechance of missing unexplored promising regions in the fea-sible space is reduced, and the evolved metamodels havemore uniform global accuracies. Nonetheless, these gainsmight be at the expense of increasing the number of
required original function evaluations and thus loweringthe speed of convergence to a promising solution [Sobesteret al., 2005]. In some cases when the original function ishighly computationally expensive to evaluate, the maxi-mum possible number of original function evaluations willbe very limited (sometimes as small as 100–200). As such,practitioners have to be satisfied with adequate solutionsthat might not be very close to the global optimum. Thereis typically a trade-off between the global search (explora-tion) capability and the efficiency of a metamodel-enabledoptimizer when computational budget is limited, especiallyin higher dimensional problems where the size of the feasi-ble space is very large. In such cases, the adaptive-recur-sive framework might be more favorable as it may find anadequate local optimum within fewer original functionevaluations than the approximation uncertainty basedframework. This statement is consistent with Regis andShoemaker’s [2007a] conclusion, after evaluating differentmetamodel-enabled optimizers involving approximationuncertainty, that ‘‘. . . more emphasis on local search is im-portant when dealing with a very limited number of functionevaluations on higher dimensional problems.’’ In this regard,Regis and Shoemaker [2007b] propose strategies to controlthe local search ability of two metamodel-enabled optimizerswith the approximation uncertainty based framework.
2.5.2. Is Initial DoE Required?[59] As demonstrated in section 2.2, most metamodel-
enabled optimizers developed in the literature start withformal DoEs to generate an initial set of design sites that isuniformly distributed in the explanatory variable space. Awell-distributed initial set helps the metamodel better repre-sent the underlying (original) function. Nonetheless, thereare metamodeling studies not involving formal DoEs. Thereare studies that only use previously evaluated points typi-cally from previous optimization attempts to initially de-velop the metamodel. Furthermore, as pointed out in section2.4.3, the studies that follow the metamodel-embedded evo-lution framework typically use the points evaluated in theearly generations of the evolutionary algorithm to developthe first metamodels.
[60] We believe that an initial DoE is in fact requiredand a sufficiently large, well-distributed initial set of designsites to develop the metamodel is a key factor to success ofa metamodeling practice. As demonstrated in section 2.4.4,metamodel-enabled optimizers can be easily deceived bymetamodels fitted to poorly distributed design sites. Typi-cally, the metamodel fitted on a set of points collected in aprevious optimization attempt would be biased toward thealready explored regions of the feasible space (probably con-taining local optima) and could be quite misleading; there-fore, the unexplored regions would likely remain unexploredwhen optimizing on such metamodels [Jin et al., 2002;Regis and Shoemaker, 2007a; Yan and Minsker, 2006].
[61] In an evolutionary algorithm the initial population isusually uniformly distributed, but the individuals in the fol-lowing generations are conditioned to the individuals in theprevious generations. As such, in the metamodel-embeddedevolution framework, the initial set of design sites to developthe metamodel, which is a collection of points evaluated inthe initial and first few generations, may not be adequatelydistributed, and therefore the resulting metamodel may not
Figure 5. A possible failure mode of approximationuncertainty based framework when the degree of non-smoothness of the original function is largely underesti-mated and as such the standard deviation of surrogatefunction, �, is misleadingly very small resulting in an ex-haustive search around the minimizer of surrogate function;yþ � and y� � series are almost on y and not visuallydistinguishable.
W07401 RAZAVI ET AL.: REVIEW W07401
16 of 32
have adequate global accuracy. In such a case, the metamo-del that is only accurate in small regions might be com-pletely misleading in the remaining parts that may containthe global optimum [Broad et al., 2005]. However, if a suffi-ciently large subset of well-distributed points exists in theinitial set of design sites (formed by the first few generationsin an evolutionary algorithm), the subset may act as if it isfrom a formal DoE.
2.5.3. Size of Initial DoE[62] The optimal size of an initial set of design sites is
highly dependent on the shape and complexity of the origi-nal response surface as well as the computational budgetavailable. The term ‘‘optimal’’ here reflects the fact that,for a given original response function, increasing the num-ber of initial design sites would enhance the accuracy of fit(a positive effect), however, after some point (which is theoptimum) this enhancement would be at the expense ofunnecessarily increasing the computational budget havingto be initially spent on DoEs while it could have been spentmore effectively in the next steps (a negative effect). Theframework through which the metamodel is used (seesections 2.4.1–2.4.4 for different frameworks) is also a fac-tor affecting the optimal size of initial DoEs; for example,smaller initial DoEs may suffice when the metamodel-enabled optimizer puts more emphasis on approximationerror (global accuracy). The optimal size also varies for dif-ferent function approximation techniques based on theirlevel of flexibility and conformability.
[63] In practice, determination of the optimal size for aparticular problem may only be possible through extensivenumerical experiments. The only prior knowledge is usu-ally the minimum limit on the number of design sites (seesection 2.6.3), which is mathematically required to use aparticular function approximation technique. There aresome suggestions in the metamodeling literature on the sizeof initial DoEs when the approximation techniques arekriging and RBFs as summarized in the following. Joneset al. [1998] find that the approximate number of initialdesign sites required, p, is
p ¼ 10D; (5)
where D is the number of dimensions of the explanatoryvariable space. Gutmann [2001] uses a two-level full facto-rial design, which samples the corners of the variablespace, and as such the number of initial design sites used is
p ¼ 2D; (6)
which becomes very large when the number of dimensionsis more than a few. Regis and Shoemaker [2007a], based onthe fact that kriging and RBFs with linear polynomialsneed at least D þ 1 design sites to fit, suggest that
p ¼ 2ðDþ 1Þ: (7)
[64] Razavi et al. [2012] relate the proper size of the ini-tial DoE to the available computational budget for optimi-zation and suggest the following equation for kriging andRBFs with linear polynomials :
p ¼ max ½2ðDþ 1Þ; 0:1n�; (8)
where n is total number of original function evaluations(which typically accounts for almost all available computa-tional budget) to be evaluated during the optimization. Forrelatively small n values, equation (8) is equivalent to equa-tion (7), but when n becomes larger, in order to design amore detailed metamodel with a better global accuracy,0.1n is used as the size of the initial DoE.
[65] Sobester et al. [2005] conduct extensive numericalexperiments with their proposed metamodel-enabled opti-mizer based on EGO (see section 2.4.4) on multiple testfunctions to study the effect of the size of initial DoEs onthe algorithm performance. They suggest that
p ¼ 0:35n: (9)
Sobester et al. [2005] also note that the size of initial DoEsfrom equation (9) is an upper bound (safe choice) suitable forvery deceptive, highly multimodal functions, and for simplerfunctions, smaller initial DoEs may be more appropriate.They also demonstrate that if the size of the initial DoEexceeds 0.60n, the metamodel-enabled algorithm becomesinefficient. Overall, as the size of initial DoEs (and the totalnumber of function evaluations) cannot typically be largewhen the original function is computationally expensive, thereis no guarantee that the initial design sites are adequately welldistributed to effectively represent the shape of the underlyingfunction (e.g., estimate locations of the regions of attraction),particularly when it is very deceptive.
2.5.4. Metamodel Refitting Frequency[66] All algorithms utilizing metamodels except those
using the basic sequential framework, aim to evolve themetamodels over time by refitting them over the newlyevaluated points (the growing set of design sites). The idealstrategy is to refit the metamodel after each new originalfunction evaluation; this strategy is the basis of the adaptive-recursive (see section 2.4.2) and approximation uncertaintybased (see section 2.4.4) frameworks. Metamodel-enabledoptimizers with the metamodel-embedded evolution frame-work do not fundamentally need to refit the metamodelfrequently (e.g., after each original function evaluation),although the higher the frequency, the more accurate themetamodel, and therefore, the better algorithm performance.Generally, the computational time required for refitting ametamodel (mostly nonlinearly) increases with an increasein the size of the set of design sites. The type of the functionapproximation technique used to build the metamodel is alsoa main factor in determining the appropriate refitting fre-quency. As such, the computational time required for themetamodel refitting substantially varies for different types offunction approximation techniques and different data sizesand may become computationally demanding and evensometimes prohibitively long. Neural networks may sufferthe most in this regard, as the neural network training pro-cess is typically computationally demanding relative to otheralternatives even for small sets of design sites. Kriging refit-ting may also become computationally demanding for largenumbers (more than a few hundreds) of design sites [Ganoet al., 2006; Razavi et al., 2012]. The maximum likelihoodestimation methodology for correlation parameter tuning isthe main computational effort in the kriging (re)fitting proce-dure. Razavi et al. [2012] propose a two-level strategy for
W07401 RAZAVI ET AL.: REVIEW W07401
17 of 32
refitting ANNs and kriging in which the first level (fast butnot very accurate) is performed after each original functionevaluation, but the frequency of performing the second level(complete refitting, computationally demanding but moreaccurate) is reduced through a function representing thecomplexity of the fitting problem as the number of designsites becomes larger. Refitting polynomials and RBFs withno correlation parameters is very fast even for moderatelylarge sets (i.e., about 1000) of design sites. SVMs, asexplained in section 2.3.4, also have two or more parametersthat are determined by trial-and-error or direct optimizationand as such their refitting might be time consuming. Theappropriate metamodel refitting frequency for a given prob-lem is also a function of the computational demand of theoriginal computationally expensive model as the computa-tional budget required for metamodel refitting may some-times be negligible and easily justified when compared tothe computational demands of the original computationallyexpensive models.
2.5.5. Optimization Constraint Function Surrogates[67] In the water resources modeling literature, as well as
the scientific literature in general, surrogate modeling hasbeen used the most in an optimization context where thesurrogate of a computationally intensive simulation modelis used to approximate either the objective function or theconstraints or both. The discussions in this paper mainlyapply to surrogate modeling when emulating the objectivefunctions and also when constraints are included in theobjective function through penalty function approaches.When a binding constraint is approximated using a surro-gate model, the approximation accuracy is highly importantas it determines the feasibility/infeasibility of a solution.
[68] A number of the optimization studies in Table 1[e.g., Broad et al., 2005; Broad et al., 2010; Kourakos andMantoglou, 2009; Yan and Minsker, 2006, 2011] apply sur-rogate models to approximate constraint functions andthese constraint functions are built into the overall objec-tive function via penalty functions. Overlooking the impor-tance of constraint satisfaction and thus failing to takespecial precautions to ensure the metamodel-enabled opti-mizer yields a feasible solution could compromise theentire metamodeling procedure [Broad et al., 2005]. As aresult, Broad et al. [2005] demonstrate an insightful threestage approach to deal with constraint function inaccuraciesand part of this approach simply involves archiving thegood quality solutions found in the course of optimizationwith the surrogate and then evaluating a set of these withthe original model after optimization if it turns out that thefinal solution is infeasible. They also note the importanceof training the surrogate model on both feasible and infeasi-ble design sites. Yan and Minsker [2011] report for theirANN surrogate model of constraints that their penalty func-tion parameters were determined by trial and error experi-ment. Although such a trial and error approach to penaltyfunction parameters can be difficult to avoid, such experi-mentation with a metamodel-enabled optimizer will presentincredible computational challenges. There are also differ-ent approaches in the broader research community to moreaccurately handle constraints with surrogates [e.g., Kimand Lee, 2010; Lee et al., 2007; Picheny et al., 2008;Viana et al., 2010]. The paper by Viana et al. [2010] nicely
overviews these general approaches (designing conserva-tive surrogates and adaptively improving surrogate accu-racy near the boundary between feasible and infeasiblesolutions).
2.5.6. Multiobjective Optimization[69] Surrogate models have been used in combination
with a variety of multiobjective optimization algorithms toapproximate the true Pareto-front within limited originalmodel evaluations. Example metamodel-enabled multiobjec-tive optimization algorithms are formed by (1) fittingresponse surface surrogate models to only one computation-ally expensive objective (or constraint) when other objectivesare fast to run [e.g., Behzadian et al., 2009], (2) aggregatingmultiple objectives into one response function (e.g., by aweighting scheme) to be approximated by a single responsesurface surrogate [e.g., di Pierro et al., 2009; Knowles,2006; Zhang et al., 2010], and (3) using multiple surrogatemodels for multiple objectives [e.g., Bau and Mayer, 2006;Keane, 2006; Li et al., 2008; Ponweiser et al., 2008]. Thedesign considerations/limitations of surrogate modeling insingle-objective optimization also typically apply to all meta-model-enabled multiobjective optimization algorithms; how-ever, the algorithms following form 3 have additionalconsiderations or limitations discussed below.
[70] The use of multiple surrogate models for multipleobjectives would have the potential to increase the prob-lems with inaccuracy and would definitely increase meta-modeling time. The multiobjective optimization algorithmsthat utilize multiple surrogates commonly assume that theapproximation errors (uncertainties) of these multiple sur-rogates are independent (no correlation) despite the factthat the objectives are typically conflicting [Wagner et al.,2010]. These considerations practically limit the metamo-del-enabled optimization algorithms in applicability toproblems with only a small number of objectives. The issueof multiple correlated outputs being approximated by surro-gate models that is discussed in section 2.6.5 becomes rele-vant for metamodel-enabled multiobjective optimizers thatapproximate two or more objectives. Recent research byBautista [2009] and Svenson [2011] address the dependen-cies between multiple objective functions when these func-tions are emulated with multivariate Gaussian processes.
[71] The metamodel-enabled frameworks outlined insections 2.4.1–2.4.4 are all applicable to multiobjectiveoptimization (see Table 1 for example applications). Whenusing the basic sequential framework, at the end of Step 3,all approximate tradeoff solutions should be evaluated withthe original computationally expensive objective functionsto determine which of these solutions are actually nondo-minated (i.e., still tradeoff solutions). The Efficient GlobalOptimization (EGO) single-objective optimization algo-rithm [Jones et al., 1998] under the approximation uncer-tainty based framework, explained in section 2.4.4, hasstimulated a great deal of research to extend the expectedimprovement concept to multiobjective optimization [Bautista,2009; Ginsbourger et al., 2010; Jeong et al., 2005; Keane,2006; Knowles, 2006; Ponweiser et al., 2008; Svenson, 2011;Wagner et al., 2010; Zhang et al., 2010]. ParEGO [Knowles,2006], which utilizes EGO to optimize a single aggregatedfunction (weighted sum with varying weights) of all theobjective functions, and SMS-EGO [Ponweiser et al., 2008],
W07401 RAZAVI ET AL.: REVIEW W07401
18 of 32
which develops multiple surrogates for multiple objectives,are among the most popular algorithms extending EGO tomultiobjective optimization. Li et al. [2008] propose anapproximation uncertainty based approach independent ofEGO to account for uncertainties of multiple surrogate mod-els for multiobjective optimization.
2.6. Limitations and Considerations of ResponseSurface Surrogates
[72] In addition to the considerations listed in section 2.5that are specific to the design of response surface surrogate-enabled search frameworks, there are more general limita-tions and considerations that are relevant in any modelinganalysis utilizing response surface surrogates. These limita-tions and considerations are discussed below in sections2.6.1 to 2.6.4.
2.6.1. High-Dimensional Problems[73] Dimensionality is a major factor affecting the suit-
ability of response surface surrogate modeling. Responsesurface surrogate modeling becomes less attractive or eveninfeasible when the number of explanatory variables islarge. In such problems, the primary issue is that the mini-mum number of design sites required to develop some func-tion approximation models can be excessively large. Forexample, to determine the coefficients of a second-orderpolynomial in a D-dimensional input space, at least p ¼(D þ 1)(D þ 2)/2 design sites are required; in a 25-variableinput space, at least 351 design sites are required. Kochet al. [1999] demonstrate that in order to obtain a reasonablyaccurate second-order polynomial, the minimum number ofdesign sites may not be sufficient and suggest that 4.5pdesign sites (1580 when D ¼ 25) are necessary, whichmight be well beyond the computational budget availablewhen dealing with computationally expensive models. Notethat this curse of dimensionality problem exists in all otherfunction approximation models that are augmented bysecond-order polynomials (e.g., RBFs and kriging used inconjunction with second-order polynomials).
[74] Most importantly, high-dimensional problems havean extremely large search space. As such, the number ofdesign sites required to reasonably cover the spacebecomes extremely large for higher number of variables.As an example, O’Hagan [2006] notes that 200 design sitesin a 25-D space yield a very sparse coverage, but the samefigure can result in a quite dense, adequate coverage formetamodeling in a 5-D space. As a result, the number ofexplanatory variables (decision variables, DVs, in optimi-zation problems) in metamodel-enabled frameworks is typ-ically not large. Among the metamodeling studies listed inTable 1, more than 65% of the metamodel applications areon functions having less than ten decision variables, andmore than 85% have less than 20. Behzadian et al. [2009]and Broad et al. [2010] are the only studies reporting suc-cessful applications of metamodeling in relatively large-sized problems (with 50 and 49 decision variables, respec-tively). Notably in both studies, the number of originalmodel evaluations are very large (�10,000), and ANNsare used as metamodels to fit the very large sets of designsites. However, di Pierro et al. [2009] report an unsuccess-ful application of a metamodel-enabled optimizer (ParEGOby Knowles [2006], see section 2.5.6) on problems having
34 and 632 DVs—the other optimizer they used (LEMMOby Jourdan et al. [2006], see section 2.4.3) considerablyoutperformed ParEGO on both problems. They point outthat they could not improve the final solution quality ofParEGO even with increasing the total number of originalfunction evaluations.
[75] Shan and Wang [2010b] survey existing strategiesto tackling the problems associated with high-dimensionalproblems in optimization. These strategies, which are alsotypically applicable to metamodel-enabled optimization,include (1) screening aiming at identifying and removingless important decision variables [e.g., Ratto et al., 2007;Young and Ratto, 2009], (2) decomposition aiming todecompose the original problem into a set of smaller-scalesubproblems [Shan and Wang, 2010b], and (3) space reduc-tion being concerned with shrinking the feasible decisionvariable space by reducing the variable ranges to only focuson more attractive regions in optimization [Shan andWang, 2004; Shin and Grandhi, 2001]. Notably, none ofthe above strategies is a complete remedy to the issues aris-ing in metamodeling in the context of high-dimensionalproblems, and as outlined by Shan and Wang [2010b], eachhas its own limitations in applicability and usefulness. Shanand Wang [2010a] develop a new function approximationmodel that is computationally efficient for larger number ofdecision variables.
2.6.2. Exact Emulators Versus Inexact Emulators[76] A question that metamodel users need to address in
any metamodeling practice is whether an exact fit (i.e.,exact emulator) to the set of design sites or an approximatefit (i.e., inexact emulator), possibly with smoothing capabil-ities, is required. Exact emulation, also referred to as inter-polation in numerical analysis, aims to construct a responsesurface surrogate representing the underlying function thatgoes through all design sites (i.e., exactly predicts alldesign sites). Kriging for computer experiments, RBFs, andGaussian emulator machines [O’Hagan, 2006] are exam-ples of exact emulators. Unlike exact emulators, there areemulation techniques that are inexact in that they produce avarying bias (deviations from the true values that are some-times unpredictable) at different design sites. Polynomials,SVMs, MARS (multivariate adaptive regression splines),and ANNs are example inexact emulators generating non-interpolating emulator of the underlying function.
[77] Under certain (usually impractical) circumstances,inexact emulators may turn to exact emulators. For exam-ple, a polynomial can exactly reproduce all design siteswhen the degree of freedom of the polynomial regression iszero—in case where there are as many coefficients in thepolynomial as there are design sites. SVMs can also behaveas if they are (almost) exact emulators when the radius ofthe "-insensitive tube is set very close to zero and theweight of the regularization term is set very small so that itbecomes nondominant (see section 2.3.4 for the details ofSVM parameters). It has been proven that single-hidden-layer ANNs are also able to exactly fit n design sites pro-vided that there are n-1 neurons in the hidden layer[Tamura and Tateishi, 1997]. ANNs with two hidden layershaving (n/2) þ 3 hidden neurons are also capable of actingas almost exact emulators [Tamura and Tateishi, 1997].Nevertheless, neither SVMs nor ANNs have been developed
W07401 RAZAVI ET AL.: REVIEW W07401
19 of 32
to apply as exact emulators and such applications would beimpractical.
[78] SVMs have been fundamentally developed for inex-act emulation with strong and direct smoothing capabilities.Although ANNs are inexact emulators, their smoothingproperties are usually unclear to the user and very hard tomanipulate [Razavi and Tolson, 2011]. Kriging with the so-called ‘‘Nugget effect’’ [Cressie, 1993] is also an inexactemulation technique with smoothing capabilities producinga statistics-based bias at design sites. Any smoothing capa-bility usually has an associated tuning parameter that con-trols the extent of smoothing.
[79] There are two general types of problems involvingfunction approximation models: physical experiments andcomputer experiments. There may exist substantial randomerrors in physical experiments due to different error sources,whereas computer simulation models are usually determin-istic (noise-free), which means observations generated by acomputer model experiment with the same set of inputs areidentical. The inexact emulators are more suitable for physi-cal experiments than computer experiments as the usualobjective is to have an approximation that is insensitive (orless sensitive) to noise. An example application where inex-act emulation is recommended is data-driven hydrologicand rainfall-runoff modeling as, e.g., neural networks havebeen extensively used in this context. Conversely, exactemulators are usually more advisable when approximatingthe deterministic response of a computer model.
[80] Figure 6 presents two real example experimentswith inexact emulators to show how the noninterpolatingbehavior can be quite misleading in optimization especiallyin the vicinity of regions of attraction. Figure 6a shows acase where the set of design sites is relatively well distrib-uted and includes a point very close to a local optimum,however, the quadratic polynomial fitted on this set is quitemisleading and returns a point (surrogate function mini-mizer) on the plateau while ignoring the already foundlocal region of attraction; evaluating the surrogate functionminimizer and refitting would not noticeably change thepolynomial. Figure 6b demonstrates a similar case where asingle-hidden-layer neural network with 5 hidden neuronsis trained to be used as the surrogate. In this case, the set ofdesign sites is intentionally very well distributed such thatthere are design sites located at both regions of attraction(one global and one local). As can be seen, in our experi-ment with this neural network, the local region of attraction(local mode on the left) is easily ignored despite the factthat there is a design site very close to the local minimum,and second the location of the global region is misinter-preted. We believe that such a misleading behavior is notunlikely as we easily observed it in this simple experimentafter a few trial-and-errors in network initialization andtraining. Notably, evaluating and adding the surrogate func-tion minimizer to the set of design sites and refitting mightnot properly change the shape of the surrogate.
[81] In search-based metamodel-enabled analyses, itmight be beneficial to have a smooth inexact emulator gen-erating a surface passing smoothly across the design sitesin the regions of the explanatory variable space with infe-rior quality as it may lead the search smoothly to the mainregions of attraction. In contrast, inexact emulators can bevery misleading in the regions of attraction (i.e., regions
containing good quality local/global optima) where evenmarginal superiority of candidate solutions over each otheris very important and the key to continue the search. Com-bining the two behaviors (i.e., exact emulation and inexactemulation) and adaptively switching from one to the othermay appear promising, although how to implement thisadaptive switching using the common function approxima-tion techniques is not trivial.
[82] Exact emulation would seem to be the most appro-priate way of approximating the deterministic response ofcomputer simulation models. To our knowledge, the issuesand shortcomings of inexact emulation for response surfacemodeling as described above have not been fully addressedin the literature, although the problems associated havebeen acknowledged to some extent in some publications,e.g., in the work of Sacks et al. [1989], Jones [2001], andRazavi et al. [2012]. For example, Jones [2001] points outthat inexact emulators are unreliable because they mightnot sufficiently capture the shape of the deterministicunderlying function. In particular, it is not clear to us fromthe water resources literature on surrogates for constraints,why the inexact emulation of a penalty function (which forlarge and sometimes continuous regions of decision spacecan be zero) is preferred or selected over an exact emulator.In contrast, for reliability-based optimization studies where
Figure 6. Example misleading responses of inexact emu-lators : (a) a quadratic polynomial, and (b) a neural networkwith 5 hidden neurons.
W07401 RAZAVI ET AL.: REVIEW W07401
20 of 32
a metamodel is fit to predict solution reliability [e.g.,Bau and Mayer, 2006; Yan and Minsker, 2011], the meta-model is usually trained to design sites with a nondetermin-istic estimate of reliability that was generated by anapproximate Monte Carlo sampling type of experiment. Inthis case, the choice between exact versus inexact emula-tion is not so clear.
2.6.3. Limits on Number of Design Sites[83] The number of design sites used for metamodel fit-
ting can be a limiting factor affecting the suitability of afunction approximation technique for a specific problem.The appropriate range (lower and upper bounds) for thisnumber varies from one function approximation techniqueto another. Generally, the more design sites used for meta-model fitting, the higher the computational expense incurredin the fitting process. The computational expense associatedwith metamodel development and fitting should be takeninto account in any metamodeling application. This expensemay be limiting and directly affect the suitability of a func-tion approximation technique for a specific problem espe-cially when the total number of original model evaluations(and accordingly the number of design sites) in a metamo-del-enabled application is relatively large.
[84] The function approximation techniques utilizing ba-sis (correlation) functions, such as kriging, RBFs, SVMs,and Gaussian Emulator Machine (GEM), are the mostprone to the limitations arising from the large numbers ofdesign sites. In these techniques, except for SVMs, thenumber of correlation functions is typically as many as thenumber of design sites, and as such, their structures and thecomputations associated for large sets of design sitesbecome excessively large. GEM may suffer the most in thisregard as the maximum number of design sites utilized inGEM applications in the literature is only 400 [Ratto et al.,2007]. Kriging has also limited applicability when thenumber of design sites is large, mostly because determiningthe kriging correlation parameters through the maximumlikelihood estimation methodology can become computa-tionally demanding for large sets. Practical numbers ofdesign sites in kriging applications are typically less than afew thousands.
[85] RBFs and SVMs can handle larger numbers ofdesign sites. Least squares methods can efficiently fit RBFseven on large sets of design sites. SVMs are also capable ofmore efficiently handling larger numbers of design sites asthe operator associated with the design site vectors in theSVM formulation is dot product [Yu et al., 2006]. However,both RBFs and SVMs may involve a relatively computa-tionally demanding parameter tuning process for the corre-lation parameters and the other two specific parameters ofSVMs.
[86] Unlike the correlation functions existing in GEM,kriging, RBFs and SVMs, each of which only responds to asmall region in the input space close to the correspondingdesign site, ANNs consist of sigmoidal units each of whichis associated with a hidden neuron having an active partover a large domain of the input space. As such, even forlarge sets of design sites, ANNs may have relatively limitednumbers of hidden neurons forming reasonably sized ANNstructures. There are ANN applications for very large setsof design sites; for example, Broad et al. [2005] use 10,000
design sites in ANN fitting and Behzadian et al. [2009]utilize ANNs in the adaptive-recursive framework with590,000 and 2,098,400 original function evaluations.
[87] As opposed to the above function approximationtechniques, which consist of a number of locally activebuilding blocks, polynomials have a single global form cov-ering the entire input space. As such, polynomial structuredoes not expand as the number of design sites increases. Pol-ynomials can be fitted very fast even over very large sets ofdesign sites. Similar to polynomials, the structure and com-plexity of multivariate adaptive regression splines (MARS)is not a function of the number of design sites, and instead, itis a function of the shape and complexity of the underlyingfunction represented by the design sites. MARS builds multi-ple piecewise linear and nonlinear regression models (basisfunctions) to emulate the underlying function in the designsites [Friedman, 1991], and its main computational effort isto search over a variety of combinations by first adding thebasis functions to the model (forward pass) and then exten-sively prunes the model (backward pass) to find a parsimoni-ous model with a satisfactory generalization ability.
[88] The minimum number of design sites required to de-velop a metamodel also varies for different functionapproximation techniques. For a polynomial, the minimumnumber equals the number of coefficients existing in thepolynomial. Thus, for zeroth-, first- and second-order poly-nomials the minimum numbers are 1, D þ 1 and (D þ 1)(D þ 2)/2, respectively, where D is the dimension of theinput space. As stated in section 2.6.2, when using theseminimum numbers, the polynomials would act as exactemulator (i.e., zero degree of freedom). The minimumnumber of design sites in kriging and RBFs depends on thepolynomials by which they are augmented; as zeroth- andfirst-order polynomials are commonly used in conjunctionwith kriging and RBFs, the minimum number of designsites in these techniques can be very small (e.g., 11 for D ¼ 10when augmented with first-order polynomials). GEM, SVM,and MARS also require reasonably small sets of design sites.For ANNs, although mathematically there is not any mini-mum limit for the number of design sites, it is commonlyaccepted that neural networks require relatively larger setsof design sites to be properly trained. The studies listed inTable 1 are consistent with this fact as the minimum numberof initial design sites for neural network training in thesestudies is 150 [from Zou et al., 2007] and the second small-est number is 300 [from Yan and Minsker, 2006], while thisnumber can be as small as 20–25 when the RBFs are actingas metamodels [from Regis and Shoemaker, 2007a].
2.6.4. Validation and Overfitting[89] Validation may be an important step in developing a
response surface surrogate and reflects how the model per-forms in terms of generalizability. When a function approx-imation model exhibits a good fit to the design sites (i.e.,zero error for exact emulators and satisfactorily smallerrors for inexact emulators), a validation measure is alsorequired to ensure that the model performs consistently forunseen areas in the model input space. Cross validationstrategies, such as k-fold cross validation and leave-one-outcross validation, are the commonly used means of valida-tion of response surface surrogates [Wang and Shan, 2007],particularly when the set of design sites is not large. The
W07401 RAZAVI ET AL.: REVIEW W07401
21 of 32
importance of validation differs for different approximationtechniques. For example, the process of developing the pol-ynomials, RBFs, or kriging approximation models is lessdependent on validation as there are studies utilizing themwithout conducting a validation step; whereas, validationis an inseparable step in developing SVMs and ANNs. Bas-tos and O’Hagan [2009] claim that there has been littleresearch on validating emulators before using them (i.e., ina surrogate-enabled analysis framework). While this state-ment, is accurate for some emulators (in particular Gaus-sian process emulators studied in the work of Bastos andO’Hagan [2009]), it is much less accurate when emulatorssuch as SVMs and ANNs are considered. The studies inTable 1 that do not utilize SVMs or ANNs as response sur-face surrogates, also show little in the way of metamodelvalidation (although most do attempt to demonstrate theutility of the overall surrogate-enabled analysis). Bastosand O’Hagan [2009] propose some diagnostics to validateGaussian process emulators.
[90] Overfitting (overtraining) degrades the generaliza-tion ability of approximation models. All approximationmodels are prone to overfitting. In statistics, overfitting isusually described as when the model fits the noise existingin the data dominantly rather than the underlying function.However, as discussed in the work of Razavi et al. [2012],surrogate models fitted on noise-free data (e.g., data gener-ated from deterministic computer models) are also prone tooverfitting, because there is another factor affecting the riskof overfitting, which is the conformability of the modelstructure with the shape of the available data. Overfittingdue to conformability is more likely when the approxi-mation model has a large degree of freedom (is overpara-meterized) compared to the amount of available data.Curve-fitting (regression analysis) practices are typicallyless prone to the negative effects of this factor, especiallyfor low orders, because they have a global prespecifiedmodel form covering the entire input variable space. But inhighly flexible approximation models, including ANNs, theproblem associated with the conformability factor can besubstantial.
[91] Neural networks are highly prone to the risk of over-fitting. However, not all of the studies involving neural net-works listed in Table 1 pay attention to overfitting/generalizability. Seven studies listed in Table 1 [Broadet al., 2005; Broad et al., 2010; Johnson and Rogers,2000; Khu and Werner, 2003; Razavi et al., 2012; Zouet al., 2007, 2009] apply the early stopping approach [Bealeet al., 2010] to avoid overtraining, and one study [Kourakosand Mantoglou, 2009] applies the Bayesian regularizationprocedure [Foresee and Hagan, 1997]. The main problemwith early stopping is that the available design sites have tobe split into a training set, a testing sets, and sometimes avalidation set resulting in fewer data available to trainANNs. In contrast, the Bayesian regularization proceduredoes not require this splitting of design sites—all can beused for training. Razavi and Tolson [2011] recently pro-posed a neural network regularization measure that alsodoes not require a testing/validation set.
[92] Overfitting due to the conformability factor mayalso occur in the function approximation techniques thatare based on basis functions such as kriging and RBFs.However, the risk and extent of overfitting in kriging
and RBFs is typically less compared to ANNs. The risk ofoverfitting is higher when there are very few design sitesrelative to the number of kriging and RBF parameters to betuned. Note that, for example, a kriging model with Gaus-sian correlation functions has D correlation function param-eters each of which is associated with one dimension in theD-dimensional input space and an RBF model with thin-plate splines has no correlation parameter to tune; as such,the number of parameters in these approximation models istypically small compared to the number of available designsites. As overfitting in kriging is not a major challenge, ithas not been directly addressed in most kriging studies. Tomitigate the possible overfitting problem in kriging, Welchet al. [1992] propose to initially keep all the correlationfunction parameters the same in all input space dimensionsin the maximum likelihood estimation process, and thenrelax them one-by-one to identify the ones resulting inhigher increase in the likelihood function and only let thembe different.
2.6.5. Emulating Multiple Outputs or MultipleFunctions
[93] The literature reviewed in Table 1 shows that eventhough the vast majority of response surface surrogate stud-ies involve a simulation model with temporally and spatiallyvarying outputs, the required number of model outputs ornumber of output functions (e.g., calibration error metrics) toapproximate with surrogates is typically limited to onlyhandful of outputs/functions (often just one). Recently,Bayesian emulation techniques have appeared that are tai-lored to approximate a time series of output (e.g., emulatinga dynamic simulator in the work of Conti and O’Hagan[2010]). According to Conti and O’Hagan [2010], there arethree approaches to emulating dynamic simulators: (1) mul-tioutput emulators that are unique because they account forcorrelations among outputs, (2) multiple single-output emu-lators, and (3) time input emulators that use time as an auxil-iary input. They conclude that the multiple single-outputemulator approach is inappropriate because of its failure toaccount for temporal correlations and believe that multiout-put emulators should eventually lead to the successful emula-tion of time series outputs. Fricker et al. [2010] also proposemultioutput emulators considering correlation between mul-tiple outputs based on multivariate Gaussian processes.
[94] In terms of the common function approximationtechniques in Table 1, since ANNs have the ability to pre-dict multiple outputs simultaneously, ANNs are a type ofmultioutput emulators. None of the other function approxi-mation techniques reviewed in detail in section 2.3.1–2.3.4can directly act as multioutput emulators. Thus, a singleANN model of multiple correlated outputs should concep-tually be able to account for these correlations among out-puts. Based on the work by Conti and O’Hagan [2010], webelieve that the ability to account for correlations amongoutputs that are significantly correlated (even multiple out-put functions such as two model calibration objective func-tions) in the response surface surrogate should conceptuallylead to increased surrogate accuracy. However, there aremultiple studies demonstrating the need for multiple ANNsurrogates to model multiple outputs (e.g., Kourakos andMantoglou [2009], Yan and Minsker [2011], and Broadet al. [2005]). Yan and Minsker [2011] approximate six
W07401 RAZAVI ET AL.: REVIEW W07401
22 of 32
outputs with three independent ANN surrogate modelswhile Broad et al. [2005] model each output of interestwith independent ANNs. Kourakos and Mantoglou [2009]utilize ANNs to approximate 34 outputs and they explainhow their single ANN to model all outputs would lead to apractically infeasible ANN training procedure as nearly2400 ANN parameters were to be specified. Instead theybuilt and trained 34 modular subnetworks to circumventthis computational bottleneck in ANN training, assumingthat the correlations between the outputs are negligible(justified based on the physics of their case study). Furtherstudies could investigate scenarios where multiple-outputANNs are beneficial and to determine all the reasons theycan fail relative to multiple single output ANNs.
3. Lower-Fidelity Physically Based Surrogates[95] In contrast to response surface surrogates, which are
data-driven techniques for approximating the response sur-face of high-fidelity (original) models based on a limitednumber of original model evaluations, lower-fidelity physi-cally based surrogates are essentially cheaper-to-run alter-native simulation models that are less faithful to the systemof interest. For any real-world system, there may exist sev-eral simulation models with different levels of fidelity (ac-curacy). A high-fidelity model refers to the most accurateand as such the most desirable model available to users. Asthe high-fidelity models may typically be computationallyintensive, there are frameworks concerned with efficientlyutilizing the high-fidelity models in conjunction withlower-fidelity models (as surrogates of high-fidelity mod-els) to enhance the overall computational efficiency; thesesurrogate modeling frameworks when applied in the fieldof optimization are also referred to as ‘‘multifidelity’’ or‘‘variable-fidelity’’ optimization [Forrester et al., 2007;Gano et al., 2006; Leary et al., 2003; Madsen and Langth-jem, 2001; Sun et al., 2010]. In some publications, low-fidelity models are called ‘‘coarse’’ models, and high-fidel-ity models are called ‘‘fine’’ models (e.g., in the work ofBandler et al. [1994]). As a simple example, a numericalmodel with very small numerical time steps may be deemeda high-fidelity model and its corresponding low-fidelitymodel may be one with larger numerical time steps. In thispaper, we often refer to ‘‘lower-fidelity physically based sur-rogates’’ as ‘‘lower-fidelity surrogates’’ for simplicity.
[96] Lower-fidelity surrogates have two immediateadvantages over the response surface surrogates: (1) theyare expected to better emulate the unexplored regions inthe explanatory variable (input) space (i.e., regions farfrom the previously evaluated points with the high-fidelitymodel) and as such perform more reliably in extrapolation,and (2) they avoid or minimize the problems associatedwith high-dimensional problems (see section 2.6.1), as theyuse domain-specific knowledge. There is a main assump-tion behind any lower-fidelity surrogate modeling practice:high-fidelity and low-fidelity models share the basic fea-tures and are correlated in some way [Kennedy andO’Hagan, 2000]. As such, the response of the low-fidelitymodel for a given input vector is expected to be reasonablyclose to the response of the high-fidelity model for the corre-sponding input vector in the high-fidelity model input space.This closeness enables the lower-fidelity model to relativelyreliably predict the performance of the high-fidelity model in
unexplored regions in the variable space. If this assumptionis violated, the surrogate modeling framework would notwork or the gains would be minimal.
[97] There are multiple strategies to reduce the numberof expensive original model evaluations when a lower-fidelity model is available. The immediate strategies includeusing the lower-fidelity model first to reduce the variables’feasible space and/or to identify unimportant variables toreduce the problem dimensionality [Madsen and Langthjem,2001]. Most studies utilizing lower-fidelity models areconcerned with developing optimization strategies wherelow- and high-fidelity models are adaptively chosen to beevaluated in the course of optimization [Forrester et al.,2007; Gano et al., 2006; Huang et al., 2006; Leary et al.,2003; Viana et al., 2009]. There are also lower-fidelity sur-rogate applications in uncertainty analysis [Allaire, 2009;Kennedy and O’Hagan, 2000].
[98] Lower-fidelity surrogate modeling has only veryrecently started to gain popularity in the water resources lit-erature. Although it is a well-established area of research inthe broader research community, formal terminologies andcommon methods available for lower-fidelity surrogatemodeling in other disciplines seem to be largely unused inwater resources literature. This section first reviews andcategorizes the research efforts for lower-fidelity surrogatemodeling in the broader research community and thenreports the research efforts accomplished in water resourcesliterature.
3.1. Types of Lower-Fidelity Physically Based Models
[99] Depending on the original model type, at least threedifferent general classes of strategies may be used to yieldlower-fidelity models. In the first class of strategies, thelower-fidelity models are fundamentally the same as theoriginal models but with reduced numerical accuracy. Forexample, in numerical simulation models of partial differ-ential equations, a lower-fidelity model can be a variationof the original model but with larger (coarser) spatial/temporal grid size [Leary et al., 2003; Madsen and Langthjem,2001; Thokala and Martins, 2007]. Finite element modelswith simpler basis functions can also be a low-fidelitymodel of an original model involving more complex basisfunctions. Whenever applicable, lower-fidelity models canbe essentially the same as the original model but with lessstrict numerical convergence tolerances. Forrester et al.[2006] employ a partially converged CFD model as alower-fidelity surrogate of a fully converged (original)model.
[100] A second class of strategies to derive lower-fidelitymodels involves model-driven approximations of the origi-nal models using model order reduction (MOR) techniques[Gugercin and Antoulas, 2000; Rewienski and White,2006; Willcox and Megretski, 2005]. MOR aims to reducethe complexity of models by deriving substitute approxima-tions of the original complex equations involved in theoriginal model. These substitute approximations are sys-tematically obtained by rigorous mathematical techniqueswithout the need of knowing the underlying system.
[101] In the third class of strategies, lower-fidelity mod-els can be simpler models of the real-world system of inter-est in which some physics modeled by the high-fidelitymodel is ignored or approximated. Strategies such as
W07401 RAZAVI ET AL.: REVIEW W07401
23 of 32
considering simpler geometry and/or boundary conditionsin the model, utilizing a lumped-parameter model in lieu ofa distributed model, and utilizing a two-dimensional modelinstead of a three-dimensional model lie under this class.For example in fluid dynamics, numerical models solvingNavier-Stokes equations are the highest-fidelity and themost expensive models, models based on Euler equationsare the lower-fidelity and less expensive models, and ana-lytical or empirical formulations are the lowest-fidelity andcheapest models [Alexandrov and Lewis, 2001; Simpsonet al., 2008; Thokala and Martins, 2007]. Note that for anyreal-world system, there may be a hierarchy of models withdifferent levels of fidelity.
[102] Thokala and Martins [2007] conduct multipleexperiments utilizing different types of lower-fidelity mod-els and conclude that the lower-fidelity models that sharethe same physical components with the original models(i.e., lower-fidelity models lying under the first and secondclasses) are more successful than those having different/simplified physical bases (i.e., lower-fidelity models lyingunder the third class), as no correction (see section 3.2.1 fordefinition of correction functions) can compensate forignoring a component of the physical characteristics of asystem.
3.2. Lower-Fidelity Model Enabled AnalysisFrameworks
3.2.1. Variable-Fidelity Models With IdenticalVariable Space
[103] Models with different levels of fidelity may bedefined over the same variable/parameter space. There aremultiple frameworks selectively utilizing lower-fidelity mod-els as substitutes of the original models to reduce the numberof expensive evaluations of the original model. These frame-works have mostly arisen from the optimization context buthave also applied for other purposes including uncertaintyanalysis. The main challenge to be addressed in these frame-works is that the response landscapes of the original andlower-fidelity models are somewhat different. Figure 7presents an illustrative hypothetical example of high- andlow-fidelity response landscapes in a one-dimensional space.As can be seen, there are discrepancies between the tworesponse landscapes; the low-fidelity function underestimates
the response on the left part of the plot and overestimates inmost of the right part. Moreover, both functions have tworegions of attractions (modes), but the global minimizer ofthe low-fidelity function coincides with the local minimizerof the high-fidelity function which is far from the globaloptimum.
[104] There are three main independent (but sometimescomplimentary) approaches to formally address the dis-crepancies between low- and high-fidelity models: correc-tion functions, space-mapping, and hybrid strategies. Themost common approach is to build a correction function tocorrect the response landscape of the lower-fidelity modeland align it with the response landscape of the originalmodel. This process in the multifidelity modeling literatureis referred to as correction, tuning, scaling, or alignment.Suppose that fhðxÞ and flðxÞ represent the response surfacesof the high- and low-fidelity models of the real-world systemof interest, respectively; the two general strategies for defin-ing a correction function are the additive approach [Ganoet al., 2006; Leary et al., 2003; Viana et al., 2009] in equa-tion (10) and the multiplicative approach [Alexandrov andLewis, 2001; Madsen and Langthjem, 2001; Thokala andMartins, 2007] in equation (11):
gaðxÞ ¼ fhðxÞ � flðxÞ; (10)
gmðxÞ ¼ fhðxÞ=flðxÞ; (11)
where gaðxÞ is the additive correction function directlyemulating the discrepancies between the high- and low-fidelity response surfaces and gmðxÞ is the multiplicativecorrection function. As the exact form of the correctionfunction is typically unknown for any given problem, anapproximate correction function is to be built by a limitednumber of high- and low-fidelity model evaluations. Then,the surrogate model can be
fsðxÞ ¼ flðxÞ þ gaðxÞ; (12)
fsðxÞ ¼ flðxÞgmðxÞ; (13)
where gaðxÞ and gmðxÞ are the approximate correction func-tions that are designed to correct the low-fidelity modelresponse by offsetting (additive) and scaling (multiplica-tive), respectively. Notably, the multiplicative form is proneto ill-conditioning when the model response approacheszero; to avoid this ill-conditioning, a constant can be addedto both the numerator and denominator of equation (11).Eldred et al. [2004] demonstrate the superiority of the addi-tive form over the multiplicative form across multiple testproblems.
[105] The process of developing the approximate correc-tion function is analogous to response surface surrogatemodeling; however, the correction function is supposedlyless complex than typical response surface surrogates, asthe response surface of a lower-fidelity model is supposedto be reasonably close to the response surface of the origi-nal model. Different approaches or tools have been pro-posed to develop the approximate correction function.Linear regression [Alexandrov and Lewis, 2001; Madsenand Langthjem, 2001; Vitali et al., 2002] and quadratic
Figure 7. A hypothetical example of lower-fidelity func-tions along with the high-fidelity (original) function.
W07401 RAZAVI ET AL.: REVIEW W07401
24 of 32
polynomials [Eldred et al., 2004; Sun et al., 2010; Vianaet al., 2009] are two common, simple approaches to buildthe correction functions. More flexible function approxima-tion models have also been used for this purpose, includingkriging [Gano et al., 2006] and neural networks [Learyet al., 2003].
[106] Note that limitations and considerations raised insection 2.6 for response surface surrogates may also holdwhen using complex function approximation models tobuild the approximate correction functions. However, thelimitations of response surface surrogates used in this con-text for high dimensional problems (see section 2.6.1) arenot as important. This is because the correction function fora good quality lower-fidelity surrogate is only of secondaryimportance (it adjusts the lower-fidelity model output). Asa result, less complex approximate correction functionsmay be more desirable in practice. Building correctionfunctions that are correlated for multiple outputs is simi-larly not as important (see section 2.6.5)
[107] The general correction-function-based frameworkutilizing lower-fidelity surrogates is as follows: the frame-work, in Step 1, starts with an initial DoE to generate sam-ple points and then evaluates them by both the original andlower-fidelity models. In Step 2, a global correction func-tion is developed to emulate the discrepancies (errors)between the responses of the original and lower-fidelitymodels at the sampled points. In Step 3, a search or sam-pling is applied on the corrected response surface of thelower-fidelity model to identify the regions of interest inthe explanatory variable space and screen out one or multi-ple points. In cases where the search algorithm is for opti-mization, this step returns the optimal/near-optimal point(or multiple high-quality points) of the corrected responsesurface of the lower-fidelity model. In Step 4, the candidatepoints from Step 3 are evaluated by the original function. Ifneeded, the framework goes back to Step 2 to modify thecorrection function and repeat the analyses in Step 3.
[108] Trust-region approaches for optimization havealso been applied in correction-function-based framework[Alexandrov and Lewis, 2001; Eldred et al., 2004; Robin-son et al., 2006]. In such a framework, an initial DoE isnot required and the framework may start with any (butdesirably a good quality) initial solution (Step 1). The ini-tial trust region size is also specified in this step. In Step2, the current solution is evaluated by both original andlower-fidelity models. In Step 3, the correction function islocally fitted around the current (best) solution. In Step 4,the corrected lower-fidelity response surface is optimizedwithin the trust region (centered at the current bestsolution) and the best solution found is evaluated by theoriginal model. In Step 5, depending on how close thishigh-fidelity response is to the low-fidelity response, thetrust region is expanded, remains the same, or is reduced.Steps 3 through 5 are repeated until convergence or stop-ping criteria are met. The trust-region based framework isprovably convergent to a local optimum of the originalmodel response surface if the corrected lower-fidelityresponse surface is at least first-order accurate at the cen-ter of the trust region [Robinson et al., 2006]. Eldredet al. [2004] demonstrate that second-order accuratecorrections can lead to more desirable convergencecharacteristics.
[109] The second main approach to tackle the discrepan-cies between low- and high-fidelity response surfaces is theso-called space mapping approach. Initially introduced byBandler et al. [1994] for optimization purposes, space map-ping aims to locally establish a relationship between theoriginal model variables and the lower-fidelity model varia-bles. By definition, space mapping can be used to make useof any sufficiently faithful lower-fidelity model even if it isdefined on a different variable space (see section 3.2.2). Toestablish a space mapping relationship, multiple points onthe original response surface and their corresponding pointson the lower-fidelity response surface are required. For anygiven point x in the original variable space, a correspond-ing point x in the lower-fidelity variable space is defined asthe point where flðxÞ is equal (or reasonably close) to fhðxÞ.To find each point required in the lower-fidelity variablespace, one optimization subproblem is to be solved withthe objective of minimizing jflðxÞ � fhðxÞj by varying x.Notably, there may exist multiple points x having the samelower-fidelity response value equal to fhðxÞ leading to thefailure of space mapping. Many approaches have been pro-posed to address this problem of nonuniqueness [Bakret al., 1999; Bandler et al., 1996; Bandler et al., 2004].Once the corresponding points in the two spaces are avail-able, different linear or nonlinear functions may be used torelate the two spaces by fitting over these points [Bandleret al., 2004]. Then any solution in the lower-fidelity vari-able space obtained in analyses with the lower-fidelitymodel can be mapped to the original variable space. Thespace mapping relationships can be updated as the algo-rithm progresses. As many optimization subproblems are tobe solved on the lower-fidelity model to adaptively estab-lish/update the mapping relationships, the total number oflower-fidelity model evaluations is relatively high. Assuch, if the low-fidelity model is not much cheaper than theoriginal model, space mapping would not be computation-ally feasible.
[110] There are hybrid strategies following a third gen-eral approach to make use of lower-fidelity models jointlywith original models to build response surface surrogates.These strategies may be used with any of the frameworksutilizing response surface surrogates (detailed in section2.4) with the main difference that there are (at least) twosets of design sites with different levels of fidelity. Thesetwo sets of design sites are used to either build a singleresponse surface surrogate formed by the sets or tworesponse surface surrogates representing the two sets inde-pendently. Forrester et al. [2007] develop a single responsesurface surrogate using cokriging, which is an exact emula-tor on the high-fidelity design sites and an inexact emulatoron the lower-fidelity design sites—such a response surfacesurrogate can capture the exact behavior of the underlyingfunction where high-fidelity design sites are available inthe variable space and only extract the trends and curva-tures in the unexplored regions from the cheaply availablelower-fidelity design sites that are far from the high-fidelitydesign sites. Leary et al. [2003] propose a heuristic way toincorporate the lower-fidelity model response into ANN-and kriging-based response surface surrogates. Huang et al.[2006] develop a methodology to incorporate the dataobtained by a lower-fidelity model to enhance the EGOalgorithm (see section 2.4.4 for EGO). Vitali et al. [2002]
W07401 RAZAVI ET AL.: REVIEW W07401
25 of 32
and Sun et al. [2010] use a fourth-order polynomial and anRBF model, respectively, to develop response surface surro-gates of their developed lower-fidelity models and then uti-lize a correction function approach to correct these responsesurface surrogates to be used in optimization problems withtheir original computationally expensive models.
3.2.2. Variable-Fidelity Models With DifferentVariable Spaces
[111] The variable space associated with a lower-fidelitymodel may differ from the original model variable space.In such cases, the number of variables associated with thelower-fidelity model can be unequal to (typically less than)the number of original variables. Since their application isnot trivial, multifidelity models when defined on differentvariable spaces have been less appealing in surrogate mod-eling literature [Simpson et al., 2008]. When the spaces aredifferent, a mapping must be established such that x ¼ PðxÞor x ¼ QðxÞ where x and x are corresponding points in theoriginal and lower-fidelity variable spaces, and P and Q aremapping functions that transform points from one space tothe other. In some cases, the space mapping relationship isclear based on the physics of the problem, and the lower-fi-delity variable vector is a subset or interpolation of the origi-nal variable vector. However, when such knowledge is notavailable, empirical relationships are to be derived. Thespace mapping approach, explained in section 3.2.1, is ameans to derive these empirical relationships [Bandler et al.,1994; Bandler et al., 2004]. The main procedure in spacemapping between different spaces is essentially the same asthe space mapping procedure when the spaces are identical.Another mapping approach has been proposed based onproper orthogonal decomposition (also called principal com-ponent analysis) and compared against space mapping in atrust-region optimization framework [Robinson, 2007; Rob-inson et al., 2006]. Robinson et al. [2006, 2008] incorporatethe idea of correction function into space mapping to matchthe gradients of the lower-fidelity and original responsesurfaces at points of interest.
3.3. Related Research in Water Resources Literature
[112] The basic idea of replacing a high-fidelity simula-tion model with a lower-fidelity model of the system forthe purposes of optimization and uncertainty or sensitivityanalysis is an intuitive concept that exists in the waterresources literature. However, the vast majority of suchstudies have applied this idea independent of the lower-fi-delity surrogate modeling studies, methods and terminol-ogy from other disciplines described in section 3.2. Thissection reviews the research efforts arising from waterresources modeling and relates them to the general method-ologies for lower-fidelity surrogate modeling. The terms inquotations below represent the terminology used in theassociated publications.
[113] There are research efforts to reduce the complexitylevel of various water resources models to be typically usedin optimization or calibration. Ulanicki et al. [1996] andMaschler and Savic [1999] develop ‘‘model reduction meth-ods’’ to simplify water distribution network models by elimi-nating less important pipes and nodes and allocating theirdemands to the neighboring nodes. Shamir and Salomons[2008] and Preis et al. [2011] utilize the model reduction
method developed by Ulanicki et al. [1996] to create‘‘reduced models’’ of water distribution networks to be usedin optimization frameworks. McPhee and Yeh [2008] de-velop a ‘‘reduced model’’ of a groundwater model and link itwith optimization in lieu of the original model for the pur-pose of groundwater management—this ‘‘reduced model’’is defined over a different parameter space based on em-pirical orthogonal functions (ordinary differential equationinstead of partial differential equation). Vermeulen et al.[2004] and Siade et al. [2010] propose methods to develop‘‘reduced models’’ of high-dimensional groundwater modelsbased on proper orthogonal decomposition. Vermeulen et al.[2005] and Vermeulen et al. [2006] utilize such ‘‘reducedmodels’’ as substitutes of the original groundwater modelsfor model inversion (calibration).
[114] The well-established idea of using response matri-ces in place of full groundwater models for groundwatermanagement (e.g., in the work of Reichard [1995]), wherethe same methods for developing response surface surro-gates are typically used, can also be classified as lower-fi-delity surrogates. Cheng et al. [2011] also refer to theresponse matrix as a ‘‘reduced model’’ replacing a ground-water model in their optimization problem formulated forgroundwater management. Pianosi and Soncini-Sessa [2009]use a ‘‘reduced model’’ of a reservoir system to improve thecomputational efficiency of stochastic dynamic program-ming for designing optimal reservoir regulation plans. Croutet al. [2009] develop a methodology to ‘‘reduce’’ waterresources models by iteratively replacing model variableswith constants.
[115] Notably, in all the above studies that are for opti-mization purposes, reduced models (i.e., lower-fidelitymodels) after being developed are treated as if they arehigh-fidelity representations of the underlying real-worldsystems and fully replace the original models (i.e., high-fidelity models) in their analyses. In other words, the dis-crepancies between the high- and low-fidelity models areignored.
[116] Simplified surrogate models have been also usedwith different Markov Chain Monte Carlo (MCMC) frame-works for uncertainty analysis of water resources models.Keating et al. [2010] develop a simplified groundwatermodel as a ‘‘surrogate’’ of a computationally intensivegroundwater model, defined over a different parameterspace. They compare the null-space Monte Carlo (NSMC)method and an MCMC for autocalibration and uncertaintyassessment of the surrogate instead of the original model,and then the NSMC, with techniques developed for the surro-gate, is used on the original model. Efendiev et al. [2005] de-velop a two-stage strategy employing a groundwater modelwith a coarse grid, referred to as ‘‘coarse-scale model,’’ as alower-fidelity surrogate of the original ‘‘fine-grid model’’ tospeed up an MCMC experiment. In their methodology, boththe surrogate and original models are defined over the sameparameter space, and the surrogate is first evaluated to deter-mine whether the original model is worth evaluating for agiven solution.
[117] There are very few surrogate modeling studies inwater resources addressing the discrepancies between theresponse surfaces of the lower-fidelity surrogate and theoriginal model. Mondal et al. [2010] develop ‘‘coarse-grid models’’ (also called ‘‘upscaled models’’) of a
W07401 RAZAVI ET AL.: REVIEW W07401
26 of 32
(high-fidelity) groundwater model to speed up MCMCcomputations for uncertainty quantification; in their study,the discrepancies are recognized and quantified by a linearcorrection function built off-line before MCMC experi-ments to avoid biased approximated posterior distribution.Cui et al. [2011] also develop a ‘‘reduced order model’’ ofa ‘‘fine model’’ by coarsening the gird structure and employit within a correction function framework to enhance the ef-ficiency of an MCMC algorithm for groundwater modelinversion; an adaptive local correction function is used intheir study in the course of MCMC sampling to improvethe accuracy.
4. Efficiency Gains of Surrogate-enabledAnalyses
[118] The most important question in assessing a surro-gate-enabled analysis is how efficient or effective it is incomparison with other efficient alternative tools withoutsurrogate modeling, especially because the computationalefficiency achieved is the main factor motivating theresearch and application of surrogate modeling. Surrogate-enabled analyses typically sacrifice accuracy for efficiencyas they utilize approximate models (less accurate than theoriginal models) to more efficiently achieve the analysisobjectives. As such, there is always a risk that surrogatemodels yield misleading results ; this risk is higher whenthe original response landscape is complex and deceptiveand is minimal for simple original response landscapes(e.g., almost negligible for smooth unimodal functionsbeing optimized). A thorough discussion of this matter inthe context of optimization is available in the work ofRazavi et al. [2012] where a comparative assessmentframework for metamodel-enabled optimizers is developedpresenting a computational budget dependent definition forthe success/failure of the metamodeling strategies. Thecareful selection of a benchmark alternative analysis or de-cision-making procedure without surrogate modeling is avital step for fair assessment of a given surrogate-enabledanalysis. To be clear, a benchmark alternative analysis hasavailable at least the same number of original model simula-tions as utilized in the surrogate-enabled analysis. AlthoughBroad et al. [2005] note that ‘‘Metamodels should only beused where time constraints prohibit the possibility of opti-mizing a problem with a simulation model.’’ In our view,the determination of such prohibition is not always clear cut.In a sampling context, one may take fewer samples thanthey would prefer, while in a search context, the algorithmcan be terminated before it converges.
[119] In an optimization or search context, the mosttangible measure of efficiency gains over a benchmark al-ternative is computational saving. For a single metamodel-enabled optimization analysis (e.g., optimal design) thiscan be calculated as
Saving ¼ 100� t � tst
h i; (14)
where t is the computational budget or time required toreach a desired solution quality through an algorithm with-out surrogate modeling, and ts is the computational budgetor time a surrogate-enabled algorithm requires to reach asolution with the same quality. Regis and Shoemaker[2007b] present such comparative assessments in terms of
efficiency by quantifying t and ts as the numbers of originalfunction evaluations the algorithms require to reach within1% of the optimal value of the original function. Computa-tional budgets may be quantified as the total CPU clocktime (as in the work of Behzadian et al. [2009], Broadet al. [2005], Kourakos and Mantoglou [2009]) or the num-ber of original function evaluations (as in the work ofMugunthan and Shoemaker [2006], Regis and Shoemaker[2007a], Zou et al. [2007, 2009]). As stated in Table 1, 15(out of 32) studies present quantitative information demon-strating the efficiency of the surrogate modeling strategiesused. Some of these studies report the associated computa-tional savings explicitly; in the other ones, savings are notclearly reported and we interpreted them based on the pub-lished results. According to Table 1, computational savingsachieved through the use of surrogate models can vary sig-nificantly, ranging from 20% of CPU time in the work ofZhang et al. [2009] to 97% in the work of Zou et al. [2007].
[120] Razavi et al. [2012] demonstrate that the failure ofmetamodel-enabled optimizers is a function of not only thedegree of complexity and deceptiveness of the originallandscape but also the available computational budget. Invery limited computational budgets surrogate modeling isexpected to be very helpful, whereas when the computa-tional budget is not severely limited, surrogate modelingmight not be as helpful, as equivalent or better solutionscan be achieved by the benchmark optimizer. We believesimilar findings are probable for all other types of metamo-del-enabled analyses. However, the details of the compara-tive efficiency assessment framework for each of theseother types of analysis (i.e., sensitivity analysis or reliabil-ity assessment), along with developing meaningful variantsfor equation (14), would need to be determined. An exam-ple variation to the efficiency assessment procedure formetamodel-enabled GLUE is demonstrated in the work ofKhu and Werner [2003].
[121] In any surrogate-enabled analysis, the availablecomputational budget or time is divided between threemain parts : (1) budget or time required to run the originalmodel, (2) budget or time required to develop, run, andupdate the surrogate model, and (3) budget or time the ana-lyst needs to identify and create an appropriate surrogate-enabled analysis framework. Parts 2 and 3, which arereferred to as ‘‘metamodeling time’’ and ‘‘analyst time,’’respectively, should consume a small portion of the avail-able computational budget leaving the majority for part 1.Nonetheless, metamodeling time and analyst time shouldideally be taken into account when assessing the computa-tional efficiency of a surrogate-enabled analysis. The com-putational and perhaps the analyst’s efforts are typicallyhigher for lower-fidelity surrogates than response surfacesurrogates. As such, it is difficult to imagine any compari-son involving lower-fidelity surrogates on the basis of thenumber of original function evaluations required as is com-monly done with response surface surrogate comparisons.When a developed surrogate is to be used in repeat applica-tions the importance of the analyst time is reduced.
[122] Any conclusion on the efficiency of a developedalgorithm with surrogate models must be based on perform-ing multiple replicates as any single application of such analgorithm (as with any other stochastic algorithm) is asingle performance level observation from a statistical
W07401 RAZAVI ET AL.: REVIEW W07401
27 of 32
population of possible performance levels. Despite theobvious computational burden of performing multiple rep-licates, it is the only way to conduct valid numerical assess-ments and comparisons.
5. Summary and Final Remarks[123] There is a large body of literature, from different
contexts and disciplines, developing and applying a wide va-riety of surrogate modeling strategies typically to improvethe computational efficiency of sampling or search-basedmodeling analyses. The surrogate modeling literature wasreviewed with an emphasis on research efforts in the field ofwater resources modeling. A set of publications including 48references on surrogate modeling in water resources prob-lems were analyzed and summarized in this paper and morethan 100 other references from other disciplines were alsoreviewed. We overviewed the components involved in asurrogate-enabled modeling analysis framework and detaildifferent framework designs.
[124] The most important observations and availableguidance on the alternative methods and surrogate model-ing frameworks that have been applied to the water resour-ces studies reviewed here are as follows:
[125] 1. It is not trivial to suggest the best function approxi-mation technique for the purpose of response surface model-ing, and metamodel developers typically pick a techniquebased on their preference and level of familiarity as well assoftware availability. Function approximation techniques thatare able to (1) act as exact emulators, (2) provide a measureof approximation uncertainty, and (3) efficiently and effec-tively handle the size of the data set (design sites) of interest,conceptually seem to be the most appealing for modeling thedeterministic response of computer simulation models.
[126] 2. The metamodel-enabled optimization frame-works that utilize metamodel approximation uncertaintyestimates (section 2.4.4) are conceptually the most robuststrategies in comparison with the other three frameworks,especially for problems with highly multimodal and decep-tive original response surfaces. In our view, this frame-work, particularly the statistics based approach [e.g., Jones,2001] is underutilized in the water resources literature asthe metamodel uncertainty characterization should proveuseful in Bayesian model calibration studies and traditionalMonte Carlo-based reliability or uncertainty analysis.
[127] 3. When evidence is available suggesting the origi-nal function is a relatively simple/unimodal function, usingthe basic sequential framework or adaptive-recursive frame-work would be the most appropriate as they would be suc-cessful and more efficient.
[128] 4. Difficulties are introduced moving from uncon-strained (or just box-constrained) surrogate-enabled singleoptimization to surrogate-enabled constrained or multi-objective optimization (see sections 2.5.5 and 2.5.6).
[129] 5. Probably the most important limitation of surro-gate modeling in applicability, especially response surfacesurrogate modeling, is when the number of dimensions in theproblem variable space is large (successful surrogate-enabledanalyses reviewed here were limited to 50 at most and typi-cally less than 20 explanatory variables). Lower-fidelity sur-rogates are much less vulnerable to this limitation.
[130] 6. Lower-fidelity models are conceptually morereliable in exploring the unseen regions in the explanatory
variable space compared to response surface surrogates.This reliability directly relates to the level of fidelity of asurrogate model and diminishes for the surrogates withvery low fidelity. As there is typically a trade-off betweenthe level of fidelity of a model and its computationaldemand, the lower-fidelity model developers should createa level of fidelity that is sufficiently faithful to the originalmodel while being efficient enough to permit the case studyspecific analysis required.
[131] 7. Lower-fidelity models have advantages whenthere is an interest in emulating multiple model outputs orin multiobjective optimization with two or more emulatedobjectives as the lower-fidelity models would inherentlyaccount for output/objective function correlations.
[132] The following remarks summarize some suggestionson future research directions, many of which are inspiredfrom ideas not commonly used in water resources literature:
[133] 1. Since the level of complexity of the originalresponse function (which is typically unknown a priori)plays a key role in the determination of an appropriatefunction approximation technique, future research may bedirected at developing methods to preanalyze the originalresponse landscapes with a very limited number of samplesto measure their level of complexity. The study by Gibbset al. [2011] is an example of research efforts for theresponse landscape analysis in the optimization context(not involving surrogate modeling).
[134] 2. The strategies utilizing lower-fidelity surrogatemodels are relatively new and seem very promising as theycircumvent many of the limitations accompanying responsesurface surrogates and although the lower-fidelity surrogateconcept is slowly making its way into the water resourcesliterature, there are multiple advanced strategies in thebroader research community that are unseen or underutil-ized in the water resources literature we reviewed here(e.g., correction functions and space mapping).
[135] 3. A recent review by Viana and Haftka [2008]argues against applying a single surrogate model andinstead suggests that multiple surrogate models should befit and even used in conjunction with one another. Thebody of multisurrogate model methods they review shouldprove useful in water resources applications.
[136] 4. Building new/revised model analysis specificcomputational efficiency assessment frameworks, similar inconcept to the one proposed in the work of Razavi et al.[2012], for Bayesian model calibration studies, sensitivityanalysis, multiobjective optimization and traditional Monte-Carlo based uncertainty analysis would help formalize meta-model-enabled methodological comparisons. In particular,such frameworks should account for the challenges noted insection 4 associated with the comparison of response surfacesurrogates with lower-fidelity modeling.
[137] Our final direction for future research is the mostimportant one. Studies demonstrating a methodology forvalidation, or perhaps a case study specific proof of con-cept, of the entire metamodel-enabled analysis would beinvaluable. In practical situations, the metamodel-enabledmodeling analysis would not be repeated without a meta-model and the analyst either hopes the metamodel-enabledanalysis yields helpful results (e.g., the list of most sensi-tive model parameters is mostly correct) or provides a bet-ter answer than they started with (e.g., the optimization
W07401 RAZAVI ET AL.: REVIEW W07401
28 of 32
solution is better than the initial solution before any meta-modeling). Such complete uncertainty about the quality oraccuracy of the final analysis result after such a time con-suming procedure does not breed confidence—in particulargiven that the success or failure of the entire metamodel-enabled analysis depends on many subjective decisions, thecomputational budget, the case study original model char-acteristics, the random number generator, etc. ! Imaginesuggesting and defending a worldwide policy decision tocombat climate change on the basis of a single metamodel-enabled analysis without rigorous validation showing theentire procedure could reliably accomplish what it wasintended to do. The real question is how to build relevantcase study specific examples for proof of concept—clearly,this would involve developing lower-fidelity models of thesystem.
[138] Acknowledgments. The authors would like to thank thereviewers, Holger Maier, Uwe Ehret, and the two anonymous reviewers,for their extensive and very helpful comments and suggestions, whichsignificantly improved this review paper.
ReferencesAlam, F. M., K. R. McNaught, and T. J. Ringrose (2004), A comparison of
experimental designs in the development of a neural network simulationmetamodel, Simul. Model. Prac. Theory, 12(7–8), 559–578.
Alexandrov, N. M., and R. M. Lewis (2001), An overview of first-orderapproximation and model management in optimization, Optim. Eng., 2,413–430.
Allaire, D. L. (2009), Uncertainty assessment of complex models withapplication to aviation environmental systems, Mass Inst. of Technol.,Cambridge, Mass.
Aly, A. H., and P. C. Peralta (1999), Comparison of a genetic algorithm andmathematical programming to the design of groundwater cleanup sys-tems, Water Resour. Res., 35(8), 2415–2425.
Audouze, C., F. De Vuyst, and P. B. Nair (2009), Reduced-order modelingof parameterized PDEs using time-space-parameter principal componentanalysis, Int. J. Numer. Methods Eng., 80(8), 1025–1057.
Bakr, M. H., J. W. Bandler, and N. Georgieva (1999), An aggressiveapproach to parameter extraction, IEEE Trans. Microwave Theory Tech.,47(12), 2428–2439.
Bandler, J. W., and K. Madsen (2001), Editorial-surrogate modelling andspace mapping for engineering optimization, Optim. Eng., 2(4), 367–368.
Bandler, J. W., R. M. Biernacki, S. H. Chen, P. A. Grobelny, and R. H.Hemmers (1994), Space mapping technique for electromagnetic optimi-zation, IEEE Trans. Microwave Theory Tech., 42(12), 2536–2544.
Bandler, J. W., R. M. Biernacki, and S. H. Chen (1996), Fully automatedspace mapping optimization of 3D structures, in 1996 IEEE Mtt-S Inter-national Microwave Symposium Digest, vols. 1–3, edited by R. G. Ransonand R. D. Pollard, Inst. of Electr. and Electr. Eng., Washington, D.C.
Bandler, J. W., Q. S. S. Cheng, S. A. Dakroury, A. S. Mohamed, M. H.Bakr, K. Madsen, and J. Sondergaard (2004), Space mapping: The stateof the art, IEEE Trans. Microwave Theory Tech., 52(1), 337–361.
Bandler, J. W., S. Koziel, and K. Madsen (2008), Editorial—Surrogatemodeling and space mapping for engineering optimization, Optim. Eng.,9(4), 307–310.
Bastos, L. S., and A. O’Hagan (2009), Diagnostics for Gaussian processemulators, Technometrics, 51(4), 425–438.
Bau, D. A., and A. S. Mayer (2006), Stochastic management of pump-and-treat strategies using surrogate functions, Adv. Water Resour.,29(12), 1901–1917.
Bautista, D. C. (2009), A sequential design for approximating the paretofront using the expected pareto improvement function, Ph.D. thesis, OhioState Univ., Columbus.
Beale, M. H., M. T. Hagan, and H. B. Demuth (2010), Neural NetworkToolboxTM 7 – User’s Guide, edited, [Available at www.mathworks.com/help/pdf_doc/nnet/nnet.pdf], MathWorks, Natick, Mass.
Behzadian, K., Z. Kapelan, D. Savic, and A. Ardeshir (2009), Stochasticsampling design using a multi-objective genetic algorithm and adaptiveneural networks, Environ. Model. Software, 24(4), 530–541.
Beven, K., and J. Freer (2001), Equifinality, data assimilation, and uncer-tainty estimation in mechanistic modelling of complex environmentalsystems using the GLUE methodology, J. Hydrol., 249(1–4), 11–29.
Bieker, H. P., O. Slupphaug, and T. A. Johansen (2007), Real-time produc-tion optimization of oil and gas production systems: A technology sur-vey, SPE Prod. Oper., 22(4), 382–391.
Blanning, R. W. (1975), Construction and implementation of metamodels,Simulation, 24(6), 177–184.
Blatman, G., and B. Sudret (2010), Efficient computation of global sensitiv-ity indices using sparse polynomial chaos expansions, Reliab. Eng. Syst.Saf., 95(11), 1216–1229.
Bliznyuk, N., D. Ruppert, C. Shoemaker, R. Regis, S. Wild, and P. Mugun-than (2008), Bayesian calibration and uncertainty analysis for computa-tionally expensive models using optimization and radial basis functionapproximation, J. Comput. Graphical Stat., 17(2), 270–294.
Borgonovo, E., W. Castaings, and S. Tarantola (2012), Model emulationand moment-independent sensitivity analysis: An application to environ-mental modelling, Environ. Model. Software, 34, 105–115.
Box, G. E. P., and K. B. Wilson (1951), On the experimental attainment ofoptimum conditions, J. R. Stat. Soc. Ser. B, 13(1), 1–45.
Broad, D. R., G. C. Dandy, and H. R. Maier (2005), Water distribution sys-tem optimization using metamodels, J. Water Resour. Plann. Manage.,131(3), 172–180.
Broad, D. R., H. R. Maier, and G. C. Dandy (2010), Optimal operation ofcomplex water distribution systems using metamodels, J. Water Resour.Plann. Manage., 136(4), 433–443.
Castelletti, A., F. Pianosi, R. Soncini-Sessa, and J. P. Antenucci (2010), Amultiobjective response surface approach for improved water qualityplanning in lakes and reservoirs, Water Resour. Res., 46, W06502,doi:10.1029/2009WR008389.
Chen, V. C. P., K. L. Tsui, R. R. Barton, and M. Meckesheimer (2006), Areview on design, modeling and applications of computer experiments,IIE Trans., 38(4), 273–291.
Cheng, W.-C., M. Putti, D. R. Kendall, and W. W. G. Yeh (2011), A real-time groundwater management model using data assimilation, WaterResour. Res., 47, W06528, doi:10.1029/2010WR009770.
Cherkassky, V., and Y. Q. Ma (2004), Practical selection of SVM parametersand noise estimation for SVM regression, Neural Networks, 17(1), 113–126.
Cieniawski, S. E., J. W. Eheart, and S. Ranjithan (1995), Using geneticalgorithms to solve a multiobjective groundwater monitoring problem,Water Resour. Res., 31(2), 399–409, doi:10.1029/94WR02039.
Conti, S., and A. O’Hagan (2010), Bayesian emulation of complex multi-output and dynamic computer models, J. Stat. Plann. Inference, 140(3),640–651.
Cressie, N. A. C. (1993), Statistics for Spatial Data, John Wiley and Sons,New York.
Crout, N. M. J., D. Tarsitano, and A. T. Wood (2009), Is my model too com-plex? Evaluating model formulation using model reduction, Environ.Modell. Software, 24(1), 1–7.
Cui, T., Fox, C., and M. J. O’Sullivan (2011), Bayesian calibration of alarge scale geothermal reservoir model by a new adaptive delayed accep-tance Metropolis Hastings algorithm, Water Resour. Res., 47, W10521,doi:10.1029/2010WR010352.
de Villiers, J., and E. Barnard (1993), Backpropagation neural nets with oneand two hidden layers, IEEE Trans. Neural Networks, 4(1), 136–141.
di Pierro, F., S. T. Khu, D. Savic, and L. Berardi (2009), Efficient multi-objective optimal design of water distribution networks on a budget ofsimulations using hybrid algorithms, Environ. Modell. Software, 24(2),202–213.
Duan, Q. Y., S. Sorooshian, and V. Gupta (1992), Effective and efficientglobal optimization for conceptual rainfall-runoff models, Water Resour.Res., 28(4), 1015–1031.
Efendiev, Y., A. Datta-Gupta, V. Ginting, X. Ma, and B. Mallick (2005), Anefficient two-stage Markov chain Monte Carlo method for dynamic dataintegration, Water Resour. Res., 41, W12423, doi:10.1029/2004WR003764.
Eldred, M. S., A. A. Giunta, and S. S. Collis (2004), Second-order correctionsfor surrogate-based optimization with model hierarchies, paper presentedat 10th AIAA/ISSMO Multidisciplinary Analysis and Optimization Con-ference, Albany, New York.
Fen, C. S., C. C. Chan, and H. C. Cheng (2009), Assessing a response sur-face-based optimization approach for soil vapor extraction systemdesign, J. Water Resour. Plann. Manage., 135(3), 198–207.
Foresee, F. D., and M. T. Hagan (1997), Gauss-Newton approximation toBayesian Regularization, in Proceedings of the 1997 International Joint
W07401 RAZAVI ET AL.: REVIEW W07401
29 of 32
Conference on Neural Networks, Houston, Texas, pp. 1930–1935, Inst. ofElectr. and Electron. Eng., New York.
Forrester, A. I. J., and A. J. Keane (2009), Recent advances in surrogate-based optimization, Prog. Aerosp. Sci., 45(1–3), 50–79.
Forrester, A. I. J., N. W. Bressloff, and A. J. Keane (2006), Optimizationusing surrogate models and partially converged computational fluid dy-namics simulations, Proc. R. Soc. A, 462(2071), 2177–2204.
Forrester, A. I. J., A. Sobester, and A. J. Keane (2007), Multi-fidelity optimi-zation via surrogate modelling, Proc. R. Soc. A, 463(2088), 3251–3269.
Fricker, T., J. Oakley, and N. M. Urban (2010), Multivariate emulators withnonseparable covariance structures, Tech. Rep. 10/06, Joint Res. Proj. onManag. Uncertainty in Complex Models, Exeter, UK.
Friedman, J. H. (1991), Multivariate adaptive regression splines, Ann. Stat.,19(1), 1–67.
Gano, S. E., J. E. Renaud, J. D. Martin, and T. W. Simpson (2006), Updatestrategies for kriging models used in variable fidelity optimization,Struct. Multidiscip. Optim., 32(4), 287–298.
Gibbs, M. S., H. R. Maier, and G. C. Dandy (2011), Relationship betweenproblem characteristics and the optimal number of genetic algorithmgenerations, Eng. Optim., 43(4), 349–376.
Ginsbourger, D., R. Riche, and L. Carraro (2010), Kriging is well-suited toparallelize optimization, Comput. Intell. Expensive Optim. Probl., 2,131–162.
Gorissen, D. (2007), Heterogeneous evolution of surrogate models, Mas-ter’s thesis, Katholieke Univ. Leuven, Leuven, Netherlands.
Gramacy, R. B., and H. K. H. Lee (2008), Bayesian treed gaussian processmodels with an application to computer modeling, J. Am. Stat. Assoc.,103(483), 1119–1130.
Gu, C. (2002), Smoothing Spline ANOVA Models, Springer, New York,USA.
Gugercin, S., and A. C. Antoulas (2000), A comparative study of 7 algo-rithms for model reduction, in Proceedings of the 39th IEEE Conferenceon Decision and Control, Vols. 1–5, pp. 2367–2372, Inst. of Electr. andElectr. Eng., Sydney, NSW, Australia.
Gutmann, H. M. (2001), A radial basis function method for global optimi-zation, J. Global Optim., 19(3), 201–227.
Hamm, L., B. W. Brorsen, and M. T. Hagan (2007), Comparison of stochas-tic global optimization methods to estimate neural network weights, Neu-ral Process. Lett., 26, 145–158.
Hemker, T., K. R. Fowler, M. W. Farthing, and O. von Stryk (2008), Amixed-integer simulation-based optimization approach with surrogatefunctions in water resources management, Optim. Eng., 9(4), 341–360.
Hornberger, G. M., and R. C. Spear (1981), An approach to the prelimi-nary-analysis of environmental systems, J. Environ. Manage., 12(1),7–18.
Hornik, K., M. Stinchcombe, and H. White (1989), Multilayer feedforwardnetworks are universal approximators, Neural Networks, 2(5), 359–366.
Huang, D., T. T. Allen, W. I. Notz, and R. A. Miller (2006), Sequentialkriging optimization using multiple-fidelity evaluations, Struct. Multidis-cip. Optim., 32(5), 369–382.
Hussain, M. F., R. R. Barton, and S. B. Joshi (2002), Metamodeling: Radialbasis functions, versus polynomials, Eur. J. Oper. Res., 138(1), 142–154.
Jeong, S., and S. Obayashi (2005), Efficient global optimization (EGO) formulti-objective problem and data mining, The 2005 IEEE Congress onEvolutionary Computation, vol. 3, pp. 2138–2145, doi:10.1109/CEC.2005.1554959.
Jin, R., W. Chen, and T. W. Simpson (2001), Comparative studies of meta-modelling techniques under multiple modelling criteria, Struct. Multidis-cip.Optim., 23(1), 1–13.
Jin, Y. (2005), A comprehensive survey of fitness approximation in evolu-tionary computation, Soft Comput., 9(1), 3–12.
Jin, Y. C., M. Olhofer, and B. Sendhoff (2002), A framework for evolution-ary optimization with approximate fitness functions, IEEE Trans. Evol.Comput., 6(5), 481–494.
Johnson, V. M., and L. L. Rogers (2000), Accuracy of neural networkapproximators in simulation-optimization, J. Water Resour. Plann.Manage., 126(2), 48–56.
Jones, D. R. (2001), A taxonomy of global optimization methods based onresponse surfaces, J. Global Optim., 21(4), 345–383.
Jones, D. R., M. Schonlau, and W. J. Welch (1998), Efficient global opti-mization of expensive black-box functions, J. Global Optim., 13(4),455–492.
Jourdan, L., D. W. Corne, D. A. Savic, and G. Walters (2006), LEMMO:Hybridising rule induction and NSGA II for multi-objective water sys-tems design, in Proceedings of the Eighth International Conference on
Computing and Control for the Water Industry, vol. 2, pp. 45–50, Univer-sity of Exeter, Exeter, UK.
Kavetski, D., G. Kuczera, and S. W. Franks (2006), Bayesian analysis ofinput uncertainty in hydrological modeling: 1. Theory, Water Resour.Res., 42, W03407, doi:10.1029/2005WR004368.
Keane, A. J. (2006), Statistical improvement criteria for use in multiobjec-tive design optimization, AIAA J., 44(4), 879–891.
Keating, E. H., J. Doherty, J. A. Vrugt, and Q. J. Kang (2010), Optimizationand uncertainty assessment of strongly nonlinear groundwater modelswith high parameter dimensionality, Water Resour. Res., 46, W10517,doi:10.1029/2009WR008584.
Kennedy, M. C., and A. O’Hagan (2000), Predicting the output from a com-plex computer code when fast approximations are available, Biometrika,87(1), 1–13.
Kennedy, M. C., and A. O’Hagan (2001), Bayesian calibration of computermodels, J. R. Stat. Soc. Ser., 63, 425–450.
Khu, S. T., and M. G. F. Werner (2003), Reduction of Monte-Carlo simula-tion runs for uncertainty estimation in hydrological modelling, Hydrol.Earth Syst. Sci., 7(5), 680–692.
Khu, S. T., D. Savic, Y. Liu, and H. Madsen (2004), A fast evolutionary-based metamodelling approach for the calibration of a rainfall-runoffmodel, paper presented at The First Biennial Meeting of the InternationalEnvironmental Modelling Software Society, Osnabruck, Germany.
Kim, C., S. Wang, and K. K. Choi (2005), Efficient response surface model-ing by using moving least-squares method and sensitivity, AIAA J.,43(11), 2404–2411.
Kim, D. W., and J. Lee (2010), An improvement of Kriging based sequen-tial approximate optimization method via extended use of design ofexperiments, Eng. Optim., 42(12), 1133–1149.
Kingston, G. B., M. F. Lambert, and H. R. Maier (2005), Bayesian trainingof artificial neural networks used for water resources modeling, WaterResour. Res., 41, W12409, doi:10.1029/2005WR004152.
Kingston, G. B., H. R. Maier, and M. F. Lambert (2008), Bayesian modelselection applied to artificial neural networks used for water resourcesmodeling, Water Resour. Res., 44, W04419, doi:10.1029/2007WR006155
Kleijnen, J. P. C. (2009), Kriging metamodeling in simulation: A review,Eur. J. Oper. Res., 192(3), 707–716.
Knowles, J. (2006), ParEGO: A hybrid algorithm with on-line landscapeapproximation for expensive multiobjective optimization problems,IEEE Trans. Evol. Comput., 10(1), 50–66.
Koch, P. N., T. W. Simpson, J. K. Allen, and F. Mistree (1999), Statisticalapproximations for multidisciplinary design optimization: The problemof size, J. Aircraft, 36(1), 275–286.
Kourakos, G., and A. Mantoglou (2009), Pumping optimization of coastalaquifers based on evolutionary algorithms and surrogate modular neuralnetwork models, Adv. Water Resour., 32(4), 507–521.
Leary, S. J., A. Bhaskar, and A. J. Keane (2003), A knowledge-basedapproach to response surface modelling in multifidelity optimization, J.Global Optim., 26(3), 297–319.
Lee, J., H. Jeong, D.-H. Choi, V. Volovoi, and D. Mavris (2007), Anenhancement of constraint feasibility in BPN based approximate opti-mization, Comput. Methods Appl. Mech. Eng., 196(17–20), 2147–2160.
Leshno, M., V. Y. Lin, A. Pinkus, and S. Schocken (1993), Multilayer feed-forward networks with a non-polynomial activation function can approx-imate any function, Neural Networks, 6(6), 861–867.
Li, M., G. Li, and S. Azarm (2008), A kriging metamodel assisted multi-objective genetic algorithm for design optimization, J. Mech. Des., 130,031401.
Liong, S. Y., S. T. Khu, and W. T. Chan (2001), Derivation of Paretofront with genetic algorithm and neural network, J. Hydrol. Eng., 6(1),52–61.
Liu, W. (2003), Development of gradient-enhanced kriging approximationsfor multidisciplinary design optimization, 177 pp., Univ. of Notre Dame,Notre Dame, Indiana.
Madsen, J. I., and M. Langthjem (2001), Multifidelity response surfaceapproximations for the optimum design of diffuser flows, Optim. Eng., 2,453–468.
Maier, H. R., and G. C. Dandy (2000), Neural networks for the predictionand forecasting of water resources variables: A review of modellingissues and applications, Environ. Model. Software, 15(1), 101–124.
Maier, H. R., A. Jain, G. C. Dandy, and K. P. Sudheer (2010), Methodsused for the development of neural networks for the prediction of waterresource variables in river systems: Current status and future directions,Environ. Model. Software, 25(8), 891–909.
W07401 RAZAVI ET AL.: REVIEW W07401
30 of 32
Maschler, T., and D. A. Savic (1999), Simplification of water supply net-work models through linearisation, Cent. Water Syst. Rep. 99/01, 119pp., Sch. of Eng., Univ. of Exeter, Exeter, U. K.
McKay, M. D., R. J. Beckman, and W. J. Conover (1979), A comparison ofthree methods for selecting values of input variables in the analysis ofoutput from a computer code, Technometrics, 21(2), 239–245.
McKinney, D. C., and M. D. Lin (1994), Genetic algorithm solution ofgroundwater-management models, Water Resour. Res., 30(6), 1897–1906.
McPhee, J., and W. W. G. Yeh (2008), Groundwater management usingmodel reduction via empirical orthogonal functions, J. Water Resour.Plann. Manage., 134(2), 161–170.
Melching, C. S., B. C. Yen, and H. G. Wenzel (1990), A reliability estima-tion in modeling watershed runoff with uncertainties, Water Resour.Res., 26(10), 2275–2286.
Michalski, R. S. (2000), Learnable Evolution ModeL: Evolutionary proc-esses guided by machine learning, Mach. Learn., 38(1–2), 9–40.
Mondal, A., Y. Efendiev, B. Mallick, and A. Datta-Gupta (2010), Bayesianuncertainty quantification for flows in heterogeneous porous media usingreversible jump Markov chain Monte Carlo methods, Adv. WaterResour., 33(3), 241–256.
Montgomery, D. C. (2008), Design and Analysis of Experiments, 7th ed.,Wiley, New York.
Mugunthan, P., and C. A. Shoemaker (2006), Assessing the impacts of pa-rameter uncertainty for computationally expensive groundwater models,Water Resour. Res., 42, W10428, doi:10.1029/2005WR004640.
Mugunthan, P., C. A. Shoemaker, and R. G. Regis (2005), Comparison offunction approximation, heuristic, and derivative-based methods forautomatic calibration of computationally expensive groundwater biore-mediation models, Water Resour. Res., 41, W11427, doi:10.1029/2005WR004134.
Mullur, A. A., and A. Messac (2006), Metamodeling using extended radialbasis functions: A comparative approach, Eng. Comput., 21(3), 203–217.
Myers, R. H., and D. C. Montgomery (2002), Response Surface Methodol-ogy, 2nd ed., Wiley-Interscience, New York.
Nakayama, H., M. Arakawa, and R. Sasaki (2002), Simulation-based opti-mization using computational intelligence, Optim. Eng., 3(2), 201–214.
O’Hagan, A. (2006), Bayesian analysis of computer code outputs: A tuto-rial, Reliab. Eng. Syst. Saf., 91(10–11), 1290–1300.
Ostfeld, A., and S. Salomons (2005), A hybrid genetic-instance based learningalgorithm for CE-QUAL-W2 calibration, J. Hydrol., 310(1–4), 122–142.
Papadrakakis, M., N. D. Lagaros, and Y. Tsompanakis (1998), Structuraloptimization using evolution strategies and neural networks, Comput.Methods Appl. Mech. Eng., 156(1–4), 309–333.
Pianosi, F., and R. Soncini-Sessa (2009), Real-time management of a multi-purpose water reservoir with a heteroscedastic inflow model, WaterResour. Res., 45, W10430, doi:10.1029/2008WR007335.
Picheny, V., N.-H. Kim, R. T. Haftka, and N. V. Queipo (2008), Conserva-tive predictions using surrogate modeling, in 49th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference,Schaumburg, IL, AIAA Paper 2008-1716, Am. Inst. of Aeronaut. andAstronaut., Reston, VA.
Ponweiser, W., T. Wagner, D. Biermann, and M. Vincze (2008), Multiobjec-tive optimization on a limited budget of evaluations using model-assistedS-metric selection, in Parallel Problem Solving from Nature - Ppsn X,Proceedings, edited by G. Rudolph et al., pp. 784–794, Springer, Berlin.
Preis, A., A. J. Whittle, A. Ostfeld, and L. Perelman (2011), Efficient hy-draulic state estimation technique using reduced models of urban waternetworks, J. Water Resour. Plann. Manage., 137(4), 343–351.
Rabitz, H., O. F. Alis, J. Shorter, and K. Shim (1999), Efficient input-outputmodel representations, Comput. Phys. Commun., 117(1–2), 11–20.
Ratto, M., and A. Pagano (2010), Using recursive algorithms for the effi-cient identification of smoothing spline ANOVA models, Adv. Stat.Anal., 94(4), 367–388.
Ratto, M., A. Pagano, and P. Young (2007), State dependent parameter meta-modelling and sensitivity analysis, Comput. Phys. Commun., 177(11),863–876.
Ratto, M., A. Castelletti, and A. Pagano (2012), Editorial - Emulation tech-niques for the reduction and sensitivity analysis of complex environmen-tal models, Environ. Modell. Software, 34(0), 1–4.
Razavi, S., and B. A. Tolson (2011), A new formulation for feedforwardneural networks, IEEE Trans. Neural Networks, 22(10), 1588–1598,doi:1510.1109/TNN.2011.2163169.
Razavi, S., B. A. Tolson, and D. H. Burn (2012), Numerical assessment ofmetamodelling strategies in computationally intensive optimization,Environ. Modell. Software, 34, 67–86
Reed, P., B. S. Minsker, and D. E. Goldberg (2003), Simplifying multiob-jective optimization: An automated design methodology for the nondo-minated sorted genetic algorithm-II, Water Resour. Res., 39(7), 1196,doi:10.1029/2002WR001483.
Reed, R. (1993), Pruning algorithms—A survey, IEEE Trans. NeuralNetworks, 4(5), 740–747.
Regis, R. G., and C. A. Shoemaker (2004), Local function approximation inevolutionary algorithms for the optimization of costly functions, IEEETrans. Evol. Comput., 8(5), 490–505.
Regis, R. G., and C. A. Shoemaker (2007a), A stochastic radial basis func-tion method for the global optimization of expensive functions,INFORMS J. Comput., 19(4), 497–509.
Regis, R. G., and C. A. Shoemaker (2007b), Improved strategies for radialbasis function methods for global optimization, J. Global Optim., 37(1),113–135.
Regis, R. G., and C. A. Shoemaker (2009), Parallel stochastic global opti-mization using radial basis functions, INFORMS J. Comput., 21(3), 411–426.
Reichard, E. G. (1995), Groundwater-surface water management with sto-chastic surface-water supplies—A simulation optimization approach,Water Resour. Res., 31(11), 2845–2865.
Rewienski, M., and J. White (2006), Model order reduction for nonlineardynamical systems based on trajectory piecewise-linear approximations,Linear Algebra Appl., 415(2–3), 426–454.
Robinson, T. D. (2007), Surrogate-based Optimization Using Multifi-delity Models with Variable Parameterization, Mass. Inst. Technol.,Cambridge.
Robinson, T. D., M. S. Eldred, K. E. Willcox, and R. Haimes (2006), Strat-egies for multifidelity optimization with variable dimensional hierarchi-cal models, in 47th AIAA/ASME/ASCE/AHS/ASC Structures, StructuralDynamics, and Materials Conference, AIAA Pap. 2006-1819, Am. Inst.of Aeronaut. and Astronaut., Reston, VA.
Robinson, T. D., M. S. Eldred, K. E. Willcox, and R. Haimes (2008), Surro-gate-based optimization using multifidelity models with variable param-eterization and corrected space mapping, AIAA J., 46(11), 2814–2822.
Ryu, J. S., M. S. Kim, K. J. Cha, T. H. Lee, and D. H. Choi (2002), Kriginginterpolation methods in geostatistics and DACE model, KSME Int. J.,16(5), 619–632.
Sacks, J., W. J. Welch, T. J. Mitchell, and H. P. Wynn (1989), Design andanalysis of computer experiments, Stat. Sci., 4(4), 409–435.
Sakata, S., F. Ashida, and M. Zako (2003), Structural optimization usingKriging approximation, Comput. Methods Appl. Mech. Eng., 192(7–8),923–939.
Saltelli, A., S. Tarantola, and F. Campolongo (2000), Sensitivity analysis asan ingredient of modeling, Stat. Sci., 15(4), 377–395.
Sasena, M. J., P. Papalambros, and P. Goovaerts (2002), Exploration ofmetamodeling sampling criteria for constrained global optimization,Eng. Optim., 34(3), 263–278.
Savic, D. A., and G. A. Walters (1997), Genetic algorithms for least-costdesign of water distribution networks, J. Water Resour. Plann. Manage.,123(2), 67–77.
Schonlau, M. (1997), Computer experiments and global optimization,Ph.D. thesis, Univ. of Waterloo, Waterloo, Canada.
Schultz, M. T., M. J. Small, R. S. Farrow, and P. S. Fischbeck (2004), Statewater pollution control policy insights from a reduced-form model, J.Water Resour. Plann. Manage., 130(2), 150–159.
Schultz, M. T., M. J. Small, P. S. Fischbeck, and R. S. Farrow (2006), Eval-uating response surface designs for uncertainty analysis and prescriptiveapplications of a large-scale water quality model, Environ. Model.Assess., 11(4), 345–359.
Shamir, U., and E. Salomons (2008), Optimal real-time operation of urbanwater distribution systems using reduced models, J. Water Resour.Plann. Manage.,, 134(2), 181–185.
Shan, S., and G. G. Wang (2004), Space exploration and global optimiza-tion for computationally intensive design problems: A rough set basedapproach, Struct. Multidiscip.Optim., 28(6), 427–441.
Shan, S., and G. G. Wang (2010a), Metamodeling for high dimensionalsimulation-based design problems, J. Mech. Des., 132(5), 051009.
Shan, S., and G. G. Wang (2010b), Survey of modeling and optimizationstrategies to solve high-dimensional design problems with computation-ally-expensive black-box functions, Struct. Multidiscip.Optim., 41(2),219–241.
Shin, Y. S., and R. V. Grandhi (2001), A global structural optimizationtechnique using an interval method, Struct. Multidiscip.Optim., 22(5),351–363.
W07401 RAZAVI ET AL.: REVIEW W07401
31 of 32
Shrestha, D. L., N. Kayastha, and D. P. Solomatine (2009), A novelapproach to parameter uncertainty analysis of hydrological models usingneural networks, Hydrol. Earth Syst. Sci., 13(7), 1235–1248.
Siade, A. J., M. Putti, and W. W. G. Yeh (2010), Snapshot selection forgroundwater model reduction using proper orthogonal decomposition,Water Resour. Res., 46, W08539, doi:10.1029/2009WR008792.
Simpson, T. W., and F. Mistree (2001), Kriging models for global approxi-mation in simulation-based multidisciplinary design optimization, AIAAJ., 39(12), 2233–2241.
Simpson, T. W., J. D. Peplinski, P. N. Koch, and J. K. Allen (2001), Meta-models for computer-based engineering design: Survey and recommen-dations, Eng. Comput., 17(2), 129–150.
Simpson, T. W., A. J. Booker, D. Ghosh, A. A. Giunta, P. N. Koch, andR. J. Yang (2004), Approximation methods in multidisciplinary analysisand optimization: A panel discussion, Struct. Multidiscip.Optim., 27(5),302–313.
Simpson, T. W., V. V. Toropov, V. Balabanov, and F. A. C. Viana (2008),Design and analysis of computer experiments in multidisciplinary designoptimization: A review of how far we have come—or not, paper pre-sented at 12th AIAA/ISSMO multidisciplinary analysis and optimizationconference, Victoria, B. C., Canada.
Skaggs, T. H., and D. A. Barry (1997), The first-order reliability method ofpredicting cumulative mass flux in heterogeneous porous formations,Water Resour. Res., 33(6), 1485–1494.
Sobester, A., S. J. Leary, and A. J. Keane (2005), On the design of optimiza-tion strategies based on global response surface approximation models,J. Global Optim., 33(1), 31–59.
Sobol, I. M. (2003), Theorems and examples on high dimensional modelrepresentation, Reliab. Eng. Syst. Saf., 79(2), 187–193.
Sreekanth, J., and B. Datta (2011), Coupled simulation-optimization modelfor coastal aquifer management using genetic programming-based en-semble surrogate models and multiple-realization optimization, WaterResour. Res., 47, W04516, doi:10.1029/2010WR009683.
Storlie, C. B., L. P. Swiler, J. C. Helton, and C. J. Sallaberry (2009), Imple-mentation and evaluation of nonparametric regression procedures forsensitivity analysis of computationally demanding models, Reliab. Eng.Syst. Saf., 94(11), 1735–1763.
Storlie, C. B., H. D. Bondell, B. J. Reich, and H. H. Zhang (2011), Surfaceestimation, variable selection, and the nonparametric oracle property,Stat. Sin., 21(2), 679–705.
Sun, G., G. Li, M. Stone, and Q. Li (2010), A two-stage multi-fidelity opti-mization procedure for honeycomb-type cellular materials, Comput.Mater. Sci., 49(3), 500–511.
Svenson, J. D. (2011), Computer experiments: Multiobjective optimizationand sensitivity analysis, Ph.D. thesis, Ohio State Univ., Columbus.
Tamura, S., and M. Tateishi (1997), Capabilities of a four-layered feedfor-ward neural network: Four layers versus three, IEEE Trans. Neural Net-works, 8(2), 251–255.
Teoh, E. J., K. C. Tan, and C. Xiang (2006), Estimating the number of hid-den neurons in a feedforward network using the singular value decompo-sition, IEEE Trans. Neural Networks, 17(6), 1623–1629.
Thokala, P., and J. R. R. A. Martins (2007), Variable-complexity optimiza-tion applied to airfoil design, Eng. Optim., 39(3), 271–286.
Ulanicki, B., A. Zehnpfund, and F. Martinez (1996), Simplification of waterdistribution network models, in Proceedings of the 2nd International Confer-ence on Hydroinformatics, pp. 493–500, Balkema, Rotterdam, Netherlands.
van Keulen, F., and K. Vervenne (2004), Gradient-enhanced response sur-face building, Struct. Multidiscip.Optim., 27(5), 337–351.
Vermeulen, P. T. M., A. W. Heemink, and C. Stroet (2004), Reduced mod-els for linear groundwater flow models using empirical orthogonal func-tions, Adv. Water Resour., 27(1), 57–69.
Vermeulen, P. T. M., A. W. Heemink, and J. R. Valstar (2005), Inversemodeling of groundwater flow using model reduction, Water Resour.Res., 41, W06003, doi:10.1029/2004WR003698.
Vermeulen, P. T. M., C. B. M. T. Stroet, and A. W. Heemink (2006), Modelinversion of transient nonlinear groundwater flow models using modelreduction, Water Resour. Res., 42, W09417, doi:10.1029/2005WR004536.
Viana, F. A. C., and R. T. Haftka (2008), Using multiple surrogates formetamodeling, in Proceedings of the 7th ASMO-UK/ISSMO Interna-tional Conference on Engineering Design Optimization, edited by V. V.Toropov et al., pp. 1–18, Int. Soc. for Struct. And Multidiscip. Optim.,Daejeon, South Korea.
Viana, F. A. C., V. Steffen, Jr., S. Butkewitsch, and M. D. F. Leal (2009), Opti-mization of aircraft structural components by using nature-inspired algo-rithms and multi-fidelity approximations, J. Global Optim., 45(3), 427–449.
Viana, F. A. C., V. Picheny, and R. T. Haftka (2010), Using cross validationto design conservative surrogates, AIAA J., 48(10), 2286–2298.
Vila, J. P., V. Wagner, and P. Neveu (2000), Bayesian nonlinear modelselection and neural networks: A conjugate prior approach, IEEE Trans.Neural Networks, 11(2), 265–278.
Vitali, R., R. T. Haftka, and B. V. Sankar (2002), Multi-fidelity design ofstiffened composite panel with a crack, Struct. Multidiscip.Optim., 23(5),347–356.
Vrugt, J. A., C. J. F. ter Braak, C. G. H. Diks, B. A. Robinson, J. M. Hyman,and D. Higdon (2009), Accelerating Markov Chain Monte Carlo simula-tion by differential evolution with self-adaptive randomized subspacesampling, Int. J. Nonlinear Sci. Numer. Simul., 10(3), 273–290.
Wagner, T., M. Emmerich, A. Deutz, and W. Ponweiser (2010), Onexpected-improvement criteria for model-based multi-objective optimi-zation, in Parallel Problems Solving from Nature - Ppsn Xi, Pt I, editedby R. Schaefer et al., pp. 718–727, Springer, Berlin.
Wahba, G. (1990), Spline Models for Observational Data, Soc. for Indust.and Appl. Math., Philadelphia, Pa.
Wang, G. G. (2003), Adaptive response surface method using inheritedLatin hypercube design points, J. Mech. Des., 125(2), 210–220.
Wang, G. G., and S. Shan (2007), Review of metamodeling techniquesin support of engineering design optimization, J. Mech. Des., 129(4),370–380.
Wang, L. Q., S. Q. Shan, and G. G. Wang (2004), Mode-pursuing samplingmethod for global optimization on expensive black-box functions, Eng.Optim., 36(4), 419–438.
Wang, Q. J. (1991), The genetic algorithm and its application to calibratingconceptual rainfall-runoff models, Water Resour. Res., 27(9), 2467–2471.
Watson, A. G., and R. J. Barnes (1995), Infill sampling criteria to lacateextremes, Math. Geol., 27(5), 589–608.
Welch, W. J., R. J. Buck, J. Sacks, H. P. Wynn, T. J. Mitchell, and M. D.Morris (1992), Screening, prediction, and computer experiments, Tech-nometrics, 34(1), 15–25.
Willcox, K., and A. Megretski (2005), Fourier series for accurate, stable,reduced-order models in large-scale linear applications, SIAM J. Sci.Comput., 26(3), 944–962.
Xiang, C., S. Q. Ding, and T. H. Lee (2005), Geometrical interpretationand architecture selection of MLP, IEEE Trans. Neural Networks,16(1), 84–96.
Xu, Y., K. W. Wong, and C. S. Leung (2006), Generalized RLS approachto the training of neural networks, IEEE Trans. Neural Networks, 17(1),19–34.
Yan, S., and B. Minsker (2006), Optimal groundwater remediation designusing an Adaptive Neural Network Genetic Algorithm, Water Resour.Res., 42, W05407, doi:10.1029/2005WR004303.
Yan, S., and B. Minsker (2011), Applying dynamic surrogate models innoisy genetic algorithms to optimize groundwater remediation designs,J. Water Resour. Plann. Manage., 137(3), 284–292.
Ye, K. Q., W. Li, and A. Sudjianto (2000), Algorithmic construction ofoptimal symmetric Latin hypercube designs, J. Stat. Plann. Inference,90(1), 145–159.
Young, P. C., and M. Ratto (2009), A unified approach to environmentalsystems modeling, Stochastic Environ. Res. Risk Assess., 23(7), 1037–1057.
Yu, P. S., S. T. Chen, and I. F. Chang (2006), Support vector regression forreal-time flood stage forecasting, J. Hydrol., 328(3–4), 704–716.
Zhang, Q., W. Liu, E. Tsang, and B. Virginas (2010), Expensive multiob-jective optimization by MOEA/D with Gaussian process model, IEEETrans. Evol. Comput., 14(3), 456–474.
Zhang, X. S., R. Srinivasan, and M. Van Liew (2009), ApproximatingSWAT model using artificial neural network and support vector machine,J. Am. Water Resour. Assoc., 45(2), 460–474.
Zou, R., W. S. Lung, and J. Wu (2007), An adaptive neural network embed-ded genetic algorithm approach for inverse water quality modeling,Water Resour. Res., 43, W08427, doi:10.1029/2006WR005158.
Zou, R., W. S. Lung, and J. Wu (2009), Multiple-pattern parameter identifi-cation and uncertainty analysis approach for water quality modeling,Ecol. Modell., 220(5), 621–629.
W07401 RAZAVI ET AL.: REVIEW W07401
32 of 32
Eos, Vol. 93, No. 40, 2 October 2012
© 2012. American Geophysical Union. All Rights Reserved.
Summarizing the field of surrogate modeling research
As computer simulations of complex phys-ical interactions grow, so do the time and expense required to operate them. Mirroring the development of such full- scale models has been the related field of surrogate mod-eling or metamodeling. Surrogate models take a variety of forms, but their shared goal is to provide a numerical output similar to that of a fully complex physical model while minimizing the computational time and cost required to calculate the result. With an eye toward introducing those unfamiliar with the practices and pitfalls of surrogate modeling to the topic and with a focus on its applica-tions to water resources research, Razavi et al. prepared a systematic review of the surrogate modeling literature.
The authors found that surrogate model-ing breaks down into two main categories: response surface models and low- fidelity models. A response surface model is a data- driven statistical emulator of the parent research model, such that rather than try-ing to model the physical system itself, the surrogate is designed to predict the parent model output. The authors describe a wide variety of techniques used for statistical
modeling, such as kriging or artificial neu-ral networks. They note that statistical emu-lators work best for systems with a minimal number of explanatory variables to take into consideration.
The other main class, low- fidelity mod-els, often referred to as “coarse” models, includes those designed with lower spatial or temporal resolution, lower dimensional-ity or simplified boundary conditions, or reduced mathematical order (simplified mathematics). Models can also be designed such that they ignore known physical pro-cesses, a technique the authors suggest is the least promising path in low- fidelity modeling. In making low- fidelity models, developers knowingly simplify the physi-cal system in favor of model efficiency and a reduced run time, a trade- off that can be valuable for those using the model opera-tionally. (Water Resources Research, doi: 10.1029/2011WR011527, 2012) —CS
Sea salt particles react with organic acids in atmosphere
Sea salt, or sodium chloride (NaCl), parti-cles lofted into the atmosphere by the motion of ocean waves affect atmospheric chemistry; these particles can undergo reactions with
trace atmospheric gases and internal mixing with anthropogenic pollutants depositing on particle surface. Several studies have found that NaCl particles in the atmosphere are depleted in chloride and have attributed this to reactions with inorganic acids. However, reactions with inorganic acids do not fully account for the observed chloride depletion in some locations; it has been suggested that organic acids, likely of anthropogenic origin, may also play a role in chloride depletion, but results have been uncertain.
To address the issue, Laskin et al. con-ducted a chemical imaging analysis of air-borne particles collected onboard aircraft during a flight in California’s Central Valley in 2010. They found that sea salt that had mixed with organic matter particles was depleted in chloride, suggesting that sea salt had reacted with anthropogenic organic acids, releas-ing volatile hydrogen chloride (HCl) into the atmosphere and leaving behind chloride- depleted sea salt particles. Corroborating the field evidence, the authors conducted sys-tematic laboratory studies in which NaCl par-ticles reacted with organic acids, resulting in chloride depletion and formation of organic salts. The authors conclude that these reac-tions, which are unique for aerosolized par-ticles, were previously overlooked and should now be included in atmospheric chemistry models. (Journal of Geophysical Research- Atmospheres, doi:10.1029/2012JD017743, 2012) —EB
Large melt channels discovered underneath Antarctic ice shelves
New radar observations reveal melt chan-nels 500 meters to 3 kilometers wide and up to 200 meters deep underneath the ice shelf buttressing the Pine Island Glacier in West Antarctica; researchers suggest that these subglacial channels could be a “prelude to eventual collapse” of the ice shelves, which cover about 10% of the areal extent of the West Antarctic Ice Sheet (WAIS). A complete
In spite of a large body of existing mea-surements of incoming short-wave solar radiation and outgoing long-wave terrestrial radiation at the surface of the Earth and, more recently, in the upper atmosphere, there are few observations documenting how radiation profiles change through the atmosphere—information that is necessary to fully quantify the greenhouse effect of Earth’s atmosphere.
Through the use of existing technology but employing improvements in observa-tional techniques it may now be possible not only to quantify but also to understand how different components of the atmo-sphere (e.g., concentration of gases, cloud cover, moisture, and aerosols) contribute to the greenhouse effect.
Using weather balloons equipped with radiosondes, Philipona et al. continuously measured radiation fluxes from the sur-face of Earth up to altitudes of 35 kilome-ters in the upper stratosphere. Combining data from flights conducted during both day and night with continuous 24-hour measure-ments made at the surface of the Earth, the researchers created radiation profiles of all four components necessary to fully cap-ture the radiation budget of Earth, namely, the upward and downward short-wave and
long-wave radiation as a function of altitude. Comparing radiation profiles between differ-ent nights in cloud-free conditions, they dem-onstrated evidence of large greenhouse forc-ing toward the Earth. They show that water vapor in the atmosphere, particularly below the tropopause (~15 kilometers altitude), increases long-wave downward radiative forcing, which accounts for increased heat-ing of Earth’s surface, by 8 watts per square meter per Kelvin. (Geophysical Research Let-ters, doi:10.1029/2012GL052087, 2012) —AB
New measurements quantify atmospheric greenhouse effect
Radiometer radiosonde package used to measure radiation fluxes.
PAGE 400
Maps of particles detected in air samples after 14–25 hours of atmospheric transport time. Blue indicates areas of original sea salt, or sodium chloride (NaCl), constituents; green is organic material that reacts with NaCl.
Eos, Vol. 93, No. 40, 2 October 2012
© 2012. American Geophysical Union. All Rights Reserved.
collapse of WAIS could increase global sea level by at least 3–4 meters.
Taking advantage of a dense network of radar lines designed to penetrate ice thick-nesses of more than 2 kilometers, Vaughan et al. combine indirect satellite observations with direct echo- sounding measurements from submersibles to show what scientists have suspected: Thinning ice shelves lead to crevassing of the ice shelves. Echo sound-ing reveals 50- to 100- meter- wide crevasses aligned with the subglacial channels; the basal crevasses penetrate up to one third of the ice thickness in the shelves buttressing the Pine Island Glacier, even leaving their mark on the surface of the glacier. Melting at the base of floating ice shelves creates subglacial channels. Using stress models, the authors show that when the subglacial
channels form sufficiently rapidly, they cre-ate crevasses, which the authors found on the surface and at the base of the ice shelf bordering the Pine Island Glacier.
The ice shelf buttressing the Pine Island Glacier has been thinning for several decades and is now undergoing basal melt-ing at an accelerated rate. This ice shelf basal melting, along with increased deliv-ery of warm ocean water into the sub- ice- shelf cavity, may cause thinning and struc-tural weakening of the ice shelf and eventual collapse of the Pine Island Glacier. (Jour-nal of Geophysical Research- Earth Surface, doi:10.1029/ 2012JF002360, 2012) —AB
Low calcification in corals in the Great Barrier Reef
Reef-building coral communities in the Great Barrier Reef—the world’s largest coral reef—may now be calcifying at only about half the rate that they did during the 1970s, even though live coral cover may not have changed over the past 40 years, a new study finds. In recent decades, coral reefs around the world, home to large numbers of fish and other marine species, have been threat-ened by such human activities as pollution, overfishing, global warming, and ocean acidification; the latter affects ambient water chemistry and availability of calcium ions, which are critical for coral communities to calcify, build, and maintain reefs.
Comparing data from reef surveys dur-ing the 1970s, 1980s, and 1990s with pres-ent-day (2009) measurements of calcifica-tion rates in One Tree Island, a coral reef covering 13 square kilometers in the south-ern part of the Great Barrier Reef, Silver-man et al. show that the total calcification rates (the rate of calcification minus the rate of dissolution) in these coral commu-nities have decreased by 44% over the past 40 years; the decrease appears to stem from a threefold reduction in calcification rates during nighttime.
The authors suggest that reduced cal-cification, particularly during nighttime, could be caused by boring and other erosional activities brought about by a change in the type or the number of bor-ing organisms (e.g., sea cucumbers) that live on these coral reefs. Another possi-bility is that warming water temperatures and ocean acidification could have caused increased dissolution of coral reefs over the past 40 years. The study is consistent with previous estimates and predictions of reduced calcification rates of coral com-munities in the Great Barrier Reef. (Journal of Geophysical Research- Biogeosciences, doi:10.1029/2012JG001974, 2012) —AB
—ErniE BalcErak, Staff Writer, and atrEyEE Bhattacharya and colin Schultz, WritersCrevassing in the Pine Island Glacier sea shelf.
Aid
an O
’Har
e/B
AS