Surrogate-assisted parallel tempering for Bayesian neural learning

Rohitash Chandraa, Konark Jaind,b, Arpit Kapoorb,c, Ashray Amanb,e

aSchool of Mathematics and Statistics, University of New South Wales, Sydney, NSW 2052, AustraliabCentre for Translational Data Science, The University of Sydney, NSW 2006, Australia

cDepartment of Computer Science and Engineering, SRM Institute of Science and Technology, Chennai, Tamil Nadu, Indiad Department of Electronics and Electrical Engineering, Indian Institute of Technology Guwahati, Assam, India

e Department of Mathematics, Indian Institute of Technology Delhi, Delhi, India

Abstract

Due to the need for robust uncertainty quantification, Bayesian neural learning has gained attention in the era of deep learning andbig data. Markov Chain Monte-Carlo (MCMC) methods typically implement Bayesian inference which faces several challengesgiven a large number of parameters, complex and multimodal posterior distributions, and computational complexity of large neuralnetwork models. Parallel tempering MCMC addresses some of these limitations given that they can sample multimodal posteriordistributions and utilize high-performance computing. However, certain challenges remain given large neural network models andbig data. Surrogate-assisted optimization features the estimation of an objective function for models which are computationallyexpensive. In this paper, we address the inefficiency of parallel tempering MCMC for large-scale problems by combining parallelcomputing features with surrogate assisted likelihood estimation that describes the plausibility of a model parameter value, givenspecific observed data. Hence, we present surrogate-assisted parallel tempering for Bayesian neural learning for simple to com-putationally expensive models. Our results demonstrate that the methodology significantly lowers the computational cost whilemaintaining quality in decision making with Bayesian neural networks. The method has applications for a Bayesian inversion anduncertainty quantification for a broad range of numerical models.

Keywords: Bayesian neural networks, parallel tempering, MCMC, Surrogate-assisted optimization, parallel computing

1. Introduction

Although neural networks have gained significant attentiondue to the deep learning revolution [1], several limitations ex-ist. The challenge widens for uncertainty quantification in de-cision making given the development of new neural networkarchitectures and learning algorithms. Bayesian neural learn-ing provides a probabilistic viewpoint with the representationof neural network weights as probability distributions [2, 3].Rather than single-point estimates by gradient-based learningmethods, the probability distributions naturally account for un-certainty in parameter estimates.

Through Bayesian neural learning, uncertainty regarding thedata and model can be propagated into the decision mak-ing process. Markov Chain Monte Carlo sampling methods(MCMC) implement Bayesian inference [4, 5, 6, 7] by con-structing a Markov chain, such that the desired distributionbecomes the equilibrium distribution after a number of steps[8, 9]. MCMC methods provide numerical approximations ofmulti-dimensional integrals [10]. MCMC methods have notgained as much attention in neural learning when compared to

Email addresses: rohitash.chandra@sydney.edu.au (RohitashChandra), konark145@gmail.com (Konark Jain),kapoor.arpit97@gmail.com (Arpit Kapoor),kapoor.arpit97@gmail.com (Ashray Aman)

URL: rohitash-chandra.github.io (Rohitash Chandra)

gradient-based counterparts since convergence becomes com-putationally expensive for a large number of model parametersand multimodal posteriors [11]. MCMC methods typically re-quire thousands of samples to be drawn depending on the modelwhich becomes a major limitation in applications such as deeplearning [1, 12, 13]. Hence, other methods for implementingBayesian inference exist, such as variational inference [14, 15]which has been used for deep learning [16]. Variational infer-ence has the advantage of faster convergence when comparedto MCMC methods for large models. However, in computa-tionally expensive models, variational inference methods wouldhave a similar problem as MCMC, since both would need toevaluate model samples for the likelihood; hence, both wouldbenefit from surrogate-based models.

Parallel tempering is an MCMC method that [17, 18, 19] fea-tures multiple replicas to provide global and local explorationwhich makes them suitable for irregular and multi-modal dis-tributions [20, 21]. During sampling, parallel tempering fea-tures the exchange of neighbouring replicas to feature explo-ration and exploitation in the search space. The replicas withhigher temperature values make sure that there is enough ex-ploration, while replicas with lower temperature values exploitthe promising areas found during the exploration. In contrastto canonical MCMC sampling methods, parallel tempering ismore easily implemented in a multi-core or parallel computingarchitecture [22]. In the case of neural networks, parallel tem-pering was used for inference of restricted Boltzmann machines

Preprint submitted to Eng. App. AI May 15, 2020

arX

iv:1

811.

0868

7v3

[cs

.LG

] 1

4 M

ay 2

020

(RBMs) [23, 24] where it was shown that [25] parallel temper-ing is more effective than Gibbs sampling by itself. Paralleltempering for RBMs has been improved by featuring the effi-cient exchange of information among the replicas [26]. Thesestudies motivated the use of parallel tempering in Bayesian neu-ral learning for pattern classification and regression tasks [27].

Surrogate assisted optimisation [28, 29] considers the use ofmachine learning methods such as Gaussian process and neuralnetwork models to estimate the objective function during op-timisation. This is handy when the evaluation of the objectivefunction is too time-consuming. In the past, metaheuristic andevolutionary optimization methods have been used in surrogateassisted optimization [30, 31]. Surrogate assisted optimizationhas been useful for the fields of engine and aerospace designto replicate computationally expensive models [32, 33, 34, 28].The optimisation literature motivates improving parallel tem-pering using a low-cost replica of the actual model via a surro-gate to lower the computational costs.

In the case of conventional Bayesian neural learning, much ofthe literature concentrated on smaller problems such as datasetsand network architecture [35, 36, 2, 3] due to computationalefficiency of MCMC methods. Therefore, parallel computinghas been used in the implementation of parallel tempering forBayesian neural learning [27], where the computational timewas significantly decreased due to parallelisation. Besides, themethod achieved better prediction accuracy and convergencedue to the exploration features of parallel tempering. We be-lieve that this can be further improved through incorporatingnotions from surrogate assisted optimisation in parallel temper-ing, where the likelihood function at certain times is estimatedrather than evaluated in a high-performance computing envi-ronment.

We note that some work has been done using surrogate as-sisted Bayesian inference. Zeng et. al [37] presented a methodfor material identification using surrogate assisted Bayesian in-ference for estimating the parameters of advanced high strengthsteel used in vehicles. Ray and Myer [38] used Gaussianprocess-based surrogate models with MCMC for geophysicalinversion problems. The benefits of surrogate assisted meth-ods for computationally expensive optimisation problems mo-tivate parallel tempering for computationally expensive mod-els. To our knowledge, there is no work on parallel temper-ing with surrogate-models implemented via parallel computingfor machine learning problems. In the case of parallel tem-pering that uses parallel computing, the challenge would be indeveloping a paradigm where different replicas can communi-cate efficiently. Besides, the task of training the surrogate modelfrom data across multiple replicas in parallel poses further chal-lenges.

In this paper, we present surrogate-assisted parallel temper-ing for Bayesian neural learning where a surrogate is used toestimate the likelihood rather than evaluating the actual modelthat feature a large number of parameters and datasets. Wepresent a framework that seamlessly incorporates the decisionmaking by a master surrogate for parallel processing cores thatexecute the respective replicas of parallel tempering MCMC.Although the framework is intended for general computation-

ally expensive models, we demonstrate its effectiveness usinga neural network model for classification problems. The majorcontribution of this paper is to address the limitations of paralleltempering given computationally expensive models.

The rest of the paper is organised as follows. Section 2 pro-vides background and related work, while Section 3 presentsthe proposed methodology. Section 4 presents experiments andresults and Section 5 concludes the paper with discussion forfuture research.

2. Related work

2.1. Bayesian neural learningIn Bayesian inference, we update the probability for a hy-

pothesis as more evidence or information becomes available[39]. We estimate the posterior distribution by sampling us-ing prior distribution and a ’likelihood function’ that evalu-ates the model with observed data. A probabilistic perspectivetreats learning as equivalent to maximum likelihood estimation(MLE) [40]. Given that the neural network is the model, webase the prior distribution on belief or expert opinion withoutobserving the evidence or training data [35]. An example of in-formation or belief for the prior in the case of neural networksis the concept of weight decay that states that smaller weightsare better for generalization [41, 2, 2, 42, 43].

Due to limitations in MCMC sampling methods, progressin development and applications of Bayesian neural learninghas been slow, especially when considering larger neural net-work architectures, big data and deep learning. Several tech-niques have been applied to improve MCMC sampling meth-ods by incorporating approaches from the optimisation litera-ture. Neal et al. [44] presented Hamiltonian dynamics thatinvolve using gradient information for constructing efficientMCMC proposals during sampling. Gradient-based learningusing Langevin dynamics refer to use of gradient informationwith Gaussian noise [45]; Chandra et al. employed Langevindynamics for Bayesian neural networks for time series predic-tion [46]. Hinton et al.[47] used complementary priors for deepbelief networks to form an undirected associative memory forhandwriting recognition. Furthermore, parallel tempering hasbeen used for the Gaussian Bernoulli Restricted Boltzmann Ma-chines (RBMs) [48]. Prior to this, Cho et al. [49] demonstratedthe efficiency of parallel tempering in RBMs. Desjardins et al.[50] utilized parallel tempering for maximum likelihood train-ing of RBMs and later used it for deep learning using RBMs[51].

In Bayesian neural learning, we estimate the posterior distri-bution by MCMC sampling using a ’likelihood function’ thatevaluates the model given the observed data and prior distribu-tion. Given input features or covariates (xt), we compute f (xt)by a feedforward neural network with one hidden layer,

f (xt) = g(δo +

H∑h=1

vhog(δh +

I∑d=1

(wdhxt))

(1)

where δo and δh are the bias for the output o and hidden h layer,respectively. vho is the weight which maps the hidden layer h to

2

the output layer. wdh is the weight which maps xt to the hiddenlayer h and g is the activation function for the hidden and outputlayer units.

Let θ = (w, v, δ, τ2), with δ = (δo, δh), and L as the numberof parameters that includes weights and biases. τ2 is a singleparameter to represent the noise in the predictions given by theneural network model. I,H,O refers to number of input, hiddenand output neurons, respectively. We assume a Gaussian priordistribution using,

log (p(θ)) = −L

2log(σ2)

−1

2σ2

H∑h=1

I∑d=1

w2dh +

H∑h=1

(δ2h + v2

h) + δ2o

(2)

where the variance (σ2) is user-defined, gathered by infor-mation (prior belief) regarding the distribution of weights oftrained neural networks in similar applications.

Given data (y), we use Bayes rule for the posterior p(θ|y)which is proportional to the likelihood p(y|θ) times the priorp(θ).

p(θ|y) ∝ p(y|θ) × p(θ)

Then, the log-posterior transforms to

log (p(θ|y)) = log (p(θ)) + log (p(y|θ))

To implement the likelihood given in Equations 1 and 2, weuse the multinomial likelihood function for the classificationproblems as shown in Equation 3.

log (p(y|θ)) =∑t∈T

K∑k=1

zt,k log πk (3)

for classes k = 1, . . . ,K, where πk is the output of the neuralnetwork after applying the transfer function, and T is the num-ber of training samples. In this case, the transfer function is thesoftmax function [52],

πk =exp( f (xp))∑K

k=1 exp( f (xk))(4)

for k = 1, . . . ,K. zt,k is an indicator variable for the given sam-ple t and the class k as given in the dataset and defined by,

zt,k =

1, if yt = k0, otherwise

(5)

2.2. Parallel tempering MCMC

Parallel tempering MCMC features an ensemble of chains(known as replicas) executed at different temperature levelsdefined by a temperature ladder that determine the extent ofexploration and exploitation [53, 21, 54, 55]. The replicaswith higher temperature values ensure that there is enough

exploration, while replicas with lower temperature values ex-ploit the promising areas found by exploration. Typically,the neighbouring replicas are exchanged given the Metropolis-Hastings criterion. In some implementations, non-neighboringreplicas are also considered for exchange [55, 56]. In suchcases, we calculate the acceptance probabilities of all possi-ble moves a priori. The specific exchange move is then se-lected, which is useful when a limited number of replicas areavailable [57]. Although geometrically uniform temperaturelevels have been typically used for the respective replicas, de-termining the optimal tempering assigned for each of the repli-cas has been a challenge that attracted some attention in theliterature [58, 59, 60, 20]. Typically, gradient-free proposalswithin chains are used for proposals for exploring multimodaland discontinuous posteriors [61, 62]; however, it is possible toincorporate gradient-based information for developing effectiveproposal distributions [27].

Although denoted ”parallel”, the replicas can be executedsequentially in a single processing unit [63]; however, multi-core or high-performance computing systems can feature par-allel implementations improving the computational time [56,64, 65]. There are challenges since parallel tempering fea-tures exchange or transition between neighbouring replicas, andwe need to consider efficient strategies that take into accountinter-process communication [65]. Parallel tempering has beenused with high-performance computing for seismic full wave-form inversion (FWI) problems [64], which is amongst the mostcomputationally intensive problems in earth science today.

Parallel tempering has also been implemented in a distributedvolunteer computing network via crowd-sourcing for multi-threading and graphic processing units [66]. Furthermore, im-plementation with field-programmable gate array (FPGA) hasshown much better performance than multi-core and graphicprocessing units (GPU) implementations [67].

In a parallel tempering MCMC ensemble, given M replicasdefined by multiple temperature levels, the state of the ensem-ble is specified by X = x1, x2, ..., xM; where xi is the ith replica,with temperature level Ti. The equilibrium distribution of theensemble X is given by,

Π(X) =

M∏i=1

exp(− 1Ti

L(xi))

Z(Ti)(6)

where L(xi) is the log-likelihood function for the replica stateat each temperature level (Ti) and Z(Ti) is an intractable nor-malization constant. At every iteration of the replica state,the replica can feature two types of transitions that includes aMetropolis transition and a replica transition.

The Metropolis transition enables the replica to perform localMonte Carlo moves defined by the energy function E(xi). Thelocal replica state x∗i is sampled using a proposal distributionqi(.|xi) and the Metropolis-Hastings acceptance probability αfor the local replica Llocal is given as,

α = min(1, exp

(−

1Ti

(L(x∗i ) − L(xi))))

(7)

3

The detailed balance condition holds for each replica, and there-fore it holds for the ensemble system.

Typically, the Metropolis-Hastings update consists of a sin-gle stochastic process that evaluates the energy of the systemand accepted is based on the temperature ladder. The selectionof the temperature ladder for the replicas is done prior to sam-pling; we use a geometric spacing methodology [68] as given,

Ti = T(i−1)

(M−1)max (8)

where i = 1, . . . ,M and Tmax is the maximum temperaturewhich is user-defined.

The Replica transition features the exchange of two neigh-bouring replica states ( xi ↔ xi+1) defined by their tempera-ture level, i and i + 1. The replica exchange is accepted by theMetropolis-Hastings criterion with replica exchange probabil-ity β by,

β = min(1, exp

(( 1Ti+1

−1Ti

)(L(xi+1) − L(xi)

))(9)

Based on the Metropolis criterion, the configuration (position inthe replica ) of neighbouring replicas at different temperaturesare exchanged. This results in a robust ensemble which cansample both low and high energy configurations.

The replica-exchange enables a replica that could be stuckat a local minimum with low-temperature level to exchangeconfiguration with a higher neighbouring temperature level andhence improve exploration. In this way, the replica-exchangecan shorten the sampling time required for convergence. Thefrequency of determining the exchange and the temperaturelevel is user-defined. For further details about the derivationfor the equations in this section, see [55, 69].

2.3. Surrogate-assisted optimization

Surrogate assistant optimisation refers to the use of machinelearning or statistical learning models to develop approximatelycomputationally inexpensive simulation of the actual model[29]. The major advantage is that the surrogate model providescomputationally efficiency when compared to the exact modelused for evolutionary algorithms and related optimisation meth-ods [30, 31]. In the optimisation literature, such approaches arealso known as response surface methodologies [70, 71] whichhave been applicable for a wide range of engineering problemssuch as reliability analysis of laterally loaded piles [72].

In the case of evolutionary computation methods, Ong et al.[30] presented parallel evolutionary optimisation for solvingcomputationally expensive functions with application to aero-dynamic wing design where surrogate models used radial basisfunctions. Zhou et al. [31] accelerated evolutionary optimiza-tion with global and local surrogate models while Lim et al.[73] presented a generalised method that accounted for uncer-tainty in estimation to unify diverse surrogate models. Jin [29]reviewed surrogate-assisted evolutionary computation that cov-ered single and multi-objective, dynamic, constrained, and mul-timodal optimisation problems. Furthermore, surrogate models

have been widely used in Earth sciences such as modelling wa-ter resources [74]. Moreover, Daz-Manrquez et al. [75] pre-sented a review of surrogate assisted multi-objective evolution-ary algorithms that showed that the method has been successfulin a wide range of application problems.

The search for the right surrogate model is a significant chal-lenge given different types of likelihood or fitness landscapegiven by the actual model. Giunta et al. [76] presented a com-parison of quadratic polynomial models with the least-squaremethod and interpolation models that featured Gaussian pro-cess regression (kriging). They discovered that the quadraticpolynomial models were more accurate in terms of errors for es-timation for the optimisation problems. Jin et al. [77] presentedanother study that compared several surrogate models that in-clude polynomial regression, multivariate adaptive regressionsplines, radial-basis functions, and kriging based on multipleperformance criteria using different classes of problems. Theauthors reported radial basis functions as one of the best forscalability and robustness, given different types of problems.They also reported kriging to be computationally expensive.

3. Surrogate-assisted multi-core parallel tempering

Surrogate models primarily learn to mimic actual or truemodels using their behaviour, i.e. how the true model respondsto a set of input parameters. A surrogate model captures therelationship between the input and output given by the truemodel. The input is the set of proposals in parallel temperingMCMC that features the weights and biases of the neural net-work model. Hence, we utilize the surrogate model to approx-imate the likelihood of the true model. We define the approxi-mation of the likelihood by the surrogate as pseudo-likelihood.We train the surrogate model on the data that is composed ofthe history of proposals for weights and biases of the neuralnetwork with the corresponding true likelihood. We do not es-timate the output of the neural network by the surrogate (sincein some problems there are many outputs); hence to limit thenumber of variables, we directly estimate the likelihood usingthe surrogate.

We implement the neighbouring replica transition or ex-change at regular intervals. The cost of inter-process communi-cation must be limited to avoid computational overhead, giventhat we execute each replica on a separate processing core. Theswap interval defines the time (number of iterations or sam-ples) after which each replica pauses and awaits for neighbour-ing replica exchange. After the exchange, the replica managerprocess enables local Metropolis transition and the process re-peats. We note that although we use a neural network model,the framework is general and we can use other models; whichcould include those from other domains such as Earth sciencemodels that are computationally expensive [78, 79].

Bayesian neural learning employs parallel tempering MCMCfor inference; this can be viewed as a training procedure forthe neural network model. The goal of the surrogate modelis to save computational time taken for evaluation of the truelikelihood function associated with computationally expensivemodels.

4

Given that the true model is represented as L = f (x), thesurrogate model provides an approximation in the form l =

f (x), such that L = l + e; where e represents the differenceor error. The surrogate model provides an estimate by thepseudo-likelihood for replacing true-likelihood when needed.The surrogate model is constructed by training from experiencewhich is given by the set of input xi,s with corresponding true-likelihoods Li,s; where s represents the sample and i representsthe replica. Hence, input features (Φ) for the surrogate is de-veloped by combining xi,s using samples generated (θ) acrossall the replicas for a given surrogate interval (ψ). The surrogateinterval defines the batch size of the training data for the sur-rogate model which goes through incremental learning. All therespective replicas sample until the surrogate interval is reachedand then the manager process collects the sampled data to cre-ate a training data batch for the surrogate model. This can beformulated as follows,

Φ = ([x1,s, . . . , x1,s+ψ], . . . , [xM,s, . . . , xM,s+ψ])λ = ([L1,s, . . . , L1,s+ψ], . . . , [LM,s, . . . , LM,s+ψ])Θ = [Φ, λ] (10)

where xi,s represents the set of parameters proposed and s,yi,s is the output from the multinomial likelihood, and M is thetotal number of replicas.

The training surrogate dataset (Θ = [Φ, λ]) consists of inputfeatures (Φ) and response (λ) for the span of each surrogate in-terval (s + ψ). Hence, we denote the pseudo-likelihood y byy = f (Θ); where f is the surrogate model. We amend the likeli-hood in training data for the temperature level since it has beenchanged by taking Llocal/Ti for given replica i. The likelihoodis amended to reflect the true likelihood rather than that rep-resented by taking into account the temperature level. All therespective replica Θi data is combined, Υ = [Θ1,Θ2, ...ΘN] andtrained using the neural network model given in Equation 1.

Algorithm 1 presents surrogate-assisted multi-core paralleltempering for Bayesian neural learning. We implement the al-gorithm using (multi-core) parallel processing where a managerprocess takes care of the ensemble of replicas that run in sep-arate processing cores. Given the parallel processing nature, itis tricky to implement when to terminate sampling distributedamongst parallel processing replicas; hence, our terminationcondition waits for all the replica processes to end. We monitorthe number of alive replica process in the master process by set-ting the number of alive replicas in the ensemble (alive = M).We note that the highlighted region of Algorithm 1 shows dif-ferent processing cores as given in Figure 2. We highlight themanager process in blue, and in pink we highlight the ensem-ble of replica processes running in parallel. We initially assignthe replicas that sample θn with values using the Gaussian priordistribution with user-defined variance (σ2 = 25) centred at themean of 0. We define the temperature level by geometric lad-der (Equation 8), and other key parameters which includes thenumber of replica samples (Rmax), swap-interval which definesafter how many samples to check for replica swap (Rswap), sur-rogate interval (ψ), and surrogate probability (S prob). The main

purpose of the surrogate interval is to collect enough data forthe surrogate model during sampling. This also can be seen asbatch-based training where we update the model after we col-lect the data at regular intervals.

Figure 2 shows how we use the manager processing unit forthe respective replicas running in parallel for the given surro-gate interval. The manager process waits for all the replicasto reach the surrogate interval. Then we calculate the replicatransition probability for the possibility of swapping the neigh-bouring replicas. Figure 2 further highlights the informationflow between the master process and the replica process viainter-process communication 1. The information flows fromthe replica process to master process using signal() given bythe replica process as shown in Stage 2.2 and 5.0 of Algorithm1.

The surrogate model is re-trained for remaining surrogate in-terval blocks until the maximum iteration (Rmax) is reached toenable better estimation for the pseudo-likelihood. The surro-gate model is trained only in the manger process, and we passa copy of the surrogate model with the trained parameters tothe ensemble of replica processes for estimating the pseudo-likelihood when needed. Note that only the samples associ-ated with the true-likelihood become part of the surrogate train-ing dataset. The surrogate training can consume a significantportion of time which is dependent on the size of the problemin terms of the number of parameters and the surrogate modelused. We evaluate the trade-off between quality of estimationby pseudo-likelihood and overall cost of computation for thetrue likelihood function for different types of problems.

In Algorithm 1, Stage 1.4 predicts the pseudo-likelihood(Lsurrogate) with given proposal θ∗s . Stage 1.5 calculates the like-lihood moving average of past three likelihood values, Lpast =

mean(Ls−1, Ls−1, Ls−2). The motivation for doing this is to com-bine univariate time series prediction approach (moving aver-age) with multivariate regression approach (surrogate model)for robust estimation where we take both current and past infor-mation into account. In Stage 1.6, we combine the likelihoodmoving average with the pseudo-likelihood to give a predictionthat considers the present replica proposal and the past behav-ior, Llocal = (0.5 * Lsurrogate) + 0.5 * Lpast. Note that although weuse the past three values for the moving average, this numbercan change for different types of problems. Once the swap in-terval (φ) is reached, Stage 2.0 prepares the replica transition inthe manager process (highlighted in blue). Stage 3.0 executesonce we reach the surrogate interval where we use data col-lected from Stage 1.8 for creating surrogate training set batchΘ as shown in Equation 10. Stage 4 shows how we execute theglobal surrogate training in the Manager process with combinedsurrogate data using the neural network model in Equation 1.We use the trained knowledge from the global surrogate model(Ψglobal) in the respective replicas local surrogate model (Ψi) asshown in Stage 1.3 and Figure 2. Once we reach the maximumnumber of samples for the given replica (Rmax, Stage 5.0 signals

1Python multiprocessing library for implementation:https://docs.python.org/2/library/multiprocessing.html

5

the manager process to decrement the number of replicas alivefor executing the termination criterion.

Finally, we execute Stage 6 in the manager process wherewe combine the respective replica predictions and proposals(weights and biases) from the ensemble by concatenating thehistory of the samples to create the posterior distributions. Thisfeatures the accepted samples and copies of the accepted sam-ples in cases of rejected samples, as shown in Stage 1.9 of Al-gorithm 1.

Furthermore, the framework features parallel tempering inthe first stage of sampling that transforms into a local mode orexploitation in the second stage where the temperature ladderis changed such that Ti = 1, for all replicas, i = 1, 2, ...,N asdone in [27, 78]. We emphasize on exploration in the first phaseand emphasize on exploitation in the second phase, as shown inStage 1.9.1 of Algorithm 1. The duration of each phase is prob-lem dependent, which we determine from trial experiments.

We validate the quality of the surrogate model prediction us-ing the root mean squared error (RMSE)

RMS E =

√√√1N

N∑i=1

(Li − Li)2

where Li and Li are the true likelihood and the pseudo-likelihood values, respectively. N is the number of cases weemploy the surrogate during sampling.

Note that instead of swapping entire replica configuration, analternative is to swap the respective temperatures. This couldsave the amount of information exchanged during inter-processcommunication as the temperature is only a double-precisionnumber, implemented by [38].

3.1. Langevin gradient-based proposal distribution

Apart from random-walk proposals, we utilize stochasticgradient Langevin dynamics [45] for the proposal distribution.It features additional noise with stochastic gradients to optimizea differentiable objective function which has been very promis-ing for neural networks [80, 27]. The proposal distribution isconstructed as follows.

θp ∼ N(θ[k],Σθ), where (11)

θ[k]

= θ[k] + r × ∇EyAD,T[θ[k]],

EyAD,T[θ[k]] =

∑t∈AD,T

(yt − f (xt)[k])2,

∇EyAD,T[θ[k]] =

(∂E∂θ1

, . . . ,∂E∂θL

)r is the learning rate, Σθ = σ2

θ IL and IL is the L × L identitymatrix. So that the newly proposed value of θp, consists of 2parts:

1. An gradient descent based weight update given by Equa-tion 11.

2. Add an amount of noise, from N(0,Σθ).

Figure 1: Data collection for training surrogate model

3.2. Surrogate model

The choice of the surrogate model needs to consider the com-putational resources taken for training the model during thesampling process. We note that Gaussian process models, neu-ral networks, and radial basis function models [81] have beenpopular choices for surrogates in the literature.

In our case, we consider the inference problem that featureshundreds to thousands of parameters; hence, the model needsto be efficiently trained without taking lots of computationalresources. Moreover, the flexibility of the model to have incre-mental training is also needed. Therefore, we rule out Gaussianprocess models since they have imitations given large trainingdatasets [82]. In our case, tens of thousands of samples couldmake the training data. Therefore, we use neural networks asthe choice of the surrogate model. The training data and neuralnetwork model can be formulated as follows.

The data given to the surrogate model is Φ and λ as in (10),where Φ is the input and λ is the desired output of the model.The prediction of the model is denoted by λ. We explain thesurrogate models used in the paper as follows.

In our surrogate model, we consider a single hidden layerfeedforward neural network as shown in Equation (1). Themain difference is that we use the rectified linear unitary ac-tivation function, g(.). In this case, after forward propagation,the errors in the estimates are evaluated using the cross-entropycost function [83] which is formulated as:

J(W,b) = −1ψ

ψ∑i=0

(λ(i)log(λ(i)) + (1 − λ(i))log(1 − λ(i)) (12)

The learning or optimization task then is to iteratively up-date the weights and biases to minimize the cross-entropy loss,J(W,b).

We note that stochastic gradient descent maintains a singlelearning rate for all weight updates and typically the learning

6

Data: Classification DatasetResult: Posterior distributions for neural network weights* Set the number of replicas (M) in ensemble as alive; alive = M* Define geometric() temperature ladder , number of replica processes (M), surrogate interval (ψ), replica swap interval (Rswap), and maximum number ofsamples for each replica (Rmax).while (alive , 0 do

Stage 0: Prepare manager process to execute each replica in parallel coresfor each i until M do

s = 0first-phase: Ti = geometric()while (s < Rmax) do

Stage 1.0: Metropolis Transitionfor each v until ψ do

for each k until Rswap do1.1 Random-walk, θ∗s = θs + ε1.2 Llocal calculate:Draw κ from a Uniform distribution [0,1]if κ < S prob and s > ψ then

Estimate Llocal from local surrogate’s prediction, Lsurrogate1.3 Copy global surrogate knowledge to local surrogate, Ψi ← Ψglobal1.4 Predict Lsurrogate value with the proposed θ∗i1.5 Lpast = mean(Ls−1, Ls−1, Ls−2)1.6 Assign Llocal = (0.5 * Lsurrogate) + 0.5 * Lpast

else1.7 Llocal = true-likelihood, given by Likelihood function in Equation (3))1.8 Save Ls = Llocal (Equation 10)

end1.9 Calculate acceptance probability α and draw u from uniform distributionif u ≤ α then

Accept replica state, θs ← θ∗sendelse

Reject and retain previous state: θs ← θ∗s−1end1.9.1 Switch sampling style by updating replica temperatureif second-phase is true then

Update temperature, Ti = 1endIncrement s

endStage 2.0: Replica Transition:2.1 Calculate acceptance probability β and draw b from a Uniform distribution [0,1]if b ≤ β then

2.2 Signal() manager process 2.3 Exchange neighboring Replica, θi ↔ θs+1end

endStage 3.0: Set Θi which features history of proposals Φ (θ) and response λ ( Llocal ). Use data collected from Stage 1.8

Stage 4.0: Global Surrogate Trainingfor each replica do

4.1 Get replica surrogate data Θi from Stage 3.0end4.2 Train global surrogate model with combined surrogate data, Υ = [Θ1,Θ2, ...ΘN ] using neural network model in Equation 14.3 Save global surrogate model parameters, Ψglobal

end

Stage 5.0: Signal() manager process5.1 decrement number of replica processes alive

endendStage 6: Combine predictions and posterior from respective replicas in the ensemble, using second-phase MCMC samples.

Algorithm 1: Surrogate-assisted parallel tempering for Bayesian neural networks. The highlighted regions of the algorithm shows differentprocessing cores. The manager process is highlighted in blue while the ensemble of replica processes running in parallel is highlighted in pink.

7

Figure 2: Surrogate-assisted multi-core parallel tempering features surrogates to estimate the likelihood function at times rather than evaluating it.

rate does not change during training. Adam (adaptive mo-ment estimation) learning algorithm [84] differs from classicalstochastic gradient descent, as the learning rate is maintainedfor each network weight and separately adapted as learning un-folds. Adam computes individual adaptive learning rates fordifferent parameters from estimates of first and second mo-ments of the gradients. Adam features the strengths of rootmean square propagation (RMSProp) [85], and adaptive gradi-ent algorithm (AdaGrad) [86]. Adam has shown better resultswhen compared to stochastic gradient descent, RMSprop andAdaGrad. Hence, we consider Adam as the designated algo-rithm for the neural network-based surrogate model.

We note that the surrogate model predictions do not provideuncertainty quantification since they are not probabilistic mod-els. The proposed framework is general and hence different sur-rogate types of models can be used in future. Different types ofsurrogate models would consume different computational timeand have certain strengths and weaknesses [77].

4. Experiments and Results

In this section, we present an experimental analysis ofsurrogate-assisted parallel tempering (SAPT) for Bayesian neu-ral learning. The experiments consider a wide range of issuesthat test the accuracy of pseudo-likelihood by the surrogate, thequality in decision making given by the classification perfor-mance, and the amount of computational time saved.

4.1. Experimental Design

We select six benchmark pattern classification problems fromthe University of California Irvine machine learning repository[87]. The problems feature different levels of computationalcomplexity and learning difficultly; in terms of the number ofinstances, the number of attributes, and the number of classes,as shown in Table 1. We use the multinomial likelihood givenin Equation 3 the selected classification problems. Moreover,we show the performance using Langevin-gradient proposalswhich takes more computational time due to cost of computinggradients when compared to random-walk proposals; however,it gives better prediction accuracy given the same number ofsamples [27]. The experimental design follows the followingstrategy in evaluating the performance of the selected parame-ters from Algorithm 1.

• Evaluate the effect of the surrogate probability (S prob onthe computational time and classification performance.

• Evaluate the effect of the surrogate interval (ψ on the com-putational time and classification performance.

• Evaluate the effect of Langevin-gradients for proposals inparallel tempering MCMC.

.We provide the parameter setting for the respective exper-

iments as follows. A burn-in time Rburn = 0.50 (50 %) ofthe samples for the respective replica. The burn-in strategy isstandard practice for MCMC sampling which ensures that the

8

Table 1: Dataset description [87]

Dataset Instances Attributes Classes Hidden Units

Iris 150 4 3 12Ionosphere 351 34 2 50Cancer 569 9 2 12Bank 11162 20 2 50Pen-Digit 10992 16 10 30Chess 28056 6 18 25

chain enters a high probability region, where the states of theMarkov chain are more representative of the posterior distri-bution. Although MCMC methods more commonly use burn-in time of 10-20 % in the literature, we use 50 % for gettingprediction performance of preferred accuracy. The maximumsampling time, Fmax = 50, 000 for all the respective problems,and number of replicas, M = 10 which run on parallel process-ing cores. The other key parameters include replica swap inter-val (Rswap = 50), surrogate interval(ψ = 50), replica samplingtime (Rmax = Fmax/M), surrogate probability (S prob = 0.25,S prob = 0.50) and maximum temperature (Tmax = 5). We deter-mined the given values for the parameters in trial experiments.We selected the maximum temperature in trial experiments bytaking into account the performance accuracy given with a fixednumber of samples. We provide details for the pattern classifi-cation datasets with details of Bayesian neural network topol-ogy (number of hidden units) in Table 1.

In the case of random-walk proposal distribution, we add aGaussian noise to the weights and biases of the network froma normal distribution with mean of 0 and standard deviation of0.025. Theuser defined constants for the priors (see Equation2) are set as σ2 = 25, ν1 = 0 and ν2 = 0.

In the respective experiments, we compare SAPT with par-allel tempering featuring random-walk proposals (PTRW) andLangevin-gradients (PTLG) taken from the literature [27].Surrogate-assisted parallel tempering also features Langevin-gradients (SAPT-LG) given in Equation (12). We use a learningrate of 0.5 for computing the weight update via the gradients.Furthermore, we apply Langevin-gradient with a probabilityLprob = 0.5, and random-walk proposal, otherwise. Note thatthe respective methods feature parallel tempering for the first50 per cent of the samples that make the burn-in period. After-wards, we use canonical MCMC where temperature T = 1 us-ing parallel computing environment featuring replica-exchangevia interprocess communication.

4.2. ImplementationWe employ multi-core parallel tempering for neural networks

2 to implement surrogate assisted parallel tempering. We useone hidden layer in the Bayesian neural network for classifica-tion problems.

We use the Keras machine learning library for implementingthe surrogate 3 with Adam learning algorithm [84]. The surro-

2Multi-core parallel tempering: https://github.com/sydney-machine-learning/parallel-tempering-neural-net

3Keras: https://keras.io/

gate neural network model architecture consists of [i, h1, h2, o];where i refers to the number of inputs that consists of the to-tal number of weights and biases used in the Bayesian neu-ral network for the given problem. h1 and h2 refers to thefirst and second hidden layers, and o represents the output thatpredicts the likelihood. In our experiments, we use hiddenunits h1 = 64, h2 = 16, for the Iris and Cancer problems.In the Ionosphere and Bank problems, we use hidden unitsh1 = 120, h2 = 40, and for Pen-Digit and Chess problems,we use hidden units h1 = 200, h2 = 50. All problems used oneoutput unit for the surrogate model, o = 1.

4.3. Results

We first present the results in terms of classification accu-racy of posterior samples for SAPT-RW with random-walk pro-posal distribution (Table 2) where we evaluate different com-binations of selected values of surrogate interval and ψ surro-gate probability S prob. Looking at the elapsed time, we findthat SAPT-RW is more costly for the Iris and Cancer problemswhen compared to PT-RW. These are smaller problems whencompared to rest given the size of the dataset shown in Table1. The Ionosphere problem saved computation time for bothinstances of SAPT-RW. A larger dataset with a Bayesian neuralnetwork model implies that there is more chance for the surro-gate to save time which is visible in the Pen-Digit and Chessproblems. The Bank problem does not save much time but re-tains the accuracy in classification performance. Furthermore,we find that instances of SAPT improve the classification ac-curacy of Iris, Ionosphere, Pen-Digit and Chess problems. InCancer and Bank problems, the performance is similar.

We provide the results for Langevin-gradient proposals(SAPT-LG) in Table 3. In general, the classification perfor-mance improves when compared to random-walk proposal dis-tribution (SAPT-RW) in Table 2. By incorporating the gradientinto the proposal, the results show faster convergence with bet-ter accuracy given the same number of samples. In a compar-ison of both forms of parallel tempering (PT-RW and PT-LG)with the surrogate-assisted framework (SAPT-RW and SAPT-LG), we observe that the elapsed time has not improved for Iris,Ionosphere and Cancer problems; however, it has improved inthe rest of the problems.

The accuracy of the surrogate in predicting the likelihood isshown in Table 4 for the smaller problems that feature Iono-sphere, Cancer and Iris. We notice that the RMSE for surrogateprediction is lower for the Iris problem when compared to theothers; however, as shown in Figure 4, Figure 5 and 6, this isrelative to the range of log-likelihood. We observe that the log-likelihood prediction by the surrogate model is much better forthe smaller problems (Iris and Cancer) when compared to thelarger problems (Pen-Digit and Chess).

Figure shows the comparison of the posterior distributionof two selected weights from input to the hidden layer of theBayesian neural network for the Cancer problem using SAPT-RW and PT-RW. We notice that although the chains convergedto different sub-optimal models in the multimodal distributions,SAPT-RW does not depreciate in terms of exploration.

9

(a) Weight 1 (b) Weight 1

(c) Weight 2 (d) Weight 2

Figure 3: Posterior distribution and trace-plot comparing surrogate-assisted parallel tempering (Panel (a) and (c)) with parallel tempering (Panel (b) and (d)) MCMCfor selected weights (1 and 2) for the Cancer problem.

Table 5 provides an evaluation of the number of replicas(cores) on the computational time (minutes) using SAPT-RWfor selected problems. We observe that in general, the com-putational time reduces as the number of replica increases bytaking advantage of parallel processing.

5. Discussion

The results, in general, have shown that surrogate-assistedparallel tempering can be beneficial for larger datasets and mod-els, demonstrated by Bayesian neural network architecture forPen-Digit and Chess classification problems. This implies thatthe method would be very useful for large scale models wherecomputational time can be lowered while maintaining perfor-mance in decision making such as classification accuracy. Weobserved that in general, the Langevin-gradients improves theaccuracy of the results. Although we used Bayesian neural net-works to demonstrate the challenge in using computationallyexpensive models with large datasets, surrogate-assisted paral-lel tempering can be used for a wide range of models acrossdifferent domains. We note that Langevin-gradients are limitedto models where gradient information is available. In large andcomputationally expensive geoscientific models such as mod-elling landscape [78] and modelling reef evolution [88], it isdifficult to obtain gradients from the models; hence, random-walk and other meta-heuristics are more applicable.

The proposed method employs transfer learning for the sur-rogate model, where we transfer the knowledge and refine thesurrogate model in the forthcoming surrogate intervals withnew data. Since each replica of parallel tempering is executedon a separate processing core, inter-process communication is

used for exchanging information between the different replicas.Inter-process communication is also used for collecting the his-tory of information in terms of proposals and associated likeli-hood for creating training datasets for the surrogate model. Theproposed method could be seen as a case for online learningthat considers a sequence of predictions from previous tasksand currently available information [89]. This is because thesurrogate is trained at every surrogate interval and the surro-gate gives an estimation of the likelihood until the next intervalis reached for further retraining based on accumulated data ofproposal and true-likelihood for the previous interval.

We observe that there is systematic under-estimation of thetrue likelihood for some of the cases (Figures 4-6). One rea-son could be that the variance is overestimated and the otheris due to the global surrogate model, where training is done byone model in the manager process and the knowledge is usedin all the replicas. The surrogate accuracy depends on the na-ture of the problem, and the neural network model, in termsof the number of parameters in the model and the size of thedataset. We find that smaller Bayesian neural network mod-els had good accuracy in surrogate estimation when comparedto others. In future work, we can consider training surrogatemodel in the local replicas which could further improve the esti-mation. The global surrogate model has the advantage of com-bining information across the different replicas, it faces chal-lenges of dealing with large surrogate training dataset whichaccumulates over sampling time. We also need to lower thetime taken for the exchange of knowledge needed for decisionmaking by the local surrogate model and the master replica. Wefound that bigger problems (such as Pen-Digit and Chess) givefurther challenges for surrogate estimation. In such cases, it

10

Table 2: Classification Results (Random-walk proposal distribution)

Dataset Method Train Accuracy Test Accuracy Elapsed TimeSAPT(S prob) [mean, std, best] [mean, std, best] (minutes)

Iris PT-RW 51.39 15.02 91.43 50.18 41.78 100.00 1.26SAPT-RW (0.25) 69.31 2.05 80.00 65.24 4.04 78.38 1.93SAPT-RW (0.50) 44.11 11.93 78.89 50.01 5.65 70.30 1.81

Ionosphere PT-RW 68.92 16.53 91.84 51.29 30.73 91.74 3.50SAPT-RW (0.25) 76.60 4.42 86.94 61.23 8.05 86.24 2.93SAPT-RW (0.50) 70.34 6.15 83.27 73.42 14.06 95.41 2.43

Cancer PT-RW 83.78 20.79 97.14 83.55 27.85 99.52 2.78SAPT-RW(0.25) 89.75 6.91 96.32 92.80 4.57 99.05 3.41SAPT-RW(0.50) 91.17 6.29 96.52 97.64 3.17 99.52 2.84

Bank PT-RW 78.39 1.34 80.11 77.49 0.90 79.45 27.71SAPT-RW(0.25) 78.44 0.67 79.69 77.79 0.63 79.60 28.67SAPT(0.50) 77.82 1.05 79.69 77.16 0.91 78.80 27.38

Pen Digit PT-RW 76.67 17.44 95.24 71.93 16.59 90.62 57.13SAPT-RW(0.25) 88.74 1.94 92.87 83.60 2.14 88.74 49.25SAPT-RW(0.50) 80.85 1.28 82.87 77.66 1.08 80.02 36.05

Chess PT-RW 89.48 17.46 100.00 90.06 15.93 100.00 252.56SAPT-RW(0.25) 97.17 8.35 100.00 97.66 6.83 100.00 197.61SAPT-RW(0.50) 90.87 13.35 100.00 90.71 13.31 100.00 143.75

would be worthwhile to take a time series approach, where thehistory of the likelihood is treated as time series. Hence, a localsurrogate could learn from the past tend of the likelihood, ratherthan the history of past proposals. This could help in address-ing computational challenges given a large number of modelparameters to be considered for surrogate training.

Although we ruled out Gaussian process models as thechoice of the surrogate model due to computational complexityin training large surrogate data, we need to consider that Gaus-sian process models naturally account for uncertainty quantifi-cation in decision making. Recent techniques to address the is-sue of training Gaussian process models for large datasets couldbe a way ahead in future studies [90]. Moreover, we note thatthere has not been much work done in the literature that em-ploys surrogate assisted machine learning. Most of the liter-ature considered surrogate assisted optimization, whereas weconsidered inference for machine learning problems. The re-sults open the road to use surrogate models for machine learn-ing. Surrogates would be helpful in case of big data problemsand cases where there are inconsistencies or noisy data. Fur-thermore, other optimization methods could be used in conjunc-tion with surrogates for big data problems rather than paralleltempering.

Finally, we use the surrogate model with a certain probabil-ity in all replicas, hence the resulting distribution is only anapproximation of the true posterior distribution. An interest-ing question would then be whether there is a way to use sur-rogates while not introducing an approximation error. This istrivially possible for sequential approaches; however, in paral-lel cases, this could be an issue since multiple replicas are usedwhich all contribute to different levels of uncertainties. Thiswould need to be addressed in future work in uncertainty quan-tification for the surrogate likelihood prediction, perhaps withthe use of Gaussian process surrogate models or by introduc-

ing an error term for each surrogate model and integrating itout via MCMC sampling. Potentially, multi-fidelity modellingwhere the synergy between low-fidelity and how fidelity dataand models could be employed for the surrogate model [91].

6. Conclusions and Future Work

We presented surrogate-assisted parallel tempering for im-plementing Bayesian inference for computationally expensiveproblems that harness the advantage of parallel processing. Weused a Bayesian neural network model to demonstrate the ef-fectiveness of the framework to depict computationally expen-sive problems. The results from the experiments reveal that themethod gives a promising performance where computationaltime is reduced for larger problems.

The surrogate-based framework is flexible and can incor-porate different surrogate models and be applied to problemsacross various domains that feature computationally expen-sive models which require parameter estimation and uncertaintyquantification. In future work, we envision strategies to furtherimprove the surrogate estimation for large models and parame-ters. A way ahead is to utilize local surrogate via time seriesprediction could help in alleviating the challenges. Further-more, the framework could be applied to problems in differentdomains such as computationally expensive geoscientific mod-els used for landscape evolution.

Software and Data

We provide an open-source implementation of the proposedalgorithm in Python along with data and sample results 4.

4Surrogate-assisted multi-core parallel tempering:https://github.com/Sydney-machine-learning/surrogate-assisted-parallel-tempering

11

Table 3: Classification Results (Langevin-gradient proposal distribution)

Dataset Method Train Accuracy Test Accuracy Elapsed Time[mean, std, best] [mean, std, best] (minutes)

Iris PT-LG 97.32 0.92 99.05 96.76 0.96 99.10 2.09SAPT-LG (0.25) 98.91 0.16 100.00 99.93 0.39 100.00 2.85SAPT-LG (0.50) 99.09 0.43 100.00 98.63 1.57 100.00 2.49

Ionosphere PT-LG 98.55 0.55 99.59 92.19 2.92 98.17 5.07SAPT-LG (0.25 ) 100.00 0.02 100.00 90.82 2.43 96.33 6.17SAPT-LG (0.50 ) 99.51 0.60 100.00 91.24 1.82 97.25 4.76

Cancer PT-LG 97.00 0.29 97.75 98.77 0.32 99.52 5.09SAPT-LG(0.25) 99.36 0.11 99.39 98.00 0.76 99.52 8.18SAPT-LG(0.50) 99.37 0.12 99.59 98.61 0.65 100.00 6.64

Bank PT-LG 80.75 1.45 85.41 79.96 0.81 82.61 86.94SAPT-LG(0.25) 79.86 0.15 80.30 80.53 0.28 79.22 75.96SAPT-LG(0.50) 80.86 0.15 80.30 81.53 0.28 79.25 65.11

Pen Digit PT-LG 84.98 7.42 96.02 81.24 6.82 91.25 86.62SAPT-LG(0.25) 82.12 7.42 94.02 82.24 6.82 93.25 66.62SAPT-LG(0.50) 83.98 7.42 95.02 83.14 6.82 92.25 56.62

Chess PT-LG 100.00 0.00 100.00 100.00 0.00 100.00 323.10SAPT-LG(0.25) 100.00 0.00 100.00 100.00 0.00 100.00 223.70SAPT-LG(0.50) 100.00 0.00 100.00 100.00 0.00 100.00 173.10

Table 4: Surrogate Accuracy

Dataset Method Surrogate Prediction Surrogate TrainingRMSE RMSE [mean, std]

Iris SAPT-RW 3.55 3.84e-05 8.40e-05Ionosphere SAPT-RW 2.63 7.20e-05 1.62e-04Cancer SAPT-RW 11.08 6.12e-05 1.58e-04Bank SAPT-RW 131.26 3.09e-05 1.53e-04Pen-Digit SAPT-RW 1246.60 3.93e-06 1.47e-05Chess SAPT-RW 4026.44 1.34e-06 3.58e-06

Credit Author Statement

Rohitash Chandra: conceptualization, methodology, experi-mentation, writing and supervision; Konark Jain: initial soft-ware development and experimentation; Arpit Kapoor: soft-ware development and experimentation; Ashray Aman: soft-ware development, experimentation, and visualisation.

Acknowledgement

The authors would like to thank Prof. Dietmar Muller andDanial Azam for discussions and support during the course ofthis research project. We sincerely thank the editors and anony-mous reviewers for their valuable comments.

References

[1] J. Schmidhuber, Deep learning in neural networks: An overview, Neuralnetworks 61 (2015) 85–117.

[2] D. J. MacKay, Probable networks and plausible predictionsa review ofpractical bayesian methods for supervised neural networks, Network:Computation in Neural Systems 6 (3) (1995) 469–505.

[3] C. Robert, Machine learning, a probabilistic perspective (2014).[4] W. K. Hastings, Monte carlo sampling methods using markov chains and

their applications, Biometrika 57 (1) (1970) 97–109.

[5] N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller,E. Teller, Equation of state calculations by fast computing machines, Thejournal of chemical physics 21 (6) (1953) 1087–1092.

[6] A. Tarantola, B. Valette, et al., Inverse problems= quest for information,Journal of geophysics 50 (1) (1982) 159–170.

[7] K. Mosegaard, A. Tarantola, Monte carlo sampling of solutions to inverseproblems, Journal of Geophysical Research: Solid Earth 100 (B7) (1995)12431–12447.

[8] A. E. Raftery, S. M. Lewis, Implementing mcmc, Markov chain MonteCarlo in practice (1996) 115–130.

[9] D. van Ravenzwaaij, P. Cassey, S. D. Brown, A simple introduction tomarkov chain monte–carlo sampling, Psychonomic bulletin & review(2016) 1–12.

[10] S. Banerjee, B. P. Carlin, A. E. Gelfand, Hierarchical modeling and anal-ysis for spatial data, Crc Press, 2014.

[11] C. P. Robert, V. Elvira, N. Tawn, C. Wu, Accelerating mcmc algorithms,Wiley Interdisciplinary Reviews: Computational Statistics 10 (5) (2018)e1435.

[12] Y. Gal, Z. Ghahramani, Dropout as a bayesian approximation: Repre-senting model uncertainty in deep learning, in: international conferenceon machine learning, 2016, pp. 1050–1059.

[13] A. Kendall, Y. Gal, What uncertainties do we need in bayesian deep learn-ing for computer vision?, in: Advances in Neural Information ProcessingSystems, 2017, pp. 5580–5590.

[14] D. M. Blei, A. Kucukelbir, J. D. McAuliffe, Variational inference: Areview for statisticians, Journal of the American Statistical Association112 (518) (2017) 859–877.

[15] A. C. Damianou, M. K. Titsias, N. D. Lawrence, Variational inference forlatent variables and uncertain inputs in gaussian processes, The Journalof Machine Learning Research 17 (1) (2016) 1425–1486.

12

Table 5: Effect of number of replica on the computational time (minutes)

Num. Replica Cancer Bank Pen-digit Chess

4 45.66 79.25 102.28 253.986 21.16 43.48 61.38 188.798 13.25 32.57 51.92 185.5210 9.17 29.75 44.7 130.57

0 5000 10000 15000 20000Samples per Replica [R-1, R-2 ..., R-N]

120

100

80

60

40

20

Log

-Lik

elih

ood

Surrogate True

0 5000 10000 15000 20000Samples per Replica [R-1, R-2 ..., R-N]

300

250

200

150

100

50

Log

-Lik

elih

ood

Surrogate True

Figure 4: The Iris (top) and Cancer (bottom) surrogate accuracy. The dashedline denotes the real likelihood function while the blue line gives the surrogatelikelihood estimation.

0 5000 10000 15000 20000Samples per Replica [R-1, R-2 ..., R-N]

180

160

140

120

100

80

60

40

Log

-Lik

elih

ood

Surrogate True

0 5000 10000 15000 20000Samples per Replica [R-1, R-2 ..., R-N]

4500

4000

3500

3000

2500

2000

1500

1000

Log

-Lik

elih

ood

Surrogate True

Figure 5: The Ionosphere (top) and Bank (bottom) surrogate accuracy. Thedashed line denotes the real likelihood function while the blue line gives thesurrogate likelihood estimation.

13

0 5000 10000 15000 20000Samples per Replica [R-1, R-2 ..., R-N]

16000

14000

12000

10000

8000

6000

4000

Log

-Lik

elih

ood

Surrogate True

0 2000 4000 6000 8000 10000Samples per Replica [R-1, R-2 ..., R-N]

45000

40000

35000

30000

25000

20000

15000

10000

Log

-Lik

elih

ood

Surrogate True

Figure 6: The Pen-Digit (top) and Chess (bottom) surrogate accuracy. Thedashed line denotes the real likelihood function while the blue line gives thesurrogate likelihood estimation.

[16] C. Blundell, J. Cornebise, K. Kavukcuoglu, D. Wierstra, Weight uncer-tainty in neural network, in: Proceedings of The 32nd International Con-ference on Machine Learning, 2015, pp. 1613–1622.

[17] R. H. Swendsen, J.-S. Wang, Nonuniversal critical dynamics in montecarlo simulations, Physical review letters 58 (2) (1987) 86.

[18] E. Marinari, G. Parisi, Simulated tempering: a new monte carlo scheme,EPL (Europhysics Letters) 19 (6) (1992) 451.

[19] C. J. Geyer, E. A. Thompson, Annealing markov chain monte carlo withapplications to ancestral inference, Journal of the American StatisticalAssociation 90 (431) (1995) 909–920.

[20] A. Patriksson, D. van der Spoel, A temperature predictor for parallel tem-pering simulations, Physical Chemistry Chemical Physics 10 (15) (2008)2073–2077.

[21] K. Hukushima, K. Nemoto, Exchange monte carlo method and applica-tion to spin glass simulations, Journal of the Physical Society of Japan65 (6) (1996) 1604–1608.

[22] L. Lamport, On interprocess communication, Distributed computing 1 (2)(1986) 86–101.

[23] R. Salakhutdinov, A. Mnih, G. Hinton, Restricted boltzmann machinesfor collaborative filtering, in: Proceedings of the 24th international con-ference on Machine learning, ACM, 2007, pp. 791–798.

[24] A. Fischer, C. Igel, A bound for the convergence rate of paral-lel tempering for sampling restricted boltzmann machines, The-oretical Computer Science 598 (2015) 102 – 117. doi:https:

//doi.org/10.1016/j.tcs.2015.05.019.URL http://www.sciencedirect.com/science/article/pii/

S0304397515004235

[25] G. Desjardins, A. Courville, Y. Bengio, P. Vincent, O. Delalleau, Tem-pered markov chain monte carlo for training of restricted boltzmann ma-chines, in: Proceedings of the thirteenth international conference on arti-ficial intelligence and statistics, 2010, pp. 145–152.

[26] P. Brakel, S. Dieleman, B. Schrauwen, Training restricted boltzmann ma-chines with multi-tempering: harnessing parallelization, in: InternationalConference on Artificial Neural Networks, Springer, 2012, pp. 92–99.

[27] R. Chandra, K. Jain, R. V. Deo, S. Cripps, Langevin-gradient paralleltempering for bayesian neural learning, Neurocomputing 359 (2019) 315– 326.

[28] R. M. Hicks, P. A. Henne, Wing design by numerical optimization, Jour-nal of Aircraft 15 (7) (1978) 407–412.

[29] Y. Jin, Surrogate-assisted evolutionary computation: Recent advances andfuture challenges, Swarm and Evolutionary Computation 1 (2) (2011) 61–70.

[30] Y. S. Ong, P. B. Nair, A. J. Keane, Evolutionary optimization of computa-tionally expensive problems via surrogate modeling, AIAA journal 41 (4)(2003) 687–696.

[31] Z. Zhou, Y. S. Ong, P. B. Nair, A. J. Keane, K. Y. Lum, Combining globaland local surrogate models to accelerate evolutionary optimization, IEEETransactions on Systems, Man, and Cybernetics, Part C (Applications andReviews) 37 (1) (2007) 66–76.

[32] Y. S. Ong, P. Nair, A. Keane, K. Wong, Surrogate-assisted evolutionaryoptimization frameworks for high-fidelity engineering design problems,in: Knowledge Incorporation in Evolutionary Computation, Springer,2005, pp. 307–331.

[33] S. Jeong, M. Murayama, K. Yamamoto, Efficient optimization designmethod using kriging model, Journal of aircraft 42 (2) (2005) 413–420.

[34] A. Samad, K.-Y. Kim, T. Goel, R. T. Haftka, W. Shyy, Multiple surro-gate modeling for axial compressor blade shape optimization, Journal ofPropulsion and Power 24 (2) (2008) 301–310.

[35] M. D. Richard, R. P. Lippmann, Neural network classifiers estimatebayesian a posteriori probabilities, Neural computation 3 (4) (1991) 461–483.

[36] D. J. MacKay, Hyperparameters: Optimize, or integrate out?, in: Maxi-mum entropy and Bayesian methods, Springer, 1996, pp. 43–59.

[37] H. Wang, Y. Zeng, X. Yu, G. Li, E. Li, Surrogate-assisted bayesian infer-ence inverse material identification method and application to advancedhigh strength steel, Inverse Problems in Science and Engineering 24 (7)(2016) 1133–1161.

[38] A. Ray, D. Myer, Bayesian geophysical inversion with trans-dimensionalGaussian process machine learning, Geophysical Journal International217 (3) (2019) 1706–1726.

[39] D. A. Freedman, On the asymptotic behavior of bayes’ estimates in the

14

discrete case, The Annals of Mathematical Statistics (1963) 1386–1403.[40] H. White, Maximum likelihood estimation of misspecified models,

Econometrica: Journal of the Econometric Society (1982) 1–25.[41] A. Krogh, J. A. Hertz, A simple weight decay can improve generalization,

in: Advances in neural information processing systems, 1992, pp. 950–957.

[42] R. M. Neal, Bayesian learning for neural networks, Vol. 118, SpringerScience & Business Media, 2012.

[43] T. Auld, A. W. Moore, S. F. Gull, Bayesian neural networks for Internettraffic classification, IEEE Transactions on neural networks 18 (1) (2007)223–239.

[44] R. M. Neal, et al., Mcmc using hamiltonian dynamics, Handbook ofMarkov Chain Monte Carlo 2 (11) (2011).

[45] M. Welling, Y. W. Teh, Bayesian learning via stochastic gradient langevindynamics, in: Proceedings of the 28th International Conference on Ma-chine Learning (ICML-11), 2011, pp. 681–688.

[46] R. Chandra, L. Azizi, S. Cripps, Bayesian neural learning via langevindynamics for chaotic time series prediction, in: D. Liu, S. Xie, Y. Li,D. Zhao, E.-S. M. El-Alfy (Eds.), Neural Information Processing, 2017.

[47] G. E. Hinton, S. Osindero, Y.-W. Teh, A fast learning algorithm for deepbelief nets, Neural computation 18 (7) (2006) 1527–1554.

[48] K. Cho, A. Ilin, T. Raiko, Improved learning of gaussian-bernoulli re-stricted boltzmann machines, in: International conference on artificialneural networks, Springer, 2011, pp. 10–17.

[49] K. Cho, T. Raiko, A. Ilin, Parallel tempering is efficient for learning re-stricted boltzmann machines, in: Neural Networks (IJCNN), The 2010International Joint Conference on, IEEE, 2010, pp. 1–8.

[50] G. Desjardins, A. Courville, Y. Bengio, Adaptive parallel temperingfor stochastic maximum likelihood learning of rbms, arXiv preprintarXiv:1012.3476 (2010).

[51] G. Desjardins, H. Luo, A. Courville, Y. Bengio, Deep tempering, arXivpreprint arXiv:1410.0123 (2014).

[52] C. Bishop, C. M. Bishop, et al., Neural networks for pattern recognition,Oxford university press, 1995.

[53] R. H. Swendsen, J.-S. Wang, Replica monte carlo simulation of spin-glasses, Physical review letters 57 (21) (1986) 2607.

[54] U. H. Hansmann, Parallel tempering algorithm for conformational studiesof biological molecules, Chemical Physics Letters 281 (1-3) (1997) 140–150.

[55] M. Sambridge, A parallel tempering algorithm for probabilistic samplingand multimodal optimization, Geophysical Journal International 196 (1)(2014) 357–374.

[56] A. Ray, A. Sekar, G. M. Hoversten, U. Albertin, Frequency domain fullwaveform elastic inversion of marine seismic data from the alba field us-ing a bayesian trans-dimensional algorithm, Geophysical Journal Interna-tional 205 (2) (2016) 915–937.

[57] F. Calvo, All-exchanges parallel tempering, The Journal of chemicalphysics 123 (12) (2005) 124106.

[58] N. Rathore, M. Chopra, J. J. de Pablo, Optimal allocation of replicas inparallel tempering simulations, The Journal of chemical physics 122 (2)(2005) 024111.

[59] H. G. Katzgraber, S. Trebst, D. A. Huse, M. Troyer, Feedback-optimizedparallel tempering monte carlo, Journal of Statistical Mechanics: Theoryand Experiment 2006 (03) (2006) P03018.

[60] E. Bittner, A. Nußbaumer, W. Janke, Make life simple: Unleash thefull power of the parallel tempering algorithm, Physical review letters101 (13) (2008) 130603.

[61] M. K. Sen, P. L. Stoffa, Bayesian inference, gibbs’ sampler and un-certainty estimation in geophysical inversion, Geophysical Prospecting44 (2) (1996) 313–350.

[62] M. Maraschini, S. Foti, A monte carlo multimodal inversion of surfacewaves, Geophysical Journal International 182 (3) (2010) 1557–1566.

[63] A. Ray, D. L. Alumbaugh, G. M. Hoversten, K. Key, Robust and acceler-ated bayesian inversion of marine controlled-source electromagnetic datausing parallel tempering, Geophysics 78 (6) (2013) E271–E280.

[64] A. Ray, S. Kaplan, J. Washbourne, U. Albertin, Low frequency full wave-form seismic inversion within a tree based bayesian framework, Geophys-ical Journal International 212 (1) (2018) 522–542.

[65] Y. Li, M. Mascagni, A. Gorin, A decentralized parallel implementationfor parallel tempering algorithm, Parallel Computing 35 (5) (2009) 269 –283.

[66] K. Karimi, N. Dickson, F. Hamze, High-performance physics simula-tions using multi-core cpus and gpgpus in a volunteer computing context,The International Journal of High Performance Computing Applications25 (1) (2011) 61–69.

[67] G. Mingas, L. Bottolo, C.-S. Bouganis, Particle mcmc algorithms and ar-chitectures for accelerating inference in state-space models, InternationalJournal of Approximate Reasoning 83 (2017) 413–433.

[68] W. Vousden, W. M. Farr, I. Mandel, Dynamic temperature selection forparallel tempering in markov chain monte carlo simulations, Monthly No-tices of the Royal Astronomical Society 455 (2) (2015) 1919–1937.

[69] D. J. Earl, M. W. Deem, Parallel tempering: Theory, applications, andnew perspectives, Physical Chemistry Chemical Physics 7 (23) (2005)3910–3916.

[70] D. C. Montgomery, J. Vernon M. Bettencourt, Multiple response surfacemethods in computer simulation, SIMULATION 29 (4) (1977) 113–121.

[71] J. D. Letsinger, R. H. Myers, M. Lentner, Response surface methods forbi-randomization structures, Journal of quality technology 28 (4) (1996)381–397.

[72] V. Tandjiria, C. I. Teh, B. K. Low, Reliability analysis of laterally loadedpiles using response surface methods, Structural Safety 22 (4) (2000)335–355.

[73] D. Lim, Y. Jin, Y.-S. Ong, B. Sendhoff, Generalizing surrogate-assistedevolutionary computation, IEEE Transactions on Evolutionary Computa-tion 14 (3) (2010) 329–355.

[74] S. Razavi, B. A. Tolson, D. H. Burn, Review of surrogate modeling inwater resources, Water Resources Research 48 (7) (2012).

[75] A. Dıaz-Manrıquez, G. Toscano, J. H. Barron-Zambrano, E. Tello-Leal,A review of surrogate assisted multiobjective evolutionary algorithms,Computational intelligence and neuroscience 2016 (2016).

[76] A. Giunta, L. Watson, A comparison of approximation model-ing techniques-polynomial versus interpolating models, in: 7thAIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysisand Optimization, p. 4758.

[77] R. Jin, W. Chen, T. W. Simpson, Comparative studies of metamodellingtechniques under multiple modelling criteria, Structural and multidisci-plinary optimization 23 (1) (2001) 1–13.

[78] R. Chandra, R. D. Muller, D. Azam, R. Deo, N. Butterworth, T. Salles,S. Cripps, Multicore parallel tempering Bayeslands for basin and land-scape evolution, Geochemistry, Geophysics, Geosystems 20 (11) (2019)5082–5104.

[79] M. Sambridge, A parallel tempering algorithm for probabilistic samplingand multimodal optimization, Geophysical Journal International 196 (1)(2013) 357–374.

[80] R. Chandra, L. Azizi, S. Cripps, Bayesian neural learning via langevin dy-namics for chaotic time series prediction, in: Neural Information Process-ing - 24th International Conference, ICONIP 2017, Guangzhou, China,November 14-18, 2017, Proceedings, Part V, 2017, pp. 564–573.

[81] D. S. Broomhead, D. Lowe, Radial basis functions, multi-variable func-tional interpolation and adaptive networks, Tech. rep., Royal Signals andRadar Establishment Malvern (United Kingdom) (1988).

[82] C. E. Rasmussen, Gaussian processes in machine learning, in: Advancedlectures on machine learning, Springer, 2004, pp. 63–71.

[83] P.-T. de Boer, D. P. Kroese, S. Mannor, R. Y. Rubinstein, A tutorial onthe cross-entropy method, Annals of Operations Research 134 (1) (2005)19–67. doi:10.1007/s10479-005-5724-z.URL https://doi.org/10.1007/s10479-005-5724-z

[84] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXivpreprint arXiv:1412.6980 (2014).

[85] T. Tieleman, G. Hinton, Lecture 6.5-rmsprop: Divide the gradient by arunning average of its recent magnitude, COURSERA: Neural networksfor machine learning 4 (2) (2012) 26–31.

[86] J. Duchi, E. Hazan, Y. Singer, Adaptive subgradient methods for onlinelearning and stochastic optimization, Journal of Machine Learning Re-search 12 (Jul) (2011) 2121–2159.

[87] D. Dua, C. Graff, UCI machine learning repository (2017).URL http://archive.ics.uci.edu/ml

[88] J. Pall, R. Chandra, D. Azam, T. Salles, J. M. Webster, R. Scalzo,S. Cripps, Bayesreef: A bayesian inference framework for modelling reefgrowth in response to environmental change and biological dynamics, En-vironmental Modelling & Software (2020) 104610.

[89] S. Shalev-Shwartz, et al., Online learning and online convex optimization,

15

Foundations and Trends R© in Machine Learning 4 (2) (2012) 107–194.[90] C. J. Moore, A. J. Chua, C. P. Berry, J. R. Gair, Fast methods for training

gaussian processes on large datasets, Royal Society open science 3 (5)(2016) 160125.

[91] B. Peherstorfer, K. Willcox, M. Gunzburger, Survey of multifidelity meth-ods in uncertainty propagation, inference, and optimization, Siam Review60 (3) (2018) 550–591.

16