sequential sampling for dynamic environment map illumination · a. ghosh, a. doucet, & w....

Eurographics Symposium on Rendering (2006)Tomas Akenine-Möller and Wolfgang Heidrich (Editors)

Sequential Sampling for Dynamic Environment MapIllumination

Abhijeet Ghosh, Arnaud Doucet, and Wolfgang Heidrich

The University of British Columbia†

Abstract

Sampling complex illumination in the form of environment maps has received a lot of attention in computer graph-ics. Recent work in this area has demonstrated that drawing samples from the product of light and BRDF producessuperior results to other sampling strategies. However, existing methods in this area consider only individualframes, and do not take advantage of coherence in animations. In this paper, we introduce a sequential samplingapproach for dynamic environment map illumination. Our algorithm efficiently samples from the product of illu-mination and BRDF, while exploiting temporal coherence. We demonstrate significant performance benefits overexisting methods.

Keywords: Methods and Applications – Monte Carlo Techniques; Rendering – Ray Tracing; Rendering – GlobalIllumination.

Categories and Subject Descriptors (according to ACM CCS): I.3.7 [COMPUTER GRAPHICS]: Three-DimensionalGraphics and Realism, Raytracing.

1. Introduction

The sampling of complex direct illumination in the formof high dynamic range (HDR) environment maps has re-ceived significant attention in recent years, with major ap-plications in realistic relighting. The best known techniquesfor direct illumination sample from the product of the in-cident illumination and the surface reflectance [BGH05,CJAMJ05, TCE05]. With advancements in HDR acquisitiontechnologies, HDR video environments are becoming in-creasingly available. This availability has spawned recentwork on sampling dynamic illumination from such HDRvideo sequences [HSK∗05, WWL05]. However, these tech-niques only take the dynamic importance of the illumina-tion into account while proposing samples for the video se-quence. Such techniques are problematic in the presence ofhigh frequencies in both the illumination as well as the sur-face BRDFs. While one can produce noise-free images by

† E-mail:ghosh, arnaud, [email protected]

using one set of samples for all surface locations, doing soeliminates the impact of dimmer light sources, which shoulddominate the reflection for certain surface orientations.

In this work, we aim to efficiently sample from the prod-uct distribution of the illumination and the BRDF in avideo sequence with dynamic illumination using a Sequen-tial Monte Carlo (SMC) sampling strategy. The basic ideais to generate samples according to the product distribu-tion in the first frame of the sequence, and thereafter to fil-ter these samples (particles) in the subsequent frames ac-cording to the dynamic product distribution. This sequentialsampling mechanism is more efficient than independentlyre-regenerating the samples for every time step (Figure 1),especially for scenes with high frequencies in both the dy-namic illumination and the BRDF. At the same time, ourmethod avoids systematic under-estimation of reflections atcertain angles, which is common to dynamic importancemethods generating point light approximations of the envi-ronment.

Our solution to sampling from the dynamic product dis-

c© The Eurographics Association 2006.

A. Ghosh, A. Doucet, & W. Heidrich / Sequential Sampling for Dynamic Environment Map Illumination

Figure 1: Quality comparison of our SMC sampling algorithm with bidirectional importance sampling for a sky probe sequence.Left: 1st frame rendered using bidirectional importance sampling in 8 seconds. Center: 5th frame of sequence rendered usingSMC sampling in 4 seconds. Right: Comparison image for 5th frame generated using bidirectional importance sampling in 4seconds.

tribution is a two-step approach. We assume that we havealready obtained a sample set according to the product dis-tribution of the previous frame. For the first frame of theanimation, we generate this sample set with bidirectionalimportance sampling [BGH05]. In the first step of our al-gorithm, samples distributed according to the product dis-tribution of the previous time step are propagated in timeusing sequential importance sampling. The product distribu-tion at the new time step is incrementally estimated usingthe weights of the sequential importance. The second stepextends the path of each of these samples using a MarkovChain Monte Carlo (MCMC) kernel whose invariant dis-tribution is the target distribution at the current time step.The MCMC kernel is implemented using the Metropolis-Hastings (MH) algorithm [MRR∗53]. No visibility tests areperformed during either of the two steps. Visibility is finallytested at the end of the second step in order to obtain a MonteCarlo estimate of the direct illumination. This approach hasthe following benefits:

• We need to propagate only a small number of samples intime to estimate the direct illumination as these samplesare distributed according to the target distribution at eachtime step. This makes sample propagation very efficient.

• The normalization constant for the product distribution ateach time step can be incrementally computed using thesequential importance weights. Thus, the normalizationconstant, i.e. the un-occluded reflected radiance at eachtime step can be estimated very efficiently without draw-ing a large number of samples.

• Sample generation cost at each time step is independent ofthe cost of sampling from the BRDF representation sincethe algorithm only requires to evaluate the BRDF but doesnot require to sample from it. Thus, any complex BRDF

representation can be used without impacting sample gen-eration cost.

• The method creates samples on the fly and does not re-quire any expensive precomputation.

2. Related Work

The computation of the direct illumination in a scene is acostly task, especially in the presence of complex real-worldlight sources such as environment maps and other image-based representations. Much effort has been focused on thedevelopment of efficient techniques for completing this taskin the presence of static as well as dynamic illumination.

2.1. Sampling from Environment Maps and BRDFs

The easiest way of dealing with this situation is by sam-pling from either the environment map or the BRDF,but not both. To sample from the energy distributionof the environment, researchers have used point relax-ation schemes [CD01, KK03], hierarchical distributionschemes [ODJ04, Deb05], or inversion of the cumula-tive density function (CDF) [LRR05, SHS02]. Agarwal etal. [ARBJ03] introduced a sampling method for environmentmaps that takes into account both the energy distribution inthe environment map and the solid angle separating the sam-ples to avoid oversampling of bright regions.

All these methods work best for high-frequency illumina-tion, but low-frequency BRDFs. In cases where the BRDFcontains high frequencies, but the illumination does not,sampling form the BRDF reduces the variance. Dependingon the BRDF representation, different strategies can be used,for example for analytical BRDFs (e.g. [Shi00]), or tabulatedBRDFs [MH97, LRR04].



When both the environment and the BRDF contain highfrequencies, neither of these methods works very well.Veach and Guibas [VG95] were the first to consider thisproblem. Their multiple importance sampling combined dif-ferent sampling distributions such as illumination and BRDFdistributions using balance heuristics.

However, as shown recently, sampling directly from theproduct of light and BRDF produces even better results.Several strategies have recently been introduced to this end.Burke et al. [BGH05] present two algorithms for samplingaccording to the product distribution, one based on rejec-tion sampling and the other based on sampling-importance-resampling (SIR), the latter of which is also used by Talbotet al. [TCE05]. Clarberg et al. [CJAMJ05] present a tech-nique for efficiently sampling the product of the illumina-tion and the BRDF using a hierarchical wavelet representa-tion. Their method is very efficient for tabulated BRDFs butrequires significant precomputation for environment mapsmaking it unsuitable for dynamic illumination. Lawrence etal. [LRR05] present an approach for compressing cumula-tive density functions for efficient inversion and apply it tosampling from many precomputed environment map PDFsfor different surface orientations. This can be seen as an ap-proximation of the product distribution.

Our proposed SMC mechanism for sampling dynamic il-lumination is very general and can be applied in combinationwith any sampling scheme discussed above for proposingsamples in the first time step. In fact, sequential samplingwould also help overcome the precomputation requirementsof some of these techniques for a dynamic sequence. In ourwork, we employ Burke et al.’s SIR algorithm for bidirec-tional importance sampling [BGH05] in the first frame ofthe sequence.

2.2. Sampling from Dynamic Illumination

Several researchers have considered the problem of sam-pling from dynamic illumination. For example, Sbert etal. [SSSK04] presented a method for reusing light pathscomputed in one frame of a light animation sequence in allother frames using multiple importance sampling. The indi-rect illumination in each frame of the sequence is approx-imated as weighted contributions from these precomputedvirtual point lights. This method works on moving pointlights, not complex environments.

Some researchers have looked at path re-usage in the con-text of global illumination for a sequence of camera anima-tion via reprojection of the primary ray hits on to the imageplane. Here, techniques that are possibly biased [HDMS03]as well as unbiased techniques [MFSSK06] using multipleimportance sampling have been proposed for the path re-usage. Our proposed SMC sampling mechanism can also beapplied to a camera animation sequence for sample propaga-tion in an unbiased manner, while being more efficient thanmultiple importance sampling.

Recent work on dynamic environment maps, includingWan et al. [WWL05] and Havran et al [HDS03], approxi-mates environments with a set of point lights that are drawnaccording to the energy distribution in the environment map,and evolve smoothly over time. However, this procedure in-troduces a systematic error for specular materials if none ofthe chosen point lights resides inside the specular lobe. Inthis case, one would expect to see a specular reflection of adimmer part of the environment, but these methods cannotproduce this result.

2.3. Metropolis Sampling for Global Illumination

Our method includes a Metropolis sampling strategy. Veach& Guibas [VG97] first applied Metropolis sampling to theproblem of image synthesis and developed a general, robustand unbiased algorithm called Metropolis Light Transport(MLT) that was well suited for hard cases for sampling be-cause of its localized exploration and path re-usage proper-ties. Fan et al. [FCL05] recently applied the Metropolis algo-rithm for efficiently sampling coherent light paths for photonmapping.

Cline et al. [CTE05] presented an efficient unbiasedmethod to solve correlated integral problems with a hy-brid algorithm that uses Metropolis sampling-like mutationstrategies in a standard Monte Carlo integration setting,overcoming the startup bias problem of MLT. They apply en-ergy redistribution over the image plane to reduce varianceof path tracing for global illumination. Our work is similar inspirit in the sense of using initial Monte Carlo sampling forproposing samples for the subsequent time step, followed byMetropolis sampling in order to explore the new direct illu-mination target distribution.

3. Sequential Monte Carlo Sampling

As mentioned in the introduction, we propose a sequentialMonte Carlo (SMC) sampling algorithm for sampling ac-cording to the dynamic product distribution of the illumi-nation and the surface reflectance during an animation se-quence. Traditional SMC algorithms in the literature dealwith the case where the target distribution of interest at timen, defined on Ωn, is of a higher dimension than the targetdistribution at time n−1 [DdFG01]. A classical example ofthis is an SMC algorithm applied to sequential Bayesian in-ference. In our case, the target distribution at every time step,i.e. the direct illumination integral, is defined on a commonspace of the hemisphere of directions Ω. Hence, we employ aclass of SMC samplers recently developed for a common do-main [dMDJ06] to the problem of sampling from the productdistribution of incident illumination and BRDF in the pres-ence of dynamic illumination.

Our SMC sampling algorithm is a two-step approach: westart with samples created according to the product distribu-tion in the previous time step, and propagate them in time



using sequential importance sampling followed by a possi-ble resampling step. We then employ an appropriate MCMCtransition kernel to redistribute the samples according to theproduct distribution at the new time step. Thereafter thesesamples, now distributed according to the new target distri-bution, are used for visibility testing.

Consider the direct illumination at a point for a given ob-server direction ωr:

Lr(ωr) =Z

Ωfr(ωi → ωr)cosθiLi(ωi)V (ωi)dωi, (1)

with Li denoting the incident illumination from an environ-ment map, fr representing the BRDF, and V being the binaryvisibility term. Note that we treat Li and fr as scalar-valuedfunctions here and throughout the text. In practice, Li and frare color-valued functions, from which we derive the scalar-valued ones by averaging the color channels.

The target distribution of interest for direct illuminationis the product distribution p of incident illumination and theBRDF:

p(ωi) :=fr(ωi → ωr)cosθiLi(ωi)

R

Ω fr(ωi → ωr)cosθiLi(ωi)dωi=

p(ωi)

Lns. (2)

Here, p(ωi) = fr(ωi → ωr)cosθiLi(ωi) is the un-normalized importance function, and Lns =

R

Ω fr(ωi →ωr)cosθiLi(ωi)dωi is the un-occluded reflected radiance inthe viewing direction and is the normalization constant ofthe target distribution. Burke et al. [BGH05] refer to Lns as"radiance, no-shadows".

Our SMC algorithm works as follows: we start with sam-ples ω( j)

i,n−1 and weights W ( j)n−1, j = 1, ...,N, such that the

weighted samples are proportional to the product distribu-tion of BRDF and illumination in frame n−1. These samplesrepresent the incident light directions for one surface point,that is, the point visible at a specific pixel. For the first frame,these samples are obtained by bidirectional importance sam-pling, and all weights are 1/N. The two steps of our methodare then sample propagation followed by MCMC transitionto adjust the samples to the product distribution in the nextframe.

Step 1: Sample Propagation: We propogate the samplesω( j)

i,n−1 in time using sequential importance sampling. Theun-normalized incremental weight wn of every sample forsequential importance at time n is given by the followingratio:

w( j)n =

pn(ω( j)i,n−1)

pn−1(ω( j)i,n−1)

, (3)

This weight is just the ratio of the target function eval-uated at the sample point at time n to that evaluated at

time n− 1 for the same point, and represents the change inweighting of a sample due to changes in the target distribu-tion. The normalized weights for the N samples at time n arethen given by:

W ( j)n =

W ( j)n−1 · w

( j)n

∑Nk=1 W (k)

n−1 · w(k)n

. (4)

This tracking of weights to represent evolving target dis-tributions is called Sequential Monte Carlo Sampling, orSMC for short [dMDJ06]. Note that we can use the SMCmechanism for sampling from a dynamic sequence even ifwe use a different proposal distribution q instead of p forthe first frame. The only difference here would be that wewould have to appropriately weight the samples of the firstframe for sequential importance. The un-normalized weightsw( j)

1 would then need to be computed as:

w( j)1 =

p1(ω( j)i,1 )

q1(ω( j)i,1 )

,

and then normalized to obtained W ( j)1 .

Step 1a: Resampling. As the variance between the proposaldistribution qn and the target distribution pn tends to in-crease with n, the variance of the un-normalized importanceweights tends to increase resulting in a potential degener-acy of the particle approximation. This degeneracy can bemeasured using the criterion of effective sample size (ESS)(∑N

j=1(W( j)n )2)−1 [LC98]. The ESS takes values between 1

and N. If the ESS is below a pre-specified threshold, sayN/2, we resample the N samples according to the weightsW ( j)

n and set the weights of the resampled samples equalto 1/N. This resampling step discards samples with lowweights while copying the samples with high weights mul-tiple times, thus keeping the samples according to the targetdistribution. Note that the mutations in the next step makesure we do not keep multiple identical samples.

Step 2: MCMC Transitions. After sample propagation andpotential resampling, we apply an MCMC kernel Kn(ωi,ω′

i)

of invariant distribution pn to every sample ω( j)i,n−1 in or-

der to obtain new samples ω( j)i,n . The new samples ωi,n are

marginally distributed according to

pn(ω′

i) =Z

Ωpn−1(ωi)Kn(ωi,ω′

i)dωi. (5)

We employ the Metropolis-Hastings algorithm (MH) forthese transitions, with a mix of local random walk movesand independent proposal moves. For a detailed descriptionof the MH algorithm, we refer the reader to the chapter



Symbol Description

ω( j)i jth sample (incident direction)

W ( j)n Weight for jth sample in frame n

w( j)n Unnormalized weight for jth sample in frame n

N Number of samples per pixelpn Target PDF for frame npn Unnormalized importance (product of illumination and BRDF)qn Proposal distribution at time npk/P

n−1:n The kth of P intermediate distributions between frame n−1 and frame n

Table 1: Summary of notation used in this paper.

on Metropolis Sampling by Pharr in the SIGGRAPH 2004course notes [DJA∗04]. When using the MH algorithm, theMCMC kernel Kn of invariant distribution pn is described interms of an acceptance probability of a proposed transition:

a(ω( j)i → ω′( j)

i ) = min1,pn(ω′( j)

i )q(ω′( j)i → ω( j)

i )

pn(ω( j)i )q(ω( j)

i → ω′( j)i )

, (6)

where ω( j)i is the current sample, ω′( j)

i is the proposed sam-ple using the transition kernel q, and a is the acceptanceprobability of the proposed transition.

We mix local walk moves with independent proposalmoves as these independent moves are required to preventthe SMC samples from getting stuck in local possibly nar-row modes of the target distribution. For high frequency dy-namic lighting, we choose local walk moves with uniformrandom directional perturbations of up to a few degrees ofthe samples, while choosing samples from the environmentmap (EM) for the independent proposal moves. When us-ing local random walk moves, there is an equal probabilityof transition between ω( j)

i and ω′( j)i , i.e. q(ω( j)

i → ω′( j)i ) =

q(ω′( j)i → ω( j)

i ). Thus, the acceptance probability alocal ofthe local walk moves is given by:

alocal(ω( j)i → ω′( j)

i ) = min1,pn(ω′( j)

i )

pn(ω( j)i )

. (7)

When using independent proposal moves from the EM,the transition probability of the proposed sample is given byq(ω( j)

i → ω′( j)i ) = Li,n(ω

′( j)i )/

R

Ω Li,ndωi. Hence the accep-tance probability of the independent move is given by:

aEM(ω( j)i → ω′( j)

i ) = min1,pn(ω

′( j)i )Li,n(ω

( j)i )/

R

Ω Li,ndωi

pn(ω( j)i )Li,n(ω

′( j)i )/

R

Ω Li,ndωi

= min1,fr(ω

′( j)i →ωr)cos θ′( j)

i

fr(ω( j)i →ωr)cos θ( j)

i

.

(8)

In practice, we propose several MCMC moves per samplefor good exploration of the target distribution.

Note the SMC algorithm as described above is unbiased:Step 1 corresponds to an importance sampling step, whichproduces the correct distribution at time n using the distribu-tion at time n− 1 as importance. The variance of this dis-tribution is reduced by applying the MCMC algorithm inStep 2. Since MCMC algorithm works on an unbiased dis-tribution, startup-bias is avoided. A formal argument can befound in [dMDJ06].

3.1. Direct Illumination Estimate

With the sample sets and weights derived above, we can nowestimate the reflected radiance at a surface location as

LN,n,smc(ωr) = Lns,n

N

∑j=1

W ( j)n ·V (ω( j)

i,n ). (9)

Equation 9 can be interpreted as the scaling of the un-occluded reflected radiance Lns,n by the weighted average ofN visibility tests performed along the directions contribut-ing most significantly to the radiance at time n. Here, thenormalization constant Lns,n at time n can be incrementallyestimated as

Lns,n

Lns,n−1=

R

Ω pn(ωi,n−1)dωi,n−1R

Ω pn−1(ωi,n−1)dωi,n−1≈

N

∑j=1

W ( j)n−1 · w

( j)n .

(10)

The derivation of this result is given in Appendix A. It isworth pointing out that this incremental estimate of the nor-malization constant Lns,n at time n according to Equation 10provides the crucial advantage for the SMC algorithm interms of computational expense over a pure MC approachsuch as bidirectional sampling where the proper estimate ofLns requires drawing large number of samples. To estimateLns,n/Lns,1, one can use the product of estimates given byEquation 10 from time t = 2 to n. Lns,1 is estimated in ourcase during bidirectional sampling for the first time step.



A summary of the method can be found in Algorithm 1.The complexity of this algorithm is in O(N).

Algorithm 1 SMC Sampling for Dynamic Illumination1: INITIALIZATION• Set n = 1.• For j = 1, ...,N draw ω( j)

i,1 ∼ p1 using bidirectional

sampling and set W ( j)1 = 1/N.

Iterate steps 2 and 3:2: WEIGHTING AND RESAMPLING• Set n = n+1.• Compute new weights W ( j)

n according to Equations 3and 4.

• If ESS < T hreshold, resample and set W ( j)n = 1/N.

3: SAMPLING• For j = 1, ...,N draw ω( j)

i,n ∼ Kn(ωi,n−1,ωi,n).• Estimate Lns,n according to Equation 10.• Estimate reflected radiance according to Equation 9.

4. Variance Reduction with Intermediate Distributions

The aim of the SMC sampling algorithm as discussed in Sec-tion 3 is to "smoothly" move samples from the target distri-bution at time n−1 to the target distribution at time n. Moreformally, we have samples

ω( j)i,n−1 ∼ pn−1(ωi) =

pn−1(ωi)

Lns,n−1,

and we want to move towards samples

ω( j)i,n ∼ pn(ωi) =

pn(ωi)

Lns,n.

This transition is smooth under the assumption thatpn−1 ≈ pn. However, this may not be true in practice, es-pecially in the case where the dynamic illumination is in theform of a high frequency HDR video environment. If thediscrepancy between the two successive distributions is toohigh, this will result in high variance in the un-normalizedincremental weights w( j)

n , and thus indirectly result in highvariance in the normalized weights W ( j)

n . The variance inW ( j)

n can be countered with the resampling step after sequen-tial importance sampling, resulting in good estimates of theposterior pn. However, the resampling step does not affectthe variance in the un-normalized weights w( j)

n , which canlead to high variance in the incremental estimate of the nor-malization constant Lns,n according to Equation 10.

In this scenario, we introduce a sequence of intermediatedistributions [GM98] between the original distribution pn−1and the new one pn in order to select a smooth transition that

the sample can follow. These intermediate distributions areblends of the original distributions:

pγn−1:n(ωi) ∝ pγ

n−1:n(ωi) = [ pn−1(ωi)]1−γ[ pn(ωi)]

γ, (11)

such that

p0n−1:n(ωi) = pn−1(ωi), p1

n−1:n(ωi) = pn(ωi).

In practice, we introduce P discrete intermediate distri-butions: pk/P

n−1:n(ωi), where k = 1, ...,P. We can use thesenew distributions to reduce variance with little additionalcost. The idea is to reduce the variance in the incrementalweights wn by computing them as a product of P incrementalweights wk of these intermediate distributions. The consecu-tive intermediate distributions pk/P

n−1:n(ωi) and p(k−1)/Pn−1:n (ωi)

are closer to each other by construction than pn−1(ωi) is topn(ωi), resulting in flatter weights wk, as compared to wn.

The SMC algorithm with P intermediate distributions re-quires slight modifications to the algorithm discussed in Sec-tion 3. Instead of first computing the un-normalized weightsw( j)

n and the normalized weights W ( j) and then doing theMCMC transitions, the algorithm for P intermediate distri-butions computes the un-normalized weights w( j)

n as a prod-uct of P intermediate un-normalized weights wk,( j) that eachinvolve an MCMC kernel of invariant distribution pk/P

n−1:n.

Here, the intermediate un-normalized weights wk,( j) arecomputed as:

wk,( j) =pk/P

n−1:n(ωk−1,( j)i )

p(k−1)/Pn−1:n (ωk−1,( j)

i ). (12)

Assuming we have samples W ( j)n−1, ω( j)

i,n−1 approximat-ing pn−1. The algorithm then proceeds as follows:

Algorithm 2 SMC with Intermediate Distributions1: INITIALIZATION:• We write ω0,( j)

i = ω( j)i,n−1 and set w( j)

n = 1.2: ITERATION: for k = 1, ...,P• Compute wk,( j) according to Equation 12.• Set w( j)

n = w( j)n · wk,( j).

• Sample ωk,( j)i ∼ Kk(ω

k−1,( j)i ,ω′k−1,( j)

i ) of invariant

distribution pk/Pn−1:n(ωi).

At the end of the P iterations of intermediate distributions,the normalized weights W ( j)

n are still computed accordingto Equation 4, and resampled if the ESS is below the pre-specified threshold. Finally, the normalization constant Lns,n



and the direct illumination estimate are computed accordingto Equations 10 and 9 respectively.

In general, there is greater benefit in terms of variancereduction in the estimate of the target distribution with theintroduction of a sequence of P intermediate distributionsinvolving one MCMC move each than with a single distribu-tion involving P MCMC moves, while being only a slightlymore expensive in terms of computation time (Figure 3 inthe results section). This benefit is because a sequence of in-termediate distributions simultaneously reduces the variancein the un-normalized weights w( j)

n while exploring the targetdistribution at time n.

5. Unoccluded Illumination Estimate with PathSampling

In this section, we explore an alternative estimate for the un-occluded radiance Lns, which can be used in place of themethod described in section 3.1. We obtain this alternate so-lution by path sampling [GM98]. In the statistics literature,the path of a sample is defined as the continuous trajectory ofa sample over time. It should not be confused with light pathsin classical global illumination literature. With this definitionof path, path sampling refers to smoothly moving samplesfrom one distribution to the next.

Considering a continuous path of distributions

pγn−1:n(ωi) =

pγn−1:n(ωi)

Lγns,n−1:n

,

the following path sampling identity holds [GM98]:

log(Lns,n

Lns,n−1) =

Z 1

0

Z d log( pγn−1:n(ωi))

dγpγ

n−1:n(ωi)dωidγ.

(13)

In our case, the logarithm of the target function p is givenby

log( pγn−1:n(ωi)) = (1− γ) log( pn−1(ωi))+ γ log( pn(ωi))

according to Equation 11.

Thus, the derivative of the logarithm of the function is

d log( pγn−1:n(ωi))

dγ= log(

pn(ωi)

pn−1(ωi)). (14)

When considering a discrete path of P intermediate distri-bution, we can approximate Equation 13 with

log(Lns,n

Lns,n−1) = 1

P ∑Pk=1

R

log(pn(ωi)

pn−1(ωi))pk/P

n−1:n(ωi)dωi

≈ 1P ∑P

k=1 ∑Nj=1 W k,( j) log(

pn(ωk,( j)i )

pn−1(ωk,( j)i )

).

(15)

Note that Equation 15 involves computing the normalizedweights W k,( j) for every intermediate distribution pk/P

n−1:n,which is not required when using the standard form of theintermediate distributions. The un-normalized intermediateweights wk,( j) are still computed according to Equation 12.

When using path sampling, we can also obtain an estimateof log(Lns/Lns,1) as follows

log(Lns,n

Lns,1) =

n

∑t=2

log(Lns,t

Lns,t−1). (16)

Computing the normalization constant using path sam-pling is a bit more expensive than the standard introductionof the P intermediate distributions as discussed in Section 4.However, the estimate of Lns according to Equation 15 gen-erally has lower variance than using P intermediate distribu-tions with the standard ratio according to Equation 10.

6. Implementation

We have implemented the algorithm described above in asystem that offers two rendering modes: a relighting mode,and a mode that allows for free camera movement.

Relighting. In relighting mode, the camera and object are infixed locations, and only the environment can change. Theinitial frame is rendered using bidirectional importance sam-pling. For all subsequent frames, the samples for each pixelare propagated and mutated as described above, and form thesample set for the same pixel in the next frame (Figures 4and 5).

Free camera movement. Once the camera is allowed tomove freely, a surface point will project to different pixelsin different frames. We take this into account by trackingthe motion of the surface points at which the samples weregenerated. When we ray-trace a sample for one frame, wealso store the corresponding information into the next frame.To this end, we compute which pixel the surface point willproject to at the next time step, and store all samples fromthe current frame into that pixel. The memory requirementfor this procedure is about 300 Bytes/pixel. When we wantto compute the illumination for the next frame, most pix-els will therefore have a sample set from the previous frameassociated with them. We propagate and mutate those as dis-cussed. Other pixels might not have a sample set due to dis-occlusion, or differences in sampling rate. We start bidirec-tional importance sampling for these pixels only (Figure 6).

General object movements can be dealt with the same wayas camera movements: by knowing where object points willbe located in the next time step, we can store sample in-formation at the appropriate pixel locations. Currently, ourimplementation does not support this kind of object motion.



Figure 2: Quality comparison of single distribution vs sequence of intermediate distributions with David model (Phong BRDFs = 50, ks = kd = 0.5) in Grace Cathedral HDR EM. Left: Single distribution with 10 MCMC moves. Center: 10 intermediatedistributions with Lns computed with standard ratio. Right: 10 intermediate distributions with Lns computed with path sampling.Rendering times are identical (5 seconds).

7. Results

In this section we compare the results of our unbiased SMCsampling algorithm with bidirectional importance samplingfor rendering from HDR video environments. Images weregenerated with a reasonably well-optimized ray tracer usinga voxel grid as the acceleration data structure for intersectionqueries. Our comparisons examine the output quality of thetwo rendering algorithms for a fixed amount of computingtime. We performed these tests on a 3.6 GHz Xeon runningLinux SuSE 9.0.

Figure 1 presents a comparison of our SMC algorithmwith bidirectional importance sampling for a sequence ofthe sky probe gallery [SJW∗04]. The image on the left isthe first frame of the sequence rendered at a high qualityusing bidirectional sampling (N = 16, M = 800) in 8 sec-onds. The image in the center is the 5th frame of the se-quence rendered using our SMC algorithm (N = 16, P = 5)with path sampling in 4 seconds. The BRDF of the Davidmodel in these images has a high specular exponent (Phongs = 50) and no diffuse component. Under these conditionsof high frequency lighting and highly specular BRDF, ourSMC algorithm does much better than bidirectional impor-tance sampling for the same computation time of 4 seconds.In this case, bidirectional sampling could only use a smallernumber of samples (M = 200) to estimate Lns for the samecompute time (right).

Figure 2 presents the quality comparison of renderingsproduced with our SMC algorithm when using a single dis-tribution (left) versus when using a sequence of intermediatedistributions with standard ratio (center) and path sampling(right). These images correspond to the first frame renderedby our SMC algorithm after rotating the EM by 1.5 alongthe radial direction simulating a small change in the HDR

illumination of the Grace Cathedral. The sequence of inter-mediate distributions greatly help in reducing the variancein the incremental computation of Lns compared to a singledistribution, while path sampling improves the quality of theestimate a bit more. Here, the BRDF of the David model hasa significant diffuse component. Hence, the incremental esti-mate of the Lns has higher variance compared to Figure 1 asthe SMC algorithm uses only a very small number of sam-ples to approximate the Lns.

0.044

0.046

0.048

0.05

0.052

0.054

0.056

0.058

0.06

0.062

2 4 6 8 10 12 14 16

RM

S e

rror

MCMC moves/Intermediate distributions

Error convergence comparison of single distribution vs. sequence of intermediate distributions

Single distributionPath sampling

Figure 3: Convergence plots of RMS errors for single dis-tribution with multiple MCMC moves and sequence of inter-mediate distributions with one MCMC move each. Note howthe RMS error reduces fast when using a sequence of inter-mediate distributions while the error does not really reducemuch with just one distribution.

In Figure 3, we present a comparison of the convergencein terms of RMS errors for a single distribution with multipleMCMC moves and a sequence of intermediate distributions



with path sampling. The RMS error plot was computed forsame frame rendered in Figure 2 with our SMC algorithm. Itis clear from the plot that multiple MCMC moves for a sin-gle distribution do not help much in reducing the variance inthe incremental computation of Lns, though they may helpin exploring the target function p. A sequence of interme-diate distributions with 1 MCMC move per distribution, onthe other hand, is effective in reducing the variance in thecomputation of Lns.

The introduction of a sequence of intermediate distribu-tions also helps in reducing the degeneracy of the samples.We tracked the sample degeneracy, in terms of ESS, forsequence of 100 frames while rendering the David model(Phong BRDF s = 50, ks = kd = 0.5) in the Grace CathedralHDR EM with rotations to the EM by 1.5 per frame. Weobserved that, on an average, the samples corresponding to36% of the pixels required resampling when using a singledistribution with 5 MCMC moves per sample. This fractionreduced to 18% when using a sequence of 5 intermediatedistributions and 1 MCMC move per sample with path sam-pling. Hence, the additional cost of computing a sequenceof intermediate distributions is offset to an extent by havingto resample fewer samples. In practice, we used only up to5−7 intermediate distributions as this was enough to reducethe variance of most pixels. However, in order to maintainthe quality of the renderings over time, we tracked the incre-mental estimate of Lns and explicitly computed Lns,n usinga large number of samples whenever the ratio Lns,n/Lns,1 al-tered significantly.

Figure 4 presents the quality comparison between bidi-rectional sampling and our SMC sampling algorithm for adynamic environment sequence from dawn to dusk from thesky probe gallery [SJW∗04]. We used the sample HDR skyprobe images from the gallery that have been captured at 10minute intervals as key frames of our sequence and interpo-lated to create 3 additional frames between each key frame.The David model in these images has the same highly spec-ular BRDF as in Figure 1, and in this situation, our SMCalgorithm performs much better than bidirectional impor-tance sampling for the same computation time of 4 seconds.The SMC samples also have a lot more temporal coherence,greatly reducing flickering in the animation (please refer thevideo).

Figure 5 presents the quality comparison between bidi-rectional sampling and our SMC sampling algorithm for thesame sky probe sequence from sunrise to sunset, except thatthe BRDF of the David model now has a significant diffusecomponent (Phong BRDF s = 50, ks = kd = 0.5). In thiscase, the difference in the quality of renderings producedby the two algorithms is not much as the SMC algorithmneeds to estimate the reflected radiance due to a wider lobewith a small number of samples. However, the images cor-responding to the SMC algorithm still have lower variance

than bidirectional sampling wherever the specular contribu-tion is high.

In Figure 6, we present example renderings with our SMCsampling algorithm adapted for changing viewpoint as dis-cussed in Section 6. We rendered a sequence with the cameramoving from right to left of the David model with 2 rota-tion of the camera between every frame. The images in thebottom row are false color visualizations of the pixels cor-responding to re-projected points that used SMC sampling(green) and new exposed pixels that used bidirectional sam-pling (red) for the two viewpoints. As shown here, in a se-quence involving a slowly moving camera, most pixels caneffectively use samples propagated from neighboring pixelsin subsequent frames.

Figure 6: SMC sampling for a camera animation sequencewith the David model in the Grace Cathedral EM. Top row:Images rendered with SMC sampling for dynamic view-points. Bottom row: False color visualization of bidirec-tional samples (red) and SMC samples (green).

The speedup that we get in the case of moving cameras islower than the speedup for relighting, since we cannot useSMC for all pixels due to occlusion, and since the reprojec-tion of samples into the next frame consumes time. Whilewe find a speedup of about a factor of 2 for relighting, thespeedup for moving cameras is only 1.6. Note that a movingcamera is in some sense the worst case scenario, since all



Figure 4: Quality comparison between bidirectional sampling and our SMC sampling algorithm for a specular BRDF in thesky probe gallery sequence. Top row: Bidirectional sampling (N = 16, M = 200). Bottom row: SMC sampling (N = 16, P = 5,path sampling). All images took the same compute time of 4 seconds.

points in the scene move. In general scenes with only fewobjects moving, one would expect a speedup somewhere be-tween the extremes of relighting and the camera movement.

8. Conclusions

In this paper we have introduced the use of Sequential MonteCarlo methods for efficiently computing direct illumina-tion in the presence of both high frequency Illuminationand BRDF. By propagating samples over time, the methodmakes efficient use of coherence across frames. We demon-strate that this approach results in significantly reduced vari-ance for the same compute time compared to other state-of-the-art methods.

Sequential Monte Carlo samplers have been the focus ofrecent research activities in statistics and machine learning.The sampling strategies used in this paper are at the lead-ing edge of methods developed in those areas. We believethat these methods are promising for solving other samplingproblems in computer graphics, for example for global illu-mination with photon maps.

9. Acknowledgments

We would like to thank David Burke for providing us his ray-tracer, Paul Debevec for the HDR sky probe gallery and en-vironment maps, Derek Bradley for helping with the video,and the Stanford Digital Michelangelo project for the Davidmodel. We also thank our anonymous reviewers for theirvaluable comments and suggestions. The first author wassupported by a UBC University Graduate Fellowship for theyear 2005-06.

References

[ARBJ03] AGARWAL S., RAMAMOORTHI R., BE-LONGIE S., JENSEN H. W.: Structured importance sam-pling of environment maps. ACM Transactions on Graph-ics (Proc. SIGGRAPH) 22, 3 (July 2003), 605–612. 2

[BGH05] BURKE D., GHOSH A., HEIDRICH W.: Bidi-rectional importance sampling for direct illumination. InProc. of Eurographics Symposium on Rendering (June2005), pp. 147–156. 1, 2, 3, 4



Figure 5: Quality comparison between bidirectional sampling and our SMC sampling algorithm for a BRDF with a diffusecomponent in the sky probe gallery sequence. Top row: Bidirectional sampling (N = 16, M = 200). Bottom row: SMC sampling(N = 16, P = 5, path sampling). All images took the same compute time of 4 seconds.

[CD01] COHEN J., DEBEVEC P.:Light-gen. HDRshop plugin, 2001.http://www.ict.usc.edu/jcohen/lightgen/lightgen.html.2

[CJAMJ05] CLARBERG P., JAROSZ W., AKENINE-MÖLLER T., JENSEN H. W.: Wavelet importance sam-pling: Efficiently evalutating products of complex func-tions. ACM Transactions on Graphics (Proc. SIGGRAPH)24, 3 (Aug. 2005), 1166–1175. 1, 3

[CTE05] CLINE D., TALBOT J., EGBERT P.: Energy re-distribution path tracing. ACM Transactions on Graphics(Proc. SIGGRAPH) 24, 3 (Aug. 2005), 1186–1195. 3

[DdFG01] DOUCET A., DE FREITAS N., GORDON N.:Sequential Monte Carlo Methods in Practice. Springer-Verlag, New York, 2001. 3

[Deb05] DEBEVEC P.: A median cut algorithm for lightprobe sampling, 2005. SIGGRAPH 2005 Poster. 2

[DJA∗04] DUTRE P., JENSEN H. W., ARVO J., BALA K.,BEKAERT P., MARSCHNER S., PHARR M.: SIGGRAPH

Course Notes: State of the Art in Monte Carlo Global Il-lumination. ACM SIGGRAPH, 2004. 5

[dMDJ06] DEL MORAL P., DOUCET A., JASRA A.: Se-quential monte carlo samplers. J. Royal Statist. Soc. B 68,3 (2006), 1–26. 3, 4, 5

[FCL05] FAN S., CHENNEY S., LAI Y.: Metropolisphoton sampling with optional user guidance. In Proc.of Eurographics Symposium on Rendering (June 2005),pp. 127–138. 3

[GM98] GELMAN A., MENG X. L.: Simulating normal-ization constants: From importance sampling to bridgesampling to path sampling. Statist. Sci. 13 (1998), 163–185. 6, 7

[HDMS03] HAVRAN V., DAMEZ C., MYSZKOWSKI K.,SEIDEL H.: An efficient spatio-temporal architecture foranimation rendering. In Proc. of Eurographics Sympo-sium on Rendering (June 2003), pp. 106–117. 3

[HDS03] HAVRAN V., DMITRIEV K., SEIDEL H.:



Goenometric diagram mapping of hemisphere. In ShortPresentations (Eurographics 2003) (2003). 3

[HSK∗05] HAVRAN V., SMYK M., KRAWCZYK G.,MYSZKOWSKI K., SEIDEL H.: Interactive system for dy-namic scene lighting using captured video environementmaps. In Proc. of Eurographics Symposium on Rendering(June 2005), pp. 43–54. 1

[KK03] KOLLIG T., KELLER A.: Efficient illuminationby high dynamic range images. In Proc. of EurographicsSymposium on Rendering (June 2003), pp. 45–51. 2

[LC98] LIU J., CHEN R.: Sequential monte carlo for dy-namic systems. J. Amer. Statist. Assoc. 93 (1998), 1032–1044. 4

[LRR04] LAWRENCE J., RUSINKIEWICZ S., RA-MAMOORTHI R.: Efficient BRDF importance samplingusing a factored representation. ACM Transactionson Graphics (Proc. SIGGRAPH) 23, 3 (Aug. 2004),496–505. 2

[LRR05] LAWRENCE J., RUSINKIEWICZ S., RA-MAMOORTHI R.: Adaptive numerical cumulativedistribution functions for efficient importance sampling.In Proc. of Eurographics Symposium on Rendering (June2005), pp. 11–20. 2, 3

[MFSSK06] MÉNDEZ-FELIU A., SBERT M., SZIRMAY-KALOS L.: Reusing frames in camera animation. Journalof WSCG, 14 (2006). 3

[MH97] MCCOOL M. D., HARWOOD P. K.: Probabilitytrees. In Proc. Graphics Interface (1997), pp. 37–46. 2

[MRR∗53] METROPOLIS N., ROSENBLUTH A. W.,ROSENBLUTH M. N., TELLER A. H., TELLER E.: Equa-tion of state calculations by fast computing machines.Journal of Chemical Physics 21, 6 (1953), 1087–1092. 2

[ODJ04] OSTROMOUKHOV V., DONOHUE C., JODOIN

P.-M.: Fast hierarchical importance sampling with bluenoise properties. ACM Transactions on Graphics (Proc.SIGGRAPH) 23, 3 (Aug. 2004), 488–495. 2

[Shi00] SHIRLEY P.: Realistic Ray Tracing. A K Peters,Natick, 2000. 2

[SHS02] SECORD A., HEIDRICH W., STREIT L.: Fastprimitive distribution for illustration. In Proc. of Euro-graphics Workshop on Rendering (June 2002), pp. 215–226. 2

[SJW∗04] STUMPFEL J., JONES A., WENGER A.,TCHOU C., HAWKINS T., DEBEVEC P.: Direct hdr cap-ture of the sun and sky, 2004. SIGGRAPH 2004 Poster.8, 9

[SSSK04] SBERT M., SZECSI L., SZIRMAY-KALOS L.:Real-time light animation. Computer Graphics Forum(Eurographics 04 Proceedings) 23, 3 (Sept. 2004), 291–299. 3

[TCE05] TALBOT J., CLINE D., EGBERT P.: Importance

resampling for global illumination. In Proc. of Euro-graphics Symposium on Rendering (June 2005), pp. 139–146. 1, 3

[VG95] VEACH E., GUIBAS L.: Optimally combiningsampling techniques for monte carlo rendering. In Proc.of ACM SIGGRAPH ’95 (Aug. 1995), pp. 419–428. 3

[VG97] VEACH E., GUIBAS L.: Metropolis light trans-port. In Proc. of ACM SIGGRAPH ’97 (1997), pp. 65–76.3

[WWL05] WAN L., WONG T., LEUNG C.: Spherical q2-tree for sampling dynamic environment sequences. InProc. of Eurographics Symposium on Rendering (June2005), pp. 21–30. 1, 3

Appendix A: Normalization Constant

The normalization constant Lns,n at time n can be in-crementally estimated according to Equation 10 in Sec-tion 3.1. This result can be explained as follows: theweighted empirical distribution W ( j)

n−1,ω( j)i,n−1:n obtained

after the MCMC sampling step is an approximation ofpn−1(ωi,n−1)Kn(ωi,n−1,ωi,n) according to Equation 5. Theexpectation E(wn) of the incremental weights wn with re-spect to this joint distribution pn−1 ·Kn is

E(wn)

=Z

Ω

Z

Ωwn pn−1(ωi,n−1)Kn(ωi,n−1,ωi,n) dωi,n dωi,n−1

=Z

Ω

Z

Ω

pn(ωi,n−1)

pn−1(ωi,n−1)pn−1(ωi,n−1)Kn(ωi,n−1,ωi,n)

dωi,n dωi,n−1

=1

Lns,n−1

Z

Ω

Z

Ωpn(ωi,n−1)Kn(ωi,n−1,ωi,n) dωi,n dωi,n−1

=1

Lns,n−1

Z

Ωpn(ωi,n−1) dωi,n−1

=Lns,n

Lns,n−1.

The Monte Carlo estimate of this expectation is given by

E(wn) ≈N

∑j=1

W ( j)n−1 · w

( j)n . (17)


sequential sampling for dynamic environment map illumination · a. ghosh, a. doucet, & w....

Documents