5d covaria nce tracing for efficient defocus and motion blur laurent belcour 1 cyril soler 2 kartic...
TRANSCRIPT
![Page 1: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/1.jpg)
5D COVARIANCE TRACINGFOR EFFICIENT DEFOCUS AND MOTION BLUR
Laurent Belcour1 Cyril Soler2 Kartic Subr3 Nicolas Holzschuch2 Frédo Durand4
1 Grenoble Université, 2 Inria, 3 UC London, 4 MIT CSAIL
![Page 2: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/2.jpg)
Blur is costly to simulate !
![Page 3: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/3.jpg)
![Page 4: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/4.jpg)
timeintegration
spacereconstruction
![Page 5: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/5.jpg)
Previous works: a posteriori Image space methods
• [Mitchell 1987], [Overbeck et al. 2009], • [Sen et al. 2011], [Rousselle et al. 2011]
Integration space• [Hachisuka et al. 2008]
Reconstruction• [Lehtinen et al. 2011], [Lehtinen et al. 2012]
Easy to plug‐ Require already dense sampling‐ Rely on point samples
![Page 6: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/6.jpg)
Previous work: a priori
First order analysis[Ramamoorthi et al. 2007]
Frequency analysis[Durand et al. 2005]
![Page 7: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/7.jpg)
Previous work: a priori
First order analysis[Ramamoorthi et al. 2007]
Frequency analysis[Durand et al. 2005]
zoom Fourier transform
![Page 8: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/8.jpg)
Previous work: a prioriPredict full spectrum
Anisotropic information− Unwieldy
Predict bounds Compact & efficient− Special cases formula
[Egan et al. 2009], [Bagher et al. 2013], [Meha et al. 2012]
[Soler et al. 2009]
None can work with full global illumination!
![Page 9: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/9.jpg)
Our idea: 5D Covariance representation
![Page 10: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/10.jpg)
5D Covariance representation Use second moments
• 5x5 matrix• Equivalent to Gaussian approx.
Formulate all interactions• Analytical matrix operators• Gaussian approx. for reflection
Nice properties• Symmetry• Additivity
space (2D)
time
angle (2D)
![Page 11: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/11.jpg)
Contributions
Unified temporal frequency analysis
Covariance tracing
Adaptive sampling & reconstruction algorithm
![Page 12: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/12.jpg)
Our algorithmAccumulate 5D Covariance
in screen space
![Page 13: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/13.jpg)
Our algorithmAccumulate 5D Covariance
in screen space
Estimate 5D sampling density
angl
e
time
time
angl
ean
gle
time
![Page 14: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/14.jpg)
Our algorithmAccumulate 5D Covariance
in screen space
Estimate 5D sampling density
Estimate 2D reconstruction filters
![Page 15: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/15.jpg)
Our algorithmAccumulate 5D Covariance
in screen space
Estimate 5D sampling density
Estimate 2D reconstruction filters
Reconstruct image
Acquire 5D samples
![Page 16: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/16.jpg)
Accumulate 5D Covariance in screen space
Estimate 5D sampling density
Estimate 2D reconstruction filters
Reconstruct image
Acquire 5D samples
![Page 17: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/17.jpg)
Covariance tracing
Add information to light paths
Update the covariance along light path
Atomic decomposition for genericity
![Page 18: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/18.jpg)
Covariance tracing
Free transport
Free transport
![Page 19: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/19.jpg)
Free transport
Covariance tracing
Reflection
![Page 20: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/20.jpg)
Covariance tracing
Free transport
Free transport
Reflection
![Page 21: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/21.jpg)
Free transport
Covariance tracing
Occlusion
Free transport
Reflection
spatial visibility
![Page 22: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/22.jpg)
Covariance tracing
Free transport
Free transport
Occlusion
![Page 23: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/23.jpg)
Covariance tracing
Reflection
Free transport
Free transport
![Page 24: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/24.jpg)
Covariance tracing
Free transport
Free transport
Reflection
![Page 25: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/25.jpg)
Just a chain of operators
Free transport Occlusion Curvature Symmetry BRDF Lens
![Page 26: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/26.jpg)
What about motion?
![Page 27: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/27.jpg)
We could rewrite all operators…
Occlusionwith moving
occluder
Curvature with moving
geometry
BRDF with moving
reflector
Lens with moving camera
Ω𝑡 Ω𝑡 Ω𝑡 Ω𝑡
Ω𝑡 Ω𝑡 Ω𝑡 Ω𝑡
![Page 28: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/28.jpg)
We will not rewrite all operators!
Occlusion Curvature BRDF Lens
Motion
Inverse Motion
![Page 29: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/29.jpg)
Motion operator
Reflection with moving reflector
space
time
angle
space
time
angle
![Page 30: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/30.jpg)
Motion operatorspace
time
angle
Reflection
Motion
![Page 31: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/31.jpg)
Motion operatorspace
time
angle
space
time
angle
Inverse Motion
Reflection
Motion
![Page 32: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/32.jpg)
Accumulate covariance
final covariance
first
ligh
t pat
hse
cond
ligh
t pat
h
![Page 33: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/33.jpg)
Accumulate 5D Covariance in screen space
Estimate 5D sampling density
Estimate 2D reconstruction filters
Reconstruct image
Acquire 5D samples
![Page 34: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/34.jpg)
Using covariance information
How can we extract bandwidth ?• Using the volume• Determinant of the covariance
How can we estimate the filter ?• Frequency analysis of integration [Durand 2011]• Slicing the equivalent Gaussian
space
time
space
𝑉
![Page 35: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/35.jpg)
Accumulate 5D Covariance in screen space
Estimate 5D sampling density
Estimate 2D reconstruction filters
Reconstruct image
Acquire 5D samples
![Page 36: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/36.jpg)
Implementation details: occlusion
Occlusion using a voxelized scene
Use the 3x3 covariance of normals distribution
Evaluate using ray marching
![Page 37: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/37.jpg)
Our algorithm
Equal time M
onte-CarloResults: the helicopter
![Page 38: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/38.jpg)
Our method
Results: the snooker
Equal-time Monte Carlo
defocus blur
motion blur
BRDF blur
![Page 39: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/39.jpg)
Results: the snooker
Our method: 25min
Eq. quality Monte Carlo: 2h25min• 200 light field samples per pixel
Covariance tracing: 2min 36s• 10 covariance per pixel
Reconstruction: 16s
![Page 40: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/40.jpg)
Conclusion Covariance tracing
• Generate better light paths• Simple formulation
Unified frequency analysis• Temporal light fields• No special case
![Page 41: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/41.jpg)
Future work
Tracing covariance has a cost• Mostly due to the local occlusion query
New operators• Participating media
![Page 42: 5D COVARIA NCE TRACING FOR EFFICIENT DEFOCUS AND MOTION BLUR Laurent Belcour 1 Cyril Soler 2 Kartic Subr 3 Nicolas Holzschuch 2 Frédo Durand 4 1 Grenoble](https://reader030.vdocuments.site/reader030/viewer/2022032723/56649d0e5503460f949e441a/html5/thumbnails/42.jpg)
GROUND IS MOVING!