3d photography - 國立臺灣大學cyy/courses/vfx/06... · 3d photography digital visual effects,...
TRANSCRIPT
![Page 1: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/1.jpg)
3D photography
Digital Visual Effects, Spring 2006Yung-Yu Chuang2005/5/24
with slides by Szymon Rusinkiewicz, Richard Szeliski, Steve Seitz and Brian Curless
![Page 2: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/2.jpg)
Announcements
• Project #2 artifacts voting• Final project will be online by the end of
tomorrow
![Page 3: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/3.jpg)
3D photography
• Acquisition of geometry and material
![Page 4: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/4.jpg)
Range acquisition
![Page 5: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/5.jpg)
Range acquisition taxonomy
rangerangeacquisitionacquisition
contactcontact
transmissivetransmissive
reflectivereflectivenonnon--opticaloptical
opticaloptical
industrial CTindustrial CT
mechanical mechanical (CMM, jointed arm)(CMM, jointed arm)
radarradarsonarsonar
ultrasoundultrasoundMRIMRI
ultrasonic trackersultrasonic trackersmagnetic trackersmagnetic trackers
inertial inertial (gyroscope, accelerometer)(gyroscope, accelerometer)
![Page 6: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/6.jpg)
Range acquisition taxonomy
opticalopticalmethodsmethods
passivepassive
activeactive
shape from X:shape from X:stereostereomotionmotionshadingshadingtexturetexturefocusfocusdefocusdefocus
active variants of passive methodsactive variants of passive methodsStereo w. projected textureStereo w. projected textureActive depth from defocusActive depth from defocusPhotometric stereoPhotometric stereo
time of flighttime of flight
triangulationtriangulation
![Page 7: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/7.jpg)
Outline
• Passive approaches– Stereo– Multiview approach
• Active approaches– Triangulation– Shadow scanning
• Active variants of passive approaches– Photometric stereo– Example-based photometric stereo
![Page 8: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/8.jpg)
Passive approaches
![Page 9: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/9.jpg)
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923
Stereo
![Page 10: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/10.jpg)
Stereo
• One distinguishable point being observed– The preimage can be found at the intersection of
the rays from the focal points to the image points
![Page 11: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/11.jpg)
Stereo
• Many points being observed– Need some method to establish correspondences
![Page 12: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/12.jpg)
Components of stereo vision systems
• Camera calibration: previous lecture• Image rectification: simplifies the search for
correspondences• Correspondence: which item in the left image
corresponds to which item in the right image• Reconstruction: recovers 3-D information from
the 2-D correspondences
![Page 13: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/13.jpg)
Epipolar geometry
• Epipolar constraint: corresponding points must lie on conjugate epipolar lines– Search for correspondences becomes a 1-D problem
![Page 14: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/14.jpg)
Image rectification
• Warp images such that conjugate epipolar lines become collinear and parallel to u axis
![Page 15: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/15.jpg)
Disparity
• With rectified images, disparity is just (horizontal) displacement of corresponding features in the two images– Disparity = 0 for distant points– Larger disparity for closer points– Depth of point proportional to 1/disparity
![Page 16: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/16.jpg)
Reconstruction
• Geometric– Construct the line segment perpendicular to R and R'
that intersects both rays and take its mid-point
![Page 17: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/17.jpg)
Basic stereo algorithm
For each epipolar lineFor each pixel in the left image
• compare with every pixel on same epipolar line in right image
• pick pixel with minimum match cost
Improvement: match windows
![Page 18: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/18.jpg)
Basic stereo algorithm
• For each pixel– For each disparity
• For each pixel in window– Compute difference
– Find disparity with minimum SSD
![Page 19: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/19.jpg)
Reverse order of loops• For each disparity
– For each pixel• For each pixel in window
– Compute difference
• Find disparity with minimum SSD at each pixel
![Page 20: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/20.jpg)
Incremental computation
• Given SSD of a window, at some disparity
Image 1Image 1
Image 2Image 2
![Page 21: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/21.jpg)
Incremental computation
• Want: SSD at next location
Image 1Image 1
Image 2Image 2
![Page 22: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/22.jpg)
Incremental computation
• Subtract contributions from leftmost column, add contributions from rightmost column
Image 1Image 1
Image 2Image 2
++++++++++
----------
----------
++++++++++
![Page 23: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/23.jpg)
Selecting window size
• Small window: more detail, but more noise• Large window: more robustness, less detail• Example:
![Page 24: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/24.jpg)
Selecting window size
3 pixel window3 pixel window 20 pixel window20 pixel window
![Page 25: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/25.jpg)
Non-square windows
• Compromise: have a large window, but higher weight near the center
• Example: Gaussian• Example: Shifted windows
![Page 26: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/26.jpg)
Ordering constraint
• Order of matching features usually the samein both images
• But not always: occlusion
![Page 27: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/27.jpg)
Dynamic programming
• Treat feature correspondence as graph problem
Left imageLeft imagefeaturesfeatures
Right image featuresRight image features
11 22 33 4411
22
33
44
Cost of edges =Cost of edges =similarity ofsimilarity of
regions betweenregions betweenimage featuresimage features
![Page 28: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/28.jpg)
Dynamic programming
• Find min-cost path through graph
Left imageLeft imagefeaturesfeatures
Right image featuresRight image features
11 22 33 4411
22
33
44
11
3344
11
22223344
![Page 29: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/29.jpg)
Energy minimization
• Another approach to improve quality of correspondences
• Assumption: disparities vary (mostly) smoothly• Minimize energy function:
Edata+λEsmoothness
• Edata: how well does disparity match data• Esmoothness: how well does disparity match
that of neighbors – regularization
![Page 30: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/30.jpg)
Energy minimization
• If data and energy terms are nice (continuous, smooth, etc.) can try to minimize via gradient descent, etc.
• In practice, disparities only piecewise smooth• Design smoothness function that doesn’t
penalize large jumps too much– Example: V(α,β)=min(|α−β|, K)
![Page 31: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/31.jpg)
Stereo as energy minimization• Matching Cost Formulated as Energy
– “data” term penalizing bad matches
– “neighborhood term” encouraging spatial smoothness
),(),(),,( ydxyxdyxD +−= JI
similar)something(or
d2 and d1 labels with pixelsadjacentofcost
21
21 ),(dd
ddV−=
=
∑∑ +=)2,2(),1,1(
2,21,1),(
, ),(),,(yxyxneighbors
yxyxyx
yx ddVdyxDE
![Page 32: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/32.jpg)
Energy minimization
• Hard to find global minima of non-smooth functions– Many local minima– Provably NP-hard
• Practical algorithms look for approximate minima (e.g., simulated annealing)
![Page 33: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/33.jpg)
Energy minimization via graph cuts
Labels (disparities)
d1
d2
d3
edge weight
edge weight
),,( 3dyxD
),( 11 ddV
![Page 34: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/34.jpg)
• Graph Cost– Matching cost between images– Neighborhood matching term– Goal: figure out which labels are connected to
which pixels
d1
d2
d3
Energy minimization via graph cuts
![Page 35: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/35.jpg)
Energy minimization via graph cuts
d1
d2
d3
• Graph Cut– Delete enough edges so that
• each pixel is (transitively) connected to exactly one label node
– Cost of a cut: sum of deleted edge weights– Finding min cost cut equivalent to finding global
minimum of energy function
![Page 36: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/36.jpg)
Computing a multiway cut
• With 2 labels: classical min-cut problem– Solvable by standard flow algorithms
• polynomial time in theory, nearly linear in practice
– More than 2 terminals: NP-hard [Dahlhaus et al., STOC ‘92]
• Efficient approximation algorithms exist– Within a factor of 2 of optimal– Computes local minimum in a strong sense
• even very large moves will not improve the energy– Yuri Boykov, Olga Veksler and Ramin Zabih, Fast Approximate Energy
Minimization via Graph Cuts, International Conference on Computer Vision, September 1999.
![Page 37: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/37.jpg)
Move examples
Starting point
Red-blue swap move
Green expansion move
![Page 38: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/38.jpg)
The swap move algorithm1. Start with an arbitrary labeling2. Cycle through every label pair (A,B) in some order
2.1 Find the lowest E labeling within a single AB-swap2.2 Go there if it’s lower E than the current labeling
3. If E did not decrease in the cycle, we’re done Otherwise, go to step 2
Original graph
A
B
AB subgraph(run min-cut on this graph)
B
A
![Page 39: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/39.jpg)
The expansion move algorithm
1. Start with an arbitrary labeling2. Cycle through every label A in some order
2.1 Find the lowest E labeling within a single A-expansion2.2 Go there if it’s lower E than the current labeling
3. If E did not decrease in the cycle, we’re done Otherwise, go to step 2
![Page 40: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/40.jpg)
Stereo results
ground truthscene
– Data from University of Tsukuba
http://cat.middlebury.edu/stereo/
![Page 41: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/41.jpg)
Results with window correlation
normalized correlation(best window size)
ground truth
![Page 42: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/42.jpg)
Results with graph cuts
ground truthgraph cuts(Potts model E,
expansion move algorithm)
![Page 43: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/43.jpg)
Volumetric multiview approaches
• Goal: find a model consistent with images• “Model-centric” (vs. image-centric)• Typically use discretized volume (voxel grid)• For each voxel, compute occupied / free
(for some algorithms, also color, etc.)
![Page 44: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/44.jpg)
Photo consistency
• Result: not necessarily correct scene• Many scenes produce the same images
All scenesAll scenes
PhotoPhoto--consistent scenesconsistent scenes
True sceneTrue scene ReconstructedReconstructedscenescene
![Page 45: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/45.jpg)
Silhouette carving
• Find silhouettes in all images• Exact version:
– Back-project all silhouettes, find intersection
Binary ImagesBinary Images
![Page 46: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/46.jpg)
Silhouette carving• Find silhouettes in all images• Exact version:
– Back-project all silhouettes, find intersection
![Page 47: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/47.jpg)
Silhouette carving
• Limit of silhouette carving is visual hull orline hull
• Complement of lines that don’t intersect object• In general not the same as object
– Can’t recover “pits” in object
• Not the same as convex hull
![Page 48: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/48.jpg)
Silhouette carving
• Discrete version:– Loop over all voxels in some volume– If projection into images lies inside all silhouettes,
mark as occupied– Else mark as free
![Page 49: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/49.jpg)
Silhouette carving
![Page 50: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/50.jpg)
Voxel coloring
• Seitz and Dyer, 1997• In addition to free / occupied, store color
at each voxel• Explicitly accounts for occlusion
![Page 51: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/51.jpg)
Voxel coloring
• Basic idea: sweep through a voxel grid– Project each voxel into each image in which
it is visible– If colors in images agree, mark voxel with color– Else, mark voxel as empty
• Agreement of colors based on comparing standard deviation of colors to threshold
![Page 52: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/52.jpg)
Voxel coloring and occlusion
• Problem: which voxels are visible?• Solution: constrain camera views
– When a voxel is considered, necessary occlusion information must be available
– Sweep occluders before occludees– Constrain camera positions to allow this sweep
![Page 53: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/53.jpg)
Occlusion handling
![Page 54: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/54.jpg)
Voxel coloring sweep order
SceneSceneTraversalTraversal
![Page 55: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/55.jpg)
Voxel coloring camera positions
Inward-lookingCameras above scene
Outward-lookingCameras inside scene
SeitzSeitz
![Page 56: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/56.jpg)
Image acquisition
•Calibrated Turntable•360° rotation (21 images)
Selected Dinosaur ImagesSelected Dinosaur Images
Selected Flower ImagesSelected Flower Images
![Page 57: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/57.jpg)
Voxel coloring results
Dinosaur ReconstructionDinosaur Reconstruction72 K voxels colored72 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
Flower ReconstructionFlower Reconstruction70 K voxels colored70 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
![Page 58: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/58.jpg)
Space carving
Image 1Image 1 Image NImage N
……......
Initialize to a volume V containing the true sceneInitialize to a volume V containing the true scene
Repeat until convergenceRepeat until convergence
Choose a voxel on the current surfaceChoose a voxel on the current surface
Carve if not photoCarve if not photo--consistentconsistentProject to visible input imagesProject to visible input images
![Page 59: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/59.jpg)
Multi-pass plane sweep
• Faster alternative:– Sweep plane in each of 6 principal directions– Consider cameras on only one side of plane– Repeat until convergence
![Page 60: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/60.jpg)
Multi-pass plane sweep
True Scene Reconstruction
![Page 61: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/61.jpg)
Multi-pass plane sweep
![Page 62: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/62.jpg)
Multi-pass plane sweep
![Page 63: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/63.jpg)
Multi-pass plane sweep
![Page 64: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/64.jpg)
Multi-pass plane sweep
![Page 65: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/65.jpg)
Input image (1 of 45) Input image (1 of 45) ReconstructionReconstruction
ReconstructionReconstructionReconstructionReconstruction
Space carving results: African violet
![Page 66: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/66.jpg)
Input imageInput image(1 of 100) (1 of 100)
ReconstructionReconstruction
Space carving results: hand
![Page 67: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/67.jpg)
Active approaches
![Page 68: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/68.jpg)
Time of flight
• Basic idea: send out pulse of light (usually laser), time how long it takes to return
tcr Δ=21 tcr Δ=21
![Page 69: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/69.jpg)
Laser scanning (triangulation)
• Optical triangulation– Project a single stripe of laser light– Scan it across the surface of the object– This is a very precise version of structured light scanning
• Other patterns are possible
Direction of travel
Object
CCD
CCD image plane
LaserCylindrical lens
Laser sheet
Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/
![Page 70: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/70.jpg)
Shadow scanning
DeskLamp
Camera
Stick orpencil
Desk
http://www.vision.caltech.edu/bouguetj/ICCV98/
![Page 71: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/71.jpg)
Basic idea
• Calibration issues:– where’s the camera wrt. ground plane?– where’s the shadow plane?
• depends on light source position, shadow edge
![Page 72: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/72.jpg)
Two Plane Version
• Advantages– don’t need to pre-calibrate the light source– shadow plane determined from two shadow edges
![Page 73: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/73.jpg)
Estimating shadow lines
![Page 74: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/74.jpg)
Shadow scanning in action
![Page 75: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/75.jpg)
accuracy: 0.1mm over 10cm ~ 0.1% error
Results
![Page 76: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/76.jpg)
Textured objects
![Page 77: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/77.jpg)
accuracy: 1mm over 50cm ~ 0.5% error
Scanning with the sun
![Page 78: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/78.jpg)
accuracy: 1cm over 2m~ 0.5% error
Scanning with the sun
![Page 79: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/79.jpg)
Active variants of passive approaches
![Page 80: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/80.jpg)
The BRDF• The Bidirectional Reflection Distribution Function
– Given an incoming ray and outgoing raywhat proportion of the incoming light is reflected along outgoing ray?
surface normal
γρ )(),( nlvlI ⋅×=
![Page 81: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/81.jpg)
Diffuse reflection (Lambertian)
��
�
�
dkvl =),(ρ albedo
Assuming that light strength is 1.
![Page 82: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/82.jpg)
Photometric stereo
N
L1
L2
V
L3
• Can write this as a matrix equation:
![Page 83: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/83.jpg)
Solving the equations
![Page 84: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/84.jpg)
More than three lights
• Get better results by using more lights
• Least squares solution:
• Solve for N, kd as before
![Page 85: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/85.jpg)
Trick for handling shadows
• Weight each equation by the pixel brightness:
• Gives weighted least-squares matrix equation:
• Solve for N, kd as before
![Page 86: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/86.jpg)
Photometric Stereo Setup
![Page 87: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/87.jpg)
Procedure
• Calibrate camera• Calibrate light directions/intensities• Photographing objects (HDR recommended)• Estimate normals• Estimate depth
![Page 88: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/88.jpg)
Estimating light directions
• Trick: place a chrome sphere in the scene
– the location of the highlight tells you where the light source is
• Use a ruler
![Page 89: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/89.jpg)
Photographing objects
![Page 90: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/90.jpg)
Normalize light intensities
![Page 91: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/91.jpg)
Estimate normals
![Page 92: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/92.jpg)
Depth from normals
![Page 93: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/93.jpg)
Results
![Page 94: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/94.jpg)
Limitations
• Big problems– doesn’t work for shiny things, semi-translucent
things– shadows, inter-reflections
• Smaller problems– calibration requirements
• measure light source directions, intensities• camera response function
![Page 95: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/95.jpg)
Example-based photometric stereo
• Estimate 3D shape by varying illumination, fixed camera
• Operating conditions– any opaque material– distant camera, lighting– reference object available– no shadows, interreflections, transparency
![Page 96: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/96.jpg)
same surface normal
“Orientation consistency”
![Page 97: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/97.jpg)
![Page 98: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/98.jpg)
![Page 99: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/99.jpg)
![Page 100: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/100.jpg)
![Page 101: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/101.jpg)
![Page 102: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/102.jpg)
![Page 103: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/103.jpg)
Virtual views
![Page 104: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/104.jpg)
Velvet
![Page 105: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/105.jpg)
Virtual Views
![Page 106: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/106.jpg)
Brushed Fur
![Page 107: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/107.jpg)
Virtual Views
![Page 108: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/108.jpg)
![Page 109: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/109.jpg)
3D Model Acquisition Pipeline
3D Scanner3D Scanner3D Scanner
![Page 110: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/110.jpg)
3D Model Acquisition Pipeline
View PlanningView PlanningView Planning
3D Scanner3D Scanner3D Scanner
![Page 111: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/111.jpg)
3D Model Acquisition Pipeline
AlignmentAlignmentAlignmentView PlanningView PlanningView Planning
3D Scanner3D Scanner3D Scanner
![Page 112: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/112.jpg)
3D Model Acquisition Pipeline
3D Scanner3D Scanner3D Scanner
AlignmentAlignmentAlignment
MergingMergingMerging
View PlanningView View PlanningPlanning
![Page 113: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/113.jpg)
Volumetric reconstruction
![Page 114: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/114.jpg)
Signed distance function
![Page 115: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/115.jpg)
Results
![Page 116: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/116.jpg)
Systems, projects and applications
![Page 117: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/117.jpg)
The Digital Michelangelo Project
• Goal: scan 10 sculptures by Michelangelo• High-resolution (“quarter-millimeter”)
geometry• Stanford University, led by Marc Levoy
![Page 118: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/118.jpg)
Scanning the David
height of gantry: 7.5 metersweight of gantry: 800 kilograms
![Page 119: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/119.jpg)
Range processing pipeline
• steps1. manual initial alignment2. ICP to one existing scan3. automatic ICP of all overlapping pairs4. global relaxation to spread out error5. merging using volumetric method
![Page 120: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/120.jpg)
• 480 individually aimed scans• 2 billion polygons• 7,000 color images• 32 gigabytes• 30 nights of scanning• 22 people
Statistics about the scan
![Page 121: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/121.jpg)
photograph 1.0 mm computer model
Comparison
![Page 122: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/122.jpg)
Results
![Page 123: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/123.jpg)
The Great Buddha Project
• Great Buddha of Kamakura• Original made of wood, completed 1243• Covered in bronze and gold leaf, 1267• Approx. 15 m tall• Goal: preservation of
cultural heritage• Institute of Industrial Science,
University of Tokyo, led byKatsushi Katsushi IkeuchiIkeuchi
![Page 124: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/124.jpg)
Scanner• Cyrax range scanner
by Cyra Technologies• Laser pulse time-of-flight• Accuracy: 4 mm• Range: 100 m
![Page 125: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/125.jpg)
Processing• 20 range images (a few million points)• Simultaneous all-to-all ICP• Variant of volumetric merging (parallelized)
![Page 126: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/126.jpg)
Results
![Page 127: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/127.jpg)
View interpolation
Bullet time video
![Page 128: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/128.jpg)
View interpolation
High-quality video view interpolation
![Page 129: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/129.jpg)
Final project
![Page 130: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/130.jpg)
Final project
• Assigned: tomorrow• Due: 6/30 Friday?• Proposal and midterm report on 6/11
![Page 131: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/131.jpg)
Final project
• Research (1-2 people)• System (1-3 people)• Film (3-4 people)
![Page 132: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/132.jpg)
Research
• Define a problem and try to solve it• You don’t need to solve it all, but have to make
a reasonable progress, for example, solve a simplified version.
• Find inspirations from SIGGRAPH /CVPR/ICCV papers
![Page 133: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/133.jpg)
System
• Implement existing algorithm into a useful system such as implementing SIGGRAPH 2006 or CVPR 2006 papers
![Page 134: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/134.jpg)
Film
• It must be an “effect” film. • You can use any tools as you want. But, I guess
that you have to write some on your own.• Find inspirations from
Gatech’s vfx course http://www.cc.gatech.edu/classes/AY2004/cs4480_spring/independent film makershttp://www.peerlessproductions.com/
• Submit two videos, final and making-of.
![Page 135: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/135.jpg)
Shameless advertisement
A web game for user-assisted ROI labeling
![Page 136: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/136.jpg)
Region of interest (ROI)
![Page 137: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/137.jpg)
User-assisted ROI labeling
http://photoshoot.csie.org
![Page 138: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/138.jpg)
![Page 139: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/139.jpg)
![Page 140: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/140.jpg)
![Page 141: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/141.jpg)
![Page 142: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/142.jpg)
What’s next?
….
machinelearning
![Page 143: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/143.jpg)
Play PhotoShoot Today!
http://photoshoot.csie.org
![Page 144: 3D photography - 國立臺灣大學cyy/courses/vfx/06... · 3D photography Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/5/24 with slides by Szymon Rusinkiewicz, Richard](https://reader035.vdocuments.site/reader035/viewer/2022063007/5fbb003c6e19933e7a774c76/html5/thumbnails/144.jpg)
References
• D. Scharstein and R. Szeliski. A Texonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithm, IJCV 2002.
• S. Seitz and C. Dyer. Photorealistic Scene Reconstruction by Voxel Coloring, CVPR 1997.
• J.-Y. Bouguet and P. Perona. 3D Photography on Your Desk, ICCV 1998.
• A. Hertzman and S. Seitz. Shape and Materials by Example: A Photometric Stereo Appraoch, CVPR 2003.