3d modeling using multi-camera and multi-lighting (mvml) dome

3D Modeling using Multi-camera and Multi-lighting (MVML) Dome

Broadband Network and Digital Media Lab (BBNC),Automation Department, Tsinghua University

Yebin Liuhttp://media.au.tsinghua.edu.cn/liuyebin.jsp

13D Modeling Using MVML Dome

23/4/21

23/4/212 3D Modeling Using MVML Dome

Introduction宽带网数字媒体技术实验室

10-year history 1999~2009

1. finished or undergoing more than 10 national major R/D projects2. Awards for National Scientific and Technological Progress

Broadband Network & Digital Media Lab

Introduction


宽带网数字媒体技术实验室Broadband Network & Digital Media Lab

• Born in Dec,1964. Ph.D, IEEE Senior Member • Distinguished Young Scholar of NSF China • Second Prize for National Scientific and Technological Progress 2008• The chief scientist of national 973 Program

Head: Prof. Qionghai Dai

16 PHD Students and 23 Master studentsMost of the PhDs are junior, and interest in Vision and Graphics

I spent 7 years in this lab. My working experiences: video streaming and coding (02’) light field and 3D display (05’) MVML dome (07’)

Homepage://media.au.tsinghua.edu.cn


Contents

Background

• 3D Studio– IBMR– FVV and performance capture

• Light Stage– IBL– Performance relighting

• Light Field– IBR– Computational photography



Multi-camera Multi-light Dome

Diameter 6m

40 flea2 camera

Lamp for color calibration

screen for chrome-keying

A Combination of 3D Studio, Light Stage and Light Field

LEDs for lighting


Topology and the Control Module

9 3D Modeling Using MVML Dome 23/4/21


• Resolution: 1024 by 768, frame rate: 25fps• FVV data and MVML data are available on http://media.au.tsinghua.edu.cn/fvv.jsp http://media.au.tsinghua.edu.cn/mvml.jsp




Contents

Multi-view Stereo (MVS): reconstruct 3D model from multiple calibrated photographs of a realistic object

Background

Available MVS methods (according to Middlebury Benchmark):

MVS in category 3,4 are accurate but not robust

1. Volumetric based MVS2. Surface evolution MVS3. Depth Map merging MVS (our work is in this kind)4. Feature propagation MVS


Former work: PCMVS• A Point Cloud (Depth Map) based FVV System1


Our work.

Furukawa’07.

FVV1

FVV2

Depth map based MVS achieve both high accuracy and robustness!

1. Y.Liu, Q.Dai, W.Xu. A Point cloud based Multi-view stereo for Free-viewpoint video . accepted in TVCG.

Former work: PCMVS

PCMVS-10 PCMVS-20 PCMVS-BINO

Under sparse array mode:

Binocular is much better

Stereo matching is crucial in MVS


Epipolar line

Target view Reference view

3D Surface

Matching Error!

2 2

( )( )

( ) ( )

i i

i i

p p q qNCC

p p q q

Why Stereo Matching Hard?


18

Popular Matching Techniques

3D Modeling Using MVML Dome 23/4/21

Optical flow Stereo Mview Stereo

Optimization mechanism Method cost define unit

We Propose ：

Proposed Pipeline

193D Modeling Using MVML Dome

23/4/21

Variational

20

Variation Goes for MVS

3D Modeling Using MVML Dome 23/4/21

( ) ( ) ( )D SE w E w E w 2

2

( ( ) ( )( )

( ( ) ( )

r t

D D

r t

I d w IE w dxdy

I d w I

x x

x x

2( )S SE w w dxdy

1.Continuous and dense Capture

2.Rotation invariant matching

3.VH and epipolar reduce the probability of local minima

Multiple Multi-scale IterationMulti-scale coarse-to-fine iteration technique*

Multi-scale image pyramid

Multiple candidates by starting the iteration from different scales


Depthmap v1 Depthmap v2

Depthmap v3Depthmap v4

*. T.Brox, A.Bruhn, N.Papenberg, and J.Weickert. High accuracy optical flow estimation based on a theory for warping. ECCV’04.

Multiple CandidatesHigher starting scale

Different

versions

generate

different

well

reconstruction

areas

well reconstruct

bad reconstruct


Target view Reference view

Candidate patches

Accurate Matching

2 2

( )( )

( ) ( )

i i

i i

p p q qNCC

p p q q

Choose the best patch from all candidates using patch based NCC measurement

Refined Depth Map Synthesis


Multi-view Depth Maps Merging


View 1 before

View 1 after View 5 after

View 5 before View 9 before

View 9 after

Combine before

Combine after

MStereo Benchmark by Middlebury college

Measured by accuracy and completeness

Reconstruction Performance


FurukawaCVPR07

High performance on detail reconstruction

LiuCVPR09


BradleyCVPR07

Ground Truth

Experimental Results


Multiple candidates + NCC metric on Continuous surface

Detail reconstruction

Robust reconstruction

CMVS Summary


Continuous Depth EstimationPros:

Cons:

Requires an initial depth map for each viewVisual hull may not necessary

Complexity: MSS optical flow is the bottleneck

(half an hour for DinoSparseRing)Optimization the OF algorithm is important


Contents

Cons: Highlights, Textureless, Cycle-texture3D

Reconstruction

Multi-view Stereo Multi-view Single-light

Multi-view Multi-light 3D Reconstruction

Background

Photometric Stereo Multi-light Single-view

Multi-light Multi-view

Cons: Continuous assumptionOnly obtain shape, not geometry



Lights

Views

Directional Illumination

Multiplex Illumination

Multi-view Photometric Stereo


• No Illumination Constraints 　－ Illuminations is represented by low order spherical harmonics*

－ No constraints on location, intensity and assumptions of the lights

*. R.Basri, D.W.Jacobs, "Lambertian Reflectance and Linear Subspaces", TPAMI'03




• No Manual Interactions － Space-light MVS provides initial normal information

－ Alternating optimization (normal and light)#

• Watertight and Robust Reconstruction － Multi-view images provide watertight reconstruction － Suitable for objects with small portion of non-Lambert. surface

• Key Model Recon. for Performance Capture Systems*

－ Bypass laser scan and model registration － Standstill for 1 second

* E.d.Aguiar, C.Stoll, C.Theobalt, N.Ahmed, H.P.Seidel, S.Thrun, "Performance Capture from Sparse Multi-view Video", SIGGRAPH'08

* D.Vlasic, I.Baran, W.Matusik, J.Popovic, "Articulated Mesh Animation from Multi-view Silhouettes", SIGGRAPH'08

# R.Basri, D.Jacobs, I.Kemelmacher, "Photometric Stereo with General, Unknown Lighting", IJCV'06

Results: Impacts of Lights


Results: Synthetic Datasets


D.Vlasic, I.Baran, W.Matusik, J.Popovic, "Articulated Mesh Animation from Multi-view Silhouettes", SIGGRAPH'08

Results: Relighting


1. How to extend to Motion Actors?2. Extend to Outdoor Scenes (Complex illumination)

3. MPS under Sparse Sampling

MPS SummaryNo illumination constraints (Unknown lights)

No manual interactions Watertight and robust reconstruction

Practical in performance capture, animation, relighting, etc.


Pros:

Future works:

√ √ √ √ √ √

√ √ √ √ √ √

√ √ √ √ √ √

√ √ √ √ √ √

light:

view

√ × √ √ √ ×

× √ × × √ √

√ × × √ × √

√ √ × √ √ √

light:

view

Contents


The Vision Field


Vision Field: the full space (1d View+1d Light+1d Time) on where vision problems define “Matching” in lower subspace vs. “Fusion” in higher subspace and full space

Vision

MVML dome is a sampling of the visual field

Vision Field: Future Works

• For free-motion object, It is impossible to capture the full Time-Light subspace.

• Only a few works have addressed this problem*

• Can be adopted in MPS and relighting motion objects.


√ √ √ √ √

√ √ √ √ √

√ √ √ √ √

√ √ √ √ √

light

time

√

√

√

√ √

light

timeImpossible mode

√ √ √ √ √

√ √ √ √ √

√ √ √ √ √

√ √ √ √ √

light

time1d Sparse sample

Matching and Completion

Reconstructed subspace

*. B.D.Decker, J.Kautz, T.Mertens, P.Bekaert, "Capturing Multiple Illumination Conditions using Time and Color Multiplexing", CVPR'09


1/30 S1/30 S 1/30 S1/30 S

Surface Matching

N Lights


N Lights

Target Model

Texture Mapping

Tracking Model

Reconstruction of full space vision field using surface matching

Surface animation techniques:


• The MVML dome achieves the samplings of the vision field.

• It is a foundation platform for the research on sampling theory of the vision field.

• We hope to extend results obtain in MVML dome to outdoor sconces and other practical applications.


Un-calibrated Un- known Un-synchronized

Conclusion

• Continuous depth map based MVS is prospective for its accuracy, robustness and flexibility

• The Vision Field and the MVML dome has still lots of interesting problems haven't been investigated


Acknowledgement

• Bennett Wilbrum

• Moshe ben-ezra



We hope for corporative researches on related topics

Welcome to visit BBNC and Tsinghua University!

Thank you!

宽带网数字媒体技术实验室Broadband Network & Digital Media Lab

3d modeling using multi-camera and multi-lighting (mvml) dome

Documents