temporally coherent 4d reconstruction of complex …cvssp.org/projects/4d/4drecon/ppt.pdf · 2016....

Armin Mustafa, Hansung Kim, Jean-Yves Guillemaut, Adrian Hilton

TEMPORALLY COHERENT 4D

RECONSTRUCTION OF COMPLEX

DYNAMIC SCENES

PROBLEM STATEMENT

2

• Reconstruct complex dynamic scenes.

• Multi-view, wide-baseline and moving handheld cameras.

• Unknown background, structure and segmentation.

3

Segmentation Depth map

• Requires accurate segmentation of foreground

• Known background and structure

PROBLEMS IN EXISTING METHODS

OVERVIEW

4

Multi-view scene

No prior

Moving cameras

Framework

Salient object

identification

Temporal coherence

4D scene reconstruction and

segmentation

FRAMEWORK

5

Multi-view

data

Frames 0 N

FRAMEWORK

6

Multi-view

data

Frames 0 N

Sparse point

cloud

Scene level

Sparse point cloud

FRAMEWORK

7

Multi-view

data

Frames 0 N

Sparse point cloud and Object Clustering

Sparse point

cloud

Object

Clustering

Scene level

FRAMEWORK

8

Multi-view

data

Frames 0 N

Initial coarse reconstruction and refinement


Sparse point

cloud

Object

Clustering

Scene level Object level

Initial coarse

reconstruction

Refinement

FRAMEWORK

9

Multi-view

data

Frames 0 N

Initial coarse reconstruction and refinement

Temporal coherence for dynamic object and refinement


Sparse point

cloud

Object

Clustering


Initial coarse

reconstruction

Temporal

coherence for

dynamic object

Refinement

FRAMEWORK

Multi-view

data

Frames 0 N


Scene level

Sparse point cloud Object clustering

Object level

Sparse point

cloud

Object

Clustering

Initial coarse

reconstruction

A. Mustafa, H. Kim, J-Y. Guillemaut and A. Hilton General Dynamic Scene Reconstruction from Multiple View Video. ICCV 2015

INITIAL COARSE RECONSTRUCTION


Sparse Cluster Triangulation Initial coarse reconstruction

FRAMEWORK

12

Multi-view

data

Frames 0 N


Sparse point

cloud

Object

Clustering


Initial coarse

reconstruction Refinement

FRAMEWORK

13

Frames 0 N

Multi-view

data



Sparse point

cloud

Object

Clustering


Initial coarse

reconstruction

Temporal

coherence for

dynamic object

19 20

TEMPORAL COHERENCE :

14

Sparse to dense reconstruction and refinement:

Previous

frame mesh Optical flow

Dense temporal

correspondence

Initial coarse

reconstruction Final mesh

FRAMEWORK

15

Frames 0 N

Multi-view

data



Sparse point

cloud

Object

Clustering


Initial coarse

reconstruction

Temporal

coherence for

dynamic object

Temporal coherence

Multi-view video

Frame 1

FRAMEWORK

Multi-view

data

Frames 0 N



Sparse point

cloud

Object

Clustering


Initial coarse

reconstruction

Temporal

coherence for

dynamic object

Refinement

• Joint segmentation and reconstruction

• Optimized based on graph cuts

J-Y. Guillemaut and A. Hilton Joint Multi-Layer Segmentation and Reconstruction for Free-Viewpoint Video Applications. IJCV 2011

REFINEMENT: SHAPE

17

where is the label and is the depth

E(l,d) = + + +

≈

REFINEMENT: SHAPE

18


E(l,d) = + + +

• Smoothness is to ensure consistency of depth between neighbouring pixels.

REFINEMENT: SEGMENTATION

19


E(l,d) = + + +

• Color and contrast information combined with geodesic star-convexity is used to

refine segmentation.

Background and Foreground GMM models


20

Geodesic star convexity(GSC):

• Shape constraints improves segmentation

Star convex object

Not star convex

V. Gulshan, C. Rother , A. Criminisi, A. Blake and A. Zisserman Geodesic Star Convexity for Interactive Image Segmentation. CVPR 2010


21

Geodesic star convexity:

• Geodesic distances instead of Euclidean

Multiple Star

Centres

V. Gulshan, C. Rother , A. Criminisi, A. Blake and A. Zisserman Geodesic Star Convexity for Interactive Image Segmentation. CVPR 2010


22


E(l,d) = + + +

• Geodesic star-convexity to refine segmentation automatically.

Subject to GSC

Without GSC With GSC


23

Geodesic star convexity:

Input Star convex Geodesic star convex No constraint

REFINEMENT:

24

Temporal coherence:

Input No temporal coherence Temporal coherence

E(l,d) = + + +

REFINEMENT:

25

Temporal coherence:

Input No temporal coherence Temporal coherence

E(l,d) = + + +

RESULTS


J-Y. Guillemaut and A. Hilton Space-time joint multi-layer segmentation and depth estimation. 3DIMPVT 2012

Y. Furukawa and J. Ponce Accurate, Dense and Robust Multi-View Stereopsis. PAMI 2010

Method No Priors Temporal

coherence

Joint refinement

(Segmentation)

Furukawa PAMI 2010 X X

Guillemaut 3DV 2012 X Mustafa ICCV 2015

X Proposed

RESULTS – SEGMENTATION:

Dance dataset

Magician dataset


J-Y. Guillemaut and A. Hilton Space-time joint multi-layer segmentation and depth estimation. 3DIMPVT 2012

RESULTS –RECONSTRUCTION:

28

Juggler dataset

Initial

reconstruction Furukawa Guillemaut Mustafa Proposed

RESULTS –RECONSTRUCTION:

29

Dance dataset

Magician dataset

RESULTS – 4D RECONSTRUCTION:

30

Input

RESULTS – TEMPORAL COHERENCE :

31

Frame 1 Frame 1

RESULTS – COMPUTATION TIME:

32

Dataset Furukawa

(s)

Guillemaut

(s)

Mustafa

(s)

Ours

(s)

Dance1 326 493 295 254

Magician 311 608 377 325

Odzemok 381 598 394 363

Office 339 533 347 291

Juggler 394 634 411 378

Dance2 312 432 323 278

CONCLUSIONS

• An automatic framework for temporally coherent 4D reconstruction.

• Sparse to dense temporal coherence to improve quality.

• Joint segmentation and reconstruction refinement using GSC.

33

With temporal W/O temporal Input

FUTURE WORK

• Extending 4D reconstruction to single view video.

• Joint semantic segmentation using recognition.

• Handle crowded dynamic scenes

34

THANK YOU! Temporally coherent 4D reconstruction of complex dynamic scenes

Armin Mustafa, Hansung Kim, Jean-Yves Guillemaut, Adrian Hilton

http://cvssp.org/projects/4d/4DRecon/

Poster number : 12

35

http://cvssp.org/projects/4d/4DRecon/

temporally coherent 4d reconstruction of complex …cvssp.org/projects/4d/4drecon/ppt.pdf · 2016....

Documents