appearance modeling: textures and ibr class 17

Appearance modeling: textures and IBR

Class 17

3D photography course schedule

Introduction

Aug 24, 26 (no course) (no course)

Aug.31,Sep.2

(no course) (no course)

Sep. 7, 9 (no course) (no course)

Sep. 14, 16 Projective Geometry Camera Model and Calibration

(assignment 1)

Feb. 21, 23 Camera Calib. and SVM Feature matching(assignment 2)

Feb. 28, 30 Feature tracking Epipolar geometry(assignment 3)

Oct. 5, 7 Computing F Triangulation and MVG

Oct. 12, 14 (university day) (fall break)

Oct. 19, 21 Stereo Active ranging

Oct. 26, 28 Structure from motion SfM and Self-calibration

Nov. 2, 4 Shape-from-silhouettes Space carving

Nov. 9, 11 3D modeling Appearance Modeling Nov.12 papers(2-3pm SN115)

Nov. 16, 18 (VMV’04) (VMV’04)

Nov. 23, 25 papers & discussion (Thanksgiving)

Nov.30,Dec.2

papers & discussion papers and discussion Dec.3 papers(2-3pm SN115)

Dec. 7? Project presentations

PapersLi Exact Voxel Occupancy with Graph Cuts

Sudipta Stereo without epipolar lines

ChrisA graph cut based adaptive structured light approach for real-time range acquisition

Nathan Space-time faces

Brian Depth-from-focus …

ChadInteractive Modeling from Dense Color and Sparse Depth

Seon Joo Outdoor calibration of active cameras

Jason spectral partitioning

Sriram Linear multi-view reconstruction

Christine 3D photography using dual …

http://www.unc.edu/courses/2004fall/comp/290b/089/papers/

Projects

Chris Wide-area display reconstruction

Nathan Structured light

Brian Depth-from-focus/defocus

Li Visual-hulls with occlusions

Chad Laser scanner for 3D environments

Seon Joo Collaborative 3D tracking

Jason SfM for long sequences

SudiptaCombining exact silhouettes and photoconsistency

Sriram Panoramic cameras self-calibration

Christine desktop lamp scanner

Multiple depth images Volumetric integration

Volumetric 3D integration

Appearance Modeling

• Texturing• Single image• Multiple image

• Image-based rendering• (Unstructured) lightfield rendering• Surface lightfields

Texture mapping 3D model

Need to estimate relative pose between camera and 3D model

Texture Mapping

• Conventional texture-mapping with texture coordinates

• Projective texture-mapping

Texture Map Synthesis I• Conventional Texture-

Mapping with Texture Coordinates• Create a triangular

texture patch for each triangle

• The texture patch is a weighted average of the image patches from multiple photographs

• Pixels that are close to image boundaries or viewed from a grazing angle obtain smaller weights

Photograph

Texture Map

3D Triangle

Texture Map Synthesis II• Allocate space for texture patches

from texture maps• Generalization of memory allocation

to 2D• Quantize edge length to a power of 2• Sort texture patches into decreasing

order and use First-Fit strategy to allocate space

First-Fit

A Texture Map Packed with Triangular Texture Patches

Appearance Modeling

texture atlas

Dealing with auto-exposure

Photometric alignment of textures (or HDR textures)

(Kim and Pollefeys, CVPR’04)

Image as texture

Depth image Triangle mesh Texture image

Textured 3DWireframe model

Affine vs. projective texture mapping (see later)

Lightfield literature

• Plenoptic function

• Lightfield (plane) and Lumigraph (some

geometry)

• Unstructered lightfield (some (view-dependent)

geometry)

• Surface lightfields (full geometry)

• Plenoptic sampling (trade-off geometry vs. images)

(Levoy&Hanrahan,Siggraph´96 Gortler et al.,Siggraph´96)

(Koch et al. ICCV´99; Heigl et al. DAGM´99; Buehler et al. Siggraph‘01)

(Chai et al.,Siggraph´00)

(Wood et al.,Siggraph´00, Chen et al., Siggraph‘02)

(Adelson&Bergen´91; McMillan&Bishop,Siggraph´95)

Lightfield rendering

focal surface

Approximate light rays by interpolating from closest light rays in lightfield

viewpoint surface

• Projection of viewpoint surface in virtual camera determines which views to get lightrays from

• Transfer from images to virtual views over focal surface determines which pixels to use

Unstructured lightfield rendering

original viewpoints

Novel view

For every pixel, combine For every pixel, combine best best rays from rays from closestclosest views views

(Koch et al.,ICCV´99; Heigl et al.,DAGM´99)

Focal surfaceFocal surface

demo

Example: desk sequence

186 images recorded with hand-held camera

Example: desk sequence

structure and motion

depth images

190 images

7000

poin

ts

Example: Desk Lightfield

Planar focal surface

(shadow artefacts)

View-dependent geometry approximation

original viewpoints

object surface

View-dependent surface approximation

Novel view

depth mapsdepth maps

Adaptation of geometry with the rendering viewpoint

View-dependent geometry approximation

Geometry subdivision

original viewpoints

object surface

View-dependent surface approximation

Novel view

depth mapsdepth maps

Note: Only necessary when depth value significantly Note: Only necessary when depth value significantly deviates from previous approximationdeviates from previous approximation

Viewpoint-geometry without subdivision

4 subdivisions

2 subdivisions

1 subdivision of viewpoint surface

Scalable geometric approximation

Example: Desk lightfield

Planar focal surface View-dependent geometry approximation(2 subdivisions)

Hardware accelerated rendering

Use blending operation similar to Gouraud shading

Use projective textures!

Demo

demo

Extrapolation(Buehler et al., Siggraph´01)

Add mesh to cover whole image (compute non-binary blending weights)

Rendered image

Blending field(courtesy Leonard McMillan)

,,,,, srfBGRI

Surface Lightfields

Surface locationSurface location Viewing directionViewing direction

Surface light field (SLF) function

Chen et al., Siggraph 2002, "Light Field Mapping: Efficient Representation and Hardware Rendering of Surface Light Fields" R. Grzeszczuk, Presentation on Light Field Mapping, SIGGRAPH 2002 Course Notes for Course “Image-based Modeling.” http://www.intel.com/research/mrl/research/lfm/

Surface Lightfields

• Partition SLF across surface primitives Pi

• Approximate SLF for each Pi individually as

• Surface light field (SLF) function

,*,,,,1

iii Pk

K

k

Pk

P hsrgsrf

Light field maps:Light field maps:stored as 2D texture mapsstored as 2D texture maps

Surface mapsSurface mapsView mapsView maps

Light Field Mapping

Data AcquisitionData Acquisition

ResamplingResampling

PartitioningPartitioning

RenderingRendering

ApproximationApproximation

CompressionCompression

Data Acquisition

• 200-400 images captured by hand-held camera

• Geometry scanned with structured lighting

• Images registered to geometry

Light Field Mapping




RenderingRendering



Partitioning• Partitioning the light field data across

small surface primitives• Individual parts add up to original SLF• Ensure continuous approximations

across neighbouring surface elements

Triangle-centered:Triangle-centered:split the light field between split the light field between individual trianglesindividual triangles

Partitioning

Triangle-centered: Triangle-centered: split the light field split the light field between individual between individual trianglestriangles ->discontinuity ->discontinuity

• Partitioning the light field data across small surface primitives• Individual parts add up to original SLF• Ensure continuous approximations

across neighbouring surface elements

Vertex-centered Partitioning

• Partition surface light field data around every vertex

Hat functionHat function

ppj sr

v,

0

a

outside

inside

ring

ring 10 a==


• Define local reference frame of the vertex

• Reparameterize each vertex light field to its local coordinate system

jjjj vvvv srfsrf ,,,,,,

Vertex light fieldVertex light field

Light Field Mapping




RenderingRendering



Resampling• Goal: Generate vertex light field

function• Visibility computation determines

unoccluded views for each triangle ring• 2 steps:

• Normalization of texture size• Resampling of viewing directions

Resampling

Each column represents a different view

j

i

jjjj vC

vvvv ffffF ...,, 32111stst view view 22ndnd view view

CCii-th view-th view

Resampling

• 1. Normalization of texture size• Each texture patch has the same

shape and size• Bilinear interpolation

• 2. Resampling of viewing directions

Resampling

1. Normalization of texture size2. Resampling of viewing directions

Projection of original viewsProjection of original views

1 2 3 4 ……. c ….. 1 2 3 4 ……. c ….. CCii11

22

……

……

..

mm

MM

Resampling


Delaunay triangulationDelaunay triangulationUniform grid of viewsUniform grid of views

MM

1 2 3 4 ……. c ….. 1 2 3 4 ……. c ….. CCii11

22

……

……

..

mm

Resampling


1 2 3 4 ……. n ….. 1 2 3 4 ……. n ….. NN11

22

……

……

..

mm

MM

Light Field Mapping




RenderingRendering



Decomposition & Approximation

• Rearrange 4-dimensional F into M*N matrix

• Decompose F using matrix factorization• Truncate the sum after K terms

iiij PN

PPv fffF ..., 21

1k

Tkk

v vuF j

N qqvk

K

kpp

vk

jj hsrg ,*,1

1

~

k

Tkk

v

vuFj

K<<NK<<N

K

Surface mapsSurface maps View mapsView maps


• Split surface maps for triangle ring into surface maps for individual triangles

• 3 surface maps for each approximation term of each triangle


11stst approximation approximation

22ndnd approximation approximation

KKthth approximation approximation

……..

Each approximationEach approximation=3 surface maps + 3 view maps=3 surface maps + 3 view maps

ff

Approximation methods

• PCA (principal component analysis)• Progressive• Arbitrary sign factors

• NMF (non-negative matrix factorization)• Parts-based representation• Non-negative factors• Easier and faster rendering

Light Field Mapping




RenderingRendering



Light Field Mapping

Tiled surface mapsTiled surface maps Tiled view mapsTiled view maps

• Light field maps are redundant• Very high compression ratio (10000:1)

Light Field Mapping




RenderingRendering



Rendering

11stst approximation approximation

22ndnd approximation approximation

KKthth approximation approximation

……..

Each approximationEach approximation=3 surface maps + 3 view maps=3 surface maps + 3 view maps

ff

Rendering

• Surface map: view-independent

• View map: • Establish vertex

coordinate system

• Project viewing vector onto view map hemisphere

Results

• Bust

• Star

• Turtle• Buddha• Horse

appearance modeling: textures and ibr class 17

Documents

texture coordinatescreate

projective texture mapping

sort texture patches

trianglethe texture

spacefirstfita texture

d trackingjasonsfm

d modelneed

desk light