Presented by Matthew Cook

INFO410 & INFO350 S2 2015

INFORMATIONSCIENCE

Paper Discussion:Dynamic 3D Avatar Creationfrom Hand-held Video Input

A Ichim, S Bouaziz & M Pauly

INFO410 S2 2015 PAPER DISCUSSION SLIDE 2INFORMATION SCIENCE

Overview Pipeline for creating rigged, personalised face from hand-

held video Makes use of

Feature tracking Multi-view stereo depth estimation Blendshape animation/modelling

New contributions Factor out environment light to create face albedo

map Build blendshape rig from single camera video input

INFO410 S2 2015 PAPER DISCUSSION SLIDE 3INFORMATION SCIENCE

Overview - Stages

INFO410 S2 2015 PAPER DISCUSSION SLIDE 4INFORMATION SCIENCE

Static Modelling From mobile phone video input, generate 2 meshes

High res model of face detail from neutral pose Low res blendshape model for animating expressions

Later will create one blendshape pose per facial expression

INFO410 S2 2015 PAPER DISCUSSION SLIDE 5INFORMATION SCIENCE

Static Modelling Point cloud

generated with ‘Structure from Motion’

Applies features of captured face to a ‘Geometric Morphable Model’

INFO410 S2 2015 PAPER DISCUSSION SLIDE 6INFORMATION SCIENCE

Static Modelling The user is asked to mark

complex ridges for increased accuracy

Eyes are created simply by applying the detected iris colour to an existing mesh

INFO410 S2 2015 PAPER DISCUSSION SLIDE 7INFORMATION SCIENCE

Albedo Map An albedo map is a texture map of just the diffuse

reflections of a surface i.e. it tries to eliminate the effect of environment light

INFO410 S2 2015 PAPER DISCUSSION SLIDE 8INFORMATION SCIENCE

Dynamic Modelling Could stop at static

modelling and include only ‘virtual’ emotions in virtual environment

But what if you want to use the user’s own expressions?

INFO410 S2 2015 PAPER DISCUSSION SLIDE 9INFORMATION SCIENCE

Blendshapes A mesh of vertices which are weighted to vertices in

another mesh As vertices from the blendshape mesh are moved, the

attached vertices from the target are moved, according to their weights

Used to smoothly animate ‘organic’ motions

http://area.autodesk.com/tutorials/dragon_slayer

INFO410 S2 2015 PAPER DISCUSSION SLIDE 10INFORMATION SCIENCE

Blendshapes The blendshape models are generated using a

combination of feature tracking and model-fitting As with point registration, users can manually adjust the

detected points

INFO410 S2 2015 PAPER DISCUSSION SLIDE 11INFORMATION SCIENCE

Detail Synthesis Additional details can be

added after animation from known information such as surface normals

INFO410 S2 2015 PAPER DISCUSSION SLIDE 12INFORMATION SCIENCE

Results

Potential Exam Questions

INFO410 S2 2015 PAPER DISCUSSION SLIDE 14INFORMATION SCIENCE

Exam Questions Describe how you can estimate the depth of a feature

using only a single video stream What is an albedo map? What is a blendshape?

INFO410 S2 2015 PAPER DISCUSSION SLIDE 15INFORMATION SCIENCE

References The Paper: http://lgg.epfl.ch/~bouaziz/pdf/Avatars_SIGG15.pdf