Learning a model of facial shape and expression from 4D scans

Tianye Li*, Timo Bolkart*,

Michael J. Black, Hao Li, Javier Romero

SIGGRAPH Asia 2017 Note: this slide is a static .pdf version (no video)For video, please see: https://youtu.be/36rPTkhiJTM

Realistic Virtual Character Warner Bros. & Paramount Pictures

Realistic Virtual Character Warner Bros. & Paramount Pictures

Consumer Application Apple 2017

Spectrum of Face Models

“Low-end” “High-end”

Spectrum of Face Models

“Low-end” “High-end”FACS-based blendshapes

Spectrum of Face Models

“Low-end” “High-end”FACS-based blendshapes

Blanz and Vetter 1999

& Basel Face Model[Paysan et al. 2009]

Spectrum of Face Models

“Low-end” “High-end”FACS-based blendshapes

Blanz and Vetter 1999

FaceWarehouse [Cao et al. 2014]

& Basel Face Model[Paysan et al. 2009]

Spectrum of Face Models

“Low-end” “High-end”FACS-based blendshapes

Blanz and Vetter 1999

FaceWarehouse [Cao et al. 2014]

Wu et al. 2016

& Basel Face Model[Paysan et al. 2009]

Spectrum of Face Models

“Low-end” “High-end”FACS-based blendshapes

Blanz and Vetter 1999

FaceWarehouse [Cao et al. 2014]

Wu et al. 2016

Digital Emily [Alexander et al. 2009]& Basel Face Model

[Paysan et al. 2009]

Spectrum of Face Models

“Low-end” “High-end”FACS-based blendshapes

Blanz and Vetter 1999

FaceWarehouse [Cao et al. 2014]

Wu et al. 2016

Digital Emily [Alexander et al. 2009]& Basel Face Model

[Paysan et al. 2009]

FLAME Face ModelIssues FLAME

FLAME Face Model

Limited identity coverage Learned from ~4000 identities

Issues FLAME

FLAME Face Model

Limited identity coverage Learned from ~4000 identities

Issues FLAME

Artist designed expression Learned from high-quality 4D expression scans

FLAME Face Model

Limited identity coverage Learned from ~4000 identities

Issues FLAME

Artist designed expression Learned from high-quality 4D expression scans

Over-complete FACS basis Orthogonal expression space

FLAME Face Model

Limited identity coverage Learned from ~4000 identities

Issues FLAME

Artist designed expression Learned from high-quality 4D expression scans

Over-complete FACS basis Orthogonal expression space

Non-linearity of jaw and neck Modeled as rotatable jointsActivated by linear blend skinningPose blendshapes further capture details

FLAME Face Model

Limited identity coverage Learned from ~4000 identities

Issues FLAME

Artist designed expression Learned from high-quality 4D expression scans

Over-complete FACS basis Orthogonal expression space

Non-linearity of jaw and neck Modeled as rotatable jointsActivated by linear blend skinningPose blendshapes further capture details

Require artist work Fully automatic registration and modeling

FLAME Face Model

FLAME Face Model

Template

FLAME Face Model

Template Shape

FLAME Face Model

Template Shape Shape+Pose

FLAME Face Model

Template Shape Shape+ Pose

+ Expression

Shape+Pose

Overview

Co-registrationHirshberg et al. 12

CAESAR dataset

Shape Data Registration Shape Model Training

MPI FacialMotion dataset

Pose Data Registration Pose Model Training

Expression Data Registration Expression Model Training

D3DFACS dataset

Initial Expression Blendshapes

Overview

Co-registrationHirshberg et al. 12

CAESAR dataset

Shape Data Registration Shape Model Training

MPI FacialMotion dataset

Pose Data Registration Pose Model Training

Expression Data Registration Expression Model Training

D3DFACS dataset

Initial Expression Blendshapes

Shape Model

Shape Data

Registration of CAESAR datasets[Robinette et al. 2002]

Learned Shape Model

Pose Model

Pose Data

Registration of MPI FacialMotion datasets for pose training

Learned Pose Model

Effect of Pose Blendshapes

Expression Model

Expression Data

Expression Data

4D Scans into Correspondence

Coarse-to-Fine Registration

>1 mm

0 mm

Stage 1: model-only

Coarse-to-Fine Registration

>1 mm

0 mm

Stage2: coupled

Coarse-to-Fine Registration

>1 mm

0 mm

Stage 3: Texture-based

Alignment

Effect of Texture-based Alignment

Effect of Texture-based Alignment

Registration Results

>1 mm

0 mm

Registration Results

>1 mm

0 mm

Detail expressions such as eye blinking are also captured

Learned Expression Model

Results

Compare on Identity Space

FaceWarehouse[Cao et al. 2014]50 components

FLAME 4949 components + 1 for gender

0 mm

>1 mm

BU-3DFE scan Fitting Scan-to-MeshDistance

Fitting Scan-to-MeshDistance

Compare on Identity Space

FLAME 198198 components + 1 for gender

0 mm

>1 mm

BU-3DFE scan Fitting Scan-to-MeshDistance

Fitting Scan-to-MeshDistance

Basel Face Model (BFM) [Paysan et al. 2009]199 components

Compare on Identity Space

FLAME 198198 components + 1 for gender

0 mm

>1 mm

BU-3DFE scan Scan-to-MeshDistance

Scan-to-MeshDistance

Basel Face Model (BFM) 199 componentsBU-3DFE scan

Compare on Identity Space

0 0.5 1 1.5 2Error [mm]

0

20

40

60

80

100

Perc

enta

geFLAME 300FLAME 198FLAME 90FLAME 49FWBFM FullBFM 91BFM 50

Compare on Identity Space

FW 50: 67%BFM 50: 69%

FLAME 49: 74%

BFM 199: 92%FLAME 198: 94%

FLAME 300: 96%

FLAME: our modelBFM: Basel Face ModelFW FaceWarehouse

Note: higher value is better

Compare on Expression Space

>3 mm

0 mm

Sparse Landmark Fitting

FLAME produces better result in 2D landmark fitting

Application: Retargeting

FLAME retargeting pipeline

FLAMEFace Model

Source Retargeted

Target Scan

Expression & pose

Identity

Application: Retargeting

Conclusion

What did we learn

• Large high-quality data

• Separation of identity, pose and expression

• Importance of face prior

• Model and data available for research purpose

Future Work

AcknowledgementTsvetelina Alexiadis, Andrea Keller, Jorge Márquez

Data Acquisition

Shunsuke Saito & Cassidy LaidlawEvaluation

Yinghao Huang, Ahmed Osman, Naureen MahmoodDiscussion

Talha ZamanVideo Recording

Alejandra Quiros-RamirezProject Website

Darren CoskerAdvice and D3DFACS Dataset

Thank You!

http://flame.is.tue.mpg.de/Registrations for D3DFACS dataset

FLAME face model (male / female) with example code