video based animation synthesis with the essential graph · video based animation synthesis with...
TRANSCRIPT
![Page 1: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/1.jpg)
Video based Animation Synthesis with the Essential Graph
Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes
![Page 2: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/2.jpg)
Goal
Input:
• Video based 4D models of elementary movements
Output:
• Novel, user guided shape and appearance animation
2
Given a set of 4D models, how to generate realistic motion from user specified constraints ?
![Page 3: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/3.jpg)
Motivation
Game Of Thrones, HBO Rise of the Tomb Raider, Cristal Dynamics
Human animation generation: Where: • Digital media production :
– Video Game industry – Motion Picture industry – Virtual Reality applications
3
![Page 4: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/4.jpg)
How: • Physical modeling:
– Computationally expensive – Model limitations
Motivation
4
Game Of Thrones, HBO Rise of the Tomb Raider, Cristal Dynamics
Human animation generation: Where: • Digital media production :
– Video Game industry – Motion Picture industry – Virtual Reality applications
![Page 5: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/5.jpg)
How: • Physical modeling:
– Computationally expensive – Model limitations
• Example data Reuse: – Key-framing data:
• Cost-wise expensive: 100-250$ / character second • Time-wise expensive: 2-3 seconds of finished character animation per day
Motivation
5
Game Of Thrones, HBO Rise of the Tomb Raider, Cristal Dynamics
Human animation generation: Where: • Digital media production :
– Video Game industry – Motion Picture industry – Virtual Reality applications
![Page 6: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/6.jpg)
How: • Physical modeling:
– Computationally expensive – Model limitations
• Example data Reuse: – Key-framing data:
• Cost-wise expensive: 100-250$ / character second • Time-wise expensive: 2-3 seconds of finished character animation per day
Motivation
Game Of Thrones, HBO Rise of the Tomb Raider, Cristal Dynamics
Synthetic Motion
Human animation generation: Where: • Digital media production :
– Video Game industry – Motion Picture industry – Virtual Reality applications
6
![Page 7: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/7.jpg)
How: • Physical modeling:
– Computationally expensive – Model limitations
• Example data Reuse: – Key-framing data:
• Cost-wise expensive: 100-250$ / character second • Time-wise expensive: 2-3 seconds of finished character animation per day
– Motion capture data: ü Real Motion
Motivation
7
Game Of Thrones, HBO Rise of the Tomb Raider, Cristal Dynamics
Synthetic Motion
CMU mocap dataset
Human animation generation: Where: • Digital media production :
– Video Game industry – Motion Picture industry – Virtual Reality applications
![Page 8: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/8.jpg)
How: • Physical modeling:
– Computationally expensive – Model limitations
• Example data Reuse: – Key-framing data:
• Cost-wise expensive: 100-250$ / character second • Time-wise expensive: 2-3 seconds of finished character animation per day
– Motion capture data: ü Real Motion
Motivation
8
Game Of Thrones, HBO Rise of the Tomb Raider, Cristal Dynamics
Synthetic Motion
Synthetic Shape
CMU mocap dataset
3D character animated with mocap
Human animation generation: Where: • Digital media production :
– Video Game industry – Motion Picture industry – Virtual Reality applications
![Page 9: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/9.jpg)
How: • Physical modeling:
– Computationally expensive – Model limitations
• Example data Reuse: – Key-framing data:
• Cost-wise expensive: 100-250$ / character second • Time-wise expensive: 2-3 seconds of finished character animation per day
– Motion capture data: ü Real Motion
– Surface capture data: ü Real Motion ü Real Shape ü Real Appearance
Motivation
9
Game Of Thrones, HBO Rise of the Tomb Raider, Cristal Dynamics
Synthetic Motion
Synthetic Shape
CMU mocap dataset
3D character animated with mocap 4D model, Thomas dataset
Human animation generation: Where: • Digital media production :
– Video Game industry – Motion Picture industry – Virtual Reality applications
![Page 10: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/10.jpg)
Surface Capture Reuse
10
Surface Reconstruction
Appearance projection on
geometry
Multi-view videos
Surface temporal tracking
• Mesh sequence with time-variant topology
• Mesh sequence with time-consistent topology
• 4D textured model
![Page 11: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/11.jpg)
Surface Capture Reuse
Data: • Cyclic human movements: walk, run, jump, etc. • Acyclic human movements: dance.
11
Surface Reconstruction
Appearance projection on
geometry
Multi-view videos
Surface temporal tracking
• Mesh sequence with time-variant topology
• Mesh sequence with time-consistent topology
• 4D textured model
Thomas, Cathy
![Page 12: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/12.jpg)
Surface Capture Reuse
Data: • Cyclic human movements: walk, run, jump, etc. • Acyclic human movements: dance.
12
Data Reuse: Generate continuous motion stream using basic operations on motion segments: • Rigid transformations • Concatenation • Smooth transition generation
Thomas, Cathy
Surface Reconstruction
Appearance projection on
geometry
Multi-view videos
Surface temporal tracking
• Mesh sequence with time-variant topology
• Mesh sequence with time-consistent topology
• 4D textured model
![Page 13: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/13.jpg)
13
Issues and challenges User control: � Intuitive formulation of user defined constraints
Data organization: � A data structure organizing the input sequences and encoding selected transitions between them
Motion synthesis: � Generating synthetic motion transitions. � Concatenating real and synthetic motion segments.
Numerical realism criterion: � Transition evaluation
![Page 14: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/14.jpg)
14
Issues and challenges Challenges :
• Limited data: • Make exhaustive use of it
• User perceptual acuity: • Reliable numerical realism criterion • Optimal results in terms of said criterion
• Sensitive data: • Robust mesh processing technique
• Complex dynamics: • Robust transition generation technique
User control: � Intuitive formulation of user defined constraints
Data organization: � A data structure organizing the input sequences and encoding selected transitions between them
Motion synthesis: � Generating synthetic motion transitions. � Concatenating real and synthetic motion segments.
Numerical realism criterion: � Transition evaluation
![Page 15: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/15.jpg)
Comparison of Graph Based Approaches
15
Input data Automatic/supervised Data organization
Motion Graph(kovar02)(Arikan02) Motion Capture automatic Motion graph
Surface Motion Graph(Huang09) 3D Surface Capture automatic Motion graph
4D Parametric Motion Graph(Casas13) 4D Surface Capture supervised Parametric
motion graph
Essential Graph(Boukhayma et. Boyer15) 4D Surface Capture automatic Essential
graph
![Page 16: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/16.jpg)
16
Contributions
• Improving realism in synthetic motion transitions through dynamic time warping and variable length blended segments
• An optimal structure for motion data organization and reuse: The essential graph
• A novel high-level constraint formulation for motion synthesis: 3D behavioral path synthesis
![Page 17: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/17.jpg)
17
Approach Motion synthesis pipeline :
![Page 18: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/18.jpg)
18
Input Data Organization
![Page 19: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/19.jpg)
Graph structure: • Node = frame/pose • Edge = transition • Edge weight = transition cost
19
Input sequences
Input Data Organization
![Page 20: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/20.jpg)
Graph structure: • Node = frame/pose • Edge = transition • Edge weight = transition cost
We need to add new transitions :
20
Input sequences
Input Data Organization
![Page 21: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/21.jpg)
Graph structure: • Node = frame/pose • Edge = transition • Edge weight = transition cost
Motion Graph(kovar02) • Adding edges
– Local minima in similarity matrix – Threshloding
21
Input sequences
Similarity matrix,
Input Data Organization
![Page 22: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/22.jpg)
Graph structure: • Node = frame/pose • Edge = transition • Edge weight = transition cost
Motion Graph(kovar02) • Adding edges
– Local minima in similarity matrix – Threshloding
22
Input sequences
Similarity matrix,
Input Data Organization
![Page 23: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/23.jpg)
Graph structure: • Node = frame/pose • Edge = transition • Edge weight = transition cost
Motion Graph(kovar02) • Adding edges
– Local minima in similarity matrix – Threshloding
23
Input sequences
Similarity matrix,
Input Data Organization
![Page 24: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/24.jpg)
Graph structure: • Node = frame/pose • Edge = transition • Edge weight = transition cost
Motion Graph(kovar02) • Adding edges
– Local minima in similarity matrix – Threshloding
• Improvements – Interpolated Motion Graph(Sofanova07) – Well-Connected Motion Graph(Zhao08) – Optimization-based Motion Graph(Ren10)
24
Input sequences
Similarity matrix,
Input Data Organization
![Page 25: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/25.jpg)
25
Essential Graph • Creating a complete digraph
– Connecting all nodes together with directed edges and transition costs as weights
Essential graph
Complete digraph
Input sequences
![Page 26: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/26.jpg)
26
Essential Graph • Creating a complete digraph
– Connecting all nodes together with directed edges and transition costs as weights
• Extracting the essential sub-graph – For each node
• Calculate the shortest path tree rooted at said node
Complete digraph
Input sequences
Essential graph
![Page 27: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/27.jpg)
27
Essential Graph • Creating a complete digraph
– Connecting all nodes together with directed edges and transition costs as weights
• Extracting the essential sub-graph – For each node
• Calculate the shortest path tree rooted at said node
shortest path tree: the path distance from the root to any vertex in the tree is the shortest path from the root to the vertex in the complete digraph
Complete digraph
Input sequences
Essential graph
![Page 28: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/28.jpg)
28
Essential Graph • Creating a complete digraph
– Connecting all nodes together with directed edges and transition costs as weights
• Extracting the essential sub-graph – For each node
• Calculate the shortest path tree rooted at said node
– Union of shortest path trees rooted at every node – This structure encodes all the optimal transitions with
respect to the transition cost
Complete digraph
Essential graph
Input sequences
shortest path tree: the path distance from the root to any vertex in the tree is the shortest path from the root to the vertex in the complete digraph
Essential graph
![Page 29: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/29.jpg)
Motion Graph vs Essential Graph
Comparative case study:
D E F
A 7 5 5
B 7 2 5
C 7 4 3
A B C
F E D
2
1
3
2
29
![Page 30: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/30.jpg)
Motion Graph vs Essential Graph
Comparative case study:
D E F
A 7 5 5
B 7 2 5
C 7 4 3
A B C
F E D
2
1
3
2
Motion graph
30
![Page 31: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/31.jpg)
Motion Graph vs Essential Graph
Comparative case study:
D E F
A 7 5 5
B 7 2 5
C 7 4 3
A B C
F E D
2
1
3
2
Motion graph
Shortest path tree rooted at A
31
![Page 32: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/32.jpg)
Motion Graph vs Essential Graph
Comparative case study:
D E F
A 7 5 5
B 7 2 5
C 7 4 3
A B C
F E D
2
1
3
2
Motion graph
Shortest path tree rooted at A
Shortest path tree rooted at B
32
![Page 33: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/33.jpg)
Motion Graph vs Essential Graph
Comparative case study:
D E F
A 7 5 5
B 7 2 5
C 7 4 3
A B C
F E D
2
1
3
2
Motion graph
Shortest path tree rooted at A
Shortest path tree rooted at B
Shortest path tree rooted at C
33
![Page 34: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/34.jpg)
Motion Graph vs Essential Graph
Comparative case study:
D E F
A 7 5 5
B 7 2 5
C 7 4 3
A B C
F E D
2
1
3
2
Motion graph
Essential graph
+ + = Shortest path tree
rooted at A
Shortest path tree rooted at B
Shortest path tree rooted at C
34
![Page 35: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/35.jpg)
Motion Graph vs Essential Graph
Comparative case study:
D E F
A 7 5 5
B 7 2 5
C 7 4 3
A B C
F E D
2
1
3
2
Motion graph
Shortest path from A to F: 6
Shortest path from D to C: 6
Essential graph
+ + = Shortest path tree
rooted at A
Shortest path tree rooted at B
Shortest path tree rooted at C
35
![Page 36: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/36.jpg)
Motion Graph vs Essential Graph
Comparative case study:
D E F
A 7 5 5
B 7 2 5
C 7 4 3
A B C
F E D
2
1
3
2
Motion graph
Shortest path from A to F: 6
Shortest path from D to C: 6
Essential graph
+ + =
Shortest path from A to F: 4
Shortest path from E to C: 5
Shortest path tree rooted at A
Shortest path tree rooted at B
Shortest path tree rooted at C
36
![Page 37: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/37.jpg)
37
Input Data Organization
Original frames Synthetic frames
Thomas dataset
![Page 38: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/38.jpg)
38
Transition Costs Motion synthesis pipeline :
![Page 39: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/39.jpg)
Transition Costs Edge weight = transition cost in terms of Realism criterion:
Ei, j
realism!
= Di, j
surface deformation!
+α Li, jduration!
39
![Page 40: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/40.jpg)
Transition Costs Edge weight = transition cost in terms of Realism criterion:
Ei, j
realism!
= Di, j
surface deformation!
+α Li, jduration!
Interpolated Transition: Source and target motion segments gradual blending • Dynamic time warping • Variable length blended segments
40
![Page 41: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/41.jpg)
Transition Costs Edge weight = transition cost in terms of Realism criterion:
Ei, j
realism!
= Di, j
surface deformation!
+α Li, jduration!
Interpolated Transition: Source and target motion segments gradual blending • Dynamic time warping • Variable length blended segments Finding segment lengths li and lj, and temporal warps wi and wj that minimize the total surface deformation cost: Function d(.,.) is a static pose distance.
D(i, j) = minli ,l j∈ lmin ,lmax[ ]
minwi ,w j
d(wi−1 (t),w j−1 (t))t∈ 0,L[ ]∑
Dynalic Time Warping! "#### $####
41
![Page 42: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/42.jpg)
Transition Costs Edge weight = transition cost in terms of Realism criterion:
Ei, j
realism!
= Di, j
surface deformation!
+α Li, jduration!
Interpolated Transition: Source and target motion segments gradual blending • Dynamic time warping • Variable length blended segments Motion synthesis level: • Pose interpolation of matched frame
along the transition • Global displacements interpolation
42
![Page 43: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/43.jpg)
43
Pose distance and interpolation Motion synthesis pipeline :
![Page 44: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/44.jpg)
Pose distance and interpolation Skeleton parameterization for Mo-cap data:
Riemannian manifold:
SO(3)njo int s
44
![Page 45: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/45.jpg)
Pose distance and interpolation Skeleton parameterization for Mo-cap data:
Riemannian manifold: • Pose metric
SO(3)njo int s
12njo int s
∑ log(R1−1R2 ) F
45
![Page 46: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/46.jpg)
Pose distance and interpolation Skeleton parameterization for Mo-cap data:
Riemannian manifold: • Pose metric • Geodesic pose interpolation
SO(3)njo int s
12njo int s
∑ log(R1−1R2 ) F
R1eλ log(R1
−1R2 )
46
![Page 47: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/47.jpg)
Pose distance and interpolation Skeleton parameterization for Mo-cap data: Skeleton parametrization for surfacic data: • Mesh Ensemble Motion Graph(James07) • One-to-Many(Zheng13)
Riemannian manifold: • Pose metric • Geodesic pose interpolation
SO(3)njo int s
12njo int s
∑ log(R1−1R2 ) F
R1eλ log(R1
−1R2 )
LBS skinning, Zheng13 Articulated plant model, James07
47
![Page 48: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/48.jpg)
Pose distance and interpolation Skeleton parameterization for Mo-cap data: Skeleton parametrization for surfacic data: • Mesh Ensemble Motion Graph(James07) • One-to-Many(Zheng13)
Drawbacks – Robust identification of an underlying skeleton structure is difficult – Realism loss due to articulated model limitation
Riemannian manifold: • Pose metric • Geodesic pose interpolation
SO(3)njo int s
12njo int s
∑ log(R1−1R2 ) F
R1eλ log(R1
−1R2 )
LBS skinning, Zheng13 Articulated plant model, James07
48
![Page 49: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/49.jpg)
Pose distance and interpolation Skeleton parameterization for Mo-cap data: Skeleton parametrization for surfacic data: • Mesh Ensemble Motion Graph(James07) • One-to-Many(Zheng13)
Drawbacks – Robust identification of an underlying skeleton structure is difficult – Realism loss due to articulated model limitation
Surface based parametrization for surfacic data: • Surface Motion Graph(Huang09) • 4D Parametric Motion Graph(Casas13) (Heck07)(Shin06)
Riemannian manifold: • Pose metric • Geodesic pose interpolation
SO(3)njo int s
12njo int s
∑ log(R1−1R2 ) F
R1eλ log(R1
−1R2 )
LBS skinning, Zheng13 Articulated plant model, James07
Parametric motion graph, Casas 13
49
![Page 50: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/50.jpg)
Pose distance and interpolation Euclidean Parametrization • Pose metric : sum of squared Euclidean distances between vertices • pose interpolation : linear interpolation of vertices coordinates
ℜ3×nvertices
50
![Page 51: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/51.jpg)
Pose distance and interpolation Euclidean Parametrization • Pose metric : sum of squared Euclidean distances between vertices • pose interpolation : linear interpolation of vertices coordinates
Drawbacks – Rigid Alignment residual error do not account for mesh geometry – Linear mesh interpolation introduces surface distortion
ℜ3×nvertices
51
![Page 52: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/52.jpg)
Pose distance and interpolation Euclidean Parametrization • Pose metric : sum of squared Euclidean distances between vertices • pose interpolation : linear interpolation of vertices coordinates
Drawbacks – Rigid Alignment residual error do not account for mesh geometry – Linear mesh interpolation introduces surface distortion
Surface Deformation based Parameterization for 3D triangular meshes:
Polar decomposition of Deformation Gradients: Riemannian manifold:
T = R.ST = (v1 − v3,v2 − v3,n)
−1.(v '1− v '3,v '2− v '3,n ')
(T1,..,Tmtriangles )∈ (SO(3)× S++3)
mtriangles
ℜ3×nvertices
52
![Page 53: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/53.jpg)
Pose distance and interpolation Euclidean Parametrization • Pose metric : sum of squared Euclidean distances between vertices • pose interpolation : linear interpolation of vertices coordinates
Drawbacks – Rigid Alignment residual error do not account for mesh geometry – Linear mesh interpolation introduces surface distortion
Surface Deformation based Parameterization for 3D triangular meshes:
Polar decomposition of Deformation Gradients: Riemannian manifold: • Pose metric
T = R.ST = (v1 − v3,v2 − v3,n)
−1.(v '1− v '3,v '2− v '3,n ')
(T1,..,Tmtriangles )∈ (SO(3)× S++3)
mtriangles
d(.,.) = 12mtriangles
∑ log(R)F+ log(S)
F( )
ℜ3×nvertices
53
![Page 54: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/54.jpg)
Pose distance and interpolation Euclidean Parametrization • Pose metric : sum of squared Euclidean distances between vertices • pose interpolation : linear interpolation of vertices coordinates
Drawbacks – Rigid Alignment residual error do not account for mesh geometry – Linear mesh interpolation introduces surface distortion
Surface Deformation based Parameterization for 3D triangular meshes:
Polar decomposition of Deformation Gradients: Riemannian manifold: • Pose metric • Pose interpolation
• Geodesic transformation interpolation
• Gradient deformation with interpolated transformation • 3D Poisson Shape reconstruction
T = R.ST = (v1 − v3,v2 − v3,n)
−1.(v '1− v '3,v '2− v '3,n ')
(T1,..,Tmtriangles )∈ (SO(3)× S++3)
mtriangles
d(.,.) = 12mtriangles
∑ log(R)F+ log(S)
F( )
!T = eλ log(R).eλ log(S )
ℜ3×nvertices
54
![Page 55: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/55.jpg)
55
User guided animation synthesis Motion synthesis pipeline :
![Page 56: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/56.jpg)
User guided animation synthesis Motion extraction casted as a graph search problem and solved using depth first search with Branch and bound
Cost function and halting condition
Motion stream Min-cost graph walk
High-level Constraint
56
![Page 57: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/57.jpg)
User guided animation synthesis
• Motion Stream – Concatenate original and interpolated
motion segments – Rigid alignment at segment junctions
Motion extraction casted as a graph search problem and solved using depth first search with Branch and bound
Cost function and halting condition
Motion stream Min-cost graph walk
High-level Constraint
57
![Page 58: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/58.jpg)
User guided animation synthesis
• Motion Stream – Concatenate original and interpolated
motion segments – Rigid alignment at segment junctions
• High Level constraints
– 3d behavioral path synthesis • Follow a 3D path, behave as precised at some parts • Adapted to locomotion data
– Pose/time constraint
• reach specific poses at specific times • Adapted to unstructured acyclic motion
Motion extraction casted as a graph search problem and solved using depth first search with Branch and bound
Cost function and halting condition
Motion stream Min-cost graph walk
High-level Constraint
3D behavioral path synthesis
2D path synthesis
58
![Page 59: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/59.jpg)
59
User guided motion synthesis
![Page 60: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/60.jpg)
60
User guided motion synthesis
![Page 61: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/61.jpg)
61
User guided motion synthesis
![Page 62: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/62.jpg)
Conclusion
• Realistic shape and appearance animation generation from user specified constraints
• Contributions: – An optimal structure for motion data organization and reuse – A method for improving realism in motion transitions – A novel high level constraint formulation
• Limitations: – Foot skate
• Future work: – Data annotation – Texture interpolation – More Data : Motion warping, Motion transfert, Merging parts motion.
62
![Page 63: Video based Animation Synthesis with the Essential Graph · Video based Animation Synthesis with the Essential Graph Adnane Boukhayma, Edmond Boyer MORPHEO INRIA Grenoble Rhône-Alpes](https://reader030.vdocuments.site/reader030/viewer/2022040516/5e748154b88f512c6012e89a/html5/thumbnails/63.jpg)
63
Thank You !