Attention-Directing Composition of
Ying Cao Rynson W.H. Lau
Elements
Look Over Here:
Antoni B. Chan
City University of Hong Kong
• Composition of manga elements (i.e., subjects and balloons )
Motivation
© Tsugumi Ohba, Takeshi Obata / Shueisha Inc.
• Manga artist guides viewer’s eyes through the page
Motivation
© Tsugumi Ohba, Takeshi Obata / Shueisha Inc.
Artist’s Guiding Path (AGP)
Viewer Attention
Motivation
• One of the most difficult steps in manga production
• No existing comic creation programs for automatic composition
Goal
1.Rabbit, I came here for gold,
2. and I'm gonna get it!
3. I gotcha, you rabbit! I'll show you!
Close-upFast
LongMedium
Close-upMedium
Big Close-upMedium
MediumMedium
You can't do this to me!
Eureka! Gold at last!
Eureka! Gold at last!
1
2
3
4
5
Talk
Related Work
• Balloon Placement
[Kurlander et al. 1996]
[Chun et al. 2006]
• Eye-tracking Techniques for Comic Analysis and Creation
[Jain et al. 2012] [Toyoura et al. 2012]
Related Work
• Computational Manga
[Qu et al. 2006]
[Qu et al. 2008][Cao et al. 2012]
Related Work
Overview
Input Storyboard Layout Probabilistic Graphical Model
Input Infer
Learn
Data
𝐆 𝐂 𝐀
Probabilistic Graphical Model
Data Acquisition
• 80 manga pages from three manga series
Annotation
[Motion state][Shot type]
[Panel Shape][Subject][Balloon]
Eye movements of viewers
Probabilistic Graphical Model
• Abstracts artist’s guiding path as a latent variable
• Used for composition synthesis
• Connects artist’s guiding path, composition and viewer attention in a probabilisitc network
𝐆Artist’s Guiding Path
𝐂Composition
𝐀Viewer Attention
Subject Placement
Balloon Placement
Viewer Attention(eye-gaze path)
Local Composition in PanelPanel Properties
Artist’s Guiding Path
Probabilistic Graphical Model
Subject Placement
Balloon Placement
Viewer Attention(eye-gaze path)
Local Composition in PanelPanel Properties
Artist’s Guiding Path
Probabilistic Graphical Model
Artist’s Guiding Path
𝐟
𝐈
mean of f ( t )
𝐈(𝑡)
actual AGP:
underlying AGP:
Subject Placement
Balloon Placement
Viewer Attention(eye-gaze path)
Local Composition in PanelPanel Properties
Artist’s Guiding Path
Probabilistic Graphical Model
Panel Properties and Local Composition Model
𝐋 𝐋={𝐱𝐋 ,𝑟 𝐋}
𝐏
𝐱𝐋𝑟𝐋
: geometric shape
long, medium, close-up, big close-up: shot type
fast, medium, slow: motion state
𝑔 ,𝑡 ,𝑚
Subject Placement
Balloon Placement
Viewer Attention(eye-gaze path)
Local Composition in PanelPanel Properties
Artist’s Guiding Path
Probabilistic Graphical Model
Subject Placement
𝐈
𝐭𝐒
𝐋
𝐱𝐏
+
𝐈(𝑡𝐒)𝐒𝐒={𝐱𝐒 ,𝑟 𝐒}
𝐱𝐒
𝑟𝐒
: panel center location
: control point index
Subject Placement
Balloon Placement
Viewer Attention(eye-gaze path)
Local Composition in PanelPanel Properties
Artist’s Guiding Path
Probabilistic Graphical Model
Balloon Placement
𝐒
𝐈
𝐭𝐁 𝐎
𝐱𝐒
𝑟𝐒
𝐱𝐁
𝑟𝐁
𝐈(𝑡𝐁)
𝐁 𝐁={𝐱𝐁 ,𝑟𝐁}
: balloon order index
: control point index
Subject Placement
Balloon Placement
Viewer Attention(eye-gaze path)
Local Composition in PanelPanel Properties
Artist’s Guiding Path
Probabilistic Graphical Model
Viewer Attention
𝐒
𝐁
𝐔
𝐱𝐒
𝑟 S
𝐱𝐁
𝑟𝐁𝐔
: viewer attention transition
Complete Model
Interactive Composition Synthesis
Interactive Composition Synthesis
• Generate a composition, subject to user-specified semantics
• Layout Generation + Composition Synthesis
1.Rabbit, I came here for gold,
2. and I'm gonna get it!
3. I gotcha, you rabbit! I'll show you!
Input:
subject & script
Close-up
Fast
shot type & motion state
Talk
inter-subject constraint
Layout Generation
…
Database of labeled pages
[# of panels ] [# of subjects] [# of balloons] [shot type] [motion state]
Composition Synthesis
Configurations of elements
• A Maximum A Posteriori inference framework
Constraint-based likelihood Conditional prior
Input elements & semantics + Layout
𝐘𝐶Constraints
Constraint-based likelihood
• : overlap term • : order term
• : boundary term
• subject relation term
Constraint-based likelihood
𝑟 𝑖 𝑟 𝑗
𝐯 ij
Interaction “Talk”
shot “medium”
Results and Evaluation
Comparison to Heuristic Method for Balloon Placement
Our approachHeuristic [Chun et al. 2006]
Comparison to Heuristic Method for Balloon Placement
Our approachHeuristic [Chun et al. 2006]
Comparison to Manual Method
• User study: How well does our approach facilitate manga composition, as compared to existing manual tool?
Participant preference voting Time for one composition
Eye-tracking Experiment and Analysis
• Is our approach effective in directing viewer attention, as compared with the alternatives?
• To measure fixation consistency across different viewers
• Similarity metrics: 1. Inlier percent
2. Root Mean Squared Distance (RMSD)
Eye-tracking Experiment and Analysis
1. Inlier percent
Viewer A Saliency Map
Viewer B
InliersClassification
RMSD,
2. RMSD
Viewer A Viewer B
Eye-tracking Experiment and Analysis
Eye-tracking Experiment and Analysis
Comparison to Existing Manga Pages
Original Synthesized© Tsugumi Ohba, Takeshi Obata / Shueisha Inc.
Summary
• A novel model for representing dependency among the artist’s guiding path, composition and viewer attention
• A novel approach for joint composition of subjects and balloons
• Enable easy and quick creation of attention-directing compositions
Limitations & Future Work
• Consider visual appearance of manga elements
• Extend to other graphic design tasks
• Understand readers’ behaviors
Thanks