attention-directing composition of ying cao rynson w.h. lau elements look over here: antoni b. chan...
TRANSCRIPT
![Page 1: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/1.jpg)
Attention-Directing Composition of
Ying Cao Rynson W.H. Lau
Elements
Look Over Here:
Antoni B. Chan
City University of Hong Kong
![Page 2: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/2.jpg)
• Composition of manga elements (i.e., subjects and balloons )
Motivation
© Tsugumi Ohba, Takeshi Obata / Shueisha Inc.
![Page 3: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/3.jpg)
• Manga artist guides viewer’s eyes through the page
Motivation
© Tsugumi Ohba, Takeshi Obata / Shueisha Inc.
Artist’s Guiding Path (AGP)
Viewer Attention
![Page 4: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/4.jpg)
Motivation
• One of the most difficult steps in manga production
• No existing comic creation programs for automatic composition
![Page 5: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/5.jpg)
Goal
1.Rabbit, I came here for gold,
2. and I'm gonna get it!
3. I gotcha, you rabbit! I'll show you!
Close-upFast
LongMedium
Close-upMedium
Big Close-upMedium
MediumMedium
You can't do this to me!
Eureka! Gold at last!
Eureka! Gold at last!
1
2
3
4
5
Talk
![Page 6: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/6.jpg)
Related Work
• Balloon Placement
[Kurlander et al. 1996]
[Chun et al. 2006]
![Page 7: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/7.jpg)
• Eye-tracking Techniques for Comic Analysis and Creation
[Jain et al. 2012] [Toyoura et al. 2012]
Related Work
![Page 8: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/8.jpg)
• Computational Manga
[Qu et al. 2006]
[Qu et al. 2008][Cao et al. 2012]
Related Work
![Page 9: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/9.jpg)
Overview
Input Storyboard Layout Probabilistic Graphical Model
Input Infer
Learn
Data
𝐆 𝐂 𝐀
![Page 10: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/10.jpg)
Probabilistic Graphical Model
![Page 11: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/11.jpg)
Data Acquisition
• 80 manga pages from three manga series
Annotation
[Motion state][Shot type]
[Panel Shape][Subject][Balloon]
Eye movements of viewers
![Page 12: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/12.jpg)
Probabilistic Graphical Model
• Abstracts artist’s guiding path as a latent variable
• Used for composition synthesis
• Connects artist’s guiding path, composition and viewer attention in a probabilisitc network
𝐆Artist’s Guiding Path
𝐂Composition
𝐀Viewer Attention
![Page 13: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/13.jpg)
Subject Placement
Balloon Placement
Viewer Attention(eye-gaze path)
Local Composition in PanelPanel Properties
Artist’s Guiding Path
Probabilistic Graphical Model
![Page 14: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/14.jpg)
Subject Placement
Balloon Placement
Viewer Attention(eye-gaze path)
Local Composition in PanelPanel Properties
Artist’s Guiding Path
Probabilistic Graphical Model
![Page 15: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/15.jpg)
Artist’s Guiding Path
𝐟
𝐈
mean of f ( t )
𝐈(𝑡)
actual AGP:
underlying AGP:
![Page 16: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/16.jpg)
Subject Placement
Balloon Placement
Viewer Attention(eye-gaze path)
Local Composition in PanelPanel Properties
Artist’s Guiding Path
Probabilistic Graphical Model
![Page 17: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/17.jpg)
Panel Properties and Local Composition Model
𝐋 𝐋={𝐱𝐋 ,𝑟 𝐋}
𝐏
𝐱𝐋𝑟𝐋
: geometric shape
long, medium, close-up, big close-up: shot type
fast, medium, slow: motion state
𝑔 ,𝑡 ,𝑚
![Page 18: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/18.jpg)
Subject Placement
Balloon Placement
Viewer Attention(eye-gaze path)
Local Composition in PanelPanel Properties
Artist’s Guiding Path
Probabilistic Graphical Model
![Page 19: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/19.jpg)
Subject Placement
𝐈
𝐭𝐒
𝐋
𝐱𝐏
+
𝐈(𝑡𝐒)𝐒𝐒={𝐱𝐒 ,𝑟 𝐒}
𝐱𝐒
𝑟𝐒
: panel center location
: control point index
![Page 20: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/20.jpg)
Subject Placement
Balloon Placement
Viewer Attention(eye-gaze path)
Local Composition in PanelPanel Properties
Artist’s Guiding Path
Probabilistic Graphical Model
![Page 21: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/21.jpg)
Balloon Placement
𝐒
𝐈
𝐭𝐁 𝐎
𝐱𝐒
𝑟𝐒
𝐱𝐁
𝑟𝐁
𝐈(𝑡𝐁)
𝐁 𝐁={𝐱𝐁 ,𝑟𝐁}
: balloon order index
: control point index
![Page 22: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/22.jpg)
Subject Placement
Balloon Placement
Viewer Attention(eye-gaze path)
Local Composition in PanelPanel Properties
Artist’s Guiding Path
Probabilistic Graphical Model
![Page 23: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/23.jpg)
Viewer Attention
𝐒
𝐁
𝐔
𝐱𝐒
𝑟 S
𝐱𝐁
𝑟𝐁𝐔
: viewer attention transition
![Page 24: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/24.jpg)
Complete Model
![Page 25: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/25.jpg)
Interactive Composition Synthesis
![Page 26: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/26.jpg)
Interactive Composition Synthesis
• Generate a composition, subject to user-specified semantics
• Layout Generation + Composition Synthesis
1.Rabbit, I came here for gold,
2. and I'm gonna get it!
3. I gotcha, you rabbit! I'll show you!
Input:
subject & script
Close-up
Fast
shot type & motion state
Talk
inter-subject constraint
![Page 27: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/27.jpg)
Layout Generation
…
Database of labeled pages
[# of panels ] [# of subjects] [# of balloons] [shot type] [motion state]
![Page 28: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/28.jpg)
Composition Synthesis
Configurations of elements
• A Maximum A Posteriori inference framework
Constraint-based likelihood Conditional prior
Input elements & semantics + Layout
𝐘𝐶Constraints
![Page 29: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/29.jpg)
Constraint-based likelihood
• : overlap term • : order term
![Page 30: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/30.jpg)
• : boundary term
• subject relation term
Constraint-based likelihood
𝑟 𝑖 𝑟 𝑗
𝐯 ij
Interaction “Talk”
shot “medium”
![Page 31: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/31.jpg)
Results and Evaluation
![Page 32: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/32.jpg)
Comparison to Heuristic Method for Balloon Placement
Our approachHeuristic [Chun et al. 2006]
![Page 33: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/33.jpg)
Comparison to Heuristic Method for Balloon Placement
Our approachHeuristic [Chun et al. 2006]
![Page 34: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/34.jpg)
Comparison to Manual Method
• User study: How well does our approach facilitate manga composition, as compared to existing manual tool?
Participant preference voting Time for one composition
![Page 35: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/35.jpg)
Eye-tracking Experiment and Analysis
• Is our approach effective in directing viewer attention, as compared with the alternatives?
• To measure fixation consistency across different viewers
• Similarity metrics: 1. Inlier percent
2. Root Mean Squared Distance (RMSD)
![Page 36: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/36.jpg)
Eye-tracking Experiment and Analysis
1. Inlier percent
Viewer A Saliency Map
Viewer B
InliersClassification
![Page 37: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/37.jpg)
RMSD,
2. RMSD
Viewer A Viewer B
Eye-tracking Experiment and Analysis
![Page 38: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/38.jpg)
Eye-tracking Experiment and Analysis
![Page 39: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/39.jpg)
Comparison to Existing Manga Pages
Original Synthesized© Tsugumi Ohba, Takeshi Obata / Shueisha Inc.
![Page 40: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/40.jpg)
Summary
• A novel model for representing dependency among the artist’s guiding path, composition and viewer attention
• A novel approach for joint composition of subjects and balloons
• Enable easy and quick creation of attention-directing compositions
![Page 41: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/41.jpg)
Limitations & Future Work
• Consider visual appearance of manga elements
• Extend to other graphic design tasks
• Understand readers’ behaviors
![Page 42: Attention-Directing Composition of Ying Cao Rynson W.H. Lau Elements Look Over Here: Antoni B. Chan City University of Hong Kong](https://reader030.vdocuments.site/reader030/viewer/2022032805/56649ee65503460f94bf63d4/html5/thumbnails/42.jpg)
Thanks