Upload File

diving deep into sentiment: understanding fine-tuned cnns for visual sentiment prediction

  • Home
  • Technology
  • Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction
59
DIVING DEEP INTO SENTIMENT: UNDERSTANDING FINE-TUNED CNNS FOR VISUAL SENTIMENT PREDICTION Víctor Campos Xavier Giró Amaia Salvador Brendan Jou

Upload: universitat-politecnica-de-catalunya

Post on 21-Jan-2018

1.057 views

Category:

Technology


0 download

Report

TRANSCRIPT

Page 1: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

DIVING DEEP INTO SENTIMENT: UNDERSTANDING FINE-TUNED CNNS FOR VISUAL SENTIMENT PREDICTION

Víctor Campos Xavier Giró Amaia Salvador Brendan Jou

Page 2: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Outline

1. Introduction2. Related work3. Methodology and results4. Conclusions5. Future work

2

Page 3: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Introduction: motivation

3

Page 4: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

4

Introduction: problem definition▷ What? ▷ How?

Page 5: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

▷ What? Predict the sentiment that an image provokes to a human▷ How?

5

Introduction: problem definition

Page 6: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

▷ What? Predict the sentiment that an image provokes to a human▷ How?

6

Introduction: problem definition

Page 7: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

▷ What? Predict the sentiment that an image provokes to a human▷ How? Using Convolutional Neural Networks (CNNs)

7

CNN

Introduction: problem definition

Page 8: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

8

CNN

Introduction: example

Page 9: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

9

CNN

Introduction: example

Page 10: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Outline

1. Introduction2. Related work3. Methodology and results4. Conclusions5. Future work

10

Page 11: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Related work: low-level descriptors

11

Siersdorfer, S., Minack, E., Deng, F., & Hare, J. (2010, October). Analyzing and predicting sentiment of images on the social web. In Proceedings of the international conference on Multimedia (pp. 715-718). ACM.

Machajdik, J., & Hanbury, A. (2010, October). Affective image classification using features inspired by psychology and art theory. In Proceedings of the international conference on Multimedia (pp. 83-92). ACM.

Page 12: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

12

Borth, D., Ji, R., Chen, T., Breuel, T., & Chang, S. F. (2013, October). Large-scale visual sentiment ontology and detectors using adjective noun pairs. In Proceedings of the 21st ACM international conference on Multimedia (pp. 223-232). ACM.

Related work: SentiBank

Page 13: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Related work: CNNs for sentiment prediction

13

You, Q., Luo, J., Jin, H., & Yang, J. (2015). Robust image sentiment analysis using progressively trained and domain transferred deep networks. In The Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI).

Page 14: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Outline

1. Introduction2. Related work3. Methodology and results

a. Convolutional Neural Networksb. Datasetsc. Experimental setup and results

4. Conclusions5. Future work

14

Page 15: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Convolutional Neural Networks

15

Krizhevsky, A.; Sutskever, I. & Hinton, G. E.: ImageNet Classification with Deep Convolutional Neural Networks. In: NIPS., 2012

Page 16: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Outline

1. Introduction2. Related work3. Methodology and results

a. Convolutional Neural Networksb. Datasetsc. Experimental setup and results

4. Conclusions5. Future work

16

Page 17: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Datasets

17

Flickr Twitter

Authors Borth et al. (2013) You et al. (2015)

Size ~500k 1269

Annotation method Textual tags5 human

annotators

Page 18: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Datasets

18

Size

Flickrdataset

Quality of the annotations

Twitter5-agreedataset

Page 19: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Datasets

19

Size

Flickrdataset

Quality of the annotations

Twitter5-agreedataset

Page 20: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Outline

1. Introduction2. Related work3. Methodology and results

a. Convolutional Neural Networksb. Datasetsc. Experimental setup and results

4. Conclusions5. Future work

20

Page 21: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

21

ARCHITECTURECaffeNet

Experimental setup: CNN

Page 22: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

22

ARCHITECTURECaffeNet

SOFTWARE[Jia’14]

Experimental setup: CNN

Page 23: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Experimental setup: CNN

23

Pre-trainedModel

ARCHITECTURECaffeNet

SOFTWARE[Jia’14]

DATASET[Deng’09]

Page 24: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Experimental setup: CNN

24

Model

ARCHITECTURECaffeNet

SOFTWARE[Jia’14]

DATASET[Deng’09]

DATASET[You’15]

Twitter 5-agree

+Fine-tuning

Pre-training

Page 25: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Experimental setup: outline

1. Fine-tuning CaffeNet2. Layer by layer analysis3. Layer ablation4. Layer addition

25

Page 26: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Fine-tuning CaffeNet

26

Page 27: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Fine-tuning CaffeNet

27

Page 28: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Fine-tuning CaffeNet

28

Page 29: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Fine-tuning CaffeNet

29

Pre-trainedmodel

Page 30: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Fine-tuning CaffeNet

30

Page 31: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Experimental setup: outline

1. Fine-tuning CaffeNet2. Layer by layer analysis3. Layer ablation4. Layer addition

31

Page 32: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Layer by layer analysis

32

Page 33: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Layer by layer analysis

33

Page 34: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Experimental setup: outline

1. Fine-tuning CaffeNet2. Layer by layer analysis3. Layer ablation4. Layer addition

34

Page 35: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Layer ablation

35

Raw ablation

2-neuron on top

Page 36: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Layer ablation

36

Page 37: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Layer ablation

37

Page 38: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Layer ablation

38

~16Mparams(~25%)

Page 39: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Experimental setup: outline

1. Fine-tuning CaffeNet2. Layer by layer analysis3. Layer ablation4. Layer addition

39

Page 40: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Layer addition

40

FC8: semantic information

Page 41: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Layer addition

41

FC8: semantic information

Page 42: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Outline

1. Introduction2. Related work3. Methodology and results4. Conclusions5. Future work

42

Page 43: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Conclusions

43

Pre-trainedmodel

Page 44: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

44

CNN

Conclusions

Page 45: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Conclusions

45

Page 46: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Outline

1. Introduction2. Related work3. Methodology and results4. Conclusions5. Future work

46

Page 47: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Future work

47

Size

Flickrdataset

Quality of the annotations

Twitterdataset

Page 48: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Future work

48

Size

Flickrdataset

Quality of the annotations

Twitterdataset

MVSOdataset

(†) B. Jou*, T. Chen*, N. Pappas*, M. Redi*, M. Topkara*, and S.-F. Chang. Visual Affect Around the World: A Large-scale Multilingual Visual Sentiment Ontology. ACM Int'l Conference on Multimedia (MM), 2015.

†

Page 49: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

49

Model

ARCHITECTURECaffeNet

SOFTWARE[Jia’14]

DATASETMVSO [Jou’15]

Future work

Page 50: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Acknowledgements

50

Financial supportTechnical support

Albert Gil Josep Pujal

Page 51: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction
Page 52: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction
Page 53: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Data augmentation (oversampling)

53

CNN

Page 54: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Data augmentation (oversampling)

54

CNN

Page 55: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Data augmentation (oversampling)

55

CNN

Page 56: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Data augmentation (oversampling)

56

CNN

Page 57: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Data augmentation (oversampling)

57

CNN

Page 58: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Data augmentation (oversampling)

58

CNN

Page 59: Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

Data augmentation (oversampling)

59

CNN


cs224n-2018-lecture12-Transformers and CNNs

Cnns reiselivsdagene arena nord

CNNS Brochure

ELASTIC: Improving CNNs with Dynamic Scaling Policies

Tuned In or Tuned Out to Your Client?

CNNs: from the Basics to Recent Advances

Artificial Neural Network Methods Applied to Sentiment ... · 2.2 Convolutional Neural Network 15 2.2.1 Architecture 16 2.2.2 CNNs for NLP 18 3 fine-grained contextual predictions

Convolutional Neural Networks (CNNs) Structured Prediction …mlcenter.postech.ac.kr/files/attach/workshop_fall_2015... ·  · 2015-10-15Structured Prediction using CNNs By Prof

apr 2014maps sentiment 0 o O O o o 00 0 0 00 sentiment opage maps sentiment opage maps maps sentiment sentiment ©page maps sentiment opage maps maps i sent sentiment

Are Transformers More Robust Than CNNs?

Do CNNs solve the CT inverse problem?

COSC579: CNNs in Computer Visionjeremybolton.georgetown.domains/courses/cv/17_COSC579_CNNsA… · COSC579: CNNs in Computer Vision Jeremy Bolton, PhD Assistant Teaching Professor

sept 2007 CardMaps · 2017. 7. 20. · you are very pager-naps pagernaps vaeœl\ues sentiment sentiment sentiment sentiment se co hello sent Th ema s sentiment sentiment rñdÞS sentime

VisualBackProp: efficient visualization of CNNs · VisualBackProp: efficient visualization of CNNs Mariusz Bojarski NVIDIA Corp. [email protected] Anna Choromanska ECE NYU …

HIERARCHICAL TEXT CLASSIFICATION USING CNNS WITH …

Robot Vision with CNNs: a Practical Example

Stay Tuned! Stay tuned! Stay Tuned! - Montana State University

Tuned Surfaces

december 2009 CardMaps · sentiment © pagemaps sentiment © pagemaps mops sentiment pagemaps Q pager-naps © pacem page Irôqps © pacema © pager-naps sentiment sentiment sentiment

3D G-CNNs for Pulmonary Nodule Detection

Poster_Diagnosis of Heart Disease via CNNs

Reinforcement Learning: CNNs and Deep Q Learning

New Learning Discriminative Sentiment Representation from … · 2019. 7. 16. · Learning Discriminative Sentiment Representation from Strongly- and Weakly-Supervised CNNs 000:3

CNNS årsmøte 2012 presentasjon

Introduction to CNNs and RNNs with PyTorchIntroduction to CNNs and RNNs with PyTorch Presented by: Adam Balint Email: [email protected]

CS11-747 Neural Networks for NLP Convolutional …...Outline 1. Feature Combinations 2. CNNs and Key Concepts 3. Case Study on Sentiment Classification 4. CNN Variants and Applications

A Bidirectional Tuned Liquid Column Damper for … · A Bidirectional Tuned Liquid Column Damper for ... and orthogonal tuned liquid column dampers ... Tuned liquid column damper,

Convolutional Neural Networks (CNNs) and Recurrent Neural

Sentiment Analysis and Opinion Miningliub/FBS/Sentiment...Sentiment Analysis and Opinion Mining 7 CHAPTER 1 Sentiment Analysis: A Fascinating Problem Sentiment analysis, also called

Auro tripathy - Localizing with CNNs

appropriate deformation stability in CNNs

Learning Lightweight Lane Detection CNNs by Self Attention ...personal.ie.cuhk.edu.hk/~ccloy/files/iccv_2019_sad.pdf · Learning Lightweight Lane Detection CNNs by Self Attention

Eupae Tuned

3D CNNs With Adaptive Temporal Feature Resolutions

E(2)-Equivariant Steerable CNNs