Revealing the Hidden Patterns of News Photos: Analysis of Millions of News Photos through GDELT and Deep Learning-based Vision APIs

Haewoon Kwak Jisun An

Qatar Computing Research InstituteHamad Bin Khalifa University

2

3

4

Roles of News Photos

● Influence people’s perception● Enhance reader’s memory● Deliver emotion otherwise hard to be

conveyed

5

6

7

Why was this photo not picked?(source: https://www.donaldjtrump.com)

8

To Understand Messages of Photos

● We need to know○ What are shown in the photos○ How they are portrayed

9

Challenges in News Photo Analysis

● Text mining has been a useful tool for analyzing news text

→ What is the appropriate tool forexamining news photos?

10

Conventional Tool for Photo Analysis

● Manual coding … hard to scale

11

Deep Learning for Image Recognition

12

Object Recognition

13

Emotion Detection

https://www.microsoft.com/cognitive-services/en-us/emotion-api 14

Deep learning enables us to study news photos in large-scale

15

Goal of This Work

● To offer a general understanding of news photos ○ What are shown in the photos?○ How are people portrayed?

■ From the perspective of emotion■ From the perspective of gender

● Case study: Portrayal of politicians

16

● We can crawl photos from news websites and analyze them

● But, setting the deep learning framework and training it take time/money/...

Data Collection

17

18

GDELT Visual GKG (VGKG)

● Collects news articles around the world● Extract photos from the articles● Calls Google Cloud Vision API to analyze

photos

● VGKG is available since 1 Jan 2016 http:

//blog.gdeltproject.org/announcing-the-new-gdelt-visual-global-knowledge-graph-vgkg/

19

Example of VGKG

20160101004500 http://www.bbc.co.uk/news/uk-35205943 http://ichef.bbci.co.uk/news/1024/cpsprodpb/B89F/production/_87436274_87436273.jpg profession<FIELD>0.95780987<FIELD>/m/063km<RECORD>person<FIELD>0.85714287<FIELD>/m/01g317<RECORD>close up<FIELD>0.82379222<FIELD>/m/02cqfm<RECORD>bishop<FIELD>0.78259438<FIELD>/m/01b7b<RECORD>bishop<FIELD>0.71334475<FIELD>/m/027k49j<RECORD>diocesan bishop<FIELD>0.64282793<FIELD>/m/09sgrf<RECORD>auxiliary bishop<FIELD>0.57118613<FIELD>/m/05mx3n<RECORD>clergy<FIELD>0.57113737<FIELD>/m/0db79 -2<FIELD>-2<FIELD>-2<FIELD>-2 0.95443642<FIELD>3.199043<FIELD>12.419704<FIELD>-7.179338<FIELD>0.621747<FIELD>433,215;575,215;575,357;433,357<FIELD>-2<FIELD>-2<FIELD>-2<FIELD>2<FIELD>-2<FIELD>-2<FIELD>-2

20

Date, Document identifier (URL), Image URL, Labels (description, confidence score, unique id), Geographic Landmarks, Logos, Safe Search, Faces (Angle, Emotion, etc.), OCR

(Potential) Limitations of GDELT

● List of news sources is not explicitly announced (also, growing) - coverage bias might exist

● Our work of comparing GDELT with another news dataset will be presented in the poster session

Two Tales of the World: Comparison of Widely Used World News Datasets - GDELT and EventRegistry Haewoon Kwak and Jisun AnICWSM'16: The 10th International Conference on Web and Social Media (poster), 2016 21

Our Dataset - Full

● GKG and VGKG in January 2016● Popularity measured by Alexa.com

22

Our Dataset - 7 Popular News Media

● Top 30 & > 1K records

23

Data Preprocessing

● Keep labels whose confidence score ≥ .8

http://i2.cdn.turner.com/cnnnext/dam/assets/160116174054-kerry-handshake-zarif-large-169.jpg

Person 0.84957772Business 0.59667766

24

What Are Shown in the Photos?Common Objects in News Photos

25

News Topics and Relevant Photos

● News photos should relate with topics of news articles→ Common objects might be different across topics

● CNN has ‘section’ info. in its URLhttp://edition.cnn.com/2016/04/07/travel/japan-best-of-wakayama/index.htmlhttp://edition.cnn.com/2016/05/05/politics/paul-ryan-donald-trump-republican-resistance/index.html 26

Person is the Most Common Object

27

But, in Travel, Person is Uncommon

28

Region-related Sections

29

● Why does this matter?

Western Media and the Third World

● Golan reports that western mass media strengthen the portrayal of the third world by reporting war, poverty, famine, conflicts, violence and conflicts and lead to negative perception (Golan 2008).

30

How CNN deals with MENA region?

31

How Are People Portrayed? From the Perspective of Emotion

32

Classification of Emotions

33https://articulation360.wordpress.com/2011/08/26/emotions-memory-game/

Google API Can Detect 4 Emotions

34https://articulation360.wordpress.com/2011/08/26/emotions-memory-game/

SURPRISE

SORROW ANGER

JOY

Neutral (75%) or Joy (24%)

● Among 11,127 faces (in 7 popular media), 2,740 faces (24.6%) have one of emotions

● Most of them (2,665 faces) express joy

35

Nonverbal & Verbal Communication

● Happy faces accelerate the cognitive processing of positive words and slow down that of negative words (Stenberg, Wiking, and Dahl 1998)

36

We Use Microsoft Face API

● Measures smiling intensity (0.0~1.0)

37

0.998 0.0 (baby)

https://www.microsoft.com/cognitive-services/en-us/face-api

Smile Comes with Positive Text

● Positive correlation between smile intensity and tone (sentiment) of the text

⍴=0.225

38

How Are People Portrayed? From the Perspective of Gender

39

Previous Studies on News Media

1. Men outnumber women2. Men and women are associated with

particular roles3. More women than men were depicted

as happy and calm.

→ We’ll verify this in large-scale

40

Again, We Use Microsoft Face API

41https://www.microsoft.com/cognitive-services/en-us/face-api

● Measures Gender and Age

Unequal Gender Representation

0.5

42

Stereotyping: Women in “Living”

0.5

43

Women Smile More Than Men

44

Younger Women, Older Men

45

Case StudyPortrayal of Politicians

46

Smiling Politicians

● Goodnow (2010) found that Obama smiles more than Clinton in photos in Time magazine

● Smile gives a positive, non-threatening impression to viewers (Goffman 1979)

47

Bias of CNN Toward Sanders?

(Smiling faces / All faces)

* CNN even uses “Sorrow” faces for Sanders

48

Pro-Clinton Media Behave Similarly

49

Summary and Future Work

50

Key Findings

● What are shown in the news photos○ People commonly appear (≥ 40.5% @top500)

● How they are portrayed○ People are neutral (75%) or smiling (24%)○ Gender representation is unequal○ Gender role stereotyping is found○ Women smile more and look younger than men

● Clinton smiles more than Sanders in some media

→We demonstrate the great potential of deep learning for computational journalism

51

Deeper Analysis on Text and Photos

● Headline and photos?● Topic and photos?● Keywords and photos?

52

Building PhotoBiasMeter.org

● Showing the preference of media outlets toward candidates over time

● Challenges○ Modeling complex dimension of

preference - “Smile” is only one dimension

53

@haewoonFull paper is available via

http://arxiv.org/abs/1603.04531

54