document-level sentiment inference with social, faction, and...

Document-level Sentiment Inference with Social, Faction, and Discourse Context

Eunsol Choi, Hannah Rashkin, Luke Zettlemoyer, Yejin Choi

ACL 2016

1

2

Targeted Sentiment InferenceExtract sentiment relation between real-world entities in news articles.

Russia Belarus

positive? negative? neutral?

Opinion Holder Opinion Target

(Stoyanov and Claire 11,Yang and Cardie 14, Deng and Wiebe 14)

Russia criticizes Belarus

This work: Document-level Sentiment Inference

3

NegativePositive

Factual Tie

Russia criticizes Belarus

Input: News article Output: Sentiment Graph

4

• Russia criticizes Belarus for … • The speaker of the Russian parliament

Friday criticized Belarus… • Saakhashvili announced on Belorussian television that he did not understand Russia’s claim

Belarus

Challenge 1: Read the entire story

Russia

positive? negative? neutral?

5

• Entities form factions inside the document.

• Subset of sentiment relations can only inferred via relationships among

entities.

Challenge 2: Inference across entities

Tim Cook

C.E.O

Apple

Katy PerryRihanna

Taylor Swift

Elie Goulding

Miley Cyrus

Challenge 3: Sentiment Beyond Sentence Boundaries

6

Russia heat, smog trigger health problems..……….………………………………… “We never care to work with a future perspective in mind,” Alexei Skripkov of the Federal Agency said. “It’s a big systemic mistake.”

Alexei Skripkov Russia

This work: Document-level Sentiment Inference

7

NegativePositive

Factual Tie

Russia criticizes Belarus

Input: News article Output: Sentiment Graph

8

Evidence: Explicit Sentiment Textual Cues

…….….Belarus for permitting Georgian President Mikheil Saakhashvili to appear on its television.…….……. Saakhashvili announced Thursday that he did not understand Russia’s claims..…….…….……s with Ge

did not understand

permit

criticize

NegativePositive

9

NegativePositiveFactionEntities in the same faction shares opinions

Homophily (Lazarsfeld and Merton, 1954)

Evidence:Sentiment Inference Through Factions

president

10

NegativePositiveFactionEnemy of an enemy is a friend.

Social Balance Theory (Heider, 1946)

Evidence:Sentiment Inference Through Relations

This work: Document-level Sentiment Inference

11

NegativePositive

Factual Tie

Russia criticizes Belarus

Input: News article Output: Sentiment Graph

Related Work

Opinion argument

Opinion attribute

Opinion expression

Chavez faced America 's hostility

SubjectiveSourceTarget

IS-FROM

IS-ABOUT

Opinion Frame

Trigger: “hostility” Polarity: negativeIntensity: highSource: “America”Target: “Chavez”

12

Corpus-level Approach (KBP)2013/4 Sentiment Task

Conan O’Brien is positive towards

David Litterman, South Korea, Pokemon

Query Entity Attitude

Answer

Fine-grained Approach (MPQA)(Deng and Wiebe 15)

Related Work

Opinion argument

Opinion attribute

Opinion expression

Chavez faced America 's hostility

SubjectiveSourceTarget

IS-FROM

IS-ABOUT

Opinion Frame

Trigger: “hostility” Polarity: negativeIntensity: highSource: “America”Target: “Chavez”

13

Corpus-level Approach (KBP)2013/4 Sentiment Task

Conan O’Brien is positive towards

David Litterman, South Korea, Pokemon

Query Entity Attitude

Answer

Fine-grained Approach (MPQA)(Deng and Wiebe 15)

This Work:

• Document-level approach• Considers all named entities• Focuses on entity-entity interactions

Dataset

14

• Document-level sentiment data collection

• Dataset Statistics

Document Annotations

15

Evaluation Dataset Collection

Evaluation Dataset Collection

Mark the relationship between the pair as one of below.

16

Positive Not Negative

Unbiased Not Positive

Negative

Evaluation Dataset Collection

Mark the relationship between the pair as one of below.

17

Positive Not Negative

Unbiased Not Positive

Negative

When sentiment is not explicitly stated but can be inferred, mark it as Inferred.

Document Output

18

Evaluation Dataset Collection

Document Output

19

Evaluation Dataset Collection

•Documents drawn from previous datasets (KBP, MPQA).

•All named entity pairs beyond sentence boundaries.

•Considers the first 15 sentences of document.

•Encouraged to capture implicit sentiment.

Dataset Statistics

20

KBP

MPQA

# of Labeled Entity Pairs Per Doc

0 30 60 90 120

11.6

5

91

44.7

9.5

11.6

Positive Unbiased Negative

Same polarity

Annotator Agreement (Cohen’s Kappa) 0.71

# Docs Avg. # of entities

MPQA 54 10.6

KBP 154 7.9

Total 15,185 entity pair labels

Data Characteristics

• High overlap (about 90%) with KBP and MPQA

• Denser (10x more) entity pair annotations than KBP/MPQA

• Cover all named entity pairs

• Capture inferred sentiment

21

Inferred Sentiment

Positive Negative

Inferred Ratio 70% 58%

The FINRA announced the fine, saying Goldman lacked adequate procedures to ensure the required disclosure.

FINRA Goldman

22

Sentiment Beyond Sentence Boundaries

~25% of entity pairs with sentiment occurs from pairs not appear in the same sentence.

23

Russia heat, smog trigger health problems..……….………………………………… “We never work with a future perspective,” Alexei Skripkov of the Federal Agency said. “It’s a big systemic mistake.”

Alexei Skripkov Russia

Model

24

• Document-level ILP model framework

• Soft constraints capturing entity-entity interactions

• Individual entity pair model

Document level ILP model

25

Objective = + + +

NegativePositive

Factual Tie

Document level ILP model

NegativePositive

Factual Tie

26

Objective = + + +

Global soft constraints capturing entity-entity interactions

Document level ILP model

27

Objective = + + +

• Russia criticizes Belarus for … • The speaker of the Russian parliament

Friday criticized Belarus… • Saakhashvili announced on Belorussian television that he did not understand Russia’s claim

Belarus Russia

Opinion Holder Opinion Target

positive? negative? neutral?

Individual entity pair sentiment classifier model scores

Entities in the same faction shares opinion, and is positive toward each other (Lazarsfeld and Merton, 1954)

Inference from Faction Relation

28

Saakhashvili Georgia

president of

Objective = + + +

Entities in the same faction shares opinion, and is positive toward each other (Lazarsfeld and Merton, 1954)

Inference from Faction Relation

29

Saakhashvili Georgia

president of

Objective = + + +

Russia

Entities in the same faction shares opinion, and is positive toward each other (Lazarsfeld and Merton, 1954)

Inference from Faction Relation

30

Saakhashvili Georgia

president of

Objective = + + +

Russia

Entities in the same faction shares opinion, and is positive toward each other (Lazarsfeld and Merton, 1954)

Inference from Faction Relation

31

Saakhashvili Georgia

president of

Objective = + + +

Russia

Inference from Faction Relation

32

Objective = + + +

Faction Detector

Heuristics to decide whether an entity belongs to the other.

• modifier, compound, possessive or appositive on dependency path

33

Saakhfshvilli

Georgiapresident of

Faction Detector

Taylor Swift

SelenaGomez

’s friend

34

Theresa May

Andrea Leadso

m

’s rival Gryzlov Russiaspeaker of

Good Examples Errors

On small annotated study, ~30% recall, ~60% precision

Inference from Sentiment Relation

35

Russia

Georgia

Belarus

Social Balance Theory(Heider, 1946): Models balance or imbalance of sentiment relation in triadic relations.

Objective = + + +

Inference from Sentiment Relation

36

Russia

Georgia

Belarus

Social Balance Theory(Heider, 1946): Models balance or imbalance of sentiment relation in triadic relations.

Objective = + + +

Russia

Georgia

Belarus

Inference from Sentiment Relation

37

Objective = + + +

38

Reciprocity (Gouldner, 1960): Opinions are often reciprocal.

Inference from Sentiment Relation

Russia Georgia Belarus

Objective = + + +

39

Inference from Sentiment Relation

Russia Georgia Belarus

Objective = + + +

Reciprocity (Gouldner, 1960): Opinions are often reciprocal.

40

Inference from Sentiment Relation

Objective = + + +

Reciprocity (Gouldner, 1960): Opinions are often reciprocal.

Data shows that these constraints are often satisfied.

5

20

35

50

65

80

Faction Balance Reciprocity

Random Data

41

• Incorporates the scores from a pairwise classifier model

42

Objective = + + +

• Factors from linear SVM classifier model

Individual Entity Pair Model

• Document Features

• Dependency Path Features

• Quotation Features

• Russia criticizes Belarus for … • The speaker of the Russian parliament

Friday criticized Belarus… • Saakhashvili announced on Belorussian television that he did not understand Russia’s claim

Russia Belarus

43

Individual Entity Pair Model

Document Features• Capture the entity salience.

• Do they appear in headline?• Do they occur together frequently?

44

The most frequently mentioned entity is 3.4 times more likely to have sentiment.

Dependency Path Features• The polarity of path by MPQA sentiment lexicon (Wilson et al 05)

• Path containing [dobj, rev_subj]

45

Skah Norway

Quotation Features

46

Russia heat, smog trigger health problems..……….………………………………… “We never care to work with a future perspective in mind,” Alexei Skripkov of the Federal Agency said. “It’s a big systemic mistake.”

Alexei Skripkov Russia

• The polarity of quotes by MPQA sentiment lexicon

Evaluation

47

Experimental Set-up

• Precision, Recall, and F1 score

• Positive and negative label sets

• Half as development set, half as a test set

48

Comparison Systems• Random (following the data distribution of the dev set)

• Sentence-level RNN classifier:

• Sentiment model on movie domain (Socher 2013)

• Sentiment labels from sentences of the pair co-occurring

• Pairwise classifier (without global inference)

49

Results (KBP)F1

Sco

re

0

10

20

30

40

Positive Negative

Random Sentence Pairwise Global

50

Ablation StudyF1

Sco

re

20

25

30

35

40

Base +Reciprocity +Balance Theory +Faction

Positive Negative

51

Contributions

• Document-level sentiment inference among pairs of entities

• Intuitions from social balance theory and reciprocity

• Models factual relationship affecting sentiment relation

52

Future work• Incorporating additional types of factual relationships

• Refine global constraints based on entity types

53

Data is available!

http://homes.cs.washington.edu/~eunsol/project_page/acl16/

Thanks!Questions?

54