1

Emotion Classification Using Massive Examples Extracted from the Web

Ryoko TOKUHISA, Kentaro INUI, Yuji MATSUMOTO

COLING’2008

Date: 2009-02-19

2

Outline

Introduction Emotion Classification Experiments Conclusion

3

Introduction

Goal: proposing a data-oriented method for inferring the emotion of a speaker conversing with a dialog system.

Method Obtaining a huge collection of emotion-provoking event

instances from the web. Decomposing the emotion classification task into two sub-steps:

Coarse-grained: sentiment polarity classification. Fine-grained: emotion classification.

4

The Basic Idea

Classification problem: a given input sentence is to be classified either into 10 emotion classes or neutral class.

Basic idea: learning what emotion is typically provoked in what situation (emotion-provoking event). Ex.: “I traveled for to get to the shop, but it was

closed” -> disappointing.

5

6

Building an EP Corpus Taking ten emotions (happiness, fear…) as emotion

classes. Building a handcrafted lexicon of emotion words

(349 emotion words) classified into the ten emotions.

7

Building an EP Corpus cont. Using 349 emotion words to find sentences in the Web corpus that poss

ibly contain emotion-provoking events. A subordinate clause was extracted as an emotion-provoking event inst

ance if: It was subordinated to a matrix clause headed by an emotion word. The relation between the subordinate and matrix clauses is marked by o

ne of the eight connectives ( ので , から , ため , て , のは , のが , ことは , ことが ).

Ex.: “I was disappointed that is suddenly started raining.” the subordinate: it suddenly started raining. connective: that.

8

Building an EP Corpus cont. Apply above emotion lexicons and patterns to collection

1.3 million events.

The evaluation of EP corpus by annotators.

9

Sentiment Polarity Classification Neutral sentences are not the majority in real Web

texts. 1000 sentences randomly sampled from the web:

Using the positive and negative examples stored in emotion-provoking corpus.

Assuming the sentence to be neutral if the output of the model is near the decision boundary.

10

Sentiment Polarity Classification cont. SVMs and the features (n-grams and the sentimen

t polarity of the word themselves).

where, the sentiment dictionary (1880 positive words and 2490 negative words) from 50 thousand most frequent words sampled from the Web.

11

Emotion Classification

Applying the KNN (k-nearest-neighbor) approach by using the EP corpus.

Similarity measure: using cosine similarity between bag-of-words vectors (Instance and EP)

||||

)(

EPI

EPII, EPsim

12

Experiment for Sentiment Classification Two test sets:

TestSet1: 31 positive utterances, 34 negative utterances, and 25 neutral utterances.

TestSet2: 1140 samples (judged Correct) are 491 positives, 649 negatives sentences and additional 501 neutral sentences.

Testing classification in both two-class and three-class setting.

Metric: F-measure

13

14

Experiment for Emotion Classification Three test sets

TestSet1 (2p, best)

TestSet1 (1p, acceptable)

TestSet2: using the results of their judgments on the correctness.

15

Baseline vs. KNN

Baseline (Pointwise Mutual Information, PMI)

where ei ∈ {angry, disgust, fear, joy, sadness, surprise,…}

cwj: each content word. Emotion class decision:

KNN: 1-NN, 3-NN and 10-NN. One step: retrieve top-k examples from the EP corpus. Two step: retrieve top-k examples from the correspon

ding sentiment pool.

)()(

)( )(

cwhitsehits

e, cwhitse, cwPMI

j jii , cwePMIEscore )( )(

16

17

Conclusion

Decomposing the emotion classification task into two sub-steps.

Word n-gram features alone are more or less sufficient to classify sentence when a very large amount of training data is available.

Two-step classification was effective for fine-grained emotion classification and outperform baseline model.