machine learning with annotator rationales to reduce annotation cost omar f. zaidan jason eisner...

Machine Learning with Annotator Rationales to Reduce Annotation Cost

Omar F. ZaidanJason Eisner

Christine D. PiatkoJohns Hopkins University

NIPS Workshop on Cost-Sensitive LearningFriday, Dec. 12, 2008

cs.jhu.edu{ |ozaidan cpiatkojason @| }

http://www.cs.jhu.edu/

mailto:[email protected]?subject=Rationales%20Question



http://www.google.com/

Zaidan, Eisner, Piatko – Annotator Rationales

Training Docs

Per

form

ance

x

x

x

xx

. . .

Supervised Learning

Annotated

Supervised Learning


Training Docs

Per

form

ance

x

x

x

xx

. . .

Supervised Learning


Richer Annotation

Training Docs

Per

form

ance

x

x

x

xx

. . .

x

xx

xx

. . .


Richer Annotation?

• Usually, an annotator indicates what the correct answer is.

• We propose the annotator also indicate why.

• Why is a related task (cf. multi-task learning), so its annotations may help learn what.– In fact, “why” is especially appropriate for this.– It means “tell me something that will help me learn what.”

• Both tasks are on the same training example.– So annotator can attempt both tasks at the same time,

which is cost-efficient.


Armageddon

This disaster flick is a disaster alright. Directed by Tony Scott (Top Gun), it's the story of an asteroid the size of Texas caught on a collision course with Earth. After a great opening, in which an American spaceship, plus NYC, are completely destroyed by a comet shower, NASA detects said asteroid and go into a frenzy. They hire the world's best oil driller (Bruce Willis), and send him and his crew up into space to fix our global problem.

The action scenes are over the top and too ludicrous for words. So much so, I had to sigh and hit my head with my notebook a couple of times. Also, to see a wonderful actor like Billy Bob Thornton in a film like this is a waste of his talents. The only real reason for making this film was to somehow out-perform Deep Impact. Bottom line is, Armageddon is a failure.

“Hey annotator,

is this review

positive or

negative?”

Useful!

Annotators are Underutilized?

Annotator says:

(y = –1)

Dataset from Pang & Lee (2004)

1000 pos & 1000 neg reviewse.g., vs.

Used their features too(bag of words – 18K features)


الحساب يوم

. الفيلم، كارثي فيلم بحق هو الكارثي الفيلم هذا ( ) ، جن توب فيلم مخرج سكوت توني إخراج منفي تكساس والية بحجم كويكب قصة يروي . مشهد بعد األرضية بالكرة لالصطدام طريقهأمريكية، فضاء سفينة فيه تدمر رائع، افتتاحيالكويكب ناسا تكتشف نيويورك، مدينة إلى إضافة . منقب بأفضل فيستعينون جنونها ويجن الذكر آنف ( ) ويرسلونه ، ويليس بروس العالم في النفط عنالعالمية مشكلتنا إلصالح الفضاء إلى وطاقمه

هذه.

وصف ويصعب بها مبالغ واآلكشن اإلثارة مشاهدكلمات بأي فيه ،سخفها نفسي وجدت حد إلى

. و مرات بضع بمدونتي جبهتي وضربت بل أتنهد،في ثورنتون بوب بيلي مثل ممثل رؤية فإن أيضا . الوحيد السبب لمواهبه مضيعة بحق هو كهذا فيلمفيلم على التفوق محاولة هو الفيلم هذا لصنع . يوم األمر، خالصة ما بطريقة امباكت ديب

. فاشل فيلم الحساب

Annotators are Underutilized!“Hey annotator,

is this review

positive or

negative?”

Useful??

Annotator says:

(y = –1)


الحساب يوم

. الفيلم، كارثي فيلم بحق هو الكارثي الفيلم هذا ( ) ، جن توب فيلم مخرج سكوت توني إخراج منفي تكساس والية بحجم كويكب قصة يروي . مشهد بعد األرضية بالكرة لالصطدام طريقهأمريكية، فضاء سفينة فيه تدمر رائع، افتتاحيالكويكب ناسا تكتشف نيويورك، مدينة إلى إضافة . منقب بأفضل فيستعينون جنونها ويجن الذكر آنف ( ) ويرسلونه ، ويليس بروس العالم في النفط عنالعالمية مشكلتنا إلصالح الفضاء إلى وطاقمه

هذه.

وصف ويصعب بها مبالغ واآلكشن اإلثارة مشاهدكلمات بأي فيه ،سخفها نفسي وجدت حد إلى

. و مرات بضع بمدونتي جبهتي وضربت بل أتنهد،في ثورنتون بوب بيلي مثل ممثل رؤية فإن أيضا . الوحيد السبب لمواهبه مضيعة بحق هو كهذا فيلمفيلم على التفوق محاولة هو الفيلم هذا لصنع . يوم األمر، خالصة ما بطريقة امباكت ديب

. فاشل فيلم الحساب


is this review

positive or

negative?”

Useful!

Annotator says:

(y = –1)


Armageddon




is this review

positive or

negative?”

Useful!

Annotator says:

(y = –1)

Our annotation UI:Boldface the rationales in an HTML editor.

(For other tasks, would be more complicated.) (cf. Peekaboom game (von Ahn et al. 2006) )


“Annotators are underutilized. Let’s ask them to do more than just annotate class.”

Related Work• Let’s ask annotator to identify relevant features.

– Haghighi & Klein (2006), Raghavan et al. (2006), Druck et al. (2008).

• But should annotators really look at feature set?– Need a human-readable notation for describing features.

– Features could be hard for annotators to understand.– We might want different features in the future.

apple apple- noun

Apple shares up 3%


. . .. . .

Non-annotated documents


. . .. . .

Saving Private Ryan

War became a reality to me after seeing Saving Private Ryan. Steve Spielberg goes beyond reality with his latest production. Keep the kids home as the R rating is for Reality. Tom Hanks is stunning as Capt John Miller, set out in France during WW II to rescue and return home a soldier, Private Ryan (Matt Damon) who lost three brothers in the war. Spielberg takes us inside the heads of these individuals as they face death during the horrific battle scenes. Private Ryan is not for everyone, but I felt the time was right for a movie like this to be made. The movie reminds us of the sacrifices made by our fighting men and women. For this I thank them and for Steve Spielberg for making a movie that I will never forget. And I’m sure the Academy will not forget Tom Hanks come April, as another well deserved Oscar with be in Tom's possession.


. . .. . .


Kundun

Martin Scorsese's Kundun has been criticized for its lack of narrative structure. Personally, I don't think it needs one: it works perfectly well as a study of Tibetan Buddhist culture and communist China. Scorsese views the Dalai Lama the way many Tibetans probably do, as a larger-than-life symbol of Buddhist spirituality and political leadership: the only glimpses into his head come from several dream sequences, but his portrayal is appropriate for a film that concentrates on the political and spiritual. The set design and cinematography were absolutely outstanding. Scorsese gets carried away with the spectacle, and it helps to augment the cultural contrast when the Dalai Lama travels to China to meet Chairman Mao. Kundun does not succumb to the temptation to start shouting slogans, delivering its message in an artistically interesting way without being overly manipulative.

. . .. . .


Volcano

Volcano starts with Tommy Lee Jones worrying about a small earthquake enough to leave his daughter at home with a baby sitter. There is one small quake then another quake. Then a geologist points out to Tommy that its takes a geologic event to heat millions of gallons of water in 12 hours. A few hours later large amount of ash start to fall. Then...it starts: the volcanic eruption!

I liked this movie...but it was not as great as I hoped. It was still good none the less. It had excellent special effects. The best view was that of the helecopters flying over the streets of volcanos. Also, there were interesting side stories that made the plot more interesting. So...it was good!!

. . .. . .


Volcano

Volcano starts with Tommy Lee Jones worrying about a small earthquake enough to leave his daughter at home with a baby sitter. There is one small quake then another quake. Then a geologist points out to Tommy that its takes a geologic event to heat millions of gallons of water in 12 hours. A few hours later large amount of ash start to fall. Then...it starts: the volcanic eruption!

I liked this movie...but it was not as great as I hoped. It was still good none the less. It had excellent special effects. The best view was that of the helecopters flying over the streets of volcanos. Also, there were interesting side stories that made the plot more interesting. So...it was good!!

. . .. . .

. . .


The Postman

Question: after the disaster that was Waterworld, what the fuck were the execs who gave Costner the money to make another movie thinking??

In this 3 hour advertisement for his new hair weave, Costner plays a nameless drifter who dons a long dead postal employee's uniform (I shit you not) and gradually turns a nuked-out USA into an idealized hippy-dippy society. (The main accomplishment of this brave new world is in re-inventing polyester.) When he's not pointing the camera directly at himself, director Costner does have a nice visual sense, but by the time the second hour rolled around, I was reduced to sitting on my hands to keep from clawing out my own eyes. Mark this one "return to sender".

. . .

. . .


Batman & Robin

I once wrote that Speed 2 was the worst film I've ever reviewed. I didn't know that I'd soon encounter Batman & Robin, the picture least worthy of your attention this summer. As directed by Joel Schumacher, B&R is one long excuse for a Taco Bell promotion.

The plot, which has Mr. Freeze and Poison Ivy planning to take over the world, is weighted down by repetitive asides about the nature of trust, partnership, blah, blah, blah. But morals are not the point of this film – topping each bloated, confusing action scene with the next one is. Only George Clooney comes out on top; he underplays nicely and pretends like he's in a real movie… . . .

. . .


Armageddon



. . .

. . .


Armageddon



. . .

. . .. . .


. . .. . .

Annotated documents

Class and also “rationales”

OK … now what??

<x,y,r>


• In our experiments, we use a linear classifier.

• Classifier is represented by a weight vector .

• Rationales will play a role when learning .

• Improvements solely due to a better-learned .

• At test time:– No change to decision rule. – No new features.– No need for rationales.

Linear Classifier

(vs. no rationales)


• At test time, when presented with document x:

• Where:– : feature vector (18k binary unigram features).– : corresponding weights (i.e. 18k weights).

Positive weights favor y = +1

Negative weights favor y = –1

Linear Classifier

)(f

0)( , 1

0)( , 1 Choose *

xf

xfy


Trees Lounge is the directoral debut from one of my favorite actors, Steve Buscemi. He gave memorable performences in in The Soup, Fargo, and Reservoir Dogs. Now he tries his hand at writing, directing and acting all in the same flick. The movie starts out awfully slow with Tommy (Buscemi) hanging around a local bar the "trees lounge" and him pestering his brother. It's obvious he a loser. But as he says "it's better I'm a loser and know I am, then being a loser and not thinking I am." Well put. The story starts to take off when his uncle dies, and Tommy, not having a job, decides to drive an ice cream truck. Well, the movie starts to pick up with him finding a love interest in a 17 year old girl named Debbie (Chloe Sevigny) and... I liked this movie alot even though it did not reach my expectation. After you've seen him in Fargo and Reservoir Dogs, you know he is capable of a better performence. I think his brother, Michael, did an excellent job for his debut performence. Mr. Buscemi is off to a good career as a director!

! " ( ) , . a acting actors after all alot am an and around as at awfully bar being better brother buscemi but capable career chloe cream debbie debut decides did dies directing director dogs drive even excellent expectation fargo favorite finding flick for from gave girl good hand hanging having he him his i ice i'm in interest is it it's job know liked local loser lounge love memorable michael movie mr my named not now obvious of off old one out pick put reach reservoir same says seen sevigny slow soup starts steve story take the then think thinking this though to tommy trees tries truck uncle up well when with writing year you you've

fand = 1, freach = 1, fterrific= 0, …

x

f (x): 0/1 vector; 18k elements


! " ( ) , . a acting actors after all alot am an and around as at awfully bar being better brother buscemi but capable

career chloe cream debbie .

debut decides did dies directing director dogs drive even excellent expectation fargo favorite

finding flick for from gave girl .

good hand hanging having he him his i ice i'm in interest is it it's job know liked local loser lounge love

memorable michael movie .

mr my named not now obvious of off old one out pick put reach reservoir same says seen sevigny slow soup

starts steve story take .

the then think thinking this though to tommy trees tries truck uncle up well when with writing year

you you've .

! " ( ) , . a acting actors after all alot am an and around as at awfully bar being better brother buscemi but capable career chloe cream debbie debut decides did dies directing director dogs drive even excellent expectation fargo favorite finding flick for from gave girl good hand hanging having he him his i ice i'm in interest is it it's job know liked local loser lounge love memorable michael movie mr my named not now obvious of off old one out pick put reach reservoir same says seen sevigny slow soup starts steve story take the then think thinking this though to tommy trees tries truck uncle up well when with writing year you you've

abc abc abc abc abc-0.2 -0.1 0.0 +0.1 +0.2

favor y = +1favor y = –1


Saving Private Ryan


How sure was the annotator that this is a positive review?


Saving Private Ryan


If a rationale is masked out, the annotator would not be as sure that this is a positive review.

Intuition: a good model should also be less sure.

How sure was the annotator that this is a positive review?


“Contrast” Examples

Original Example

Contrast ExamplesObtain a contrast by

masking out a rationale

Intuition: a good model should be less sure of a positive classification on contrasts than on the original.

Our work: modified SVM that takes this intuition into account.


Standard SVM

Class +1 examples

Class -1 examples


Class +1 examples

Class -1 examples

Support vectors

Standard SVM

Class +1 examples

Class -1 examples


Standard SVM


w1

1x

Standard SVM2x


w1

1x

Standard SVM2x

:Minimize

2

2

1w

:subject to

1 ixw


12v11vj1v

1 ixw

:Minimize

2

2

1w

:subject to

w1

1x

Incorporating Contrasts

. . . . . .

2x

. . . . . .22v

21vj2v


12v11vj1v

1 ixw

:Minimize

2

2

1w

:subject to

w1

1x


. . . . . .

2x

. . . . . .22v

21vj2v

Original (x)

Contrast (v)


w1

1x


. . . . . .

12v11vj1v

2x

. . . . . .22v

21vj2v

1 ixw

:Minimize

2

2

1w

:subject to

Original (x)

Contrast (v)


w1

1x


. . . . . .

12v11vj1v

2x

. . . . . .22v

21vj2v

1 ixw

:Minimize:subject to

ALSO

2

2

1w

:subject to

iji vwxw

Original (x)

Contrast (v)


1

iji vxw

w1

1x


. . . . . .

12v11vj1v

2x

. . . . . .22v

21vj2v

1 ixw


ALSO

2

2

1w

:subject to

iji vwxw

Original (x)

Contrast (v)


w1

1x

. . . . . .

12v11vj1v

2x

. . . . . .22v

21vj2v

1 ixw 1 ijxw


ALSO

2

2

1w

:subject to

iji vwxw

1

iji vxw

Original (x)

Contrast (v)

def



1 ixw i 1 ijxw


ALSO

2

2

1w

:subject to

w1

1x

Slack Variables

. . . . . .

12v11vj1v

2x

. . . . . .22v

21vj2v

iji vwxw

1

iji vxw

Original (x)

Contrast (v)

i

iC


ij1 ixw i 1 ijxw


ALSO

2

2

1w

:subject to

w1

1x

Slack Variables

. . . . . .

12v11vj1v

2x

. . . . . .22v

21vj2v

iji vwxw

1

iji vxw ij

)1( ij

Original (x)

Contrast (v)

ji

ijcontrastC,

i

iC


) (iy 1 ixw i ij) (iy 1 ijxw

jii

ijcontrasti CC,


ALSO

2

2

1w

:subject to

w1

1x

((Include Negative Examples))

. . . . . .

12v11vj1v

2x

. . . . . .22v

21vj2v

iji vwxw

1

iji vxw ij

)1( ij) (iy

) (iy

Original (x)

Contrast (v)


ij) (iy 1 ijxw


ALSO

2

2

1w

:subject to

w1

1x

The Modified SVM

. . . . . .

12v11vj1v

2x

. . . . . .22v

21vj2v

iji vwxw

1

iji vxw ij

)1( ij) (iy

) (iy 1 ixw i

) (iy

Original (x)

Contrast (v)

jii

ijcontrasti CC,


What this Means in Practice



Standard SVM cares about this margin




Modified SVM cares about both margins




Modified SVM cares about both margins

i.e. a hyperplane w that might reduce standard margin (to help the other margin)


Recap

• Training examples: (x1,y1), (x2,y2), …

• yi has ni rationales: ri1,ri2,…,rin

• xi gives ni contrast examples: vi1,vi2,…,vin

(obtain jth contrast by masking out jth rationale.)

• We extend the SVM to determine best hyperplane subject to:– Constraints for standard margin,

and also– Constraints for original/contrast separating margin.


What this is not• In tasks like digit recognition, one can “generate” more

training data from the existing examples

Class-preserving transformations

?


What this is not

New examples have original class

Automatic (rescale, deslant, etc)

?New examples have other class, or have original class less confidently

Manual (new info via human insight)


Results


0 400 800 1200 1600

Training Set Size (Documents)

93

91

89

87

85

83

81

79

Ac

cu

rac

y (

%)

Baseline

Contrasts Introduced

Standard vs. Modified SVMS

VM

as constrasts:+

Mod

ified

SV

M

Significantly different


0 400 800 1200 1600


Ac

cu

rac

y (

%)

Baseline



Baseline


SV

M

as constrasts:+

Mod

ified

SV

M


Baseline


Rationales Removed

SV

MS

VM

as constrasts:+

Mod

ified

SV

M

Removing rationales

hurts baseline


Baseline


Rationales Removed

SV

MS

VM

as constrasts:+

Mod

ified

SV

M


Baseline


Rationales Removed

Rationales Only (as concatenations)

SV

MS

VM

SV

M

as constrasts:+

Mod

ified

SV

M

Keeping only rationales

hurts performance


Baseline


Rationales Removed


SV

MS

VM

SV

M

as constrasts:+

Mod

ified

SV

M


Baseline


Rationales Removed

Rationales Only (as individual docs)


SV

M

SV

M

SV

MS

VM

as constrasts:+

Mod

ified

SV

M


Baseline


Rationales Removed

Rationales Only (as individual docs)


SV

M

SV

M

SV

MS

VM

as constrasts:+

Mod

ified

SV

M

Pieces to solving classification puzzlecannot be found solely in the rationales


We got better performance, but at what increment to the cost?


• With a fixed budget, better to annotate rationales, or just annotate more documents?

• We did timing studies on 3 annotation tasks.– 50 docs/task given to four annotators

Annotation Time


– T1: given document, annotate the class.– T2: given document and gold standard class,

annotate the rationales.– T3: given document, annotate both the class

and the rationales.

• We found that Time(T3) ≈ 2 x Time(T1)

• Even though on average 8.3 rationales/doc + class!– This task has unusually high why/what ratio– Added cost would presumably be lower for other tasks

• How come it’s so fast?– Annotator already needs to find rationales to determine class.

Extra work is only to make them explicit:

Time(T3) < Time(T1)+Time(T2) by about 20%

Annotation Time


Using Rationales fromsome (and not all) Documents

• To speed up, can we get most of the benefit by using rationales from some training documents instead of all training documents?

Baseline


Explore this space


0 400 800 1200 1600

Training Set Size T (Documents)

93

91

89

87

85

83

81

79

Ac

cu

rac

y (

%)

Baseline



No Rationales

0 400 800 1200 1600


93

91

89

87

85

83

81

79

Ac

cu

rac

y (

%)


0 400 800 1200 1600


93

91

89

87

85

83

81

79

Ac

cu

rac

y (

%)

Rationales from Only 200 Documents

T = 800: Class annotation from 800 documents.

R = 200: Rationales from 200 documents only.


0 400 800 1200 1600


93

91

89

87

85

83

81

79

Ac

cu

rac

y (

%)

Use Rationales from R Documents

Class annotation from T documents.

Rationales from R documents only.


• Observation #1: much of the benefit can be obtained without annotating 100% of the documents– e.g. (0%, 50%, 100%) for T = 800 and T = 1600– Recommendation: If rationales are expensive, still collect some.



• Observation #2: Adding more rationales is often more useful than adding more documents.– E.g., adding rationales provides a fresh benefit even after

conventional annotation flattens out (diminishing returns)– Recommendation: Adaptively decide when to solicit rationales,

using marginal cost/benefitmeasurements.

(Could combine with

active learning.)



Rationales from Documents

• Observation #3: Lazy annotators would likewise be faster– Fewer rationales per doc (instead of fewer docs with rationales)– Keep random fraction per doc similar results to before

some (and not all)Using


Rationales from Documents

• Observation #3: Lazy annotators would likewise be faster– Fewer rationales per doc (instead of fewer docs with rationales)– Keep random fraction per doc similar results to before– A real live lazy annotator may be even better

• Keeps the obvious rationales: disproportionately fast and useful?

some (and not all)Using

0 400 800 1200 1600


Diligent Lazy (simulated)

The two (T=800,R=200) points are comparable (same # of rationales)


Problem with previous method:In general, how to construct feature

vectors for contrast examples?• Not obvious which features to turn off if we

“mask out” these rationale regions of the image.– (Ideally, would impute the missing data: expensive!)

Annotator: “this is a one because there is a lot of empty space here and here”


Why are rationales useful, anyway?

The Annotator

Class labelsRationales

Two kinds of annotation

Both give evidence about same parameters


–1! " ( ) , . a acting actors after all alot am an and around as at awfully bar being better brother buscemi but capable









you you've .

This was learned in a standard way:

choose that models class labels well

(of training data)

(incorrect)

logistic regression – probabilistic equiv. of SVM


choose that models class labels & rationales well

Our work: another way to learn :

(of training data)

! “ ( ) , . a acting actors after all alot am an and around as at awfully bar being better brother buscemi but capable career chloe

cream debbie .

debut decides did dies directing director dogs drive even excellent expectation fargo favorite finding flick for from gave girl .

good hand hanging having he him his I ice i'm in interest is it it's job know liked local loser lounge love memorable michael movie .

mr my named not now obvious of off old one out pick put reach reservoir same says seen sevigny slow soup starts steve story take .

the then think thinking this though to tommy trees tries truck uncle up well when with writing year you you've .


+1


Our work: another way to learn :

(of training data)

! “ ( ) , . a acting actors after all alot am an and around as at awfully bar being better brother buscemi but

capable career chloe cream debbie .


finding flick for from gave girl

good hand hanging having he him his I ice i'm in interest is it it's job know liked local loser lounge love


mr my named not now obvious of off old one out pick put reach reservoir same says seen sevigny slow

soup starts steve story take .

the then think thinking this though to tommy trees tries truck uncle up well when with writing

year you you've .


–1! " ( ) , . a acting actors after all alot am an and around as at awfully bar being better brother buscemi but capable









you you've .

This was learned in a standard way:

(of training data)

choose that models class labels well


+1 ! “ ( ) , . a acting actors after all alot am an and around as at awfully bar being better brother buscemi but









year you you've .

This was learned in a novel way (our algorithm!):


(of training data)


+1 ! “ ( ) , . a acting actors after all alot am an and around as at awfully bar being better brother buscemi but









year you you've .

This was learned in a novel way (our algorithm!):

(of training data)



n

iii xyp

1

),|(maxarg Choose

),,|(

iii xyrp


n

iii xyp

1

),|(maxarg Choose

),,|(

iii xyrp

From a review for “Prince of Egypt” ( y = +1 ):

O O O O O I I I I O

51 weeks into '98 , a champ has emerged .

O O O O I I I I I O

the prince of egypt succeeds where other movies failed .


n

iii xyp

1

),|(maxarg Choose

x

x

),,|(

iii xyrp


O O O O O I I I I O


O O O O I I I I I O



n

iii xyp

1

),|(maxarg Choose

x

x

),,|(

iii xyrp

r

r


O O O O O I I I I O


O O O O I I I I I O



n

iii xyp

1

),|(maxarg Choose

x

x

),,|(

iii xyrp

r

r


O O O O O I I I I O


O O O O I I I I I O


• We encode rationales as a tag sequence.


n

iii xyp

1

),|(maxarg Choose

x

x

),,|(

iii xyrp

r

r


O O O O O I I I I O


O O O O I I I I I O


• We encode rationales as a tag sequence.

• The model should predict tag sequence with some help from . Let’s describe it…

),,|(

xyrp


…r1

x1

r2

x2

rM

xM

r3

x3

…

(Tags)

(Words)

Designing the Model .• Mission: model {I,O} tag sequence with ’s help.

Hmm… has no role here.


• Makes sense: if for some word w, , then annotator is more likely to mark it I.

• How much more likely? Depends on annotator.• Nonlinear function, parameterized by .• is learned jointly with from annotator’s data.

…r1

x1

r2

x2

rM

xM

r3

x3

…

…

(Tags)

(Words)


( )


• Problem with this model: it would like to explain every I tag via high for its word.

• BUT:

( )

… temptation to start shouting slogans, delivering its message in an artistically interesting way without being overly manipulative.

…r1

x1

r2

x2

rM

xM

r3

x3

…

…

(Tags)

(Words)



• So, learn 4 more weights:

( )

… temptation to start shouting slogans, delivering its message in an artistically interesting way without being overly manipulative.

…r1

x1

r2

x2

rM

xM

r3

x3

…

…

(Tags)

(Words)



• Mission: model {I,O} tag sequence with ’s help.

Designing the Model .

• So, learn 4 more weights:

– These model the length and quantity of rationales.

• Better! Now: “w is marked with I either because of high or being around others marked with I.”

• CRF: Used in other labeling tasks too (POS, NER).

…r1

x1

r2

x2

rM

xM

r3

x3

…

…

(Tags)

(Words)

LM CM

( )


• Linear classifier:

• Our work: choose that models all of annotator’s data well: both class labels and rationales.

• Remember, at test time:– No change to decision rule.

– No new features.

– No need for rationales.

Recap

Learned jointly : Standard log-linear model

: CRF predicting I/O tag sequence

Improvement due solely to better-learned .


Results


Discriminative Baseline (SVMs)

Generative Baseline (log-lin.)

Discriminative Rationales

Generative Rationales

0 400 800 1200 1600


95

93

91

89

87

85

83

81

79

Acc

urac

y (%

)




• Remember this?

• Actually, we used an expanded set that includes about 100 “conditional” transition features, e.g.:– How often do you see I-O around punctuation?– How often do you see I-O around syntactic boundaries?

• Extra features model rationales better, but don’t help learn any better than basic feature set.

Fancy -Features

…r1

x1

r2

x2

rM

xM

r3

x3

…

…

(Tags)

(Words)


• is a noisy channel, parameterized by .

• Noisy? Annotators use r to tell us about , but they don’t do it “perfectly”:– They may mark too little (or too much).– They may prefer to start rationales at syntactic

boundaries, and/or end around punctuation.

• To what degree? Remember that we train both models jointly:

• So, learned captures style of a specific annotator.

. Captures Annotator’s Style


• So, learned captures style of a specific annotator.

• “So, perhaps you got lucky with A0 (whose data you used in experiments). Would this work with others?”

. Captures Annotator’s Style

SVM baseline 72.0 72.0 72.0 70.0

“SVM Contrast” method 75.0 73.0 74.0 72.0

Smaller experiments (100 documents only) A0 A3 A4 A5

Log-linear baseline 71.0 73.0 71.0 70.0

Generative method 76.0 76.0 77.0 74.0

Each annotator’s own works best. (So there really were different annotation styles to

learn!)


• 2 baseline methods: SVM, logistic regression.• Improved each by using rationales at training time.

• Our second approach is better, and generalizes:– To other classifiers: incorporate in objective.– To other domains: e.g., model how relevant portions of

image tend to trigger rationales (“channel model”) and what size/shape rationales tend to be (“language model”).

• Doing an annotation project? Collect rationales!– Even a small number could help.– “You may get more bang for the buck!”

Conclusions


On The Internets

• The enriched dataset (and slides and papers):http://cs.jhu.edu/~ozaidan/rationales

Thank you!


Thank you!


Machine Learning with Annotator Rationalesto Reduce Annotation Cost

This talk presented an interesting idea; it does seem to me that rationales would help. I think the authors are on the right track. However the talk was quite long. The animation was OK, I suppose, though I think some people might be put off by it. At any rate, I’d be interested to know more about this research. It should be noted that other ML methods may benefit from this approach even more than SVM’s, if such a learning method emulates the human decision process more than an SVM (e.g. a decision tree). An interesting idea overall though more experimental results are needed to convince me completely.

Thank you!

machine learning with annotator rationales to reduce annotation cost omar f. zaidan jason eisner...

Documents

reviewpositive ornegative

armageddonthis disaster

billy bob thornton

global problem

wonderful actor

size of texas

american spaceship

collision course