exploiting timelines to enhance multi-document summarization

Exploiting Timelines to Enhance Multi-document

SummarizationJun-Ping Ng, Yan Chen, Min-Yen Kan and

Zhoujun Li

National University of SingaporeBeihang University

Cyclone Sidr

2007, JTWC designation: 06B

Cyclone Sidr

2007, JTWC designation: 06B

Image Courtesy: Univ. Wisconsin-Madison

Image Courtesy: Univ. Wisconsin-Madison

“A fierce cyclone

packing extreme

winds and torrential

rain smashed into

Bangladesh’s

southwestern coast

Thursday, …”

“A fierce cyclone

packing extreme

winds and torrential

rain smashed into

Bangladesh’s

southwestern coast

Thursday, …”

24 Jun 2014ACL 2014 - Timelines in

Summarization2


Summarization3

Image Courtesy: US Navy / Wikipedia


“… wiping out

homes and trees in

what officials

described as the

worst storm in

years.”

“… wiping out

homes and trees in

what officials

described as the

worst storm in

years.”


Summarization4

Image Courtesy: US State Department / Wikipedia

Image Courtesy: US State Department / Wikipedia

“More than 100,000

coastal villagers

have been

evacuated before

the cyclone made

landfall.”

“More than 100,000

coastal villagers

have been

evacuated before

the cyclone made

landfall.”



1991 Bangladesh Cyclone

1991 Bangladesh Cyclone

“The storm matched one in 1991 that sparked a tidal wave that killed an estimated 138,000 people, Karmakar told AFP.”

“The storm matched one in 1991 that sparked a tidal wave that killed an estimated 138,000 people, Karmakar told AFP.”


Summarization5


Summarization6

[3] “The storm matched one in 1991 that sparked a tidal wave that killed an estimated 138,000 people, Karmakar told AFP.”


[2] “More than 100,000

coastal villagers have been

evacuated before the

cyclone made landfall.”

[2] “More than 100,000




[1] “A fierce

cyclone packing

extreme winds and

torrential rain

smashed into

Bangladesh’s

southwestern coast

Thursday, wiping

out homes and

trees in what

officials described

as the worst storm

in years.”

[1] “A fierce

cyclone packing

extreme winds and

torrential rain

smashed into

Bangladesh’s

southwestern coast

Thursday, wiping

out homes and

trees in what

officials described

as the worst storm

in years.”


Summarization7


[2] “More than 100,000




[1] “A fierce

cyclone packing

extreme winds and

torrential rain

smashed into

Bangladesh’s

southwestern coast

Thursday, wiping

out homes and

trees in what

officials described

as the worst storm

in years.”

Timelines from Text


Summarization8


[2] “More than 100,000 coastal villagers

have been evacuated before the


[1] “A fierce cyclone packing extreme

winds and torrential rain smashed into

Bangladesh’s southwestern coast

Thursday, wiping out homes and trees

in what officials described as the worst

storm in years.”

Key time spans are summary worthy


Summarization9


[2] “More than 100,000 coastal villagers

have been evacuated before the


[1] “A fierce cyclone packing extreme

winds and torrential rain smashed into

Bangladesh’s southwestern coast

Thursday, wiping out homes and trees

in what officials described as the worst

storm in years.”

Timelines + Summarization

Timelines (per input document)

SummarizationSystem

Summary

Lexical and positional features

Timeline-derived features


Summarization

Timelines + Summarization

Outline• Goal and Motivation• Timeline Generation• Integrating Timelines– In Scoring: (Contextual) Importance,

Density– In Re-ordering: TimeMMR

• Experiments• Discussion


Summarization

Timeline Generation


Summarization

1. Event-Event Temporal Classification


Summarization

(Ng et al., 2013; EMNLP)

2. Event-Timex Temporal Classification


Summarization

(Ng and Kan, 2012; COLING)

3. Timex Normalization

15

“Today” June 6, 2014


Summarization

(HeidelTime; Strötgen and Gertz, 2013)

Timeline Construction1. Map normalized timexes to timeline2. Place events which OVERLAP with timexes onto

timeline3. Place events which OVERLAP with other events

onto the timeline4. Insert rest of events based on BEFORE/AFTER

ordering

16

1999


Summarization

Integrating Timelines into SWING

17

Time Span ImportanceContextualTime Span ImportanceSentence

Temporal Coverage Density

Time MMR


Summarization

Temporal Processing

Summarization PipelineSWING (Ng et al., COLING 2012,

TAC 2011)

State-of-the-art open-source extractive summarizerhttps://github.com/WING-NUS/SWING

Basic, k of n sentence

summaries

1. Time Span Importance (TSI)

• Time spans which contain many events are more salient

• Sentences which references events in these time spans are thus better candidates for a summary


Summarization18

2. Contextual Time Span Importance (CTSI)

• Time spans near to important time spans are important

• Search left and right for local peaks


Summarization19

, where

3. Sentence Temporal Coverage Density (TCD)

• Favour sentences which– contain more events– covering a wide

variety of time spans


Summarization20

Identifying Redundancies

• SWING makes use of the Maximal Marginal Relevance (MMR) algorithm to identify redundancies in selected sentences

• MMR is based largely on surface lexical similarities

Idea: Let’s use time as a basis to penalize the selection of sentences from redundant time periods.


Summarization

TimeMMR• Beyond lexical similarities, identify sentences

which contain substantial time span overlap.• Candidate sentences which share many time

spans with selected sentences are penalized.

22

(1) An official in Barisal, 120 kilometres south of Dhaka, spoke of severe destruction as the 500 kilometre-wide mass of cloud passed overhead.

(2) “Many trees have been uprooted and houses and schools blown away,” Mostofa Kamal, a district relief and rehabilitation officer, told AFP by telephone.

(3) “Mud huts have been damaged and the roofs of several houses blown off,” said the state’s relief minister, Mortaza Hossain.

Lexic

ally

dis

sim

ilar

but

redu

nd

ant


Summarization

Proportion of overlap

Experiments• Data– TAC 2010 dataset for training– TAC 2011 dataset for testing

• Temporal Processing Systems– HeidelTime (Strötgen and Gertz, 2013)– E-T temporal classification (Ng and Kan,

2012)– E-E temporal classification (Ng et al., 2013)

• Summarization baseline– SWING (Ng et al., 2012)


Summarization

Results

24

# Configuration R-2

R SWING 0.1339

B1 CLASSY 0.1278

1 SWING + Timeline Features 0.1394*

2 SWING + Timeline Features + TimeMMR 0.1389


Summarization

Doesn’t seem very effective!

* = p < 0.1, ** = p < 0.05, against R row

Analysis: Timelines contain errors

• Errors from underlying temporal processing systems

• Simplifying assumptions made in timeline construction

• Lack of consistency checking and validation

For effective use, we must identify good timelines• Identify timelines which potentially contain more

errors• Exclude these when performing summarization


Summarization

Reliability Filtering• Short timelines can result when the system fails

to extract or relate events and timexes• Features derived from short timelines are prone

to have extreme values

• Use the length of a timeline as a gauge of its accuracy

• Don’t use timelines shorter than average(as computed over the whole collection)


Summarization

With Reliability Filtering

27

# Configuration R-2

R SWING 0.1339

B1 CLASSY 0.1278

1 SWING + Timeline Features 0.1394*

2 SWING + Timeline Features + TimeMMR 0.1389

3 SWING + Timeline Features [Filtered] 0.1418**

4SWING + Timeline Features + TimeMMR [Filtered]

0.1402**


Summarization

TimeMMR doesn’t seem effective! Why?

* = p < 0.1, ** = p < 0.05, against R row

Does TimeMMR actually help?

28

L1 An Iraqi reporter threw his shoes at visiting U.S. President George W. Bush and called him a ”dog” in Arabic during a news conference with Iraqi Prime Minister Nuri al-Maliki in Baghdad

R1

L2 ”All I can report is it is a size 10,. R2

L3 Muntadhar al-Zaidi, reporter of Baghdadiya television jumped and threw his two shoes one by one at the president, who ducked and thus narrowly missed being struck, raising chaos in the hall in Baghdad’s heavily fortified green Zone.

The incident occurred as Bush was appearing with Iraqi Prime Minister Nouri al-Maliki.

R3

L4 The president lowered his head and the first shoe hit the American and Iraqi flags behind the two leaders.

Muntadhar al-Zaidi, reporter of Baghdadiya television jumped and threw his two shoes one by one at the president, who ducked and thus narrowly missed being struck, raising chaos in the hall in Baghdad’s heavily fortified green Zone.

R4

L5 The The president lowered his head and the R5

R-2: 0.2643, worse by R-2R-2: 0.2772, better by R-2


Summarization

Possibly Redundant?

=

Could an (automated) evaluation metric cater for time?

Conclusion• Use of automatic timeline generation• Integration of timelines into

summarization– Sentence scoring via timeline features– Sentence re-ordering via TimeMMR– Length based timeline filtering helps to

ameliorate errors


Summarization

For details on temporal processing, see:Jun Ping’s work at COLING 2012, EMNLP 2013 and his doctoral thesis (2014)

Questions? If not, ask for more detailed analysis!

Additional Slides


Summarization

Related Work• For Sentence Reordering– Barzilay et al., 1999

• Recency as an indicator of salience– Goldstein et al., 2000;Wan, 2007;

Demartini et al., 2010– Liu et al., 2009 (“Temporal Graph”)– Wu, 2008 (“Largest Cluster”)

• TREC Temporal Summarization Track– Not as relevant; about monitoring an event

over time


Summarization31

Close to our TSI


Summarization

Baseline; worse

With time features; better

TSI: A crane accident

With TSI, the cause of the accident in this summary is included; the alternative R1 sentence is background information and does not occur at any key time span.


Summarization33

With TSI; betterWithout TSI;

worse


Summarization

With CTSI; better

Without CTSI; worse

With CTSI, the “warn” and “disappear” events were promoted in importance due to their proximity with peak P

CTSI: Coral Reef Preservation

Timeline Caveats• Some events span a long period of time (i.e.,

“1999”)

• Events are ordered based on the start of the duration

• Timeline captures relative order

• Construction algorithm does not attempt to reconcile contradictions


Summarization35

Timex Normalization

Source:Bethard, 2013


Summarization36

References• Jun-Ping Ng, Interpreting Text with Time, Doctoral Thesis, National University of Singapore, 2014

• Jun-Ping Ng, Min-Yen Kan, Ziheng Lin, Wei Feng, Bin Chen, Jian Su, Chew-Lim Tan, Exploiting Discourse Analysis for Article-Wide Temporal Classification, EMNLP 2013

• Jun-Ping Ng, Praveen Bysani, Ziheng Lin, Min-Yen Kan, Chew-Lim Tan, Exploiting Category-Specific Information for Multi-Document Summarization, COLING 2012

• Jun-Ping Ng, Min-Yen Kan, Improved Temporal Relation Classification using Dependency Parses and Selective Crowdsourced Annotations, COLING 2012


Summarization37

exploiting timelines to enhance multi-document summarization

Documents

worst storm

torrential rain

extreme winds

bangladeshs southwestern

coastal villagers

tidal wave

bimage courtesy

summarizationimage courtesy