![Page 1: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/1.jpg)
1
Evaluating Summary Evaluating Summary Content SelectionContent Selection
Pyramid Method: Work in Pyramid Method: Work in ProgressProgress
Rebecca Passonneau
Ani Nenkova
![Page 2: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/2.jpg)
2
OUTLINEOUTLINE
1. Motivation
2. Problems
3. DUC Evaluations
4. Pyramid Method: Current Status
5. Open Issues
6. Conclusions
![Page 3: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/3.jpg)
3
EVALUATION GOALSEVALUATION GOALS Define parameters of the problem
o What is summarization?
Compare systemso Is the metric meaningful?
Track progresso When does output improve?
Cost Effectivenesso Can it be (partly) automated?
![Page 4: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/4.jpg)
4
PICTURING CONTENT PICTURING CONTENT “OVERLAP”“OVERLAP”
Philippine Airlines (PAL) experienced a crisis in 1998. Unable to make payments on a $2.1 billion debt, it was faced by a pilot's strike in June and the region's currency problems which reduced passenger numbers and inflated costs. On September 23 PAL shut down after the ground crew union turned down a settlement which it accepted two . . .
Starting in May 1998, Philippine Airlines (PAL) laid off 5000 of its 13,000 workers. A 3-week pilots' strike in June and a currency crisis that reduced passenger numbers made payments on PAL's $2 billion debt debt impossible. President Estrada brokered an agreement to suspend collective bargaining for 10 years in exchange for 20% of PAL stock and union seats on its board.The large ground crew union initially voted no.After PAL shut down operations for 13 days starting Sept. 23rd, leaving much of the country without air service and foreign . . .
![Page 5: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/5.jpg)
5
OBSTACLESOBSTACLES
Humans select different content
Humans present same content differently
Lack clear standard of “good” summary
[Contrasts with translation: L1(C)L2(C)]
Need objective method to get at subjective notion of what a summary IS
![Page 6: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/6.jpg)
6
PREVIOUS WORK: PessimismPREVIOUS WORK: PessimismHuman Judgments
Extraction Low Agreement (Rath, 1961; Salton et al, 1997) Inconsistent over time (Rath, 1961; Lin & Hovy,
2002)
Abstraction (Depends on individual’s orientation (Gerrig et al1991)
Automated Evaluation
Extraction (Pastra & Saggion, 2003 EACL) 3-humans; multiple “models”; inconclusive
Abstraction (Lin & Hovy, 2002 ACL) Accepts inconsistent judgments as target Difficult to extend
![Page 7: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/7.jpg)
7
PREVIOUS WORK: OptimismPREVIOUS WORK: Optimism
Good design methodology leads to better understanding areas of agreement
High compression rate leads to high agreement (Jing et al., 1998)
Content variation offset by logarithmic growth in pool of distinct content units (Halteren & Teufel,2003)
Content can be reliably annotated (Beck et al., 1991)
![Page 8: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/8.jpg)
8
HOW TO GET AT “CONTENT” HOW TO GET AT “CONTENT” FROM ITS “EXPRESSION”FROM ITS “EXPRESSION”
1. ADAPT BLEU MT EVALUATIONa) Collect multiple “model” summariesb) Quantify ngram overlap
2. IDENTIFY ABSTRACT CONTENT UNITSa) DUC
b) Reading Comprehension
3. A THIRD WAYa) Content unit “level”b) Multiple expressions of same content
unit
![Page 9: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/9.jpg)
9
DUC: THE CURRENT DUC: THE CURRENT APPROACHAPPROACH
Yearly evaluation of systems on new data sets
NIST evaluations performed by humans
Widely cited results
Does it work?• Compare current systems • Track individual system progress • Track community progress from year to year• Identify specific strengths/weaknesses• Can it eventually be automated?
![Page 10: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/10.jpg)
10
DUC SCORING METHODDUC SCORING METHOD
Datasets: human/machine summaries
Designate “model” human summary
(Automatically) identify content units in “model” summary
Split “peer” summaries into sentences
Human judges evaluate “peer” against model
![Page 11: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/11.jpg)
11
COMPUTE DUC SCORESCOMPUTE DUC SCORES
1. For each EDU:a) Does peer sentence express any partb) How much? (0, 20, 40, 60, 80, 100%)
2. Average EDU percent overlap scores
3. Resulting score ranges from 0 to 1
![Page 12: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/12.jpg)
12
DRAWBACKS TO DUC DRAWBACKS TO DUC SCORESSCORES
• Very sensitive to choice of “model”
• All “model” units created equal
• Difficult to interpret scoreso Human summary scores as low as 0.1o Scores vary for same summarizero Scores vary for same summary
• Systems cannot be differentiated
![Page 13: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/13.jpg)
13
DUC SCATTERPLOTDUC SCATTERPLOT
10 DUC Summary Evaluators
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 10 20 30 40
Summary ID (1 to 30)
Sco
re
Summarizer A
Summarizer B
Summarizer C
Summarizer D
Summarizer E
Summarizer F
Summarizer G
Summarizer H
Summarizer I
Summarizer J
![Page 14: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/14.jpg)
14
FOUNDATION OF PYRAMIDFOUNDATION OF PYRAMID
A few CUs appear in many summaries
Humans can identify same/different CUs
Weight CUs differentially
![Page 15: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/15.jpg)
15
MULTIPLE GOOD SUMMARIESMULTIPLE GOOD SUMMARIES
This pyramid predicts 6 different good summaries consisting of 4 SCUs:
![Page 16: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/16.jpg)
16
SCU ANNOTATION EXAMPLESCU ANNOTATION EXAMPLE
A.2 Unable to make payments on a $2 billion debt
H.2 made payments on PAL’s $2 billion debt impossible
I.1 With a rising $2.1 billion debt J .3 PAL is buried under a $2.2 billion dollar debt
it cannot repay
SCU1 W=4 PAL has a debt of over $2 billion
SCU2 W=3 PAL cannot make its payments
![Page 17: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/17.jpg)
17
PAL PYRAMID TIER: W=3 PAL PYRAMID TIER: W=3 (N=4)(N=4)
SCU1: PAL has $2.1 billion debt
H2 [PAL’s $2 billion debt]1
I1 [and with a rising $2.1 billion debt,]1
J3 [PAL is buried under a $2.2 billion dollar debt]1
SCU2: PAL enforced a shutdown
H5 [After PAL shut down operations]2
I1 [stopped all operations]2
J5 [by a]2 [shutdown]2
SCU3: PAL in crisis
H1 [Philippine Airlines]3
I1 [Philippines Airlines (PAL),]3 [devastated]3
J1 [The fate]3 [is uncertain.]3
![Page 18: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/18.jpg)
18
PAL PYRAMID TIER: W=2 PAL PYRAMID TIER: W=2 (N=8)(N=8)
SCU5: PAL unable to repay debtH2 [made payments on]5 [impossible.]5J3 [it cannot repay]5 SCU6: PAL experienced pilots' strikeH2 [A]5 [pilots' strike]6I1 [by pilot]5 [strikes]6 SCU7: this PAL crisis occurred in 1988
H1 [1998,]7I1 [in 1998]7
. . .
![Page 19: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/19.jpg)
19
ANNOTATION: KEEPING ANNOTATION: KEEPING TRACKTRACK
H1 [Starting in May]23 [1998,]7 [Philippine Airlines]3
[laid off 5000 of its 13,000 workers.]24
H2 [A]6 [3-week]25 [pilots' strike]6 [in June]11 [and a
currency crisis]12 [that reduced passenger numbers]13
H3 [President Estrada brokered an agreement to suspend
collective bargaining for 10 years]17 [in exchange
for 20% of PAL stock and union seats on its board.]26
H4 [The large ground crew union initially voted no.]18
H5 [After PAL shut down operations]2 [for 13 days]4
[starting Sept. 23rd,]8 [leaving much of the country
without air service]27 [and foreign carriers flying
some domestic routes,]9 [61% voted yes.]19
. . .
![Page 20: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/20.jpg)
20
RELIABILITYRELIABILITY
Two Annotators Consensus Annotation
Number of SCUs: 33 versus 37 35
Count of Pairwise Agreements (PAs) SCU Label SCU Members
Comparison of Annotations to Consensus Recall/Precision not valid 65/69 PAs Most “disagreements” due to membership size Only 2 “conflicts”
![Page 21: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/21.jpg)
21
ANOTHER CONSISTENCY ANOTHER CONSISTENCY TESTTEST
Pyramid A H C J
Consensus .95 .89 .85 .76
Annotation1 .97 .87 .83 .82
Annotation2 .94 .87 .84 .74
![Page 22: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/22.jpg)
22
PYRAMID SCORE PART 1PYRAMID SCORE PART 1 1. For N summaries, score each “peer”
against a pyramid with N-1 tiers2. “Peer” annotation
a) Gives SCU “size”
b) Yields a residue of SCUs not in pyramid
3. Compute D (Observed distribution) where D=sum of weights of SCUs
EG: Summary A (D30042), size=20D=(6x3) + (6x2) + (4x1) + (4x0) = 34
![Page 23: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/23.jpg)
23
PYRAMID SCORE PART IIPYRAMID SCORE PART II
1. Compute Max = Ideal Sum of weights of SCUs, given the summary SCU size
2. Pyramid of H,I,J:
a) 9 SCUs in tier, w=3b) 10 SCUs in tier, w=2c) 12 SCUs in tier, w=1
3. Size=20, Max=(9x3) + (10x2) + (1x1)=48
4. P=D/Max PA= 34/48=.71
![Page 24: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/24.jpg)
24
COMPARISON TO DUC COMPARISON TO DUC SCORES:SCORES:
HUMAN SUMMARIESHUMAN SUMMARIES
Lockerbie A B C D DUC n.a. .82 .54 .74 Pyramid .71 .82 .71 .81 PAL A H I J DUC .30 n.a. .30 .10 Pyramid .76 .72 .60 .45 China C D D F DUC n.a. .28 .27 .13 Pyramid .52 .65 .73 .62
![Page 25: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/25.jpg)
25
MACHINE SUMMARY MACHINE SUMMARY EXAMPLEEXAMPLE
African countries voted in June to ignore the U.N. flight ban which was imposed in 1992 to try and force Libya to hand over for trial two suspects wanted in the 1988 bombing of an American airliner over Lockerbie, Scotland. The reported jailing of the three officials comes as Gadhafi is under pressure to accept a plan to turn over for trial two other Libyans wanted for the 1988 bombing of Pan am flight 103 over Lockerbie, Scotland, that led to 270 deaths. The visit was Farrakhan's …
![Page 26: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/26.jpg)
26
COMPARISON TO DUC COMPARISON TO DUC SCORES:SCORES:
MACHINE SUMMARIESMACHINE SUMMARIESSYSTEM DUC PYRAMID
Sys06* .30 .79
Sys13 .03 .24
Sys14 .25 .51
Sys16* .25 .26
Sys17* .03 .17
Sys18 .03 .20
Sys20 .10 .64
![Page 27: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/27.jpg)
27
MACHINE SUMMARIESMACHINE SUMMARIES
System 6
PAL, Asia’s oldest airline, has been unable to make payments on dlrs 2.1 billion debt after being devasted by a pilot’s strike and by Asia’s currency crisis. PAL earlier accepted a preliminary investment offer from Cathay Pacific, Ailing Philippine Airlines and prospective investor Cathy Pacific Airways have clashed over . . .
![Page 28: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/28.jpg)
28
MACHINE SUMMARIESMACHINE SUMMARIES
System 16
President Joseph Estrada on Saturday urged militant unionists at Philippine Airlines to accept a vote by workers approving a 10-year no-strike deal to revive the debt-laden airline. President Joseph Estrada said Saturday the financially troubled airlines will resume its international flights on Sunday by flying him to Singapore . . .
![Page 29: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/29.jpg)
29
MACHINE SUMMARIESMACHINE SUMMARIES
System 17
Christmas is a sacred holiday in the Philippines, and nowhere is that more evident than at the headquarters of Philippine Airlines. But Ramos, who was intent on privatizing the economy, opened the industry to competition, licensing rivals like Air Philippines, Cebu Pacific, and Grand Air. PAL closed for nearly 2 weeks on Sep. 23 after . . .
![Page 30: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/30.jpg)
30
OPEN ISSUESOPEN ISSUES
Distribution of SCUs NOT an independent variableOrderingKnowledgeInformational Goal
Can Pyramid Scoring be Automated?
![Page 31: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/31.jpg)
31
SCU INTERDEPENDENCIESSCU INTERDEPENDENCIES
1. SCU4 presupposes SCU1:
SCU1 (w=4): PAL has a debt > 2 billion
SCU4 (w=3): PAL cannot make its debt payments
2. SCU7, SCU8 depend on SCU2
SCU2 (w=4): PAL shutdown operations
SCU7 (w=3): shutdown began on 9/23
SCU8 (w=3): shutdown lasted 2 weeks
![Page 32: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/32.jpg)
32
SCUs and DEPENDENCY/TAG SCUs and DEPENDENCY/TAG GRGRA3
[On September 23]7
[PAL shut down]2
[after the ground crew union turned down a
settlement]18
[which it accepted two weeks later.]19 SCU71 On IN 5 shut t02 September NNP 4 PAL t2 3 23 CD 4 PAL t2
![Page 33: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/33.jpg)
33
““LARGE” CONSTITUENTSLARGE” CONSTITUENTS
1. PAL experienced a crisis in 1998.
2. Unable to make payments on a $2.1 billion debt,
3. it was faced by a pilot's strike in June
4. and the region's currency problems
5. which reduced passenger numbers and inflated costs.
6. On September 23 pal shut down
7. after the ground crew union turned down a settlement
8. which it accepted two weeks later.
9. PAL resumed domestic flights on October 7
10. and [resumed] international flights on October 26.
11. Resolution of the basic financial problems was elusive,
however,
12. and as of December 18 pal was still $2.2 billion in
debt
13. and [pal was] losing close to $1 million a day.
![Page 34: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/34.jpg)
34
DOCSET TF*IDF DOCSET TF*IDF
TERMS: $2, airline, billion, day, debt, pal (6 of 13 LCs) 1 1. Philippine Airlines (pal) experienced a crisis in
1998.SCU3 w=3
3 2. Unable to make payments on a $2.1 billion debt,SCU1 w=4
1 6. On September 23 pal shut downSCU2 w=4 & SCU7 w=3
1 9. pal resumed domestic flights on October 7SCU10 w=2
4 12. and as of December 18 pal was still $2.2 billion in debtNO SCU
1 13. and losing close to $1 million a day.SCU15 w=2
![Page 35: 1 Evaluating Summary Content Selection Pyramid Method: Work in Progress Rebecca Passonneau Ani Nenkova](https://reader036.vdocuments.site/reader036/viewer/2022062421/56649d385503460f94a10cee/html5/thumbnails/35.jpg)
35
CONCLUSIONSCONCLUSIONS
Define parameters of the problemo What is summarization?
Compare systems and/or humanso Is the metric meaningful?
Track progresso When does output improve?
Cost Effectivenesso Can it be (partly) automated?