r comparing presentation summaries: slides vs. reading vs. listening liwei he, elizabeth sanocki...

Comparing Presentation Summaries:Slides vs. Reading vs. Listening

Liwei He, Elizabeth SanockiLiwei He, Elizabeth Sanocki

Anoop Gupta, Jonathan GrudinAnoop Gupta, Jonathan Grudin

Collaboration and Multimedia GroupCollaboration and Multimedia Group

Microsoft ResearchMicrosoft Research

Liwei He, Elizabeth SanockiLiwei He, Elizabeth Sanocki

Anoop Gupta, Jonathan GrudinAnoop Gupta, Jonathan Grudin

Collaboration and Multimedia GroupCollaboration and Multimedia Group

Microsoft ResearchMicrosoft Research

CHI 2000 2

Motivation

Multimedia presentations are being Multimedia presentations are being archived for on-demand accessarchived for on-demand access

University coursesUniversity courses

Corporate training and seminarsCorporate training and seminars

Effective summarization and skimming Effective summarization and skimming can help users utilize time bettercan help users utilize time better

Multimedia presentations are being Multimedia presentations are being archived for on-demand accessarchived for on-demand access

University coursesUniversity courses

Corporate training and seminarsCorporate training and seminars

Effective summarization and skimming Effective summarization and skimming can help users utilize time bettercan help users utilize time better

CHI 2000 3

Video Skimming TechniquesTime compressionTime compression

1.5 – 2.5 saving factor at most1.5 – 2.5 saving factor at most

Video summaryVideo summary

2.5+ saving factor is possible2.5+ saving factor is possible

Time compressionTime compression

1.5 – 2.5 saving factor at most1.5 – 2.5 saving factor at most

Video summaryVideo summary

2.5+ saving factor is possible2.5+ saving factor is possible

O rig ina l

S um m ary

CHI 2000 4

Previous Summarization Study

Compared 4 video summary techniquesCompared 4 video summary techniques

1 by authors of the presentation1 by authors of the presentation

3 by computer algorithms3 by computer algorithms

Pre- and post quizzes and subjective ratingsPre- and post quizzes and subjective ratings

More details in our paper in ACM Multimedia 99More details in our paper in ACM Multimedia 99

““Auto-Summarization of Audio-Video Presentations”Auto-Summarization of Audio-Video Presentations”

Compared 4 video summary techniquesCompared 4 video summary techniques

1 by authors of the presentation1 by authors of the presentation

3 by computer algorithms3 by computer algorithms

Pre- and post quizzes and subjective ratingsPre- and post quizzes and subjective ratings

More details in our paper in ACM Multimedia 99More details in our paper in ACM Multimedia 99

““Auto-Summarization of Audio-Video Presentations”Auto-Summarization of Audio-Video Presentations”

CHI 2000 5

Auto Summary Study Results

All four summaries improve quiz scoresAll four summaries improve quiz scores

Human-generated summary is significantly Human-generated summary is significantly better than computersbetter than computers

No difference among computer-generated No difference among computer-generated summariessummaries

Overall, all are appreciated by subjectsOverall, all are appreciated by subjects

All four summaries improve quiz scoresAll four summaries improve quiz scores

Human-generated summary is significantly Human-generated summary is significantly better than computersbetter than computers

No difference among computer-generated No difference among computer-generated summariessummaries

Overall, all are appreciated by subjectsOverall, all are appreciated by subjects

CHI 2000 6

Questions Raised

What about other forms of summary?What about other forms of summary?

Amount of information from slides?Amount of information from slides?

Skimming text transcript vs. watching video?Skimming text transcript vs. watching video?

Transcripts with key points highlighted vs. video Transcripts with key points highlighted vs. video summaries?summaries?

What about other forms of summary?What about other forms of summary?

Amount of information from slides?Amount of information from slides?

Skimming text transcript vs. watching video?Skimming text transcript vs. watching video?

Transcripts with key points highlighted vs. video Transcripts with key points highlighted vs. video summaries?summaries?

CHI 2000 7

Experimental Design (1)

4 summarization techniques4 summarization techniques

PowerPoint slides onlyPowerPoint slides only

Raw text transcriptsRaw text transcripts

Transcripts with key points highlightedTranscripts with key points highlighted

Author-generated video summariesAuthor-generated video summaries

4 summarization techniques4 summarization techniques

PowerPoint slides onlyPowerPoint slides only

Raw text transcriptsRaw text transcripts

Transcripts with key points highlightedTranscripts with key points highlighted

Author-generated video summariesAuthor-generated video summaries

CHI 2000 8


To compare summarization techniquesTo compare summarization techniques

Objective measure: quiz score improvement Objective measure: quiz score improvement before and after watching a summarybefore and after watching a summary

Subjective measure: user ratingsSubjective measure: user ratings

4 talks chosen from Microsoft training site4 talks chosen from Microsoft training site

Original presenters wrote quiz questionsOriginal presenters wrote quiz questions

To compare summarization techniquesTo compare summarization techniques

Objective measure: quiz score improvement Objective measure: quiz score improvement before and after watching a summarybefore and after watching a summary

Subjective measure: user ratingsSubjective measure: user ratings

4 talks chosen from Microsoft training site4 talks chosen from Microsoft training site

Original presenters wrote quiz questionsOriginal presenters wrote quiz questions

CHI 2000 9


24 Microsoft employees were subjects24 Microsoft employees were subjects

Quiz before and after each summaryQuiz before and after each summary

Watches 4 summaries, each of a different typeWatches 4 summaries, each of a different type

Summary types and talks are counter-balanced Summary types and talks are counter-balanced within each subjectwithin each subject

24 Microsoft employees were subjects24 Microsoft employees were subjects

Quiz before and after each summaryQuiz before and after each summary

Watches 4 summaries, each of a different typeWatches 4 summaries, each of a different type

Summary types and talks are counter-balanced Summary types and talks are counter-balanced within each subjectwithin each subject

CHI 2000 10

PowerPoint Slides Only

CHI 2000 11

Raw Text Transcript

CHI 2000 12

Text Transcript w/ Highlights

CHI 2000 13

Video Summary

CHI 2000 14

Four Presentations Used

P1P1 P2P2 P3P3 P4P4

Length (min)Length (min) 7171 4141 4747 7171

# of slides# of slides 1717 1818 2727 5252

# of pages# of pages 1515 1010 88 1515

HighlightedHighlighted 19%19% 24%24% 25%25% 20%20%

CHI 2000 15

Quiz Score Improvement (1)

Plot by summary typesPlot by summary typesPlot by summary typesPlot by summary types

0

1

2

3

4

5

Slide only Raw text Highlighttext

Videosummary

Sc

ore

dif

fere

nc

e

CHI 2000 16


Highlight text and video summary Highlight text and video summary

> (at p<0.001)> (at p<0.001)

slide only and raw textslide only and raw text


> (at p<0.001)> (at p<0.001)


0

1

2

3

4

5


Videosummary

Sc

ore

dif

fere

nc

e

CHI 2000 17


0

1

2

3

4

5


Videosummary

Sc

ore

dif

fere

nc

eVideo summary > highlight text ?Video summary > highlight text ?

p = 0.087p = 0.087

Video summary > highlight text ?Video summary > highlight text ?

p = 0.087p = 0.087

CHI 2000 18


-1

0

1

2

3

4

5

P1 P2 P3 P4Sco

re d

iffer

ence

Slide only Raw text Highlight text Video summary

Plot by presentationsPlot by presentationsPlot by presentationsPlot by presentations

CHI 2000 19

Subjective Ratings (1)

Slide Slide onlyonly

Raw Raw texttext

Highlight Highlight texttext

Video Video summarysummary

SynopsisSynopsis 3.133.13 3.583.58 4.704.70 4.964.96

Key pointsKey points 41%41% 62%62% 64%64% 69%69%

Skip talkSkip talk 1.961.96 3.503.50 4.524.52 4.414.41

ConciseConcise 2.922.92 3.503.50 4.524.52 5.135.13

CoherentCoherent 2.832.83 4.174.17 4.354.35 4.134.13

Table by summarization techniquesTable by summarization techniquesTable by summarization techniquesTable by summarization techniques

CHI 2000 20



> (at p=0.01)> (at p=0.01)



> (at p=0.01)> (at p=0.01)



Raw Raw texttext






ConciseConcise 2.922.92 3.503.50 4.524.52 5.135.13


CHI 2000 21



Raw Raw texttext






ConciseConcise 2.922.92 3.503.50 4.524.52 5.135.13



are not significantly different (at p=0.05)are not significantly different (at p=0.05)


are not significantly different (at p=0.05)are not significantly different (at p=0.05)

CHI 2000 22



Raw Raw texttext






ConciseConcise 2.922.92 3.503.50 4.524.52 5.135.13


Raw text, highlight text, and video summary Raw text, highlight text, and video summary

> (at p=0.05)> (at p=0.05)

slide onlyslide only

Raw text, highlight text, and video summary Raw text, highlight text, and video summary

> (at p=0.05)> (at p=0.05)

slide onlyslide only

CHI 2000 23

User Comments

13 out of 24 like video summaries13 out of 24 like video summaries

““It is more enjoyable listening and seeing the It is more enjoyable listening and seeing the presenter.”presenter.”

11 prefer highlighted transcripts11 prefer highlighted transcripts

““I liked having the option of being able to get I liked having the option of being able to get more detailed info when I need it.”more detailed info when I need it.”

13 out of 24 like video summaries13 out of 24 like video summaries

““It is more enjoyable listening and seeing the It is more enjoyable listening and seeing the presenter.”presenter.”

11 prefer highlighted transcripts11 prefer highlighted transcripts

““I liked having the option of being able to get I liked having the option of being able to get more detailed info when I need it.”more detailed info when I need it.”

CHI 2000 24

Conclusions

Effective summary techniques are keyEffective summary techniques are key

This study compared 4 summarization This study compared 4 summarization techniquestechniques

Slide only does not work well for most talksSlide only does not work well for most talks

Raw text transcript is hard to readRaw text transcript is hard to read

Human produced summaries work betterHuman produced summaries work better

Effective summary techniques are keyEffective summary techniques are key

This study compared 4 summarization This study compared 4 summarization techniquestechniques

Slide only does not work well for most talksSlide only does not work well for most talks

Raw text transcript is hard to readRaw text transcript is hard to read

Human produced summaries work betterHuman produced summaries work better

CHI 2000 25

Conclusions (cont.)

Slide authoring style makes a differenceSlide authoring style makes a difference

What to do?What to do?

For authors: tools to generate summariesFor authors: tools to generate summaries

For users: interactive and intelligent video For users: interactive and intelligent video browserbrowser

Slide authoring style makes a differenceSlide authoring style makes a difference

What to do?What to do?

For authors: tools to generate summariesFor authors: tools to generate summaries

For users: interactive and intelligent video For users: interactive and intelligent video browserbrowser

CHI 2000 26

Compare with AutoSum Study (1)

Current study and auto summary study Current study and auto summary study are comparableare comparable

4 talks and quiz are the same4 talks and quiz are the same

Both have author-generated summaryBoth have author-generated summary

Slides are shown in all conditions for bothSlides are shown in all conditions for both

Evaluation methods are the same Evaluation methods are the same

Current study and auto summary study Current study and auto summary study are comparableare comparable

4 talks and quiz are the same4 talks and quiz are the same

Both have author-generated summaryBoth have author-generated summary

Slides are shown in all conditions for bothSlides are shown in all conditions for both

Evaluation methods are the same Evaluation methods are the same

CHI 2000 27


-1

0

1

2

3

4

5

SO T TH A

Ave

rage

qui

z sc

ore

diffe

renc

e

UI DH IE MT

-1.0

0.0

1.0

2.0

3.0

4.0

5.0

S P SPU A

Ave

rage

qui

z sc

ore

diffe

renc

e

UI DH IE MT

AutoSum StudyAutoSum StudyAutoSum StudyAutoSum Study Current StudyCurrent StudyCurrent StudyCurrent Study

CHI 2000 28


SS A*A* AA SOSO




ConciseConcise 4.084.08 5.635.63 5.135.13 2.922.92


Subject ratings (AutoSum vs. Current)Subject ratings (AutoSum vs. Current)Subject ratings (AutoSum vs. Current)Subject ratings (AutoSum vs. Current)

CHI 2000 29


SS A*A* AA SOSO




ConciseConcise 4.084.08 5.635.63 5.135.13 2.922.92


A* in AutoSum consistently > AA* in AutoSum consistently > AA* in AutoSum consistently > AA* in AutoSum consistently > A

CHI 2000 30


SS A*A* AA SOSO




ConciseConcise 4.084.08 5.635.63 5.135.13 2.922.92


Slide-based summary (S) > slide only (SO)Slide-based summary (S) > slide only (SO)Slide-based summary (S) > slide only (SO)Slide-based summary (S) > slide only (SO)

r comparing presentation summaries: slides vs. reading vs. listening liwei he, elizabeth sanocki...

Documents

video summary techniques

forms of summary

summary subjective measure

audiovideo presentations

presentation summaries

time better slide

computergenerated summaries

possible slide