lecturing with digital ink richard anderson university of washington
Post on 21-Dec-2015
216 views
TRANSCRIPT
Lecturing with Digital Ink
Richard AndersonUniversity of Washington
Lessons learned from the Classroom Presenter project
Classroom Pedagogy Teaching with ink
HCI Ink based presentation
Multimedia Analysis of lecture artifacts
Classroom Presenter
Integration of slides and digital ink using Tablet PC
Key ideas: Ink overlay on images Distributed application
Many other systems also support ink and slides
Classroom Presenter as a distributed application Designed as
distributed application for distance learning
Enables many scenarios
Mobility Walking and talking
Sharing materials with students
Note taking Classroom interaction
Student submissions
Deployments
Estimated use in at least 100 courses Wide use inside of computer science Push for adoption outside of CS Lecture archives from UW
Professional Master’s Program Several hundred hours of recorded audio,
video, and ink.
Distance Learning Classes
“Typical ink usage”
“Typical ink usage”
Planning for ink usage
Ink use in presentation
Cognitive load Only limited attention available for
computer while lecturing Linkage with speech
Close tie between ink and speech
Cognitive load Limited feature use
Even color change unusual User interface must be simple (and
robust) Cannot give feedback to user Many actions appear to minimize
mental effort Color change only for contrast Reliance on screen erase
Understanding Attentional Marks Properties
Brief, simple markings Occur with speech Augment meaning of speech Ad hoc form
Is there a linguistic context in which to understand these marks?
Spontaneous Hand Gestures
Spontaneous Hand gestures [McNeill]: are synchronous w/speech are co-expressive w/speech lack standard of form
Attentional marks share these properties.
Gesture Types: Iconic
Gesture Types: Deictic & Cohesive
Analysis of digital ink Understand ink usage Motivation: inform development of
ink based applications Archiving
Search, Summarization, Transcription Lecture based
Improved rendering, note taking, accessibility
Ink classification Textual Diagrammatic Attentional
% of strokes % of episodes
B C B+C B C B+C
Attentional 49 53 51 77 74 76
Diagram 9 7 8 8 8 8
Writing 41 38 40 14 16 15
Other 1 2 1 2 2 2
Coding of six hours of lecture
Goals
Understand usage “in the wild” Cannot expect lecturers to modify
behavior Determine opportunities for
automatic analysis Identify challenges
Methodology
Study of recorded classes Best data set: Professional
Master’s Program Distance courses Audio, Video, Ink archives HCI, Compilers, Programming
Languages, AI, Transaction Processing
Attentional ink Problem – content
matching Identify slide content
referred to by ink Study
Implement basic algorithms to match attention marks to slide content
Compare results with human coders
Attentional ink
Determine the lecturer’s intent:
Determine level to parse the content
Attentional ink Challenges
Recognition of attentional ink on text
Difficult example:
Handwriting
How well does handwriting recognition work on “typical” instructor writing? Domain has many challenges
Recognition Study Studied isolated
words/phrases written on slides
Removed non-textual ink
Fed through the Microsoft Handwriting Recognizer
No training
Recognition Examples The Good:
The Bad:
The Ugly:
Handwriting Reco Results
Exact Alternate
Close None
Prof. A
16 (88%) 1 (6%) 0 (0%) 1 (6%)
Prof. B
146 (59%)
26 (10%) 6 (2%) 71 (29%)
Prof. C
18 (42%) 5 (11%) 1 (3%) 19 (44%)
Prof. D
262 (61%)
45 (11%) 9 (2%)111
(26%)
Prof. E
408 (79%)
46 (9%) 2 <(1%) 58 (11%)
Total 850 (68%)
123 (10%)
18 (1%)260
(21%)
Joint Writing and Speech Recognition Can we use handwriting recognition
with speech recognition together to improve accuracy?
Co-expression of ink and speech Are written words spoken as well?
Can speech disambiguate handwriting?
Can handwriting disambiguate speech?
Examples Difficult for Speech and Ink Recognition
Difficult Written Abbreviations
Speech/Ink Used to Disambiguate Ink/Speech
Experiment Examined instances of isolated word
writing Selected word writing episodes at random
but uniformly from the various instructors Generated transcripts manually from the
audio Checked whether the instructor spoke the
exact word written Measured the time between the written
and spoken word
Speech/Text Co-occurrence Results
Exact Approx None Simul 0-2s > 2s
A 1 (100%) 0 (0%) 0 (0%) 1 (100%) 0 (0%) 0 (0%)
B 9 (75%) 3 (25%) 0 (0%) 12 (100%) 0 (0%) 0 (0%)
C 9 (82%) 2 (18%) 0 (0%) 10 (91%) 1 (9%) 0 (0%)
D 12 (86%) 2 (14%) 0 (0%) 10 (71%) 4 (29%) 0 (0%)
E 9 (56%) 7 (44%) 0 (0%) 7 (44%) 4 (25%) 5 (31%)
Total 40 (74%) 14 (26%) 0 (0%) 40 (74%) 9 (17%) 5 (9%)
Activity Recognition Identifying slide corrections
Example Results
Diagrammatic ink
How do instructors use diagrams Basic legibility Observed behaviors
Diagram phasing Locality of expression
Typical diagram Basic, irregular
shapes Difficult labels Attentional ink
More examples
Zipf diagram
Stroke order
Diagram phasing
More phasing
Top arrows: “Not there”
Separate wins indicated together
Locality in diagrams
Summary
Pedagogy with ink How is ink used in conjunction with
content and speech to express information
Presentation with ink Low attention task
Analysis of ink usage Extracting meaning from archived
lectures
Resources
cs.washington.edu/education/dl/presenter/ Software Downloads Papers
Contact info Richard Anderson,
[email protected] Ruth Anderson, [email protected] Craig Prince, [email protected]