click to edit the title text format methodology for authoring dialogues pamela jordan university of...
Post on 21-Dec-2015
219 views
TRANSCRIPT
Click to edit the title text formatMethodology for Authoring Dialogues
Pamela Jordan
University of Pittsburgh
Learning Research and Development Center
Agenda
Methodology for authoring dialogues Some lessons learned on authoring
computer-mediated dialogues Next steps for projects & discussion
Authoring preparation methodologies
Corpus-based Theory-based Corpus-inspired Incremental refinement
Corpus-based authoring
Collect corpus of humans interacting on task Computer mediated Non-interruptible turns
Analyze for goals/topics & adjust for learning objectives Analyze goals/topics identified for student responses, look for
answer categories of: Partially correct/incomplete Partially incorrect Overly vague Overly specific Correct but premature
Identify tutor tactics for each answer category Analyze student language
Form tactics
Pump: can you say more about X? Hint & reask: fill in a possible missing piece then
try again Socratic: lead through line of reasoning Simulation: lead through an example & abstract For additional ones, see chapters 7 & 8 of Evens
& Michael (2006), One-on-One Tutoring by Humans and Computers
ProPL student language analysis
Use to define response concepts Strategy: pick a minimal set of key words that will
distinguish between responses
Authoring preparation methodologies
Corpus-based Theory-based Corpus-inspired Incremental refinement
Theory-based authoring
Based on theories about domain/task & learning
Examples of theoretical conceptual tactics:Definitions & applications of concepts (e.g.
distinguish technical & lay senses of terms)Conceptual variant of a domain principle (e.g.
boundary conditions)Variant of problem
Authoring preparation methodologies
Corpus-based Theory-based Corpus-inspired Incremental refinement
Corpus-inspired authoring
Combination of corpus-based & theory-based Locate related corpus Identify theoretical goals Search for some of those goals within a corpus
& refine relative to what can find Identify theoretically expected student responses Refine relative to those response instances can
find in corpus
Authoring preparation methodologies
Corpus-based Theory-based Corpus-inspired Incremental refinement
Incremental refinement
Author main-path dialogues w/ correct answers
Refine according to answer categories Author responses to answer categories Pilot dialogues Analyze logs & refine authored dialogues
Agenda
Methodology for authoring dialogues Some lessons learned on authoring
computer-mediated dialogues Next steps for projects & discussion
Recognizing student responses Language recognizer uses simple technique of minimum edit
distance Minimum edit distance is smallest number of “edits”
(insertions/deletions) needed for student response to match a response phrase
For each set of alternative NL phrases (concept) for all responses for the current question, find the minimum edit distance
Select concept with smallest minimum edit distance If that edit distance is within threshold (default of < .5) then select
that concept as the response Else the student response is not recognized, so follow the
unanticipated response path Beginnings & endings of unmatched parts of responses are not
penalized Stop words (e.g. of, the) are not penalized Strategy for authoring a response: pick a minimal set of key
words that will distinguish between responses for a question
Advice on computer-mediated dialogues
Students prone to refusal to answer e.g., “I don’t know”, “who cares” Don’t always bottom out Prod student to try (e.g. “Make your best guess”)
Avoid interrogation: remember to address coherency; include short recaps, turn and topic transitions, make some abstractions, meta-information explicit e.g., “Let’s break it down some
more”, “First, we’ll identify the givens”. Assess understanding:
Avoid explicit “do you understand?” Use trick questions; after success check strength of assertion
“Are you sure?” “What other forces are there?” (when answer is no more)
Don’t be interactive just for sake of being interactive, instead use it to adapt to individual Interact in order to diagnose what the student needs Dialogue slow if cover everything; figure out what can be skipped
Agenda
Methodology for authoring dialogues Some lessons learned on authoring
computer-mediated dialogues Next steps for projects & discussion
Next steps for projects
Look at dialogue samples/corpus for yur project and identify goals to cover in dialogueavailable corpora:
http://andes3.lrdc.pitt.edu/TuTalk/corpora/ For each goal author main path with only
correct responses and unanticipated response follow-ups