Click to edit the title text format

Methodology & Basics of Authoring TuTalk Dialogue Agents

Pamela Jordan

University of Pittsburgh

Learning Research and Development Center

Session Team Members

Pam Jordan Yue Cui (Jenny) Moses Hall Mike Ringenberg

Agenda

Review of basic authoring Methodology for creating dialogue agents Advice on computer-mediated dialogues A closer look at basic authoring

What is TuTalk?

Dialogue system construction tool for learning applicationsAuthoring tools for setting up an artificial

dialogue partner/agentDialogue agent that conducts:

Agent-led or mixed-initiative dialogues and Tutorial or conversational dialogues

Core modules: capabilities and limitations

Understanding: minimum edit distance, costs of insertions, deletions, substitutions, adjustable threshold for costs allowed

Generation: stored phrases Student Model:

uses dialogue history to assess student performance Dialogue Manager:

Supports authoring features that balance expressiveness and ease of use

Limited mixed initiative handling All can be replaced (e.g. LSA for Understanding)

Finite state dialogue management

As a runner pushes a ball away, what horizontal forces act on it?

{runner’s}{gravitational, *}

<subdialog > <subdialog>Good!

After the push ends, what forces….?

<anything else>

Example dialogue

T: Let’s consider the forces acting on objects after they have been thrown. As a runner pushes a ball away, what horizontal forces are acting on it while she is pushing it?

S: Gravity?T: In what direction does gravity act?S: downwardT: Yes. So is a downward force a horizontal or vertical

force?S: It is verticalT: Right. I asked for the horizontal forces instead and the

only one is from the runner. After the push ends, what horizontal forces are acting

on the ball?

Subdialogue

Authoring basics

Write recipes and concepts Concepts are like the ingredient list in a cooking

recipe Recipe uses a simple referring expression (concept

label) and must look at ingredient list for the details

Recipe has one to many steps Step is an

Initiation that is optionally paired with an expected Response

Can be a pointer to another recipe

Examples of concept specifications (abstract)

ask_share_appetizer [So, should we share an appetizer?] [I’d like to share an appetizer. What looks good

to you?]

skip_appetizer [I don’t want an appetizer] [Let’s skip the appetizer]

Example of a dialogue script (abstract)

Response action: push to recipe named

possible responses

Recipe: select-appetizer

Step: enthuse_about_appetizers

Step: ask_share_appetizer

[agree_to_share_appetizer]

[skip_appetizer abort, ask-soup]

[unknown abort, loose-temper]

Step: agree-on-appetizer

initiation

Concept to realize or recognize

Subrecipe: push to recipe named

goal name

Authoring preparation methodologies

Corpus-based Theory-based Corpus-inspired Incremental refinement

Corpus-based authoring

Collect corpus of humans interacting on task Computer mediated Non-interruptible turns

Analyze for goals/topics & adjust for learning objectives Analyze goals/topics identified for student responses, look for answer

categories of: Partially correct/incomplete Partially incorrect Overly vague Overly specific Correct but premature

Identify tutor tactics for each answer category Analyze student language

Tutoring tactics in ProPl

Form tactics

Pump: can you say more about X? Hint & reask: fill in a possible missing piece then

try again Socratic: lead through line of reasoning Simulation: lead through an example & abstract For additional ones, see chapters 7 & 8 of Evens

& Michael (2006), One-on-One Tutoring by Humans and Computers

Applying tactics in ProPl

ProPl student language analysis

Theory-based authoring

Same as corpus-based but Based on theories about task & learning Skip corpus collection

Examples of theoretical conceptual tactics: Definitions & applications of concepts (e.g. distinguish

technical & lay senses of terms) Conceptual variant of a domain principle (e.g.

boundary conditions) Variant of problem

Corpus-inspired authoring

Combination of corpus-based & theory-based

Locate related corpus Identify theoretical goals & refine w/

relevant ones find in corpus Identify theoretical expected responses &

refine relative to corpus

Incremental refinement

Author main-path dialogues w/ correct answers

Refine for answer categories Author response to answer categories &

attach to answer categories Pilot dialogues Analyze logs & refine authored dialogues

Author dialogues: Import a corpus

Import a corpus to authoring tool as in demoAdjust clusters to identify topics/goalsManually extract answer categories per goal

Write main-path dialogues in text editor relative to goals then import to tool

Append topics to template, pair turns and annotate goals & concepts

Author dialogues: alternative approaches

Write xml directly (see preliminary documentation and dtd at http://andes3.lrdc.pitt.edu/TuTalk/#papers)

Skip external authoring of main-path dialogues and author all directly in tool

Advice on computer-mediated dialogues

Students prone to “refusal to answer”: I don’t know, who cares Don’t always bottom out Prod to try

Avoid interrogation: remember coherency w/ short recaps, turn and topic transitions, make some abstractions, meta-info explicit

Assess understanding: Avoid explicit “do you understand” Use trick questions, after success check strength of assertion

Are you sure? What other forces (when answer is no more)

Don’t be interactive for sake of being interactive but for sake of adapting to individual Dialogue slow if cover everything

Additional authoring options

Turn transitions/feedback Mixed initiative Optional steps:

skip if in recent history

Turn Transitions in xml

In xml add truth-val attribute to initiation and response (values = yes,no,partial,unknown)

<initiation truth-val=“yes”> </initiation>

Globally enable/disable (default is enabled)

Say feature in authoring tool overrides automatic transitions (currently does both)

Simple XML Script

<goal name=“select-appetizer”>

<step>

<initiation>enthuse_about_appetizers</initiation>

</step>

<step>

<initiation>ask_share_appetizer</initiation>

<response>agree_to_share_appetizer</response>

<response push=“abort ask-soup”>skip_appetizer</response>

<response push=“abort lose-temper”>unanticipated_response</response>

</step>

<step>

<subgoal>agree-on-appetizer</subgoal>

</step>

</goal>

XML Script w/ truth-val

<goal name=“select-appetizer”>

<step>

<initiation>enthuse_about_appetizers</initiation>

</step>

<step>

<initiation>ask_share_appetizer</initiation>

<response>agree_to_share_appetizer</response>

<response truth=“unknown” push=“abort ask-soup”>skip_appetizer</response>

<response push=“abort lose-temper”>unanticipated_response</response>

</step>

<step>

<subgoal>agree-on-appetizer</subgoal>

</step>

</goal>

Mixed Initiative

Limited mixed initiative: allow student to initiate a topic/question off by default

T: What is the net force on the egg?S: What is the difference between net force and

force?T: Net force is the sum of all the forces on a body.S: okayT: What is the net force on the egg?

Peer agent

As a runner pushes a ball away, what horizontal forces act on it?

{runner’s}{gravitational, *}

<subdialog > <subdialog>Good!

After the push ends, what forces….?

<anything else>

If student says

Agent picks an arc

Optional Steps

As a runner pushes a ball away, what horizontal forces act on it?

{runner’s}{gravitational, *}

<subdialog > <subdialog>Good!

After the push ends, what forces….?

<anything else>

Any others?Any others?

not said

said

Example of optional steps

T: … what horizontal forces are acting on it while she is pushing it?S: Gravity?T: In what direction does gravity act? .T: So are there any other forces on the ball?S: noT: What about the runner? .T: Okay. After the push ends, what forces…

T: … what horizontal forces are acting on it while she is pushing it?S: The runner’sT: Right! So are there any other forces on the ball?S: noT: Good. After the push ends, what forces…

subdialogue

Script with an optional step and semantic labels in XML

<goal name=“ask-appetizer” sem=“ask-appetizer”>

<step optional=“yes”>

<initiation sem=“enthuse-about-appetizers”>enthuse-about-appetizers</initiation>

</step>

<step>

<initiation>ask-appetizer</initiation>

<response sem=“skip-appetizer-order” push=“abort soup”>skip-appetizer</response>

<response push=“abort soup”>no</response>

<response push=“lose-temper”>unanticipated_response</response>

</step>

<step>

<subgoal>order-appetizer</subgoal>

</step>

</goal>

Configuring built-in dialogue behavior

Automatic feedback Initiative policies:

always ignore (default) or always accept