Towards a Theoretical Framework for the Integration of Dialogue Models

into Human-Agent Interaction

John R. LeeAssistive Intelligence Inc.

Andrew B. WilliamsSpelman College

Motivation

• How should an intelligent agent incorporate communication?

• How does communication and behavior integrate within an agent model?

• How can ideas from many different dialogue models and conversation examples by incorporated?

• How can one validate the correctness of an agent conversational model?

Motivation

Negotiation

Persuasion

Formal Argumentation

Informal Argumentation

Integrated Model

Cooperative Planning

Belief Grounding

Dialogue Agent Paradigm

• Embedded dialogue manager– Perception processing

• Embedded behavior model

DIALOGUE CAPABLE AGENT

ActionsPercepts

DialogueManager

Behavioral Model

Conversation

Internal API or Language

Goal

• A unified conversational architecture for intelligent agents and assistants– Representation of communication– Incorporation of communication within

behavior– Incorporating a variety of models and ideas

into a single integrated model– Validation of the conversational model– Independent of dialogue interpreter or agent

Focus

Com

munication

Behavior

Agent Implementation

Bring behavior to communication as much as possible

Focus

Com

munication

Behavior

Agent Implementation

• Separate dialogue interpreter from agent– Parallel development of each– Interchangeable components

INTELLIGENT AGENTHUMAN USER

VOCAL CHORDSAND EAR DRUM

SPEAKER ANDMICROPHONE

PHYSICAL LAYER

(SOUND WAVES)

PHONEMES PHONEMESINDIVIDUAL PHONEMES

SOUND CAPTUREAND GENERATION

MUSCLE AND NERVE INFORMATION

UTTERANCE UTTERANCEUTTERANCES AND SENTENCES

(UH-HUH, “OKAY”, “LET’S DO THAT”)

SPEECH RECOGNITION AND GENERATION

PRONUNCIATION AND VOCABULARY RECOGNITION

SEMANTICS SEMANTICSUTTERANCE UNDERSTANDING

(LANGUAGE INDEPENDENT MEANING)

CONTEXTUALIZING AND REFERENCE RESOLUTION

CONTEXT AND REFERENCE RESOLUTION

PRAGMATICS PRAGMATICSDIALOGUE UNDERSTANDING

(TAKE TURN, CLARIFY, ACKNOWLEDGE)

DIALOGUE AND SPEECH ACT THEORY

INTERACTION ABSTRACTION

TASK MANIPULATION MODEL

LOW-LEVEL INTERACTION

(ADOPT, SELECT, IDENTIFY, EVALUATE)

INTERACTION RECOGNITIONAND APPLICATION

SYNTAX SYNTAXLANGUAGE DEPENDENT ANALYSIS

(CONTEXT INVARIANT MEANING)

SENTENCE FORMATION AND UNDERSTANDING

SENTENCE PARSING AND GENERATION

COMMITMENT, BELIEFS, INTENTIONS KNOWLEDGE

COMMITMENT, BELIEFS, INTENTIONS KNOWLEDGE

INTERACTION MANAGEMENT

TASK ABSTRACTIONTASK MODEL

ABSTRACTION

TASK CONTEXT UNDERSTANDING

(OBJECTIVE, RESOURCE, ACTION)

TASK RECOGNITION AND APPLICATION

HIGH-LEVEL INTERACTION

(NEGOTIATE, EXPLAIN, PERSUADE)

Hum

an Interpreter

The Practical CommunicationLanguage (PCL) Hypothesis

There exists a language between that of a human conversational participant and that of an intelligent agent.

This language is capable of abstracting away the complexity of human language while yet maintaining the practical information of the conversation.

Adding to The Practical Dialogue Hypothesis and

The Domain-independence Hypothesis stated in Allen 2000.

Current Utterance-Based Languages

• Application Programmer Interfaces (API)– Task Management Interface

• Specialized Languages– Artificial Discourse Language– Universal Communication Language (Interlingua)– Parameterized Action Representation

• Discourse and Speech Act Tags• Agent Communication Languages

– Foundation for Intelligent Physical Agents (FIPA-ACL)– Knowledge Query Manipulation Language (KQML)

Searching for the language…

True PCL is ideal and volatileEver expanding definition of ‘practical’

PCL should be abstracted* of

1. Region and dialect aspects of language.

2. Informal, Colloquial, Slang and Idiomatic expressions.

3. Modality (Spoken, Written, Gestural, GUI)

*Translated or Incorporated not discarded.

Approach

• Task Communication Language (TCL)– Messages to/from Dialogue Interpreter

Task ModelTask

Communication Model

Interaction Model

TCL Message

– Set of integrated models

TCL Messages

• Header– Generator: Generated utterance or gesture

– Addressee: Intended Receivers of message

– Receiver: Participants who saw or heard

– Uncertainty in all above fields– Interpretation Stack

• Information obtained at al levels of translation– Used by feedback mechanism for improving interpreters

– Content• Meaning-Action Concept

Conversational Paradigms

Human Agent

Human Observation

Manager / Assistant

Teacher / Student

Coach / Player

Peer / Peer

AgentAgent Communication

Semantic Web

Conversational Paradigms

Single Agent Multiple Agent

Single

HumanCurrent Trend

Simulation and Training

Consumer Products

Multiple

Human

Mediator

Discussion Leader

Team Coordinator

Referee

Semantic Web

Marketplaces

Teamwork

Conversational Paradigms

• Not just endpoint to endpoint

• Multiple segmentations– A conversation between people listening in on

another conversation

Meaning-Action Concept

• Meaning of utterance or gesture• Possible association with action.

• “propose( action: )”• “propose( goal: )”• “reject( goal: )”• “counter-propose( action: )”• “query( justification( action: ) )”

Meaning-Action Concepts (MAC)

• Defined in ontological format– Allows for rollback to known concepts

– Manageable growth of concept space

Proposal( )

Counter-Proposal( )

Commit( )

Commit( confidence:30 )

Task Communication Expression

• First-order logic expression of MAC.– Conjunction: Multiple Meanings– Disjunction: Ambiguity– Expressiveness and complexity

Focus

Com

munication

Behavior

Task Model

• Task Concepts:

• Objectives • Recipes • Actions

• Resources • Situations • States

• Constraints • Beliefs • Intentions

• Metrics • Priorities

Task Model

• Task Operations:

• Adoption • Selection • Deferment

• Abandonment • Release • Identification

• Evaluation • Modification

Task Model

Com

munication

Behavior

Task Model

Task-Communication Model

• Integrate the task concepts and operators– Communication with a dialgue interpreter– Task manipulation of an intelligent agent

• Modeling can be language independent– CFSM, CPetriNet, Inference-Based, BDI...

Task-Communication Model

• Nested task operators

• Lower layers:– Persuasion, inquiry, deliberation, formal

argumentation, informal argumentation, clarification, explanation…

• Higher layers:– Negotiation, cooperative planning, learning

through orders, command and control…

Task-Communication Model

Com

munication

Behavior

Task Model

Task-Communication Model

Task-Communication Model

Trivial Example:

• Communicative acts– TCL Messages

• Behavioral acts– Agent integration

[IN]: Propose( Action A )

Evaluate( Action A )

[OUT]: Reject( Action A )

[OUT]: Accept( Action A )

Integration Model

• Generated automatically through tracing the Task-Communication Model

• Represents incoming and outgoing message sequences and possibilities

Task-Communication Model

[IN]: Propose( Action A )

Evaluate( Action A )

[OUT]: Reject( Action A )

[OUT]: Accept( Action A )

Interaction Model

• Extraction of Input-Output Sequences

[IN]: Propose( Action A ) [OUT]: Accept( Action A )

[OUT]: Reject( Action A )

[OUT]: Refine( Action A )

[OUT]: Clarify( Goal G )

[OUT]: Counter( Action A )

Interaction Model

Com

munication

Behavior

Task Model

Interaction Model

Task-Communication Model

Interaction Model

• Validated with known TCL sequences– If sequence is covered, path validated– If sequence is missing, update model

• Assists in integration of models

• Prove various properties– Turn Taking– Liveness– Sanity checks

Mixed-Initiative Control

• No longer in hands of dialogue interpreter– Also managed by ‘task communication model’

• Task-communication model must– Initiate dialogue sequences– Manage

• turn-taking• context tracking• autonomy

Stratagus

• Open source real-time strategy engine– Multiple data sets for varying games

• Dynamically changing environment

• Real time resource management

Discussion