multimodal communication in the staging virtual farm
Post on 15-Jan-2016
59 Views
Preview:
DESCRIPTION
TRANSCRIPT
Multimodal Communication in the Staging Virtual farm
Patrizia Paggio and Bart Jongejan
Center for Sprogteknologi
MUMIN workshopHelsinki 2002
Paggio and Jongejan - Helsinki ‘02 2
The Staging project (www.staging.dk)
Interdisciplinary Danish project: nature and use of 3D applications populated with autonomous agents.
CST’s work: multimodal communication components of a 3D virtual farm.
Focus: multimodal integration, mixed-initiative dialogue, interaction between dialogue and other behaviours.
Paggio and Jongejan - Helsinki ‘02 3
The Staging VEThe VE is in charge of simulating the world provides the agents with sensory information processes requests from the agents (move
objects, produce sounds, play animations)
Staging VE developed at CVMT (Aalborg University)CST has developed a mock-up for testing purposes.
Paggio and Jongejan - Helsinki ‘02 4
Agents
Agents carry out behaviours in reaction to external stimuli and according to
their inner state (hunger, tiredness…) based on strength of activation level
Engaging in a dialogue with the user’s avatar is also a behaviour.
Dialogue behaviour has strong degree of activation for the farmer agent.
Paggio and Jongejan - Helsinki ‘02 5
The Aalborg VE
Paggio and Jongejan - Helsinki ‘02 6
The CST farm
Her skal vises et billede af vores VE
Paggio and Jongejan - Helsinki ‘02 8
Multimodal communicationUser can interact with agents via various devices:
microphone, keyboard, touch screen, data glove.
Commercial speech technology, dedicated gesture recogniser (Karin Husballe Munk at CVMT).
Speech can be combined with deictic, iconic and turn-taking gestures (Cassell and Prevost 1996). Gestures and speech merged by multimodal parser.
Multimodal integration
Speech recognition
Parsing
Semantic mapping
Communication management
Chart initialisation
SpeechHand movements
Gesture recognition
Action
pointing, size
turn-taking
Paggio and Jongejan - Helsinki ‘02 10
More integrationGesture and word are paired:
Feed that cow$1|cowGesture adds information to lexicon entry.
Word and gesture must be (nearly) synchronous Syntactic constraints:
deictic (pointing) requires noun or pronoun iconic (size) requires noun
Semantic constraints: semantic types must be compatible
Paggio and Jongejan - Helsinki ‘02 11
Example
Feed that cow$1|animal.pointgesture := <object-type>$<internal-id>
{act=request, predicate=feed, arg3={reln=animal, semtype=animal, objectid=cow$1}} reln and object type unified, semtype compatible,
objectid added.
Paggio and Jongejan - Helsinki ‘02 12
Contradiction example
Feed that cow$1|apple.{act=request, predicate=feed, arg3={reln=animal, semtype=animal, objectid=cow$1}} gesture and noun semantic types incompatible; only the
interpretation provided by the gesture is compatible with the semantics of the predicate and survives.
Paggio and Jongejan - Helsinki ‘02 13
Examples
Deictic gesturesU: Feed an animal, please. A: Which animal shall I feed?U: Take that cow (+ pointing)
Iconic gesturesU: Feed the sheep, please. A: Which food shall I take?U: The small apple (+ size)
Turn-giving and taking gesturesU: Hi (+give turn) A: Shall we...
Paggio and Jongejan - Helsinki ‘02 14
The Communication Manager
Interprets user’s dialogue moves Builds dialogue trees Interprets references not resolved by gestures Decides agent’s dialogue moves based on
preceding dialogue and on changes in the VE
Dialogue goals arising from scenario combined with dialogue obligations created by preceding dialogue.
Paggio and Jongejan - Helsinki ‘02 15
Dialogue goals
Dialogue goals are created based on domain-specific action templates (Badler et al 1999).
A template specifies actions with related semantic arguments, corresponding attribute name in the semantic representation, necessary preconditions.
FeedAction(Topic=Feed, Animal=<arg3>,
Food=<arg2>, Tool=<instr>,
Precondition=Hungry(Animal))
Paggio and Jongejan - Helsinki ‘02 16
Example: feed action
U: Hi come hereA: Okay, I’ll do itU: Feed an animal.A: Which animal shall I take?U: That cow$1|cow.A: Which food shall I take?U: (Take) a small$|small apple.A: Which tool shall I take?U: Take the pitchfork.A: Okay, I’ll do it.
Paggio and Jongejan - Helsinki ‘02 17
Example: precondition not met
U: Give that brown cow$2|cow an apple, please....A: The cow is not hungry.
Paggio and Jongejan - Helsinki ‘02 18
Example: agent initiative
A: Shall I feed the brown cows and the sheep?U: Yes, give the animals a carrot.A: Which tool shall I take?U: The pitchfork.A: Okay, I’ll do it.
Paggio and Jongejan - Helsinki ‘02 19
Dialogue obligations
Set of condition/obligation pairs model valid speech act sequences.
E.g.: Request/Accept, RejectWhque/Answer, Inform
Used to produce a correct reaction to a user move interpret a user move as either closing a
dialogue segment or opening a new one
Paggio and Jongejan - Helsinki ‘02 20
Dialogue trees
request U: Give the white cow an apple please. whque A: Which tool shall I use?
whque U: Where is the pitchfork?inform A: The pitchfork is in front of
the tree. request U: Take the pitchfork then.accept A: Okay, I’ll do it.
Paggio and Jongejan - Helsinki ‘02 21
Relaxing the rulesCondition/obligation pairs do not always fit.Speech acts can be implied:
A: HiU: Feed the animals please
They can be coerced:
U: Feed an animal.A: Which animal shall I take?U: Feed the brown cow then.
Paggio and Jongejan - Helsinki ‘02 22
Conclusions
Staging has made an initial attempt at giving an agent multimodal dialogue abilities to allow for mixed-initiative dialogues.
Future research: more advanced gesture recognition better understanding of how gestures and
speech can complement each other repairs and self-repairs interaction between dialogue and other
behaviours
Paggio and Jongejan - Helsinki ‘02 23
FILM
top related