Introduction to the TrindiKit

Dialogue Systems 2GSLT spring 2003

Staffan Larssonsl@ling.gu.se

What is TrindiKit?

• a toolkit for – building and experimenting with dialogue move engines and systems, – based on the information state approach

• not a dialogue system!

• architecture & concepts• what’s in TrindiKit?• extending TrindiKit• building a system

This lecture

Architecture & concepts

module1 module…

Total Information State (TIS)•Information state proper (IS)•Module Interface Variables•Resource Interface Variables

resource1

control

modulei modulej module… modulen

resource… resourcem

DME

• an abstract data structure (record, DRS, set, stack etc.)

• accessed by modules using conditions and operations

• the Total Information State (TIS) includes– Information State proper (IS)– Module Interface variables– Resource Interface variables

Information State (IS)

• module or group of modules responsible for – updating the IS based on observed moves– selecting moves to be performed

• dialogue moves are associated with IS updates using IS update rules– there are also update rules no directly associated with any

move (e.g. for reasoning and planning)

• update rules: rules for updating the TIS– rule name and class– preconditon list: conditions on TIS– effect list: operations on TIS

• update rules are coordinated by update algorithms

Dialogue Move Engine (DME)

• Modules (dialogue move engine, input, interpretation, generation, output etc.) – access the information state– no direct communication between modules

• only via module interface variables in TIS• modules don’t have to know anything about other modules• increases modularity, reusability, reconfigurability

– may interact with user or external processes

• Resources (device interface, lexicons, domain knowledge etc.)– hooked up to the information state (TIS) – accessed by modules– defined as object of some type (e.g. ”lexicon”)

Modules and resources

What’s in TrindiKit?

What does TrindiKit provide?• High-level formalism and interpreter for

implementing dialogue systems– promotes transparency, reusability, plug-and-play,

etc.– allows implementation and comparison of dialogue

theories – hides low-level software engineering issues

• GUI, WWW-demo • Ready-made modules and resources

– speech– interfaces to databases, devices, etc.– reasoning, planning

• a library of datatype definitions (records, DRSs, sets, stacks etc.)– user extendible

• a language for writing information state update rules

• GUI: methods and tools for visualising the information state

• debugging facilities– typechecking– logs of communication modules-TIS– etc.

TrindiKit contents (1)

• A language for defining update algorithms used by TrindiKit modules to coordinate update rule application

• A language for defining basic control structure, to coordinate modules

• A library of basic ready-made modules for input/output, interpretation, generation etc.;

• A library of ready-made resources and resource interfaces, e.g. to hook up databases, domain knowledge, devices etc.

TrindiKit contents (2)

Special modules and resources included with TrindiKit

• OAA interface resource– enables interaction with existing software

and languages other than Prolog

• Speech recognition and synthesis modules– TrindiKit shells for off-the-shelf recognisers– currently only ViaVoice, but more on the way

• Possible future modules:– planning and reasoning modules– multimodal input and output

Asynchronous TrindiKit

• Internal communication uses either– OAA (Open Agent Architecture) from SRI, or– AE (Agent Environment), a stripped-down

version of OAA, implemented for TrindiKit

• enables asynchronous dialogue management– e.g.: system can listen and interpret, plan

the dialogue, and talk at the same time

How to build a system

TrindiKitinformation state approach

How to use TrindiKit

• We start from TrindiKit– Implements the information state

approach– Takes care of low-level programming:

dataflow, datastructures etc.

TrindiKit

basicdialogue theory

basic system

information state approach

How to build a basic system

• Formulate a basic dialogue theory – Information state– Dialogue moves– Update rules

• Add appropriate modules (speech recognition etc)

TrindiKit

basicdialogue theory

basic system

information state approach

genre-specific theoryadditions

genre-specific system

How to build a genre-specific system

• Add genre-dependent IS components, moves and rules

TrindiKit

basicdialogue theory

domain & languageresources

basic system

application

information state approach

genre-specific theoryadditions

genre-specific system

How to build an application

• Add application-specific resources

• Come up with a nice theory of dialogue• Formalise the theory, i.e. decide on

– Type of information state (DRS, record, set of propositions, frame, ...)

– A set of dialogue moves– Information state update rules, including

rules for integrating and selecting moves– DME Module algorithm(s) and basic control

algorithm – any extra datatypes (e.g. for semantics:

proposition, question, etc.)

Building a domain-independent Dialogue Move Engine

Domain independence of the Dialogue Move Engine

• The DME is domain independent, given a certain type of dialogue– information-seeking – instructional– negotiative– ...

• Domain independence of DME is not enforced by TrindiKit, but is good practice– promotes reuse of components– forces abstraction from domain-specific details,

resulting in a more general theory of dialogue

Specifying Infostate type

• the Total Information State contains a number of Information State Variables– IS, the Information State ”proper”– Interface Variables

• used for communication between modules

– Resource Variables• used for hooking up resources to the TIS, thus

making them accessible from to modules

• use prespecified or new datatypes

sample infostate type declaration

infostate_variable_of_type( is, IS ) :- IS = record( [

private : Private, shared : Shared ] ),

Shared = record( [ com : set( proposition ), qud : stack( question ), lm : set( move ) ] ),

Private = record( [ agenda: stack( action ), plan : stackset( action ), bel : set( proposition ),

tmp : Shared ] ) ] ).

resulting infostate type

PRIVATE :

PLAN : stackset( Action )

AGENDA : stack( Action )

SHARED :

BEL : set( Prop )

TMP : (same type as SHARED)

COM : set( Prop )

QUD : stack( Question )

LU: [SPEAKER:dp, MOVES:set( Move )]

Sample interface variable type declarations

interface_variable_of_type( input, string ).

interface_variable_of_type( output, string ).

interface_variable_of_type( latest_speaker, speaker ).

interface_variable_of_type( latest_moves, set(move) ).

interface_variable_of_type( next_moves, set(move) ).

Specifying a set of moves

• amounts to specifying objects of type move (a reserved type)– there may be type constraints on the

arguments of moves

• preconditions and effects of moves– formalised in update rules, not in the move

definition itself– a move may have different effects on the IS

depending e.g. on who performed it

sample move specifications

% Social

of_type( quit, move ).

of_type( greet, move ).

of_type( thank, move ) .

% Q&A

of_type( ask(Q), move ) <- of_type( Q, question ).

of_type(inform(P), move ) <- of_type( P, proposition).

of_type( answer(R), move ) <-

of_type( R, proposition) or

of_type( R, ellipsis ).

Writing rules

• rule = conditions + updates– if the rule is applied to the IS and its

conditions are true, the operations will be applied to the IS

– conditions may bind variables with scope over the rule (prolog variables, with unification and backtracking)

A sample rule

rule( integrateUsrAnswer, [ $/shared/lu/speaker = usr,

assoc( $/shared/lu/moves, answer(R), false ),

fst( $/shared/qud, Q ),

$domain : relevant_answer( Q, R ),

$domain : reduce(Q, R, P)

], [

set_assoc( /shared/lu/moves, answer(R),true),

shared/qud := $$pop( $/shared/qud ),

add( /shared/com, P ) ] ).

A sample rule (old syntax)

• rule( integrateUsrAnswer, [ – val#rec( shared^lu^speaker, usr ), – assoc#rec( shared^lu^moves, answer( R ), false ),– fst#rec( shared^qud, Q ), – domain :: relevant_answer( Q, R ), – domain :: reduce(Q, R, P) – ], [ – set_assoc#rec( shared^lu^moves, answer(R),true), – pop#rec( shared^qud ), – add#rec( shared^com, P ) ] ).

Writing rules

• available conditions and operations depend on the infostate type– the infostate is declared to be of a certain

(usually complex) type

• datatype definitions provide– selectors: Sel(InObj,SelObj)– relations: Rel(Arg1, …, ArgN)– functions: Fun(Arg1, …, ArgN,Result)– operations:

Op(ObjIn,Arg1, …, ArgN,ObjOut)

• New datatypes may be added

Writing rules: locations in TIS

• objects may be specified by giving a path to a location in the infostate; – paths are specified using selectors, which are similar to

functions• $$Sel2($$Sel1) ~ $Sel1/Sel2 • $$fst($/shared/qud) ~ $/shared/qud/fst

– ”$” evaluates a path and gives the object at the location specified

– ”$$” evaluates a function– $$fst($/shared/qud) = $/shared/qud/fst

• example: – is/shared/com is a path, pointing to a location in the TIS– $is/shared/com is the object in that location– the is can be left out, giving $/shared/com

Writing rules: conditions (1)• conditions do not change the information state• if a condition fails, backtracking ensues• condition syntax (incomplete)

– Rel(Arg1, … , ArgN), e.g. • fst($/shared/qud,Q)

– Arg1:Rel(Arg2,…,ArgN), e.g. • $/shared/qud:fst(Q)• $domain:relevant_answer(Q,A)

– Arg1 = Arg2• Q = $$fst($/shared/qud)

– Cond1 and Cond2– Cond1 or Cond2– not Cond1– forall(Cond1, Cond2)– (Arg is object or prolog variable)

Writing rules: conditions (2)

• quantification, binding and backtracking– if an instantiation a of a variable V in a condition C is

found that makes condition C true, V is bound to a– backtracking occurs until a successful instantiation of

all variables in the list of conditions has been found

• example list of conditionsfst($/shared/qud,Q), in($/shared/com,P),$domain:relevant_answer(P,Q)

• Explicit quantificationQ.P. fst($/shared/qud,Q) & in($/shared/com,P) & $domain:relevant_answer(P,Q)

Writing rules: updates

• operations change the information state• if an operation fails, an error is reported• variable bindings survive from conditions to

operations• operation syntax (incomplete)

– Op(Path,Arg1,…,ArgN)• push(/shared/qud, Q)

– Path : Op(Arg1, … ,ArgN)• /shared/qud : push(Q)

– Store := Fun(Obj,Arg1,…,ArgN)• /private/tmp/qud := $$push($/shared/qud,Q)

Specifying update algorithms

• uses rule classes• constructs include

– Rule– RuleClass– if Cond then S else T– repeat R until C– repeat R– try R– R orelse S– test C– SubAlgorithm

Sample update algorithmgrounding, if $latest_speaker == sys then

try integrate, try database, repeat downdate_agenda, store

else repeat

integrate orelse accommodate orelse find_plan orelse

if (empty ( $/private/agenda ) then manage_plan else downdate_agenda

repeat downdate_agendaif empty($/private/agenda))then repeat manage_plan

repeat refill_agendarepeat store_nim try downdate_qud

Specifying serial control algorithms

• serial constructs include– Module{:Algorithm}– if Cond then S else T– repeat R until C– repeat R– try R– R orelse S– test C– SubAlgorithm

Specifying concurrent control algorithms

• Agent1 | Agent2 | … | AgentN

• where Agenti is • AgentName : {

– import Module1 ,–

… – import Modulep ,– Trigger1 => SerialAlgoritm1 ,– …– Triggerm => SerialAlgoritmm }

• triggers:– condition(C) (C is a subset of the full condition set)– init– new_data(Stream)

Sample control algorithm (1)

repeat ( [ select,

generate,

output,

update,

test( $program_state == run ),

input,

interpret,

update ] )

Sample control algorithm (2)input: {

init => input:display_prompt, new_data(user_input) => input }

| interpretation: { import interpret, condition(is_set(input)) => [ interpret, print_state ] }

| dme: { import update, import select, init => [ select ], condition(not empty(latest_moves)) => [

update, if $latest_speaker == usr then select

] }

| generation: { condition(is_set(next_moves)) => generate }

| output: { condition(is_set(output)) => output } )).

From DME to dialogue system

Build or select from existing components: • Modules, e.g.

– input– interpretation– generation– output

• Still domain independent• the choice of modules determines e.g.

the format of the grammar and lexicon

Domain-specific system

Build or select from existing components:• Resources, e.g.

– domain (device/database) interface– dialog-related domain knowledge, e.g. plan

libraries etc.– grammars, lexicons

Extending TrindiKit

You can add

• Datatypes– Whatever you need

• Modules– e.g. General interfaces to speech

recognizers and synthesizers

• Resources– E.g. General interfaces to (passive) devices

• Important that all things added are reasonably general, so they can be reused in other systsems

Datatype definitions

• relations– relations between objects; true or

false– format: relation(Rel,Args).– Example

• definition: relation(fst,[stack([E|S]),E]).• condition: fst($/shared/qud,Q)

Datatype definitions

• functions– functions from arguments to result– format: function(Fun,Args,Result).– Example

• definition: function(fst,[stack([E|S])],E). • in condition:

– Q = $$fst($/shared/qud)– Q = $/shared/qud/fst

• in effect: – next_move/content := $$fst($/shared/qud)

– every function corresponds to a relation relation(Fun,[Args@[Result]]).

Datatype definitions (2)

• selectors – selects an object (Obj) embedded in

another object (Arg)– selector(Sel,Arg,Obj,ArgWithHole,Hole).

– e.g. selector(fst,stack([E|S]),E,stack([H|S]),H).

– Every selector corresponds to a function function(Sel,[Arg],Object).

Datatype definitions (3)

• operations– operation(Op,InObj,Args,OutObj).

– e.g. operation(push,stack(S),[E],stack([E|S])).

– every operation corresponds to a relation relation(Op,[InObj|Args]@[OutObj]).

– push($/shared/qud,Q,$/shared/qud).

Building modules• DME modules

– Specific to a certain theory of dialogue management

– Best implemented using rules and algorithms

• Other modules– Should be more general, less specific to certain

theory of dialogue management– May be easier to implement directly in prolog or

other language• DME-ADL only covers checking and updating the infostate• These modules may also need to interact with other

programs or devices

Building resources

• Resource– the resource itself; exports a set of predicates

• Resource interface– defines the resource as a datatype T, i.e. in terms of

relations, functions and operations

• Resource interface variable– a TIS variable whose value is an object of the type T

• By changing the value of the variable, resources can be switched dynamically– change laguage– change domain

sample resource variable type declarations (incl. resource interface)

resource_type( lexiconT ).

resource_variable_of_type( lexicon, lexiconT ).

of_type( lexicon_travel_english, lexiconT ).

of_type( lexicon_autoroute_english, lexiconT ).

of_type( lexicon_travel_svenska, lexiconT ).

of_type( lexicon_cellphone_svenska, lexiconT ).

resource_condition(lexiconT,input_form(Phrase,Move),Lexicon) :- Lexicon : input_form( Phrase, Move ).

resource_condition(lexiconT,output_form(Phrase,Move),Lexicon):-

Lexicon : output_form( Phrase, Move ).

Conclusion

• explicit information state datastructure – makes systems more transparent – enable e.g. context sensitive interpretation,

distributed decision making, asynchronous interaction

• update rules – provide an intuitive way of formalising

theories in a way which can be used by a system

– represent domain-independent dialogue management strategies

TrindiKit features

TrindiKit features cont’d

• resources– represent domain-specific knowledge– can be switched dynamically

• e.g. switching language on-line in GoDiS

• modular architecture promotes reuse– basic system -> genre-specific systems– genre-specific system -> applications

Theoretical advantages of TrindiKit

• theory-independent– allows implementation and comparison of

competing theories– promotes exploration of middle ground

between simplistic and very complex theories of dialogue

• intuitive formalisation and implementation of dialogue theories– the implementation is close to the theory

Practical advantages of TrindiKit

• promotes reuse and reconfigurability on multiple levels

• general solutions to general phenomena enables rapid prototyping of applications

• allows dealing with more complex dialogue phenomena not handled by current commercial systems

availability• version 3.0a avaliable!

– SIRIDUS deliverable D6.4

• TrindiKit website– www.ling.gu.se/projects/trindi/trindikit

• SourceForge project– development versions available– developer community?

• licensed under GPL• more info in

– Larsson & Traum: NLE Special Issue on Best Practice in Dialogue Systems Design, 2000

– TrindiKit manual (available from website)

Conclusions: TrindiKit & Information State Approach

• a toolkit for dialogue systems R&D• freely available to researchers• close the gap between theory and

practive of dialogue systems• theory-independent• promotes reuse and

reconfigurability on multiple levels

TrindiKit and VoiceXML

• VoiceXML– industry standard– form-based dialogue manager– web/telephony infrastructure– requires scripting dialogues in detail

• Theory-specific?– VoiceXML implements a specific theory of

dialogue– TrindiKit allows implementation of several

different theories of dialogue– More complex dialogue phenomena hard to deal

with in VoiceXML

TrindiKit and VoiceXML, cont’d

• Combine VoiceXML with TrindiKit?– future research area – support connecting TrindiKit to

VoiceXML infrastructure– use TrindiKit system as VoiceXML

server, dynamically building VoiceXML scripts

– convert VoiceXML specifications to e.g. GoDiS dialogue plans

Possible future modifications

• regler i klasser väljs indeterministiskt

• test C -> if C then halt / break (gå ur loop)

• ta bort möjlighet att ha flera regler med samma namn