BIWST 2009 1

Formal Approaches to Modelling HCI

David Duce

Oxford Brookes University

daduce@brookes.ac.uk

BIWST 2009 2

Contents

• Syndetic modelling (Duke, Barnard, Duce, May)• Process algebraic modelling (Bowman, Barnard et al)• Verification-guided modelling … (Curzon, Blandford et al)

BIWST 2009 3

Syndetic Modelling

• David Duke, David Duce (1999). “The Formalization of a Cognitive

Architecture and its Application to Reasoning About Human Computer

Interaction”, Formal Aspects of Computing, 11, pp. 665-689

• Philip Barnard, Jon May, David Duke, David Duce (2000). “Systems,

Interactions, and Macrotheory”, ACM ToCHI, 7(2), 222-262

• David Duce and David Duke (2001), “Syndetic Modelling: Computer

Science Meets Cognitive Psychology”, Electronic Notes in Theoretical

Computer Science, Volume 43, May 2001, Pages 50-74. Available in

ScienceDirect (invited talk at workshop FM-Elsewhere)

BIWST 2009 4

Research problems

• How does interaction work?

• How can we build better interactive systems?

Interaction = computation + cognition

State

Interface

Interpretation

Perception / Action

BIWST 2009 5

• Implicit modelling of cognition– Design rationale approaches– Why does a problem occur?– How can it be addressed?

• Explicit models– Which cognitive model?– How can it be represented?– How can we work with it?

Mind the gap

BIWST 2009 6

Bridging the gap

• Syndetic modelling

– Duke, Barnard, Duce and May

– Faconti, Bowman and Massink

– mathematical model as common language

• Cognitive model

– Interacting Cognitive Subsystems

– Barnard, 1979 -

BIWST 2009 7

Building a syndetic model

Mathematicalmodel of ICS

Device modelCognitive modelof domain user

Syndetic model

Domain ortechnologymodel?

BIWST 2009 8

Syndesis in practice

• MATIS example– Laurence Nigay, Joelle Coutaz (IMAG - Grenoble)– experimental platform for multi-modal interaction– deictic reference

BIWST 2009 9

The device ...

interactor MATIS

attributes

vis fields : qnr name value - query content

vis current : qnr - current query

mouse : seq data - data stream from mouse

speech : seq slot - data (& holes) from speech

result : name data - outcome of resolving deixis

actions

art speak : name value - articulate a data value

lim select : data - select a data value

fuse - fuse input streams

fill - fill in slots on a query form

axioms

1 speech = X [speak(nm,d)] speech = X^(nm, d)2 mouse = M [select(d)] mouse = M^d

BIWST 2009 10

ICS

• Interacting Cognitive Subsystems, Barnard et al, MRC, Cambridge

• Highly parallel architecture

• Control of system wide interactions is decentralised

• Rich behaviour arises from interaction among multiple subsystems

• Computational models must capture interactions between mental subsystems at an abstract level

BIWST 2009 11

ICS architecture

OBJVIS

AC MPL ART

BS

LIM

IMPLIC

PROP

BIWST 2009 12

Subsystem operation

Incomingrepresentations

Blending atinput array

Transformed intooutput representations

Copied intoepisodic memory ...

… and revived

BIWST 2009 13

ICS configuration

BIWST 2009 14

Formalising ICS

interactor ICS

attributes

sources : tr P tr

stable : P tr

input : tr repr

_on_ : repr tr

_@_ : repr sys

coherent : tr tr B

buffered : tr

config : Config

flows : P Flow

actionsengage : tr trdisengage : tr trbuffer : trtrans

BIWST 2009 15

From principles to axioms

axioms

1 coherent(t1, t2) dest(t1) = dest(t2) p, q : repr p on t1 q on t2 p q

2 t stable s1, s2 : sources(t)

coherent(s1, s2)

(t = buffered sources(t) stable)

3 t config (t stable src(t) {art, lim} s: tr • t sources(s))

Etc.

BIWST 2009 16

… the user ...

interactor MATIS-User

MATIS - include the MATIS spec

ICS - and the ICS framework

actions

read : data - observe the MATIS presentation

axioms

1 per(read(d)) d in MATIS vis‑obj:, :obj‑mpl:, :mpl‑prop: flows

2 per(select(d)) d on-flow word‑search d in MATIS

3 per(speak(s)) s on-flow speech

BIWST 2009 17

wordsearch and speech

BIWST 2009 18

… and the consequences.

• reasoning about interaction

MATIS-User per(speak(s)&select(d))

{ proof steps }

s d

• specification points to cause of difficulties– inspectable argument critical assumptions

– inspectable model theoretical grounds

– inspectable theory freedom for change

BIWST 2009 19

Achievements …

• Syndesis allows reasoning about properties of interaction• Chain of reasoning is explicit• Points to theoretically grounded reasons• Approach is independent of particular cognitive theory

– ICS has breadth and depth

– wide range of applications

– ICS expert system

– commensurate level of abstraction to FDTs

BIWST 2009 20

…. limitations

• Representations are abstract, uninterpreted– properties addressed only through axioms– insight ceases at level of axioms– why might representations be coherent?

• ICS captured at one level of granularity– but ICS deals with multiple levels

• Flows and configurations• Representations and transformations

– duration calculus, time-based process calculi might address• but, require more detail than cognitive science can supply• require more engagement with mathematical structures

BIWST 2009 21

Prospects - dynamics

• Dynamic aspects of ICS include– overall configuration and stability of data streams from moment to moment– mental representations as inputs to processes– process (if any) that can draw on contents of image record (buffered)

• Issues– what representations are in memory?– how are they used in tasks?– how is product of revival related to contents of memory?– how to deal with multiple levels of detail, variable levels of temporal granularity?

BIWST 2009 22

Prospects – mental representations

• Acknowledge existence of representations carried in flows, but not structure– cognitive theory unclear (modelling perspective)

– lack the ‘right’ mathematics

• Human visual, auditory systems extensively studied– signal processing theory

– ICS assumes that all subsystems based on common architecture

– is there a common mathematical model for all subsystems?

BIWST 2009 23

… directions ???

• Interaction between representations (blending) suggests wave-like model

• Analogies with quantum mechanics

• Electric circuit models (voltage-controlled oscillators, etc) used to model low-level neural behaviours

• Multiple frequencies processed in parallel in visual system – scale-space models

• Can different levels be linked by refinement?

BIWST 2009 24

Prospects – social context

• Safety-critical systems involve human as well as computer agents;

need to reason about both

• Systems increasingly involve groups of users, not just single users

• Distant future: treatment of emergent group behaviour

BIWST 2009 25

Conclusions (Syndetic modelling)

• Crystallise our understanding of human information processing through mathematical structures

• Reveals where understanding incomplete or vague• Moderate success in reasoning about flows• Need deeper understanding of representations• Opening up levels of detail may shed light on utilisation of information

in the external world• Can we generalise to higher levels of interaction between humans in

groups?

BIWST 2009 26

Contents

• Syndetic modelling (Duke, Barnard, Duce, May)• Process algebraic modelling (Bowman, Barnard et al)• Verification-guided modelling … (Curzon, Blandford et al)

BIWST 2009 27

Bowman, Barnard et al

• Li Su, Howard Bowman, Philip Barnard, Brad Wyble (2008). “Process algebraic modelling of attentional capture and human electrophysiology in interactive systems”, Formal Aspects of Computing, DOI 10.1007/s00165-008-0094-3

• Latest in a series of papers

• Explore process algebra formalism; alternative formalism for describing ICS

BIWST 2009 28

Process algebra

• Processes connected by communication channels

• Interact by message exchange along channels

• Processes can be hierarchically nested

• LOTOS is one common process algebra formalism; there are many others

BIWST 2009 29

Variant of Attentional Blink (Barnard et al)

• Subjects were asked to report a word if it referred to a job or profession for which people get paid, e.g. waitress

• Embedded amongst distractors (background words) that all belonged to same category, e.g. nature words

• But also stream contained a key distractor item, semantically related to target category, e.g. tourist, vegetarian

• Serial position that target appeared after key distractor was varied in experiments

BIWST 2009 30

Results

• Target report accuracy w.r.t lag of target relative to key distractor

• Key distractor drew attention away from target with a clear temporal profile

• Diagram from Su et al (2008)

BIWST 2009 31

Two subsystem model

• Su et al, figure 5. (CLOCK process and control channels omitted)

• Subsystems perform salience assessments as items pass through pipeline

• Word composed of six constitutent representations (not letters) – meaning builds up over time

BIWST 2009 32

Attention allocation

• Can only assess salience at a subsystem when attention is engaged• Attention can only be engaged at one subsystem at a time• Cannot glance at one item while looking at and scrutinising another• When attention engaged, the subsystem is buffered• Buffer ensures serial allocation of resources, while items pass

concurrently

BIWST 2009 33

Salience assignment

• Subsystems assign salience based on constituent representations entering

• Word composed of six constituent representations – ca. 110 ms presentation time used in experiments

• Semantic meaning builds up over time, not letter by letter• Assume 3 constituent time slots (60 ms) required for extraction of

useful meaning• Numbers consistent with earlier experiments

BIWST 2009 34

LOTOS specification (extract)

process SOURCE[device_source,source_implic,tick,debug] : noexit :=(* Initialise the delay line with 4 empty constituent representations. *)let vis_dl : Dline = (* Specification omitted *) inVis[device_source,source_implic,tick,debug](vis_dl)whereprocess Vis[device_source,source_implic,tick,debug](dl:Dline) :noexit :=tick?y:Nat!false;(* Synchronise with the global clock at the start of each cycle. *)(((device_source?ia:Rep; (* Recieve visual stimuli from device andstore it in ia (input array). *)exit(ia))|||(source_implic!Get(dl); (* Get() returns the last item in the delay line. *)exit(any Rep)))(* The above processes are performed in parallel, and when both finish,the sequential operator >> ensures that stimuli in the input array arepassed to the next step. *)>> accept ia:Rep in([(y mod 20) eq 0] ->(* Update delay line with the new stimuli when time is a multiple of 20. *)(let dl:Dline = Put(ia,Push(dl)) inVis[device_source,source_implic,tick,debug](dl))[][(y mod 20) ne 0] ->(* Otherwise, do noting. *)(Vis[device_source,source_implic,tick,debug](dl))))endprocendproc

BIWST 2009 35

Contents

• Syndetic modelling (Duke, Barnard, Duce, May)• Process algebraic modelling (Bowman, Barnard et al)• Verification-guided modelling … (Curzon, Blandford et al)

BIWST 2009 36

Approach

• Rimvydas Ruksenas, Jonathan Back, Paul Curzon, Ann Blandford (2009), “Verification-guided modelling of salience and cognitive load”, Formal Aspects of Computing. DIO 10.1007/s00165-009-0102-7

• Higher order formalisation of properties of cognitive architecture• (Not ICS)• Formalisation of salience and dependency on cognitive load• Original formalisation in HOL; most recent paper used SAL (Symbolic

Analysis Laboratory)

BIWST 2009 37

Aims

• Well-defined interfaces use procedural and sensory cues, to increase salience of appropriate actions

• But cognitive load can influence strength of the cues• Formalises relationship between salience and cognitive load revealed

by empirical data

• Remembering to collect original document after making photocopies• Remembering to take bank card after balance enquiry

• Well-designed interfaces increase sensory salience of signals used to cue actions that are frequently forgotten, or performed in wrong sequence

• Evidence that sensory cues are not always noticed under high workload senarios

BIWST 2009 38

Cognitive principles

• Non-determinism – any one of several cognitively plausible behaviours might be taken

• Relevance – given several options, person chooses one that seems relevant to task goals

• Salience – affects user choices, even though non-deterministic

• Mental versus physical actions – delay between moment person commits mentally to action and moment when physical action is taken; each physical action modelled is associated with an internal mental action committing to it

• Pre-determined goals – user engages an interaction with knowledge of task, and task-dependent sub-goals

• Reactive behaviour – user may react to external stimulus, e.g. flashing light, insert coins in adjacent slot

• Voluntary task completion – person may decide to terminate interaction

• Forced task termination – if no apparent action person can take that will help complete task, e.g. ticket machine doesn’t sell the right kind of ticket

BIWST 2009 39

SAL

• Symbolic Analysis Laboratory• “The heart of SAL is a language, developed in collaboration with

Stanford and Berkeley, for specifying concurrent systems in a compositional way. It is supported by a tool suite that includes state of the art symbolic (BDD-based) and bounded (SAT-based) model checkers, an experimental "Witness" model checker, and a unique "infinite" bounded model checker based on SMT solving. Auxiliary tools include a simulator, deadlock checker and an automated test generator.”

• Higher-order language• SAL specifications are transition systems• Constructs include functions, update, non-deterministic choice and

guarded commands

BIWST 2009 40

Cognitive architecture in SAL

• Main concept is user goals• Organised as hierarchical tree, nodes are compound or atomic• Atomic goals map to an action• A goal, g, is modelled as

– guard predicate, specifies when goal enabled, e.g. device prompts– choice models high-level ordering of goals by specifying when goal can be

chosen– achieved – predicate specifying main task goal– salience – specifies sensory salience of g– cueing – function, for each goal g, returns strength of atomic goal h as

procedural cue for g– load – specifies intrinsic load associated with execution of g– subgoals – data structure specifying subgoals of goal

• State space consists of input variable in, output variable out and global variables (memory) mem, and env (environment)

BIWST 2009 41

Fire engine dispatch task (from FAC paper)

BIWST 2009 42

Experiment

• Modelled task and compared to experimental results• Errors investigated

– Initialisation – clicking on “Start next call” without prioritising calls

– Confirmation 1 – Start next call should only be clicked when both new call ID and confirm priority change have been clicked. Forgetting to click “confirm priority change” was confirmation error 1.

– Mode – constructing route using wrong mechanism – user behaves as if system in the other mode

– Termination – clicking get/send route information without selecting a backup unit was considered to be erroneous

– Confirmation 2 – required to click on route complete after selecting backup unit

BIWST 2009 43

Error rates (FAC paper)

BIWST 2009 44

Results

• Refined architecture; taking account of procedural and sensory cues

BIWST 2009 45

Ruksenas et al conclusions

• Model was refined based on empirical data; needs further work to confirm that the refinements are generic

• Irrespective of that, led to deeper understanding of the empirical data– Model allowed team to probe data more deeply than would otherwise have been possible– Refinements followed from discussions between cognitive science and formal modelling

specialists in the team; relied on cognitive mechanisms and factors known from the literature

• Formal development of salience and load rules suggested new experimental hypotheses

– Salience hierarchy “We hypothesize that participants executing a routine procedure will make significantly less errors than participants executing a non-routine procedure (where procedural cues are absent), regardless of whether they possess a high or low level knowledge of the domain.

– Sensory salience “We hypothesize that the sensory salience of an action only captures attention if the semantic meaning of the object has been encoded or the procedure has been learnt by following spatial mappings.

– Cognitive cueing “We hypothesize that cognitive cueing is sensitive to both intrinsic and extraneous load providing that the semantic meaning of objects are interpreted correctly.”

• “.. Provides a good example of the cyclic nature of our interdisciplinary research methodology based on the mutual benefits of bridging cognitive science and formal methods in computer science”

BIWST 2009 46

Acknowledgements (Syndetic Modelling)

• Phil Barnard

• Jon May

• Giorgio Faconti

• Howard Bowman

• Mieke Massink

• Ann Blandford

• Amodeus-2

• TACIT

BIWST 2009 47

Thank You!

Any Questions?

BIWST 2009 48

Overall architecture

vis-objlim-hand

copy

mpl-art art-speech

prop-mplprop-obj

obj-limobj-prop