biwst 20091 formal approaches to modelling hci david duce oxford brookes university...
Post on 28-Mar-2015
215 Views
Preview:
TRANSCRIPT
BIWST 2009 1
Formal Approaches to Modelling HCI
David Duce
Oxford Brookes University
daduce@brookes.ac.uk
BIWST 2009 2
Contents
• Syndetic modelling (Duke, Barnard, Duce, May)• Process algebraic modelling (Bowman, Barnard et al)• Verification-guided modelling … (Curzon, Blandford et al)
BIWST 2009 3
Syndetic Modelling
• David Duke, David Duce (1999). “The Formalization of a Cognitive
Architecture and its Application to Reasoning About Human Computer
Interaction”, Formal Aspects of Computing, 11, pp. 665-689
• Philip Barnard, Jon May, David Duke, David Duce (2000). “Systems,
Interactions, and Macrotheory”, ACM ToCHI, 7(2), 222-262
• David Duce and David Duke (2001), “Syndetic Modelling: Computer
Science Meets Cognitive Psychology”, Electronic Notes in Theoretical
Computer Science, Volume 43, May 2001, Pages 50-74. Available in
ScienceDirect (invited talk at workshop FM-Elsewhere)
BIWST 2009 4
Research problems
• How does interaction work?
• How can we build better interactive systems?
Interaction = computation + cognition
State
Interface
Interpretation
Perception / Action
BIWST 2009 5
• Implicit modelling of cognition– Design rationale approaches– Why does a problem occur?– How can it be addressed?
• Explicit models– Which cognitive model?– How can it be represented?– How can we work with it?
Mind the gap
BIWST 2009 6
Bridging the gap
• Syndetic modelling
– Duke, Barnard, Duce and May
– Faconti, Bowman and Massink
– mathematical model as common language
• Cognitive model
– Interacting Cognitive Subsystems
– Barnard, 1979 -
BIWST 2009 7
Building a syndetic model
Mathematicalmodel of ICS
Device modelCognitive modelof domain user
Syndetic model
Domain ortechnologymodel?
BIWST 2009 8
Syndesis in practice
• MATIS example– Laurence Nigay, Joelle Coutaz (IMAG - Grenoble)– experimental platform for multi-modal interaction– deictic reference
BIWST 2009 9
The device ...
interactor MATIS
attributes
vis fields : qnr name value - query content
vis current : qnr - current query
mouse : seq data - data stream from mouse
speech : seq slot - data (& holes) from speech
result : name data - outcome of resolving deixis
actions
art speak : name value - articulate a data value
lim select : data - select a data value
fuse - fuse input streams
fill - fill in slots on a query form
axioms
1 speech = X [speak(nm,d)] speech = X^(nm, d)2 mouse = M [select(d)] mouse = M^d
BIWST 2009 10
ICS
• Interacting Cognitive Subsystems, Barnard et al, MRC, Cambridge
• Highly parallel architecture
• Control of system wide interactions is decentralised
• Rich behaviour arises from interaction among multiple subsystems
• Computational models must capture interactions between mental subsystems at an abstract level
BIWST 2009 11
ICS architecture
OBJVIS
AC MPL ART
BS
LIM
IMPLIC
PROP
BIWST 2009 12
Subsystem operation
Incomingrepresentations
Blending atinput array
Transformed intooutput representations
Copied intoepisodic memory ...
… and revived
BIWST 2009 13
ICS configuration
BIWST 2009 14
Formalising ICS
interactor ICS
attributes
sources : tr P tr
stable : P tr
input : tr repr
_on_ : repr tr
_@_ : repr sys
coherent : tr tr B
buffered : tr
config : Config
flows : P Flow
actionsengage : tr trdisengage : tr trbuffer : trtrans
BIWST 2009 15
From principles to axioms
axioms
1 coherent(t1, t2) dest(t1) = dest(t2) p, q : repr p on t1 q on t2 p q
2 t stable s1, s2 : sources(t)
coherent(s1, s2)
(t = buffered sources(t) stable)
3 t config (t stable src(t) {art, lim} s: tr • t sources(s))
Etc.
BIWST 2009 16
… the user ...
interactor MATIS-User
MATIS - include the MATIS spec
ICS - and the ICS framework
actions
read : data - observe the MATIS presentation
axioms
1 per(read(d)) d in MATIS vis‑obj:, :obj‑mpl:, :mpl‑prop: flows
2 per(select(d)) d on-flow word‑search d in MATIS
3 per(speak(s)) s on-flow speech
BIWST 2009 17
wordsearch and speech
BIWST 2009 18
… and the consequences.
• reasoning about interaction
MATIS-User per(speak(s)&select(d))
{ proof steps }
s d
• specification points to cause of difficulties– inspectable argument critical assumptions
– inspectable model theoretical grounds
– inspectable theory freedom for change
BIWST 2009 19
Achievements …
• Syndesis allows reasoning about properties of interaction• Chain of reasoning is explicit• Points to theoretically grounded reasons• Approach is independent of particular cognitive theory
– ICS has breadth and depth
– wide range of applications
– ICS expert system
– commensurate level of abstraction to FDTs
BIWST 2009 20
…. limitations
• Representations are abstract, uninterpreted– properties addressed only through axioms– insight ceases at level of axioms– why might representations be coherent?
• ICS captured at one level of granularity– but ICS deals with multiple levels
• Flows and configurations• Representations and transformations
– duration calculus, time-based process calculi might address• but, require more detail than cognitive science can supply• require more engagement with mathematical structures
BIWST 2009 21
Prospects - dynamics
• Dynamic aspects of ICS include– overall configuration and stability of data streams from moment to moment– mental representations as inputs to processes– process (if any) that can draw on contents of image record (buffered)
• Issues– what representations are in memory?– how are they used in tasks?– how is product of revival related to contents of memory?– how to deal with multiple levels of detail, variable levels of temporal granularity?
BIWST 2009 22
Prospects – mental representations
• Acknowledge existence of representations carried in flows, but not structure– cognitive theory unclear (modelling perspective)
– lack the ‘right’ mathematics
• Human visual, auditory systems extensively studied– signal processing theory
– ICS assumes that all subsystems based on common architecture
– is there a common mathematical model for all subsystems?
BIWST 2009 23
… directions ???
• Interaction between representations (blending) suggests wave-like model
• Analogies with quantum mechanics
• Electric circuit models (voltage-controlled oscillators, etc) used to model low-level neural behaviours
• Multiple frequencies processed in parallel in visual system – scale-space models
• Can different levels be linked by refinement?
BIWST 2009 24
Prospects – social context
• Safety-critical systems involve human as well as computer agents;
need to reason about both
• Systems increasingly involve groups of users, not just single users
• Distant future: treatment of emergent group behaviour
BIWST 2009 25
Conclusions (Syndetic modelling)
• Crystallise our understanding of human information processing through mathematical structures
• Reveals where understanding incomplete or vague• Moderate success in reasoning about flows• Need deeper understanding of representations• Opening up levels of detail may shed light on utilisation of information
in the external world• Can we generalise to higher levels of interaction between humans in
groups?
BIWST 2009 26
Contents
• Syndetic modelling (Duke, Barnard, Duce, May)• Process algebraic modelling (Bowman, Barnard et al)• Verification-guided modelling … (Curzon, Blandford et al)
BIWST 2009 27
Bowman, Barnard et al
• Li Su, Howard Bowman, Philip Barnard, Brad Wyble (2008). “Process algebraic modelling of attentional capture and human electrophysiology in interactive systems”, Formal Aspects of Computing, DOI 10.1007/s00165-008-0094-3
• Latest in a series of papers
• Explore process algebra formalism; alternative formalism for describing ICS
BIWST 2009 28
Process algebra
• Processes connected by communication channels
• Interact by message exchange along channels
• Processes can be hierarchically nested
• LOTOS is one common process algebra formalism; there are many others
BIWST 2009 29
Variant of Attentional Blink (Barnard et al)
• Subjects were asked to report a word if it referred to a job or profession for which people get paid, e.g. waitress
• Embedded amongst distractors (background words) that all belonged to same category, e.g. nature words
• But also stream contained a key distractor item, semantically related to target category, e.g. tourist, vegetarian
• Serial position that target appeared after key distractor was varied in experiments
BIWST 2009 30
Results
• Target report accuracy w.r.t lag of target relative to key distractor
• Key distractor drew attention away from target with a clear temporal profile
• Diagram from Su et al (2008)
BIWST 2009 31
Two subsystem model
• Su et al, figure 5. (CLOCK process and control channels omitted)
• Subsystems perform salience assessments as items pass through pipeline
• Word composed of six constitutent representations (not letters) – meaning builds up over time
BIWST 2009 32
Attention allocation
• Can only assess salience at a subsystem when attention is engaged• Attention can only be engaged at one subsystem at a time• Cannot glance at one item while looking at and scrutinising another• When attention engaged, the subsystem is buffered• Buffer ensures serial allocation of resources, while items pass
concurrently
BIWST 2009 33
Salience assignment
• Subsystems assign salience based on constituent representations entering
• Word composed of six constituent representations – ca. 110 ms presentation time used in experiments
• Semantic meaning builds up over time, not letter by letter• Assume 3 constituent time slots (60 ms) required for extraction of
useful meaning• Numbers consistent with earlier experiments
BIWST 2009 34
LOTOS specification (extract)
process SOURCE[device_source,source_implic,tick,debug] : noexit :=(* Initialise the delay line with 4 empty constituent representations. *)let vis_dl : Dline = (* Specification omitted *) inVis[device_source,source_implic,tick,debug](vis_dl)whereprocess Vis[device_source,source_implic,tick,debug](dl:Dline) :noexit :=tick?y:Nat!false;(* Synchronise with the global clock at the start of each cycle. *)(((device_source?ia:Rep; (* Recieve visual stimuli from device andstore it in ia (input array). *)exit(ia))|||(source_implic!Get(dl); (* Get() returns the last item in the delay line. *)exit(any Rep)))(* The above processes are performed in parallel, and when both finish,the sequential operator >> ensures that stimuli in the input array arepassed to the next step. *)>> accept ia:Rep in([(y mod 20) eq 0] ->(* Update delay line with the new stimuli when time is a multiple of 20. *)(let dl:Dline = Put(ia,Push(dl)) inVis[device_source,source_implic,tick,debug](dl))[][(y mod 20) ne 0] ->(* Otherwise, do noting. *)(Vis[device_source,source_implic,tick,debug](dl))))endprocendproc
BIWST 2009 35
Contents
• Syndetic modelling (Duke, Barnard, Duce, May)• Process algebraic modelling (Bowman, Barnard et al)• Verification-guided modelling … (Curzon, Blandford et al)
BIWST 2009 36
Approach
• Rimvydas Ruksenas, Jonathan Back, Paul Curzon, Ann Blandford (2009), “Verification-guided modelling of salience and cognitive load”, Formal Aspects of Computing. DIO 10.1007/s00165-009-0102-7
• Higher order formalisation of properties of cognitive architecture• (Not ICS)• Formalisation of salience and dependency on cognitive load• Original formalisation in HOL; most recent paper used SAL (Symbolic
Analysis Laboratory)
BIWST 2009 37
Aims
• Well-defined interfaces use procedural and sensory cues, to increase salience of appropriate actions
• But cognitive load can influence strength of the cues• Formalises relationship between salience and cognitive load revealed
by empirical data
• Remembering to collect original document after making photocopies• Remembering to take bank card after balance enquiry
• Well-designed interfaces increase sensory salience of signals used to cue actions that are frequently forgotten, or performed in wrong sequence
• Evidence that sensory cues are not always noticed under high workload senarios
BIWST 2009 38
Cognitive principles
• Non-determinism – any one of several cognitively plausible behaviours might be taken
• Relevance – given several options, person chooses one that seems relevant to task goals
• Salience – affects user choices, even though non-deterministic
• Mental versus physical actions – delay between moment person commits mentally to action and moment when physical action is taken; each physical action modelled is associated with an internal mental action committing to it
• Pre-determined goals – user engages an interaction with knowledge of task, and task-dependent sub-goals
• Reactive behaviour – user may react to external stimulus, e.g. flashing light, insert coins in adjacent slot
• Voluntary task completion – person may decide to terminate interaction
• Forced task termination – if no apparent action person can take that will help complete task, e.g. ticket machine doesn’t sell the right kind of ticket
BIWST 2009 39
SAL
• Symbolic Analysis Laboratory• “The heart of SAL is a language, developed in collaboration with
Stanford and Berkeley, for specifying concurrent systems in a compositional way. It is supported by a tool suite that includes state of the art symbolic (BDD-based) and bounded (SAT-based) model checkers, an experimental "Witness" model checker, and a unique "infinite" bounded model checker based on SMT solving. Auxiliary tools include a simulator, deadlock checker and an automated test generator.”
• Higher-order language• SAL specifications are transition systems• Constructs include functions, update, non-deterministic choice and
guarded commands
BIWST 2009 40
Cognitive architecture in SAL
• Main concept is user goals• Organised as hierarchical tree, nodes are compound or atomic• Atomic goals map to an action• A goal, g, is modelled as
– guard predicate, specifies when goal enabled, e.g. device prompts– choice models high-level ordering of goals by specifying when goal can be
chosen– achieved – predicate specifying main task goal– salience – specifies sensory salience of g– cueing – function, for each goal g, returns strength of atomic goal h as
procedural cue for g– load – specifies intrinsic load associated with execution of g– subgoals – data structure specifying subgoals of goal
• State space consists of input variable in, output variable out and global variables (memory) mem, and env (environment)
BIWST 2009 41
Fire engine dispatch task (from FAC paper)
BIWST 2009 42
Experiment
• Modelled task and compared to experimental results• Errors investigated
– Initialisation – clicking on “Start next call” without prioritising calls
– Confirmation 1 – Start next call should only be clicked when both new call ID and confirm priority change have been clicked. Forgetting to click “confirm priority change” was confirmation error 1.
– Mode – constructing route using wrong mechanism – user behaves as if system in the other mode
– Termination – clicking get/send route information without selecting a backup unit was considered to be erroneous
– Confirmation 2 – required to click on route complete after selecting backup unit
BIWST 2009 43
Error rates (FAC paper)
BIWST 2009 44
Results
• Refined architecture; taking account of procedural and sensory cues
BIWST 2009 45
Ruksenas et al conclusions
• Model was refined based on empirical data; needs further work to confirm that the refinements are generic
• Irrespective of that, led to deeper understanding of the empirical data– Model allowed team to probe data more deeply than would otherwise have been possible– Refinements followed from discussions between cognitive science and formal modelling
specialists in the team; relied on cognitive mechanisms and factors known from the literature
• Formal development of salience and load rules suggested new experimental hypotheses
– Salience hierarchy “We hypothesize that participants executing a routine procedure will make significantly less errors than participants executing a non-routine procedure (where procedural cues are absent), regardless of whether they possess a high or low level knowledge of the domain.
– Sensory salience “We hypothesize that the sensory salience of an action only captures attention if the semantic meaning of the object has been encoded or the procedure has been learnt by following spatial mappings.
– Cognitive cueing “We hypothesize that cognitive cueing is sensitive to both intrinsic and extraneous load providing that the semantic meaning of objects are interpreted correctly.”
• “.. Provides a good example of the cyclic nature of our interdisciplinary research methodology based on the mutual benefits of bridging cognitive science and formal methods in computer science”
BIWST 2009 46
Acknowledgements (Syndetic Modelling)
• Phil Barnard
• Jon May
• Giorgio Faconti
• Howard Bowman
• Mieke Massink
• Ann Blandford
• Amodeus-2
• TACIT
BIWST 2009 47
Thank You!
Any Questions?
BIWST 2009 48
Overall architecture
vis-objlim-hand
copy
mpl-art art-speech
prop-mplprop-obj
obj-limobj-prop
top related