principles of language use for user centered design of software user interfaces

Principles of Language Use for User Centered Design

of Software User Interfaces

Derek BrockNavy Center for Applied Research in

Artificial IntelligenceNaval Research Laboratory

230 July 2002Derek Brock, Navy Center for Applied Research in Artificial Intelligence

A different design perspective

• Designing a software user interface is like writing a reference book with many chapters and subsections– The Designer is the author – Users are the readers

• Interaction starting points in the user interface—menu choices, documents, tool bar icons, etc.—are like chapters and subsections

• The author’s goal in each chapter or subsection is to provide readers with:– Clear and useful expositions– Pointers to dependent and related references


What’s the point?

• Software is designed by people to be used by people– The purpose of a user interface is to coordinate what

the designer thinks users need to understand—and do—to use the software

– But few users can read a designer’s mind

• Any form of coordination between people requires some form of communication

• Successful communication requires—– Collaboration (a willingness to work together)– Meaning and understanding– The use of various kinds of signals– A shared basis for the use of those signals


What goes on when we read a book?

• A successful communication takes place—– The author and the reader carry out a kind of

collaboration—both have to work at it– The author’s meaning (ideally) becomes the reader’s

understanding– A variety of linguistic and nonlinguistic signals are

used—text, diagrams, illustrations, etc.– The author and the reader begin with a communally

shared basis for the signals used…• …which becomes a personally shared basis when the

communication is complete


The same ideas apply to software use

• When software use is successful—– The designer and the user carry out a kind of

collaboration—again, both have to work at it– The designer’s meaning (ideally) becomes the user’s

understanding– A variety of linguistic and nonlinguistic signals are

used—labels, icons, actions, sounds, messages, etc.• The user also makes use of input signals—more on this

later

– The designer and the user begin with a communally shared basis for the signals used…• …which becomes a personally shared basis when the

communication is complete


The larger picture

• Signal use is really language use• People use language to do things with each other • People’s language use skills are their greatest

resource whenever meaning and understanding are involved

• The design of human-computer interaction is a representation problem that primarily involves the designer’s meaning and the user’s understanding

• The principles of language use are a foundational framework for this design problem


Introducing the study of language use

• This is not the study of linguistics– Linguistics is the study of the development and

structure of languages, particularly spoken and written languages

• Instead, it’s the study of what’s involved in how people use signals to do things together– How do people convey and grasp each other’s

intentions?– How do people succeed in coordinating this?– How does this work when people aren’t face-to-face?– What are the cognitive and social processes

involved?– How is language used in different conceptual

domains of action?


First concepts and terms of art

• Language use requires participants and raises the notions of speaker and addressee

• In addition to signals, language use always involves speaker’s meaning and addressee’s understanding

• Joint actions are atomic instances of language use– They are not composed of disjoint actions that are simply

carried out by speaker and addressee independently– But, instead, are the coordinated result of actions that

speaker and addressee carry out together, as a duet—even when this must happen across time and space

• Joint activities emerge from sequences of joint actions– Interactive joint activities can never be fully planned in

advance– This is the proper level for specifying roles, social purposes,

goal hierarchies, etc.• Language use always takes place in a setting

– Settings shape how language is used


Settings

• The notion of a setting in language use combines notions of scene and medium– Scene—where the language is used– Medium—the manner of language use (e.g., spoken,

written, recorded, gestural, a mixture, etc.)

• Settings range in type, in their properties, and in what they require of people and the roles each person plays

• The fundamental paradigms are:– Face-to-face settings—conversation, lecture, wedding…– Written settings—books, letters, movies, recorded music…

• A joint activity’s setting determines how joint actions are coordinated and what skills are needed– Face-to-face settings are the most basic (or natural)– Written settings are the most demanding (or contrived)


Joint actions

• Complete joint actions involve the coordination of a causally related ordering of cognitive, physical, and perceptual subactions (speakers addressees)– Proposing an intention considering it (engagement) – Conceptually signaling the intention (meaning) recognizing

it (understanding)– Presenting the signal identifying it– Executing the presentation attending to it

• All coordinated joint actions between people require these subactions in some form; some examples—– A handshake, paddling a canoe, dancing, making music, etc.– Asking for directions, posting a notice, driving in traffic, etc.

• To coordinate these subactions as parallel parts of the whole, people depend on their intuitions of downward evidence and upward completion


But wait…

• How are signals used to convey meaning and achieve understanding?– How can a speaker be confident a particular signal

will convey his or her intended meaning– How does an addressee converge on the intended

meaning represented by the signal

• It turns out this is a deep problem for computer science, but something that nature has solved rather well

• Meaning and understanding depend on the notion of common ground


Characterizing common ground informally

• Common ground is knowledge people intuitively assume they can use with each other on a presumed basis of shared information– The idea is that when Alan and Zoe go to a movie…– …afterwards, they can presume they had mostly

similar experiences of its content…– …and can assume that each will now understand the

other’s use of this newly shared information

• Shared experience naturally becomes a shared basis of information– In fact, this is exactly how people introduce and then

use new ideas with each other

• But even more is implied by this notion of a shared basis…


Awareness of a shared basis is the essence of common ground

• Verification or inference of similar experiences also builds shared bases– Alan and Zoe don’t even need to see a movie

together to use their knowledge of it with each other—they can simply find out if they have both seen it

– Common ground acquired in this way often contains many discrepancies—but all common ground contains discrepancies

– And note: Alan and Zoe’s separate knowledge of a movie isn’t common ground at all until they have evidence that confirms they’ve both seen it—that their knowledge is actually shared

• In general, people can usually account for what they take to be common ground


Building common ground

• Common ground is built in joint activities– People expect each other to reason similarly and to

maintain a similar awareness of what they are doing together

– People expect each other to be engaged• Common ground in joint activities can be divided into at

least three parts:1. Knowledge initially taken to be held in common2. Knowledge of the activity’s current state3. Knowledge of what has conspicuously taken place so far

• When common ground is missing, meaning and understanding immediately break down– But people are very good at repairing misunderstandings

and filling in missing common ground• Building common ground is a lifelong social and

personal process


Using common ground in joint actions: posing coordination problems

• People implicitly justify their common ground by using what they take to be shared references to it– Shared references are simply signals that refer to elements of

common ground; they function as coordination devices

• Joint actions involving meaning and understanding are really coordination problems people pose and agree to solve together

• People use coordination devices to pose coordination problems, focus their intent, and to indicate their solution

• Most coordination problems in language use are familiar and most are readily solved

• To make a coordination problem easy to solve—– It should have a straightforward solution when it is posed

– Its coordination devices should fully indicate the solution

– The solution should be obvious from what is common ground


For example…

• What does Zoe do if she and Alan are sharing popcorn at the movies and she wants a sip from Alan’s drink?– She can signal Alan either by asking him for a sip or by simply

pointing to his drink in a way that draws his attention– By asking or gesturing, Zoe makes a clear and obvious shared

reference to her common ground with Alan – Either ploy is an excellent coordination device because—

• Zoe has a straightforward solution in mind that Alan can fulfill• Either device explicitly indicates her solution • Her solution is obvious given her common ground with Alan

– In agreeing to Zoe’s coordination problem, Alan correspondingly expects that—• Zoe has an intended solution in mind (what he’s supposed to do)• Her signal contains enough information to solve the problem• He can figure out the solution on the basis of what is obvious in their

common ground


More about coordination problems

• Codifying the joint premises for solving coordination problems:– Solvability (a straightforward solution exists)– Sufficiency (the solution is fully indicated)– Salience (the solution is obvious from common ground)

• Working out coordination problems is a dominant part of people’s language use skills.– These skills have their origins in the perceptual richness of

face-to-face settings • In choosing the right shared basis to use as a

coordination device, joint salience is usually the most important consideration. – Appealing to this aspect of people’s attention in context

greatly increases the likelihood of their converging on the intended solution.


Salient signals

• Signals that make salient coordination devices with respect to common ground include:– Actions (including sounds and gestures)

• These command perceptual attention

– External representations• These are physical aspects of a language use setting that have

conceptual meaning(s) for a joint activity; for example—– Playing cards, the elements of a board game, ceremonial objects, etc.

• External representations enjoy perceptual immediacy

– Conventions of use• Conventions are representations of standing solutions to

frequently occurring coordination problems• When a convention is used as a coordination device, addressees

usually know how to proceed immediately• Languages are full blown systems of conventions• Conventions of use give written settings their power


Layers—conceptual domains of action

• Layers are shifts in the conceptual domain of action (and/or its setting) that shape the focus of common ground for purposes of solving particular kinds of coordination problems in joint activities

• Layers depend on people’s abilities to change perspective• Storytelling illustrates the basic paradigm—

– Primary layer• A storyteller and an audience use language in a real world setting

as themselves– Secondary layer

• The characters of the story use language in the story world setting

• Note that the audience has no trouble shifting its common ground into the story world

– Further layers within the story may arise• Layers can involve play acting, removes in space, time,

and reality, persons not present, changes in setting, etc.


Summary of language use principles

• Language use takes place in settings (e.g., face-to-face, written) that shape its coordination and its required skills

• Language use is a joint activity; it is participatory as opposed to an individual activity

• Joint actions are the unit of language use– They involve cognitive, physical, and perceptual levels of

subactions– They are coordination problems

• Coordination devices are signals people use to pose, focus, and solve coordination problems– The joint premises for solving coordination problems are

solvability, sufficiency, and salience– Salient sources for coordination devices include actions,

external representations, and conventions of use• Layers are conceptual shifts of perspective in joint

activities that shape the focus of common ground


Language use in human-computer interaction

• Speaker and addressee in human-computer interaction are designer and user– Software use is their joint activity– This joint activity fully involves designer’s meaning

and user’s understanding– Software use takes place in a written setting and

makes use of an important secondary layer

• Joint actions are the unit of software use– An entire class of joint actions in software use are

called user interactions– User interactions and other joint actions in software

use are simply coordination problems– Designing an effective system of coordination devices

highlights the problem of building common ground


Thinking about software as awritten setting

• As with any written setting, a software user interface is designed and produced in advance of when it is taken up by its user– It is useful to think of a written setting as a type of pre-

coordinated joint activity

• Conventional written settings build common ground through a static, serial narrative whose domain typically defines a secondary layer of action

• The written setting of software use differs from this paradigm in three important ways– It has a non-serial narrative– Its presentation is dynamic (non-static)– Its secondary layer of action places the user in a type of

face-to-face interactive setting


Characterizing the first two layers of software use

• In the primary layer of software use—– The designer and the user pursue their joint activity

(software use) as themselves in a written setting– In effect the user “reads” the designer’s presentation

—the display portion of the user interface

• In the secondary layer of software use—– The user participates in a face-to-face interaction

with the computer (as opposed to the designer)

• Particularly because of the change in setting, each layer makes different demands of the user’s language use skills


Characterizing the primary layer of software use in language use terms• The design poses a fixed set of coordination problems • Most software uses a cafeteria paradigm for user interaction

– This paradigm implies a non-serial narrative– The design pre-coordinates the joint activity, but the user has

opportunistic control of how the software use proceeds– The user interface generalizes to a huge menu of starting points

• The presentation uses a language of coordination devices

made up of a variety of linguistic and nonlinguistic signals, – These include elements of natural language, visual artifacts,

and behaviors (i.e., actions and procedures)– Many elements of this language are (rightly) presumed to be

conventions• The presentation serves, in part, as an external

representation of the joint activity – Manipulable display elements are generally given specific

conceptual meanings in the context of the software use


Building common ground in the primary layer of software use

In a cafeteria-style presentation:• Both the designer and the user strive to find and build

common ground, but…• The designer’s control over the process of building a

complete body of common ground with the user in the planned manner of a serial narrative is sacrificed

• The user’s common ground with the designer builds contingently (irregularly) on the basis of—– Opportunistically solving coordination problems posed in

the interaction design – And, more generally, opportunistically becoming familiar

with the presentation language • Despite the presentation’s non-serial narrative—

– Over time, users generally develop a sense of where common ground is missing and will look for it unless the effort proves to be too costly


Designing coordination problems in the primary layer of software use

• Coordination problems in a software design deliberately anticipate the user’s domain goals – But addressees’ knowledge in written settings can never fully

anticipated– Users would benefit from an account of what the software can

and cannot do• All coordination problems posed by a design should honor

users’ expectations of salience, sufficiency, and solvability– From a practical standpoint, this is not always possible

• Nontrivial designs generally involve multidimensional conceptual dependencies

• Opportunistic user control pre-empts the designer’s control over the introduction of critical material

• When users are unfamiliar with relevant supporting concepts, they may not know what to do through no fault of their own

– Working solution: identify where common ground may be missing and provide users with immediate, context dependent access to it


Other primary layer issues in software use

• Serial narratives naturally correspond to the third part of common ground—people’s record of what has openly occurred so far in their joint activity– Serial narratives in hard copy are—

• Intuitively indexed; viewed as external representations, they easily represent their current state in this way

• Readily open to review when misunderstandings, questions, or lapses of memory arise

• These advantages are de-emphasized in the written setting of software use– The presentation may not always represent the current state

of the joint activity– External representations in the presentation are generally

dynamic (non-static) and actions are evanescent– Cafeteria organization of entry points obfuscates the user’s

intuitive ability to index the presentation and/or locate particular functions

– Little or no narrative record is kept for the user to consult


Characterizing the secondary layer of software use in language use terms• What we call “human-computer interaction” takes place in

the secondary layer of software use– In this layer, the user participates in a direct interaction with

the computer (as opposed to the designer)• The interaction is designed to resemble a joint activity in a

face-to-face setting…– But this setting is not equivalent with a face-to-face setting

between people– The user and the computer use different communication

languages…– …and their perceptors are mismatched– External representations in the presentation are technically

available to the computer but are generally unused by the system to justify common ground beyond, at most, small numbers of interactions

• Joint actions between the user and the computer are designed to closely resemble joint actions between people


Digression—knowledge people accumulate about activities in general

Claim: In any interactive activity, the knowledge people accumulate and are able to justify corresponds structurally to the three parts of common ground:1. Knowledge initially held about the activity2. Knowledge of the activity’s current state3. Knowledge of what has conspicuously taken place so far in

the activity

• This accumulated knowledge of an activity— – Is common ground when the interaction takes place

between people and shared informational bases exist to justify it as common ground

– Is only an individual’s conception of the activity when the interaction takes place with elements of the environment that accrue no such knowledge

– Is part common ground and part individual conception in human-computer interaction


Building common ground in the secondary layer of software use

When the user engages the computer through a software user interface:

• The interaction is designed to resemble a joint activity in a face-to-face setting

• The user actively keeps track of contingent interaction knowledge corresponding to three parts of common ground as well as informational bases to justify it

• At most, software keeps track of some of the same knowledge and shares informational bases in limited, ad hoc ways, such as undo mechanisms and other kinds of interaction histories

• Contingent interaction knowledge the software fails to be able to justify and use cannot be construed as common ground in the secondary layer’s joint activity

• Consequently, some of the user’s most valuable face-to-face language use skills go largely unused


Key points about language use for the design of software user interfaces• The written setting of software use differs from

conventional, serially narrated written settings in several critical ways—– Its non-serial narrative structure builds common ground

irregularly and introduces problems associated with conceptual dependencies

– Dynamic and evanescent aspects of the presentation and its cafeteria organization reduce its hard-copy utility for the user for purposes of reviewing the software use

• User interactions and other joint actions in software use are simply coordination problems– Almost all display elements of a user interface should be

understood as coordination devices– Users expect all coordination problems posed by a design to

honor the joint premises of salience, sufficiency, and solvability

• Strategies for the accumulation of common ground with the user in the secondary layer remain ad hoc, at best


Practical language use advice for user centered design

• Focus on coordination problems from a primary layer perspective

• Identify coordination problems where common ground may be missing for the user– Provide the user with immediate links to context sensitive help;

make it easy for the user to return from help to the task– Provide explanatory drop-down labels for icons (“tool tips”) and

other external representations; do this even for cases where a terse standing label may be initially insufficient

– Give users a simple, point-of-interaction facility for annotating their software use

• Provide a hyperlinked index to the software’s set of functions (this can be part of a help mechanism)

• Provide a deep undo mechanism• Be aware of the disorienting effect of interface clutter, the

value of meaningful organization, the harm of unexplained evanescent behaviors, and the inevitable wrath of the user


Take home messages

When the normal function of common ground in a language use setting is impaired, people are forced to work harder cognitively to accomplish their goals

Designers of software user interfaces must strive to compensate users for a paradigm that, in primary layer of software use, inherently builds irregular common ground

Building true common ground between the user and the computer in the secondary layer of software use awaits the future


Language useful references

Brock, D. (2002). A language use perspective on the design of human-computer interaction. Proceedings: Workshop on Cognitive Elements of Effective Collaboration. Arlington, VA: Office of Naval Research, Code 342.

Alterman, R. and Garland, A. (2001). Convention in joint activity. Cognitive Science (25)4,2001.

Clark, H. H. (1996). Using language. New York, NY: Cambridge University Press. (read this if nothing else!)

Hutchins, E. (1996). Cognition in the wild. Cambridge, MA: MIT Press.

Norman, D. A. (1992). Turn signals are the facial expressions of automobiles. Reading, MA: Addison-Wesley.

Norman, D. A. and Draper, S. W., eds. (1986). User centered system design. Hillsdale, NJ: Lawrence Erlbaum.

Winograd, T. and Flores, F. (1986). Understanding computers and cognition. Reading, MA: Addison-Wesley.

principles of language use for user centered design of software user interfaces

Documents

applied research

software usewhen software

peoples language use

artificial intelligencewhats

use of input signalsmore

author users

design problemderek

complete derek brock