A p RI L I 3 - I 8' I 9 9 6 CHI 96 PAPERS

Structuring Information With Mental Models: A Tour of Boston

Ishantha Lokuge

MIT Media Lab

20 Ames Street

Cambridge, MA 02139

ishi@media.mit.edu

Stephen A. Gilbert

Brain & Cognitive Sciences MIT, E 10-120

Cambridge, MA 02139

stephen@psyche.mit.edu

Whitman Richards MIT Media Lab

20 Ames Street

Cambridge, MA 02139

whit@media.mit.edu

ABSTRACT We present a new systematic method of structuring information using mental models. This method can be used both to evaluate the efficiency of an information structure and to build user-centered information structures. In this paper we present the method using Boston tourist attractions as an example domain. We describe several interfaces that take advantage of our mental models with an activation spreading network. Multidimensional Scaling and Trajectory Mapping are used to build our mental models. Because of the robustness of the technique, it is easy to compare individual difference in mental models and to customize interfaces for individual models.

Keywords Cognitive models, multidimensional scaling, visualization, interaction design, evaluation.

INTRODUCTION We all know that a curious person can more efficiently absorb infonnation when it is well structured than when it is arbitrarily scattered. The question that every information architect then asks is, "How might I best organize the information for that person?" This question contains three issues: what structures are useful for organizing information in general, what structures are useful for organizing that information, and what structures are useful for organizing information for that person.

To answer these above questions, we propose (i) collecting experimental data from a number of subjects, (ii) analyzing the mental models of those subjects with Multidimensional Scaling (MDS) [7, 15, 16) and Trajectory Mapping (TM) [4, 9, 13], and then (iii) using those models to design information structures. Such models should lead more quickly to suitable interfaces rather than beginning with trial and error explorations.

Permission to make digital/hard copies of all or part �f this material fo_rpersonal or classroom use is granted without fee �rov1ded that the copiesare not made or distributed for profit or commercial advantage, the_ copyright notice, the title of the publication and its date appear, and nollce is_ given that copyright is by pennission of �e. ACM, Ii:ic, To

copy othe�1se,to republish, to post on servers or to red1stnbute to hsts, requires specific

permission and/or fee. CHI 96 Vancouver, BC Canada @ 1996 ACM 0-89791-777-4/96/04 .. $3.50

Before continuing on to details, we offer definitions of "information structure" and "mental model." By "information structure" we mean an arrangement of pieces of information. The arrangement might be a 2-D array on a table or a screen, as in a card game or a spreadsheet. It might also be a 1-D ordering of items, like a shopping list. It could also consist of a set of information nodes with connecting association links, such as a hypertext. All the pieces of information should belong to the same conceptual type category, such as numbers to add, tasks to do, or words to remember. By "mental model" we mean not the explanatory model offered by Johnson-Laird [5], but rather a more general definition: the cognitive layout that a person uses to organize information in his or her memory.

THE EXAMPLE DOMAIN Consider the map of Boston shown in Figure la. We have chosen fifteen different sites of activity taken from a tour guide book (listed in Figure lb). This set of information pieces (or "stimuli," as we will call them) is a very highdimensional in feature space, and it has a relatively large variance across people. That is, there are many features

Figure l a: Geographic layout of activities

413

Stimulus Set: Boston Tourist Attractions

I. Sports Museum

2. Children's Museum

3. Science Museum

4. Aquarium5. Swan Boats (boat tour of garden)

6. Newbury Street (elegant shopping)7. Quincy Market (outdoor mall)

8. Trinity Church (historic site)

9. Magic Show

10. Salem (nearby historic town)

1 l. Harvard University

12. Museum of Fine Arts

13. Zoo 14. Fenway Park (baseball stadium)

15. Arboretum (nature preserve)

Note: Activities 9-15 are outside the scale of the map.

Figure 1 b: The stimuli for the MDS and TM experiments.

that can be used to describe the stimuli, and people do so in significantly different ways. As a contrasting example, a deck of playing cards contains very few features, namely the number of a card, its suit, and perhaps its color, and different people describe playing cards similarly.

MDS AND TM METHODS

A person who lives in Boston and knows the sites will have at least two mental models of them, one based on their geographic locations, and one based on their content. In order

• ooe 0 • • • 1 2 4 4 6 5

0 6 5 4 3 1 0 2 3 5 1

• 2 5 4 0 1 2

• 4

•

Figure 2a: Typical input data for the MDS algorithm; sim

ilarity data for black and white circles of different sizes. 7 = very similar; 1 = very dissimilar.

414

CHI 96 A p R ! L I 3 - I 8. I 9 9 6

to illustrate the capabilities of Multidimensional Scaling (MDS), first used by Torgerson [16], Shepard [15], and Kruskal [7], we collected data from two subjects for each of these mental models (Figure 3). Note that the judgments based on geographic similarity (3a) are completely different from judgments based on content (3b). The input data for MDS takes the form of pairwise similarity judgments, and the output is an arrangement of the stimuli in a metric space.

To clarify the procedure for obtaining these MDS plots, we offer a simple example; consider a collection of seven black and white circles of different sizes. In Figure 2a, these circles are arranged randomly along the edges of a similarity matrix. The numbers show a subject's similarity rating on a scale of I to 7. From these numbers the MDS algorithm arranges the circles as shown in Figure 4, with neighboring circles being the most similar.

The distances between the points in the output is usually a non-linear transformation of the values in the similarity matrix. For Figure 3a, we asked the subjects for every pair of stimuli, "On a scale from l to 7, how similar are stimuli X and Y in terms of their distance from each other?" For Figure 3b, we asked, "How similar are stimuli X and Y in terms of their content or theme?" We then used KYST2 [6] to run the MDS and generate the graphs shown.

As one might expect, the distance-based MDS plot is similar to the actual map of Boston, though somewhat warped; the warping could stem from differing familiarity with the sites [11] or from thinking of distance as travel time instead of geographic distance. In the content-based MDS plot,

Figure 2b: Typical input data for the TM algorithm; extrapolations and interpolations for black and white cir

cles of different sizes. Here are 12 of 21 possible pairs.

A p RI L I 3 - Is' I 9 9 6 CHI 96

Arboretum +Zoo+-

•

Salem Magic Show • ··---- 1'_ __________ t -------------

Sports • • Science Museum Museum

Harvard

Swan

Aquarium •

• Quincy Market

Boals;; ------

Newbury St.., • Trinity Church

Children's" Museum •

Figure 3a: MDS plot based on geographic similarity

similar activity sites like the Aquarium and the Zoo appear near each other, as do shopping areas Newbury Street and Quincy Market. One can use the groupings of points in an MDS plot to assign features to clusters of points, as we have done in Figure 3b with the features, "historical," etc. Likewise, many researchers who use MDS would attempt to assign meaning to the axes of the plot, suggesting perhaps that the X-axis runs from "playful to serious" and Y-axis runs from "outdoors to indoors". Such labels or category assignments must be done carefully and should be verified with separate experiments, however, because they are often biased by the experimenter's a priori knowledge of the data.

The reader might wonder what would be the outcome if we had asked subjects to give general similarity ratings without specifying what type of similarity. Would the resulting MDS plot have been an unfortunate mixture of the two plots in Figure 3? This issue raises the question of how we could

,,✓,20s+'"fefic;,','

•.¾f;;t;;:

C ~ " ~ @ ~ APRIL t 3 - 1 8 , 1 9 9 6

in the two "ex" columns, and the interpolant in the "int" col- umn. As well as using a stimulus for each slot, the subject may also enter an "X" or a "...". An "X" indicates that the subject did not reel comfortable choosing a stimulus for that spot, and the "..." indicates that the subject could imagine a stimulus that would fit there, but that such a stimulus was not present in the given data set to choose from.

From this set of quintuples, we can now extract a connected graph of the stimuli in which the maximum number of quin- tuples fit. For example, in the top row of Figure 2b, we have an ordered set of white circles ending in a small dot in one column, and a set of black circles ending in the same small dot in the other column. If these quintuples are equally con- sidered as constraints on the graph, the ending trajectory map will contain a path from the large white circle to the large black circle, going through the small dot, as shown in Figure 4.

BOSTON DATA Figure 5 shows a trajectory map for tourist site data gathered from the authors. Note here that the positions of the nodes are not important; the mental model lies in the connections between the nodes, i.e. the topology of the graph. The weights on the links are based not on similarity, but rather on the robustness of that link across a gamut of parameters within the TM algorithm. The weight on a link can thus be thought of as the strength of the connection in the mental model. (A TM algorithm is being fine-tuned for release by Gilbert.)

By pruning the trajectory map to only its strongest links, one can see rough feature clusters emerge (see the heavier links

Mag c Show .8~;4;~ ~4 ~;~, . . . . . . . . . . . . . . . . . Salem

; 4 : ~&

( " i

lildren's mourn

Science , M i i~..11 i m i

Museum ~iiiiii ~

O 0

Newbury St.

Arboretum Quincy Market

,:. ~,C? -0! ( ) ~ , ,

Figure 6: The mental models combined: a rough TM path superimposed over the MDS content plot.

in Figure 5). In our example, the clusters are roughly simi- lar to the estimated clusters in the MDS plot (Figure 2b). In a trajectory map, however, the s~imuli are ordered within each cluster, e.g. Arboretum, Swan Boats, Aquarium/Zoo. Figure 6 shows the combined mental models, a rough TM path drawn over the MDS content-based plot.

Because the tourist sites have several possible features that could be used by subjects to report their TM extrapolations and interpolations, one can discover the most salient fea- tures of the data set by running the algorithm across many

m Fenway Park

Aquarium

O

Z q

Figure 7a: Children's activities energized by the region- al activation spreading network.

m

F ¢nw,~tf Pw~

i~ ::~L ̧

I I tmi

Fine Arts Museum Trinity Church

Z ~

m

Figure 7b: Adults' activities energized by the regional activation spreading network.

416

A p RI L I 3- I 8' I 9 9 6 CHI 96

II II

•-

II ..

FenwayPw�

II

•-

PAPERS

Figure 8: The network activates the path segments when the user examines the sites Sports Museum, Fenway Park and Aquarium. The activation levels are mapped to image and typographic size, enabling the system to suggest to the user an appropriate next step based on the MDS and TM results.

subjects. It is likely with this tourist site information, for example, that some subjects would do the TM based on geographic distance, while others would do it based on the content of the sites. This feature difference would eventually manifest itself from obviously disparate trajectory maps, and lead to the phrasing of the MOS questions described earlier: similarity in terms of distance and similarity in terms of content. Examples of different trajectory maps for the same domain can be seen in [12].

EXPERIMENTATION

The mental models from TM and MDS provide an excellent basis for measuring the efficiency of an information structure that has been built from the models. One might arrange the stimuli serially according to the TM paths or distribute them according to the MDS plot. After collecting data from a new pool of subjects for measures of readability, ease of remembering, etc. (see [2, 3] for a typical array of memory measures), the information structure can be systematically varied by changing the parameters that produce the MOS and TM models.

EXAMPLE INTERFACES

If one considers the set of tourist attractions as an information space to be explored, we can also use the mental models to give the user well-founded suggestions as to his or her next step of exploration. As a demonstration of feasibility, we have designed a visualization system that allows the exploration of the Boston tourist attractions. Since we are no longer restrained to the geography of Boston, as an actual tourist would be, we must now answer the question, "How should we order the various sites?" To explore the different possible answers, we have built three different interfaces for the system, each of which defines an "attentional window" as the current region of interest in the information space. An activation spreading network based on the mental models defines the size of the attentional window as the user explores. Thus, if the aquarium node of the network is currently activated, and the models suggest that the swan

boats node is closely related, then the activation will spread to swan boats, leading the attentional window to include swan boats as a potential next focus.

In the first two interfaces, the 15 sites are arranged twodimensionally according to the MDS plot in Figure 3b. Whichever sites fall within the attentional window are rendered larger than the others. In one interface, the attentional window envelops MDS regions (Figure 7), and in the other, the window spreads across TM paths (Figure 8). Figure 7a shows the display of regional activation with emphasis on children's activities. Figure 7b shows the activities in an adult context. In contrast, Figure 8 shows several successive frames of path activation: when the user investigates the Sports Museum, the network suggests that Fenway Park or the Children's Museum might be a good next choice (those two sites are slightly enlarged in Figure 8a). As the user shifts attention to the next event, past activities fade away and activities further along the path become more conspicuous.

The difference between these two styles of exploration can be characterized by the width of the attentional window: narrow in the case of paths, and wide in the case of regions. By smoothly varying the width, one might change smoothly between modes of exploration.

The third interface attempts to combine the path and region following ideas of the first two. This interface depicts the sites as semitransparent multimedia cubes in a 3-D space (Figure 9). The cubes are arranged along the TM paths, but at each cube, other cubes within the same region can be seen orbiting the current cube nearby. Thus, the user can "fly" smoothly through the space along the TM routes, or branch off to a similar site within the current region. Also, the floor of each cube displays a geographic map with the site's location high-lighted, thus incorporating all of the features that we have discussed thus far.

417

CHI 96 A p RI L I 3- I 8, I 9 9 6

aquarium W11h vivid banners !lying overhead and a Susumu Shingu sculpture dancing in the wind. the New England Aqu�r�'?Piil,16hed at !he edge of Bos Ion harbor. is easy 10-spo1. In an ouldoor lank near the enlrance. harbour seals f1olicalmost close enough 10 louch. The inside scene is even more colorful and act,ve. The centorp,ece of the aquarium. a massive 187.000 gallon ocean tank. is one of the world s larges! cyllnd11cal sail

Figure 9: An interface that combines path and region following; the sites are represented by cubes in a 3-D space. The cubes lie along the TM paths, and the other sites from the MDS region encircle the cube within the attentional window.

The activation spreading network was designed by Lokuge and Ishizaki [10], and the visualization system is implemented on a Silicon Graphics Onyx workstation. This network links nodes in a space (Figure 10) and uses the ordering in the space to control the activation level of a node [ 1, 12]. When attention "jumps" to a new node, the network partially reduces the activation levels in the original set of nodes, leaving a faint trace as a history of the attentional sequence.

DISCUSSION

We have designed this visualization system with its various interfaces both as an existence proof that such a system could be devised and as an illustration of our proposed methodology for structuring information. We mentioned above the fact that a variety of mental maps exist for each data set, both within and across individuals. A further development would be an intelligent activation spreading network, i.e. one which contained the full repertoire of different mental models for a given data set and chose the best model based on the user's behavior. Such a repertoire could be described as a "hyper-mental-map" and could likely formed by gathering MDS and TM data on the mental mod-

Goal: Kids' Tour

of Boston

' 1

Sports Museum

� / \enway Park

�/ Chnd='• •• :.rn

l

s�, """=

Magic Show l

� Salem

Figure 10: Schematic diagram of a regional activation spreading network. Activation energy spreads in the direction of the arrows initiated by the Goal: Kids' Tour of Boston.

418

els as stimuli themselves.

We have proposed and implemented a method of organizing information space around cognitive maps. We plan next to explore the degree to which this method can be used with more generalized domains of knowledge, but examples from previous MDS and TM papers lends us hope that the method could become generally applicable. By accommodating an individual's search context (e.g. through paths or regions) and his or her particular model of a domain (e.g. a particular MDS or TM model), one can offer both a more personalized tour of the information space and a more easily absorbable mass of information.

ACKNOWLEDGMENTS

This work was in part sponsored by ARPA, JNIDS, NYNEX and Alenia. However, the views and conclusions expressed here are those of the authors and do not necessarily represent that of the sponsors.

REFERENCES

l . Anderson, J. A. Spreading activation theory of memory. Journal of Learning and Verbal Behavior 22, 261-295,1983.

2. Atkinson, R.C. & Shiffrin, R.M. The control of shortterm memory. Scientific American, 225, 82-90, 1971.

3. Craik, F.I.M. & Lockhart, R.S. Levels of processing: Aframework for memory research. Journal of VerbalLearning and Verbal Behavior, 11, 671-684, 1972.

4. Gilbert, S. A. & Richards, W. Using trajectory mappingto analyze musical intervals. Proceedings of the SixteenthAnnual Conference of the Cognitive Science Society.1994.

5. Johnson-Laird, P.N. Mental models. Cambridge, MA:Harvard University Press, 1983.

6. Kruskal, J.B. kyst2a. AT&T Bell Laboratories,Computing Science Research Center, Netlib repository of software. http://plan9.att.com/netlib/mds/index.html.

i

APRIL 13-18, 1996 Cl" l | 9 6

7. Kruskal, J.B. Nonmetric multidimensional scaling: A numerical approach. Psychometrika, 29, 28-42, 1964.

8. Lokuge, I. Dynamic magical environments: Engaging interaction based on the art of illusion. Masters Thesis, MIT Media Lab, 1995. To receive a copy email docsOmit.edu or call MITDocument Services at 6171253-5650.

12.

13.

Maes, P. Situated agents can have goals in Designing Autonomous Agents: Theory and Practice from Biology to Engineering and Back, ed. P. Maes, MIT Press/Bradford Books, 1992.

Richards, W. & Koenderink, J.J. Trajectory mapping ("TM"): A new non-metric scaling technique. Proceedings of European Conference on Visual Perception, Edinburgh (Abstract), 1993. In press Perception.

9. Lokuge, I., Gilbert, S.A. & Richards, W. Visualizing information spaces: A tour through Boston. Presented at European Conference on Visual Perception '95, Ttibingen. Abstract in Perception, 24 Supplement, p. 76.

14. Saint-Arnaud, N. Class~cafion of Sound Textures. Masters Thesis, MIT Media Lab, 1995. To receive a copy email d o c s @ m i t . e d u or call MITDocument Services at 617/253-5650.

10. Lokuge, I. & Ishizaki, S. GeoSpace: An interactive visu- alization system for exploring complex information spaces. Proceedings of ACM SIGCHI'95, Denver, Colorado, 1994.

15. Shepard, R.N. The analysis of proximities: Multi- dimensional scaling with an unknown distance func- tion. I. & II. Psychometrika, 27, 125-140, 219-246, 1962.

11. Lynch, K. Image of the City. Cambridge, MA: MIT Press, 1960.

16. Torgerson, W.S. Multidimensional scaling: I. Theory and method. Psychometrika, 4, 401-419, 1952.

419