Touch Research

A project for Communication Design M1

HTWG Constance

Summer Term 2008

Part 3: Reading a Research Paper

How Bodies Matter: Five Themes for Interaction Design, DIS 2006, June 26–28, 2006

With the graphical user interface (GUI) so popular these days ...

... this reduced “body” symbolizes how we use our computers. With

keyboard (fingers),

mouse (fingers),

monitor (eyes), and

speakers (ears).

Are richer interaction paradigms possible?

The tangible user interface (TUI)!

A tangible user interface (TUI) is a user interface in which a person interacts with digital information through the physical environment. The initial name was graspable user interface, which no longer is used.

Image: Nintendo Wii

Bodies matter!

Consider riding a bicycle: one is simultaneously navigating, balancing, steering, and pedaling; yet it is not possible for bicyclists to articulate all of the nuances of an activity that they successfully perform.

Perhaps the most remarkable aspect of this is that riding a bicycle is just one of thousands of activities that our bodies can do.

Our physical bodies play a central role in shaping human experience in the world, understanding of the world, and interactions in the world.

This paper draws on theories of embodiment — from psychology, sociology, and philosophy — synthesizing five themes we believe are particularly salient for interaction design.

Contrast the richness, subtlety, and coordination of tasks at several levels of concern that bicycling offers with the graphical user interface that we use today.

Douglas C. Engelbart, oN-Line System (NLS) and computer mouse, 1960s

A revolutionary computer collaboration system designed by Douglas Engelbart and the researchers at the Augmentation Research Center (ARC) at the Stanford Research Institute (SRI).

The first system to employ the practical use of hypertext links, the mouse (co-invented by Engelbart and colleague Bill English), raster-scan video monitors, information organized by relevance, screen windowing, presentation programs, and other modern computing concepts.

Features: The mouse, 2-dimensional display editing, in-file object addressing, linking, hypermedia, outline processing, flexible view control, multiple windows, cross-file editing, integrated hypermedia email, hypermedia publishing, document version control, shared-screen teleconferencing, computer-aided meetings, formatting directives, context-sensitive help, distributed client-server architecture, uniform command syntax, universal "user interface" front-end module, multi-tool integration, grammar-driven command language interpreter, protocols for virtual terminals, remote procedure call protocols, compilable "Command Meta Language”

Altair 8800, 1975

The MITS Altair 8800 was a microcomputer design from 1975 based on the Intel 8080 CPU and sold by mail order through advertisements in Popular Electronics, Radio-Electronics and other hobbyist magazines.

The Altair is widely recognized as the spark that led to the microcomputer revolution of the next few years: The computer bus designed for the Altair was to become a de facto standard in the form of the S-100 bus, and the first programming language for the machine was Microsoft's founding product, Altair BASIC.

Programming the Altair was an extremely tedious process. The user toggled the switches to positions corresponding to an 8080 microprocessor instruction or opcode in binary, then used an 'enter' switch to load the code into the machine's memory, and then repeated this step until all the opcodes of a presumably complete and correct program were in place.

When the machine first shipped the switches and lights were the only interface, and all one could do with the machine was make programs to make the lights blink.

Xerox Star, 1981

The Star workstation, officially known as the Xerox 8010 Information System, was introduced by Xerox Corporation in 1981. It was the first commercial system to incorporate various technologies that today have become commonplace in personal computers, including a bitmapped display, a window-based graphical user interface, icons, folders, mouse, Ethernet networking, file servers, print servers and e-mail.

The key philosophy of the user interface was to mimic the office paradigm as much as possible in order to make it intuitive for users. The concept of WYSIWYG was considered paramount. Text would be displayed as black on a white background, just like paper, and the printer would replicate the screen using InterPress, a page description language developed at PARC.

The user would see a desktop that contained documents and folders, with different icons representing different types of documents. Clicking any icon would open a window. Users would not start programs first (e.g. a text editor, graphics program or spreadsheet software), they would simply open the file and the appropriate application would appear.

Apple iMac, 2008 edition

Members of the Apple Lisa engineering team saw Star at its introduction at the National Computer Conference (NCC '81) and returned to Cupertino where they converted their desktop manager to an icon-based interface modeled on the Star. Among the developers of the Gypsy editor, Larry Tesler left Xerox to join Apple in 1980 and Charles Simonyi left to join Microsoft in 1981 (whereupon Bill Gates spent $100,000 on a Xerox Star and laser printer),[9] and several other defectors from PARC followed Simonyi to Microsoft in 1983.

Some people feel that Apple, Microsoft, and others plagiarized the GUI and other innovations from the Xerox Star, and believe that Xerox didn't properly protect its intellectual property.

Dan Saffer, reviewing the paper:

“With the current keyboard-mouse-monitor set-up, we do every task, no matter if it is writing a paper or editing a movie or even playing a game, all the same way. Pointing, clicking, dragging and dropping, etc. The work has become ‘homogenized‘.”

http://www.odannyboy.com/blog/new_archives/2008/05/review_five_the.html

Five themes for interaction design:

1. Thinking through doing 2. Performance 3. Visibility 4. Risk 5. Thickness of practice

1. Thinking through doing

describes how thought (mind) and action (body) are deeply integrated and how they co-produce learning and reasoning. Individual corporeality.

Unlike theories of information processing and human cognition that focus primarily on thought as something that only happens in the head, theories and research of embodied cognition regard bodily activity as being essential to understanding human cognition.

Dan Saffer: “There are a lot of skills you simply cannot learn by reading or listening alone. You have to try them out. Gestures aren't just for embellishment to communication, they can also be an aid to learning and understanding. Manipulation of items allows for greater understanding of the item. Artifacts have their own characteristics, and their "backtalk" uncovers problems or can suggest new designs.”

1.1 Learning Through Doing

Montessori toys employ bodily engagement with physical objects to facilitate active learning.

Being able to move around in the world and interact with pieces of the world enables learning in ways that reading books and listening to words do not.

Having the hands stuck on a keyboard are likely to hinder the thinking and communication.

1.2 The Role of Gesture

From studies of gesturing in face-to-face interactions, we know that people use gesture to conceptually plan speech production and to communicate thoughts that are not easily verbalized.

Gesturing has been shown to lighten cognitive load for both adults and children.

Less constraining interaction styles are likely to help users think and communicate.

1.3 Epistemic Action

Distinguishing pragmatic action — manipulating artifacts to directly accomplish a task — from epistemic action — manipulating artifacts to better understand the task’s context.

Puzzle players manipulate pieces to understand how different options would work.

1.4 Thinking through Prototyping

Reflective practice, the framing and evaluation of a design challenge by working it through, rather than just thinking it through, points out that physical action and cognition are interconnected.

Successful product designs result from a series of “conversations with materials.”

The epistemic production of concrete prototypes provides the crucial element of surprise, unexpected realizations that the designer could not have arrived at without producing a concrete manifestation of her ideas.

1.5 On Representation

The representation of a task can radically affect our reasoning abilities and performance.

Tangibility offers both direct familiarity and a set of common metaphors to leverage in interaction.

The most common stated purpose of tangibility is that these interfaces provide “natural” mappings and leverage our familiarity with the real world, e.g., virtual objects are positioned in virtual space by moving physical handles in physical space.

2. Performance

describes the rich actions our bodies are capable of, and how physical action can be both faster and more nuanced than symbolic cognition . Individual corporeality.

One of the most powerful human capabilities relevant to designers is the intimate incorporation of an artifact into bodily practice to the point where people perceive that artifact as an extension of themselves; they act through it rather than on it.

While much of the recent TUI literature has focused on “walk up and use” scenarios which require a low use threshold, this section describes how designing for skilled bodies can yield interfaces for expert performance. Physical interfaces with dedicated (i.e., spatially multiplexed) controls and dedicated actions can leverage this skill to improve interaction speed and reliability.

Dan Saffer: “We should design products for expert users, able to use their hands and motor memory to perform action-centered skills. Thinking can be too slow; experiential cognition (learned skillful behavior like driving a car) can be more rapid and powerful than reflective cognition.”

2.1 Action-centered Skills

The tacit knowledge that many physical situations afford plays an important role in expert behavior.

We draw attention to the importance of tacit knowledge because computerization can, often accidentally, inhibit it.

2.2 Motor Memory

We are able to sense, store and recall our own muscular effort, body position and movement to build skill.

It is this motor, or kinesthetic, memory that is involved in knowing how to ride a bicycle, how to swim, how to improvise on the piano.

Traditional GUI interfaces employ the same bodily actions for a wide variety of tasks - this universality is both a strength and a weakness.

For any given application, kinesthetic memory can only be leveraged to a limited extent since the underlying actions are the same across applications.

2.3 Reflective Reasoning

Beyond reliability and robustness of kinesthetic recall, speed of execution also favors bodily skill for a class of interactive systems that require tight integration of a human performer “in the loop.”

Many daily actions such as driving a car or motorcycle, operating power tools, or engaging in athletic activities require complex yet rapid bodily responses for which planning through explicit cognition is simply too slow.

Tangible interfaces that engage the body can leverage body-centric experiential cognition. To date, computer game controllers have been the most commercially successful example of such interfaces.

2.4 Hands

Simultaneously a means for complex expression and sensation:

Hands allow for complicated movement but their skin also has the highest tactile acuity of our extremities.

Many of the complex motions that we perform are bi-manual and asymmetric.

3. Visibility

describes the role of artifacts in collaboration and cooperation. Social affordance.

The extent to which the activities of a practice are made visible to colleagues and onlookers through the performance of the activity.

Dan Saffer: “Through the performance of an activity, that activity can be made visible to others easily, so that collaboration and situated learning can occur spontaneously. ”

Visibility facilities

3.1 Coordination

The visibility provided through collocated practice with task-specific artifacts is also successful in supporting synchronous collaboration, and can be especially useful in mission-critical systems.

3.2 Situated Learning

The invisibility of work practice that the GUI has brought about inhibits peripheral participation.

A child watching to see what an adult is doing in front of a laptop, and then copying those motions is not learning anything.

With the graphical interface, there is no mechanism to be aware of the practices of experts; it all looks the same.

That’s what

3.3 Live Performance

is about

The value we place in visibility of creative production is exemplified by live musical performance.

While the music itself is more intricate and polished in studio recordings, audiences still pack concert venues because live perform-ances permit listeners to witness the act of performance as well as co-produce the event (musician and audience respond to each other through mutual feedback).

4. Risk

explores how the uncertainty and risk of physical co-presence shapes interpersonal and human-computer interactions. Social affordance.

Dan Saffer: “Most products are designed to decrease risk, but retaining some risk can be beneficial. With risk comes trust, responsibility, and attention.”

4.1 Physical Action is Characterized by Risk

Risk is having to choose an action which cannot be undone while the consequences of the action are not fully knowable ahead of time.

One cannot undo a social faux pas in face to face interactions; technology mitigates against this risk: one can delete sentences before sending them to friends over IM or email.

4.2 Trust and Commitment

Though risk can make people feel more anxious about interactions with others, it can also engender the kind of trust necessary for successful distance collaborations.

Situations that involve more risk can also stimulate more committed involvement by participants of the interaction.

Painting in watercolor requires more commitment to each stroke than working in Adobe Photoshop.

4.3 Personal Responsibility

There are situations where the decision-makers should not be subject to the overwhelming repercussions of their decisions, e.g., natural disaster response planning.

4.4 Attention

Situations of higher risk cause people to feel more emotion-ally negative and, therefore, more focused, paying closer attention to detail, while situations of low risk allow people to feel more emotionally positive, relaxed, curious, and creative.

5. Thickness of practice

suggests that because the pursuit of digital verisimilitude is more difficult than it might seem, embodied interaction is a more prudent path.

Dan Saffer: “Because there is so much benefit to the real world, we should be careful with replacing physical artifacts with digital ones. The best case scenario is to augment the physical world with digital behaviors, and thus "admitting the improvisations of practice that the physical world offers."”

5.1 Final Scratch

From a design perspective, solutions that carefully integrate the physical and digital worlds — leaving the physical world alone to the extent possible — are likely to be more successful by admitting the improvisations of practice that the physical world offers.

Clearly, the digital world can provide advantages. To temper that, we argue that because there is so much benefit in the physical world, we should take great care before unreflectively replacing it.

5.2 “Embodied Virtuality” Rather Than Virtual Reality.

Designing interactions that are the real world instead of ones that simulate or replicate it hedges against simulacra that have neglected an important practice.

“In this paper we developed our view of the affordances of physicality and concreteness for the design of interactive systems.

We believe the five themes presented in this paper will be of value both generatively — helping designers come up with new solutions — and for evaluation — providing a rich set of axes for analyzing the benefits of systems.”

Scott R. Klemmer, Björn Hartmann, Leila Takayama

Credits

Via Creative Commons on flickr:

Javier Martínez www.flickr.com/hyoga/1165367241/

raysto www.flickr.com/raysto/252630833/

duncan c www.flickr.com/duncan/92623025/

Kit Cowan www.flickr.com/kitcowan/712113879/

Chel's piccies www.flickr.com/mich_chel/1858616605/

Stefan Sonntag www.flickr.com/zerega/1029076197/

foreversouls/ tree & j hensdill www.flickr.com/foreversouls/2083165757/

Derrick Mealiffe www.flickr.com/dmealiffe/171720479/

Thomas Hawk www.flickr.com/thomashawk/2492298772/

François @ Edito.qc.ca www.flickr.com/francois/1161267539/

Philip Milne www.flickr.com/pamilne/1392285543/

Tom Adriaenssen www.flickr.com/inferis/266391949/

missionbycicles www.flickr.com/mission_bicycles/2108803461/

Chris Gansen www.flickr.com/daychokesnight/2041048614/

Connie Arida/ Connie Sec www.flickr.com/conniesec/2211700732/

Bruce Beh www.flickr.com/brucebeh/120075758/

all in green/ Shana www.flickr.com/isoldesmom/446843362/

Via Creative Commons on flickr:

SOCIALisBETTER www.flickr.com/27620885@N02/2597329151/

"Cowboy" Ben Alman www.flickr.com/rj3/2562634563/

Loran Tatooine www.flickr.com/noloran/641931694/

Luca Casamassima www.flickr.com/tyler89/2364065361/

Axel Pfaender www.flickr.com/axor/2295802927/

rickz www.flickr.com/rickz/3071093/

Jenny Downing www.flickr.com/jenny-pics/2583485800/

Daniel Y. Go www.flickr.com/danielygo/1781777706/

Marc A. Garrett www.flickr.com/since1968/9923894/

Cortiça CenaCarioca www.flickr.com/cenacarioca/355888627/

Cheon Fong Liew www.flickr.com/liewcf/894035077/

Luca Mascaro www.flickr.com/lucamascaro/523822075/

Additional thanks to:

Wonderbrains.com

Apple.com

OLPC

Perceptive Pixel

How Bodies Matter: Five Themes for Interaction Design

Scott R. Klemmer, Björn Hartmann, Stanford University HCI Group, Computer Science Department

Leila Takayama, Stanford University CHIMe Lab, Communication Department

DIS 2006, June 26–28, 2006, University Park, Pennsylvania, USA. Copyright 2006 ACM 1-59593-341-7/06/0006.

http://hci.stanford.edu/publications/2006/HowBodiesMatter-DIS2006.pdf

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage.

Dan Saffer, reviewing the paper

http://www.odannyboy.com/blog/new_archives/2008/05/review_five_the.html

All the rest

http://wikipedia.org

University of Applied Sciences Constance, Faculty for Communication Design, Project “Touch Research”

http://www.htwg-konstanz.de

http://www.kd.fh-konstanz.de/dina8/daten_e.php?wodenn=will

http://www.felgner.ch/2008/04/touch_research.html