1

Beyond Mouse & Keyboard: Post-WIMP and Novel Forms of Interaction

Jacques ChuekeOpen Day, Centre for HCI Design

London, UK, April 2011

Master in Design, PUC-Rio, RJ, BrazilPhD Researcher at the Centre for HCI DesignFaculty of Informatics, City University London

2

“The most profound technologies are those that disappear. They wave themselves into the fabric of everyday life until they are indistinguishable from it.” Weiser (1991)

Is this possible?Is this desirable?

3

It’s not supposed to be like this…

Gmail Motion, April 2011

4

• In computing post-WIMP refers to work on user interfaces, mostly graphical user interfaces, which attempt to go beyond the paradigm of Windows, Icons, Menus and a Pointing device. E.g. virtual (VR) and augmented reality (AR), Tangible User-Interface (TUI), ubiquitous and pervasive computing, context-aware computing, mobile devices, console gaming, Affective Computing, Reality-Based Interactions (RBI), Organic-User Interfaces (OUI), Brain-Computer Interface (BCI) and Natural-User Interface (NUI).

Post-WIMP

BumpTop 3D, 2007 - 2010

5

• Defined by van Dam as interfaces “containing at least one interaction technique not dependent on classical 2D widgets such as menus and icons”. Ultimately it will involve all senses in parallel, natural language communication and multiple users. Communications of ACM, 1997

Post-WIMP

MIT Media Lab: DepthJS – 2011

6

• Regarding the visual UI (User-Interface), are Perceptible Affordances within Post-WIMP interfaces (with NUI modes of interaction) efficiently signaling possible interactions? Regular WIMP-GUI might not be accomplishing this task efficiently.

The Question

7

• "Affordance" means what you can do to an object. For example, a checkbox affords turning on and off, and a slider affords moving up or down. "Perceived affordances" are actions you understand just by looking at the object, before you start using it (or feeling it, if it's a physical device rather than an on-screen UI element). All of this is discussed in Don Norman's book The Design of Everyday Things. (a.k.a POET: Psychology of Everyday Things). Jakob Nielsen's Alertbox, February 19, 2008: Top-10 Application-Design Mistakeshttp://www.useit.com/alertbox/application-mistakes.html

Perceptible Affordances • The concept of affordance has been used in HCI to solve problems related to the usability of designed systems. The concept was originally coined by Gibson (1986) and introduced to the HCI field by Norman (1988) and was further appropriated by Gaver (1991), Bærentsen & Trettvik (2002), amongst others. Vyas, D. (2006)

8

Post-WIMP GUI [INTERFACE LAYER]

PERCEPTIBLE AFFORDANCE

OUTPUT

INPUT

TUI/NUI/RBI/VR/AR/BMI/OUI [MODE OF INTERACTION LAYER]

Perceptible Affordances in Post-WIMP

- LESS SYMBOLIC- MORE INTUITIVE (USER)- MORE REACTIVE (COMPUTER)

9

• The paradigms that define GUI and establish the conventions for manipulation, besides the presentation of the visual interface itself -within different digital environments (e.g. Mac and Windows OS and the WWW) - do not match and better display/encompass the most suitable use of Post–WIMP interactions.

Hypothesis

Pranav Mistry, inventor of SixthSense, 2009

10

(…) The notion of the invisible interface correctly identifies the inflexible obtrusiveness of conventional interfaces as problematic. Embodied interaction provides some conceptual tolls for understanding how the interface might move into the background without disappearing altogether. Dourish (2005: 202-203)

11

TUI: Embodiment• Activity Centered Design: The tool is the way (Norman, 2005) + Embodiment (Heidegger ‘being in the world’ and Dourish): We’re the tool.

Microsoft Surface, 2007

12

NUI: Intuitive• Assistive Technology: Tobii P-10 at the SmartLab (UEL).

Tobii P-10 equipment, Oct. 2010

13

Social Aspects: Installations

Live Wall 02, Jacques Chueke. Plymouth, 2010

Live Wall 01, Jacques Chueke. Plymouth, 2010

• Live Wall: The objectives are to create an installation, where I'm interested on the ethnographic and sociological aspects of the experience that takes place between the beholder and the reactive environment. All interaction will occur with regular gestures, eye gaze, movement and sound issued by the observer/participant.

14

Digital Art [INTERFACE LAYER]

SYMBOLIC / SYNESTHESIC / ABSTRACT

OUTPUT

INPUT

TUI/NUI/RBI/VR/AR/BMI/OUI [MODE OF INTERACTION LAYER]

Perceptible Affordances in Digital Art

- No Prior-Learning, besides knowledge of the world- Reactive

15

Conclusions

• We propose a view that identifies some fraction of a user interface as based on the Post-WIMP theme (1) plus some other fraction that provides computer-only functionality (2) that is not realistic. As a design approach or metric, the goal would be to make the first category as large as possible and use the second only as necessary, highlighting the tradeoff explicitly. Jacob at al, 2008

• Novel technologies for interaction will be used to elicit user exploration of new and visually unfamiliar digital interfaces to understand how users visually scan such interface to obtain the gist of its interactive potential. Perceptible Affordances theory, within HCI and Cognitive Psychology will be used to better understand those issues.

16

BibliographyBeaudouin-Lafon, M. (November 2000). "Instrumental Interaction: An Interaction Model for Designing Post-WIMP User Interfaces". CHI '00: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. The Hague, The Netherlands: ACM Press. pp. 446–453. doi:10.1145/332040.332473. ISBN 1-58113-216-6. http://www.daimi.au.dk/CPnets/CPN2000/download/chi2000.pdf. 

Breeze, James. Eye Tracking: Best Way to Test Rich App Usability. UX Magazine, access on 25 November 2010. (http://www.uxmag.com/technology/eye-tracking-the-best-way-to-test-rich-app-usability)

Buxton, W. (2001). Less is More (More or Less), in P. Denning (Ed.), The Invisible Future: The seamless integration of technology in everyday life. New York: McGraw Hill, 145–179 ITU Internet Reports 2005: The Internet of Things – Executive Summary.

Dam, A. (February 1997). "POST-WIMP User Interfaces". Communications of the ACM (ACM Press) 40 (2): pp. 63–67. doi:10.1145/253671.253708. 

Dourish, P. Where the Action Is: The Foundations of Embodied Interaction. A Bradford Book: The MIT Press, USA, 2004.

Gaver, W. Technology Affordances. Copyright 1991 ACM 0-89791-383-3/91/0004/0079.

Gentner, D. and Nielsen, J. (April 1993). "The Anti-Mac Interface". Communications of the ACM (ACM Press) 39 (8): pp. 70–82. http://www.useit.com/papers/anti-mac.html. 

Jacob, R. et al. (2008). "Reality-Based Interaction: A Framework for Post-WIMP Interfaces". CHI '08: Proceedings of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems. Florence, Italy: ACM. pp. 201–210. doi:http://doi.acm.org.ezproxy.lib.ucf.edu/10.1145/1357054.1357089. ISBN 978-1-60558-011-1.

17

BibliographyMcGrenere, J., Ho, W. (2000). Affordances: Clarifying and Evolving a Concept. Procs. of Graphic Interfaces 2000, Montreal, May 2000.

McNaughton, J. Utilizing Emerging Multi-touch Table Designs. Technology Enhanced Learning Research Group - Durham University. TR-TEL-10-01.

Nielsen, J. (April 1993). "Noncommand User Interfaces". Communications of the ACM (ACM Press) 36 (4): pp. 83–99. doi:10.1145/255950.153582. http://www.useit.com/papers/noncommand.html. 

Norman, D. (1999). Affordance, Conventions and Design. In ACM Interactions, (May + June, 1999), 38-42. Picard, R. Affective Computing. The MIT Press, Cambridge, Massachusetts. London, England, 1998.

Ramduny-Ellis, D.; Dix, A.; Hare, J.; Gill, S. Physicality: Towards a Less-GUI Interface (Preface). Procs. Third International Workshop on Physicality. Cambridge, England, 2009.

Sorensen, M. Making a Case for Biological and Tangible Interfaces. Proceedings of the Third International Workshop on Physicality. Cambridge, England, 2009.

Sternberg, R. Cognitive Psychology. Wadsworth, Cengage Learning. Belmont, CA, USA, 2009, 2006.

Vyas, D., Chisalita, C. Veer, G. Affordance in Interaction. ECCE '06 Proceedings of the 13th Eurpoean conference on Cognitive ergonomics: trust and control in complex socio-technical systems. ACM New York, NY, USA ©2006 ISBN: 978-3-906509-23-5