1

Organic interaction technologies:

from stone to skin

By Jun Rekimoto (Professor in the Interfaculty Initiative in Information Studies at The University of Tokyo, and

a director of the Interaction Laboratory at Sony Computer Science Laboratories, Inc. in Tokyo.)

Ubiquitous Computing 2014/05/12 #3

Naemura Laboratory M1 49-146317 Takefumi HIRAKI

2

Today's Theme

● Know what is the Organic UI (OUI).● Study the examples of OUI.● Understand the research issues of OUI.● Think what is the future of interaction”. “

3

Today's Theme - (1)

● Know what is the Organic UI (OUI).● Study the examples of OUI.● Understand the research issues of OUI.● Think what is the future of interaction”. “

4

What is the Organic UI ?

● Interfaces which seek to incorporate human manipulation skills into human-computer interaction.

● They more closely resemble natural human-physical and human-human interaction than traditionals.

5

Traditional UI v.s. Organic UI [1]

6

Today's Theme - (2)

● Know what is the Organic UI (OUI).● Study the examples of OUI.● Understand the research issues of OUI.● Think what is the future of interaction”. “

7

HoloWall (1997) - About

● Camera-based interactive wall/table system.

● By using IR reflection of objects, it can distinguish multiple objects.

Sensor configuration [1]

8

HoloWall - Applications

Ping-pong game demo [1]

9

SmartSkin (2002) - About

SmartSkin [2] Sensor configuration [2]

10

SmartSkin - Prototype(1)

Top: potential field

Bottom: arms on a table and a corresponding potential field [2]

Interactive Table Prototype with SmartSkin [2]

11

SmartSkin - Interaction techniques(1)

● The simple emulation of a mouse-like interface

● Shape-based object manipulation

– The potential field created by the hand’s proximity to the table is used to move objects.

Mouse emulation by using calculated hand position (Drug) [2]

Shape-based object manipulation [2]

12

SmartSkin - Prototype(2)

Gesture recongnition Pad Prototype by SmartSkin [2] Gestures and corresponding

sensor values [2]

13

SmartSkin - Interaction techniques(2)

● Interaction systems by using fingers and hand gestures.

● Zooming, Scrolling, and choosing the menu commands by palm and finger position.

Application of multiple finger user interface [2]

14

DiamondTouch (2003) - About

DiamondTouch [3] Surface configuration [3]

15

DiamondTouch - Principles

Sensor configuration [3] Equivalent Circuit [3]

16

DiamondTouch - Merits

● Multipoint: Detects multiple, simultaneous touches.● Identifying: Detects which user is touching each point.● Debris Tolerant: Objects left on the surface do not interfere with

normal operation.● Durable: Able to withstand normal use without frequent repair

or re-calibration.● Unencumbering: No additional devices should be required for use.● Inexpensive to manufacture.

17

DiamondTouch - Demerits

● If two X and two Y coordinates are given, the system cannot tell if the intended touches are (X1, Y1) and (X2, Y2) or (X1, Y2) and (X2, Y1).

● This ambiguity is not a problem when performing simple interactions.

18

(cf.)SmartSkin v.s. DiamondTouch

Sensor configuration of SmartSkin [2]

Sensor configuration of DiamondTouch [3]

● SmartSkin: Being able to detect many points (includes finger points and palms).

● DiamondTouch: Being able to identify each users.

19

PreSense (2006) - AboutⅡ

PreSense [4]Ⅱ Sensor configuration [4]

20

PreSense - PrinciplesⅡ

The graph of multi-level button operation and corresponding pressure value traces [4]

21

PreSense - ApplicationsⅡ

Sample applications [4]

● Scrolling by sliding a finger on a touchpad and zooming (in and out) by bi-directional finger pressure.

● Only scrolling but speed is controlled by bi-directional finger pressure

22

Today's Theme - (3)

● Know what is the Organic UI (OUI).● Study the examples of OUI.● Understand the research issues of OUI.● Think what is the future of interaction”. “

23

Research issues of organic UIs

● There are three Research issues.

– Interaction techniques for Organic UIs

– Stone(Tool) v.s. skin

– Other modalities for interaction

24

Interaction techniques for Organic UIs

● GUI v.s. OUIs(Organic UIs)

● Some new techniques

– As-Rigid-As-Possible Shape Manipulation [5]

GUI OUIs

Interface Mouse (MultiTouch)

Interaction tech(first time) Point at objects Zooming

Interaction tech(next time)

Pop-up menus / scrollbars NONE!!!

25

As-Rigid-As-Possible Shape Manipulation [5]

● Interactive system that lets a user move and deform a two-dimensional shape

● With a multiple-point input device, even beginners can easily move, rotate, and deform shapes at will.

Sample applications [6]

26

Stone(Tool) v.s. skin

● TUIs(Tangible UIs) v.s. OUIs(Organic UIs)

TUIs OUIs

Action Physical manipulation Directlyinteraction

Compatibillity to application Domain-specific Less application-oriented

Being compared to real world Physical instruments Direct contact

Conclusion Logical, manipulation-oriented Emotional, communication-oriented

27

Other modalities for interaction

● Now : mainly hands● Should be able to use other parts.

– Eye gaze, Blowing, ...● Controllable, Conveying emotion

● BYU-BYU-View System [7]

– Adds air movement in the interaction.

– Delivering information directly to the skin.

28

Today's Theme - (4)

● Know what is the Organic UI (OUI).● Study the examples of OUI.● Understand the research issues of OUI.● Think what is the future of interaction”. “

29

What is the future of interation” ?“

● “interaction”– Now : between a human and a conputer

– Next : between the real (physical) world and computers● Controlling more physical experience (illumination, air, temperature,

humidity, energy)

● ex) Stabilizing room condition by reacting to human gesture.● “Future interactive systems may more seamlessly interact with and

control our physical environments.”

30

References

1. Rekimoto, J. Organic interaction technologies: from stone to skin. Commun. ACM 51, 6. Jun. 2008. 38-44.

2. Rekimoto, J. SmartSkin: An infrastructure for freehand manipulation on interactive surfaces. In Proceedings of the ACM Conference on Computer-Human Interaction (Minneapolis, MN, Apr. 20–25). ACM Press, New York, 2002, 113–120.

3. Dietz, P. and Leigh, D. DiamondTouch: A multiuser touch technology. In Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology (Orlando, FL, Nov. 11–14). ACM Press, New York, 2001, 219–226.

4. Rekimoto, J. and Schewsig, C. PreSenseII: Bi-directional Touch and Pressure Sensing Interactions with Tactile Feedback. CHI 2006 extended abstracts on Human factors in computing systems, 2006, 1253-1258.

5. Igarashi, T., Moscovich, T., and Hughes, J. As-Rigid-As-Possible Shape Manipulation. In Proceedings of the SIGGRAPH Conference (Los Angeles, July 31–Aug. 4). ACM Press, New York, 2005, 1134–1141.

6. http://www-ui.is.s.u-tokyo.ac.jp/~takeo/research/rigid/anim.gif

7. Sawada, E., Ida, S., Awaji, T., Morishita, K., Aruga, T., Takeichi, R., Fujii, T., Kimura, H., Nakamura, T., Furukawa, M., Shimizu, N., Tokiwa, T., Nii, H., Sugimoto, M., and Inami, M. BYU-BYU-View: A Wind Communication Interface. In the Emerging Technologies Exhibition at SIGGRAPH (San Diego, Aug. 5–9, 2007); www.siggraph.org/s2007/attendees/etech/4.html.