Multi Touch & Gesture - Event Interface and Types

June, 2010 TC Meeting in Minneapolis, MN

Eunil SEO / seoei2@tobesoft.com Siman KIM / smkim@tobesoft.com

Sangchual CHA / chasc@tobesoft.com

1

Agenda

• UI Event - Mouse Event in W3C

• Multi Touch and Gesture

• Event Issues

• W3C DOM Event Model Extension

2

Purpose of this presentation

• To collect current technical status on multi-touch and gesture event.

• To reveal issues about multi-touch and gesture event.

• To suggest multi-touch and gesture event model with architecture, interface and types.

3

Event Model in W3C

• Document Object Model (DOM) Level 3 Events Specification (Sep. 2009)

http://www.w3.org/TR/DOM-Level-3-Events/

• Event Architecture• Event Interface• Event Module

– Event Types– Event Module Definition

4

DOM Lvl3 Event Architecture

5

• Event Dispatch Mechanism• Event Flow

• Capture Phase• Bubble Phase• Target Phase

DOM Lvl3 Event Interface

6

http://www.w3.org/TR/DOM-Level-3-Events/#dom-event-architecture

DOM Lvl3 MouseEvent Interface

interface MouseEvent : UIEvent { readonly attribute long screenX; readonly attribute long screenY; readonly attribute long clientX; readonly attribute long clientY; readonly attribute boolean ctrlKey; readonly attribute boolean shiftKey; readonly attribute boolean altKey; readonly attribute boolean metaKey; readonly attribute unsigned short button; readonly attribute EventTarget relatedTarget; … …};

7

DOM Lvl3 MouseEvent Interface Cont’d

interface MouseEvent : UIEvent { … … void initMouseEvent(in DOMString typeArg, in boolean canBubbleArg, in boolean cancelableArg, in views::AbstractView viewArg, in long detailArg, in long screenXArg, in long screenYArg, in long clientXArg, in long clientYArg, in boolean ctrlKeyArg, in boolean altKeyArg, in boolean shiftKeyArg, in boolean metaKeyArg, in unsigned short buttonArg, in EventTarget relatedTargetArg); };

8

DOM Lvl3 Event Types

9

Event Iterfaces• MouseEvent

• TextEvent

• KeyboardEvent

• CompositionEvent

• MutationName Event

• MouseWheelEvent

• WheelEvent

Event Iterfaces• MouseEvent

• TextEvent

• KeyboardEvent

• CompositionEvent

• MutationName Event

• MouseWheelEvent

• WheelEvent

Mouse Event Types• click event

• dblclick event

• mousedown event

• mouseenter event

• mouseleave event

• mousemove event

• mouseover event

• mouseout event

• mouseup event

• mousewheel Event

• wheel Event

Mouse Event Types• click event

• dblclick event

• mousedown event

• mouseenter event

• mouseleave event

• mousemove event

• mouseover event

• mouseout event

• mouseup event

• mousewheel Event

• wheel Event

RIA Event Spectrum

10

Device

O/S

RIA Platform

Application

Event Interpreter

Device Event

Device Event

Device Event

Event Interpreter

Event Interpreter

Interpreted Event

Interpreted Event

* Physical device with device driver

* With Application Framework

Device

* RIA Platform includes Application FrameworkRIA Platform

RIA Event Spectrum

• Flow through Device, O/S, RIA Platform, RIA Application Framework and RIA Application.

• Device Event– Events origin from device.– Keep its context through spectrum.

• Interpreted Event– Combination of device events and interpreted events.– Represent with State Diagram.

11

New Event Issues

• Single Touch - Click vs. Tap– Click with stylus vs. Tap with Fingers – Interval vs. Duration – Point(x, y) vs. Region(x, y, width, height)

• Array of Touch Evet– Sequence of touch vs. Set of touch

• Gesture – How to understand the intension of action.– Gesture in Space (Gesture with Glove, Camera, etc)

• Shake and rotate (Gyro Sensor)12

Multi-Touch Event

• Multi-touch is an enhancement to touch-screen technology, which provides the user with the ability to apply multiple finger gestures simultaneously onto the electronic visual display to send complex commands to the device. (Wikipedia)

• Multi Touch = Sequence of Touch (Down, Up, Move)

13

Gesture Event

• Any physical movement that can be sensed and responded to by a digital system without the aid of a traditional input device such as a mouse or stylus.

• To understand user intensions by recognizing multi-touch events, from the first finger down event to the last finger up event.

• Multi-touch gesture, Mouse gesture, Motion gesture, etc

14

Sample Gestures in Touch Device

15

Tap

Two finger zoom out Two finger zoom in

Double Tap

Touch Down

Touch Up

One finger drag

Pinch zoom

Two finger tap

One finger swipe Two finger swipe

Two finger double tap

Two finger horizontal scroll Two finger

vertical scroll

Gestureworks Open Souce Gesture Library (http://gestureworks.com/about/open-source-gesture-library/

)

Gestures in iPhone O/S

• Gesture Types– Tapping (any number of tabs)– Pinching in and out (for zooming a view)– Panning or dragging– Swing (in any direction)– Rotation (fingers moving in opposite

directions)– Long Press (aks touch and hold)

16

http://developer.apple.com/iphone/library/documentation/General/Conceptual/iPadProgrammingGuide/GestureSupport/GestureSupport.html

Gesture Recognizers

http://developer.apple.com/iphone/library/documentation/General/Conceptual/Devpedia-CocoaApp/EventHandlingiPhone.html Events (iPhone OS)

Gestures in iPhone O/S cont’d

17

• Device (O/S) detect touch event• Interpret Touch Event in Cocoa

Framework• UITouch Event Class

Gestures in iPhone O/S cont’d

18

Gestures in iPhone O/S cont’d

19

• touchesBegan:withEvent: is called for touch objects in the Began phase.• touchesMoved:withEvent: is called for touch objects in the Moved phase.• touchesEnded:withEvent: is called for touch objects in the Ended phase.• touchesCancelled:withEvent: is called when some external event (e.g. Incomming

Call).

Gestures in Android

• Event Listeners– OnClickListner– OnLongClickListner– OnTouchListener

• Touch Event– public abstract boolean onTouch (View v, MotionEvent event)

• TouchDown• TouchMove• TouchUp• TouchTap

Gestures in Android cont’d

• OnGestureListeners– OnGesture– OnOnGestureCancelled– OnGestureEnded– OnGestureStarted

OnGestureStartedOnGestureStarted

OnGestureCancelledOnGestureCancelled

ReadyReady

OnGestureEndedOnGestureEnded

OnGestureOnGesture

OnTouchDownOnTouchDownOnTouchUpOnTouchTap

OnTouchUpOnTouchTap

Cancelling EventEx) CallingEvent

Cancelling EventEx) CallingEvent

OnTouchDownOnTouchUp

OnTouchMoveOnTouchTap

OnTouchDownOnTouchUp

OnTouchMoveOnTouchTap

Gestures in Android cont’d

• GestureDetector (SimpleGestureDetector)– onDoubleTap– onDown– onFling– onLongPress– onScroll– onShowPress– onSingTapUp– onGestureRotate– onGestureScroll– onGestureScale

Reference : How to use Multi-touch in Android 2 in ZDNET

Gestures in MS Windows 7• System gesture Synthesized equivalent

message• Hover (when supported) >> Mouse hover• Tap (down and up) >> Mouse left-click• Double tap (down and up twice)>> Mouse double left-

click• Press and hold (down, pause, up)>>Mouse right-click

• Drag (down, move, up) >> Mouse left-drag• Press, hold, and drag (down, pause, move, up) >>

Mouse right-drag• Select (down, move over selectable objects, up)

>> Mouse select

23

MS Windows - Panning

• Entry state: One or two fingers in contact with the screen.

• Motion: Drag, with any additional fingers remaining in same position relative to each other.

• Exit state: Last finger up ends the gesture.• Effect: Move the underlying object directly

and immediately as the fingers move. Be sure to keep the contact point under the finger throughout the gesture.

24

MS Windows - Zoom

• Entry state: Two fingers in contact with the screen at the same time.

• Motion: Fingers move apart or together (pinch) along an axis.

• Exit state: Any finger up ends the gesture or the fingers break the axis.

• Effect: Zoom the underlying object in or out directly and immediately as the fingers separate or approach on the axis. Be sure to keep the contact points under the finger throughout the gesture.

25

MS Windows - Rotate

• Entry state: Two fingers in contact with the screen at the same time.

• Motion: One or both fingers rotate around the other, moving perpendicular to the line between them.

• Exit state: Any finger up ends the gesture. • Effect: Rotate the underlying object the

same amount as the fingers have rotated. Be sure to keep the contact points under the finger throughout the gesture.

26

MS Windows - Two-finger tap

• Entry state: Two fingers in contact with the screen at the same time.

• Motion: No motion. • Exit state: Any finger up ends the

gesture. • Effect: Alternatively zooms or restores

the default view for the object between the fingers.

27

MS Windows - Press and tap

• Entry state: One finger in contact with the screen, followed by a second finger.

• Motion: No motion. • Exit state: Second finger up ends the

gesture. • Effect: Performs a right click for the

object under the first finger

28

Related Issues

• Lack of standard : Different understanding in user actions.

• Event Sharing between components : Inter-component event dispatch. (e.g. Drag & Drop)

• We need RIA standard gesture event model.

• W3C DOM Level 3 Event Model doesn’t support Gesture event. Browser War !!

29

Reference

• UXGuide - http://msdn.microsoft.com/en-us/library/aa511258.aspx

30