magPenMagnetically Driven Pen Interaction

On and Around Conventional Smartphones

Presenter: S.J. HwangPh.D. Candidate | 6th Semester

BACKGROUND

Touch screens provide intuitive and effective input. However, flat touchscreens haveseveral interaction issues: small interaction area, fat finger problem, and virtualcontrols lack tactile clues.

BACKGROUND

Pen-like interfaces provide users with higher precision and less occlusion, as well as tactile clues already familiar to most users (Engelhard, 2008)

Pen-like interface

BACKGROUND

UnmousePad(Rosenberg et al., 2009)

Papiercraft(Liao et al., 2005)

N-Tring(Engelhard et al., 2008)

Pen interaction techniques

BACKGROUND

Rolling

Expansion of the design space of pen interaction

Tilting Gripping Contacting

Pen Rolling (Bi et al., 2008)

Tilt Menu(Tian et al., 2008)

MTPen(Song et al., 2011)

Conté(Vogelet al., 2011)

BACKGROUND

Technique (Author, Year) surface customization pen customization P/A (styli)

Papiercraft (Liao et al., 2005) dot patterns IR camera Active

UnmousePad (Rosenberg et al., 2009) a force sensor grid - Passive

N-Tring (Engelhard et al., 2008) an electrostatic grid capacitive probes Active

Pen Rolling (Bi et al., 2008) Cameras Vicon markers Passive

Tilt Menu (Tian et al., 2008) - a tilt sensor Active

MTPen (Song et al., 2011) - multi-touch sensors Active

Conté (Vogelet al., 2011) A camera IR LEDs Active

Although these methods have successfully enriched users’ experiences with styli,they require either special sensory surfaces or pen customization withmechanical parts.

Previous methods for enabling pen interaction

BACKGROUND

Magnets are very cheap, small, and robust. It do not require power, andcan be easily detected wirelessly using a magnetometer alreadyembedded in modern phones. Moreover, they do not easily degradeover time.

Magnetic Input Techniques

BACKGROUND

Magnetic Input Techniques

Abracadabra(Harrison et al., 2009)

MagiWrite(Ketabdar et al., 2010)

Digital Music Performance (Ketabdar et al., 2011)

Magnetic Appcessories (Bianchi et al., 2013) GaussSense (Liang et al., 2012)

BACKGROUND

Technique (Author, Year) Task Limitation

Abracadabra(Harrison et al., 2009)

accurate selections for small screens No tactile clue

MagiWrite(Ketabdar et al., 2010) writing system No tactile clue

Digital Music Performance(Ketabdar et al., 2011) musical performance No tactile clue

GaussSense(Liang et al., 2012) Pen interaction

Need additional mechanical hardware.(a board with 192 magnetic sensors and USB connection).

Malfunctioning for ferromagnetic materials.

Magnetic Input Techniques

While these studies definitely improve usability, some lack tactile clues (i.e., drawingin the “air” as opposed to drawing on a “surface”) and some need special sensoryhardware.

OUR METHOD

a capacitive pen

1. provide tactile clues.

+a passive magnet

2. increase input expressiveness and expand input area (e.g., touchscreen).

+a magnetometer

3. using a magnetometer that already installed on current mobile devices.

OUR METHODHardware

MagPen is a capacitive pen augmented with a magnet. It is made from a plastic cylinder (8-12ø, 12-15cm) covered with conductive tape and physically connected to two conductive rubber tips (8 mm) at the pen’s extremes.

N

S

a coin-shaped magnet

conductive tape

a plastic cylinder

a conductive rubber tip

a conductive rubber tip

OUR METHODHardware

MagPen comes in three different forms: basic, identifiable, and pressure sensitive type.

OUR METHOD

Hardware – basic type

120mm

60mm

8ø

3T/5ø x 1, 500 uTesla

For the basic pen, a single coin-shaped permanent magnet (3T/5ø,about 500 uT) is embedded in the center of the pen, with the magnetfacing parallel to the screen.

OUR METHODHardware – identifiable type

For the identifiable pen, we attached magnets with different magneticproperties (intensity and position).

OUR METHODHardware – pressure-sensitive type

For the identifiable pen, we designed the pressure-sensitive pen to adapt its length depending on the pressure being applied to the tip. When pressure is applied to the tip, the inner spring is squeezed, bringing the magnet to the device (a change of about 250 mm).

OUR METHOD

Software

- We implemented a magPen toolkit on the Android 4.0.4. (SHV-E160S, Samsung Galaxy Note).

- We also built a 3D visualizer using OpenGL to examine changes inthe magnetic field.

- The inertial three-axis magnetometer sensed the magnetic field ata sampling frequency of 100 Hz.

- Used a buffer to reduce sampling noise; the movement vector fromthe last reading to its predecessor was computed and normalized.

OUR METHOD

a) detecting the orientation that the stylus is pointing to.b) selecting colors using locations beyond screen boundaries.c) recognizing different spinning gestures associated with different actions.d) inferring the pressure being applied to the pen.e) identifying various pens associated with different operational modes.

MagPen Interactions

To verify some of these techniques, five participants (2 women) aged between 28 and 34 (average 31.4, SD 2.4) were involved.

OUR METHOD(A) Determining Pen Orientation

Five participants trained our machine learning software by free drawing with bothsides of the simple pen for 10 seconds. 10% cross validation showed that ourapplication correctly determined which of the two tips was being used up to 99%(99.8%, K=0.99 using J48 Decision Tree).

by using the bipolar property of magnets

OUR METHOD

OUR METHOD(B) Dragging Gesture on the Device Frame

It is possible to estimate the position of the pen outside of the screen frame. Theadvantage of this technique is that it alleviates occlusion on the touchscreen,resulting in a larger interaction area.

by using a vector calculation that compares the incoming magnetic value to the values stored when calibrating.

OUR METHOD

OUR METHOD

(C) Spin Gesture Recognition

Spinning the pen between the fingers is a common activity among many users who are concentrating. We tried to explore how this common activity could be encoded in a rich vocabulary of recognizable gestures

OUR METHOD

OUR METHOD(C) Spin Gesture Recognition

Three different spinning gestures were classified using a machine-learning approach (J48 decision tree) with a set of features that included the average movement speed, speed variance, size of the gesture, shape, and trajectory.

OUR METHOD(C) Spin Gesture Recognition

To verify our prototype, we conducted a pilot study and sampled 10 trials for eachgesture from among five participants (5 participants × 10 trials × 3 gestures). Wethen conducted a 10% cross-validation and found that the gestures wereclassified correctly with 99.3% accuracy (K= 0.99).

Charge Twisted Sonic Zigzaghttp://en.wikipedia.org/wiki/Pen_spinning

OUR METHOD

OUR METHOD(D) Identifying Different Pens

OUR METHOD(D) Identifying Different Pens

The magnetic intensities for the three pens, and the gaps among them, decreaseas the distance increases. We used these curves as standards to identity each pen.

250 uT

70mm

OUR METHOD(D) Identifying Different Pens

The participants were asked to make 10 horizontal strokes, starting from the top and going to the bottom for each pen (5 participants × 10 trials × 3 pens). The results showed that the penswere classified correctly with 92.6% accuracy and that all errors occurred at distances of more than 83.5 mm.

Error occurred

Error occurred

OUR METHOD

OUR METHOD(E) Determining Applied Pressure

OUR METHOD(E) Determining Applied Pressure

We used a relative position between the two curves as a coarse proxy of pressure (The farther the distance is, the denser the pressure levels).

OUR METHOD

LIMITATION

- The magnetism involved might damage the objects when the objects are very close to the magnets.

- The magnetometer we used occasionally reports a large offset in the data or a stuck pointing in one direction. In this case, user should wave the device in figure eight to get back on track.

However, magnets do not affect flash memories on modern mobile devices or IC chips on credit cards.

This technical glitch, however, will be undoubtedly solved as sensor technology improves.

CONCLUSION

- We have shown how MagPen provides users with high input expressiveness, a wide input area, and rich tactile clues without requiring power or wireless connections.

- We believe that MagPen will open a large area for novel pen interaction designs.

- Future work will explore how to further extend the additional degrees of freedom (e.g., pen hovering or tilting).

- We plan to run several user studies to gauge the feasibility of this interface and determine how it compares with previous, more complex hardware solutions

THANK YOU

Q&ASpin Gesture Recognition

Three different spinning gestures were classified using a machine-learning approach (J48 decision tree) with a set of features that included the average movement speed, speed variance, size of the gesture, shape, and trajectory.

Q&A

EXAMPLE USE