Exploring Multi-dimensional Data on Mobile Devices with Single Hand Motion and Orientation Gestures

Abstract ScatterDice Mobile (SDM) is a novel visualization system that leverages embodied motion and orientation gestures for intuitive and effective exploration of multi-dimensional data on mobile devices. Inspired by Elmqivist et al’s recent work, SDM uses the gyroscope sensor available on mobile devices to establish an orientation aware “dice rolling” metaphor for browsing scatterplot matrix visualizations mapped to a cube on mainstream mobile devices without any hardware modification. SDM has the potential for applications that require prompt access and exploration of large scale, multi-dimensional data on mobile devices anytime, anywhere.

Author Keywords Motion Sensing; Mobile Devices; Smart Phones; Gesture; Gyroscope; Visualization; Scatterplot.

ACM Classification Keywords H.5.2 [Information interfaces and presentation]: User Interfaces. - Graphical user interfaces; Input devices and strategies, Theory and methods.

Copyright is held by the author/owner(s).

MobileHCI’12, September 21–24, 2012, San Francisco, CA, USA.

ACM 978-1-4503-1443-5/12/09.

Jesse Thomason Department of Computer Science, University of Pittsburgh Pittsburgh, PA 15213, USA jdt34@pitt.edu Jingtao Wang Department of Computer Science, University of Pittsburgh Pittsburgh, PA 15213, USA jingtaow@cs.pitt.edu

173

Introduction Visualizing, exploring and making sense of large scale, multi-dimensional data has always been a key challenge for researchers and practitioners due to technological advances in sensing, communication and storage. This problem is even harder on mobile devices, given the much smaller screen size, limited input modality/capability, and extra distractions in diversified surrounding environments.

Although great strides have been made by the research community towards visualizing multi-dimensional data, the scatterplot matrix, one of the simplest and oldest visualization techniques, remains to be widely used by people in different backgrounds. Elmqivist and colleagues recently invented a novel visualization system and related interaction techniques (i.e. ScatterDice [1]), which greatly improved the usability of scatterplot matrix in an interactive setting. ScatterDice combines both the simplicity and intuitiveness of scatterplot matrix visualization via a focus+contex display and a dice rolling mapping metaphor. However, the original ScatterDice system was designed and optimized for desktop computers with large displays, and some techniques require desktop only input devices such as a mouse or a stylus with push buttons.

Projects like pCubee [5], Peephole Display [7] and Virtual Shelves [4] already demonstrated the feasibility and promising interaction scenarios of spatial/orientation aware interfaces. However, all these projects require external, expensive motion capture devices to simulate the intended effects, which made proposed techniques not available for everyday use at this moment.

In this paper, we present ScatterDice Mobile (SDM), a novel system for exploring multi-dimensional data on mobile devices with orientation aware interactions. By leveraging real-time orientation estimations from the built-in gyroscope sensor and taking advantage of the multi-touch capabilities of mobile devices, SDM makes it intuitive to explore scatterplot matrix interactively via the “dice rolling” interaction metaphor [1] with even a single hand. Paradoxically, feeling like manipulating data at one’s fingertips, the orientation/spatial aware “dice rolling” metaphor works even more naturally in a mobile setting.

Related Work Accelerometers and gyroscopes are becoming the default motion sensors on smart phones nowadays. Although the usage of accelerometers has been well studied in HCI community [3], our knowledge of gyroscopes in mobile interaction design is still very limited, in part because gyroscopes are only becoming ubiquitous in late 2010 thanks to the recent advances in MEMS (Micro-Electro-Mechanical Systems) technology. Vibrating structure gyroscopes manufactured in MEMS leverage the Coriolis Effect [2] rather than the angular momentum of a spinning wheel in the mechanical gyroscope to determine relative device rotation. Such a design leads to tremendous reduction in cost, size and power consumption.

An accelerometer measures the linear acceleration and the gravitational field of the earth. With the cost of drifting errors and noises, both velocity and position changes can be derived with single and double integral operations. In contrast, the gyroscope detects the angular velocity of a device in up to three orthogonal axes. With a single integral operation, angular

Figure 1. ScatterDice Mobile running on an iPhone 4. Changing the device orientation will reveal corresponding perspective-corrected visualization.

174

distances and orientation information can be derived with less noise due to one less integral operation needed. Camera phone based motion sensing [6] could achieve more accurate estimations than accelerometers on transitional movements, but the refresh rate is bounded by the frame rate of the camera, which is usually around 15 – 30 HZ for common camera phones.

Design of Scatter Dice Mobile

Many design ideas of ScatterDice Mobile were drawn from Elmqivist and colleagues’ recent implementation [1]. We also designed unique features to streamline the experience of interactive scatterplot matrix exploration on mobile devices.

Overlay Based Mapping As shown in figure 2, we rely on a semi-transparent scatterplot matrix overlay to establish the mapping from the scatterplot matrix to the 3D cube planes (figure 2). The current active region is highlighted in purple in the global scatterplot matrix overlay. We have enabled both touch-based and gyroscope-based orientation aware local navigation (i.e. switching the active view to an adjacent view in the scatter plot matrix) in SDM. Zooming in and out can be achieved via pinching the multi-touch screen of the device.

Global Navigation With the scatter matrix displayed in the foreground (figure 2, middle), the user can use the touchscreen to tap on a cell hence triggering a hyperjump. In the hyperjump, the current cell will first rotate continuously to the proper x position of the destination cell. Then, it will rotate to the proper y position, thus reaching the target cell in at most two animations. This approach

gives the user an efficient way to navigate to areas of interest that are far away from the current active cell on the scatterplot matrix overlay.

Figure 2. Using a semi-transparent overlay to establish the conceptual mapping between the scatterplot matrix and the cube plane (left, middle) The overlay can be moved to background (right) and can be dismissed during interactions.

Data Queries In order to segregate data, SDM allows users to specify multiple constraints in multiple perspectives to highlight points of interest (figure 3, up, middle). The user first taps one of the query buttons at the bottom of the screen, and can then drag a query selection on the scatter plot. This type of data segregation not only makes clusters of data on the current plot easier to distinguish, it also lends the data extra dimensionality. Tapping on a data point or cluster of data points will draw a small inspection box around the area. The names of all the points that fall within the inspection area will be displayed, as in figure 3 (down). This allows the user to look up data by name, and in

Figure 3. Queries persist over rotations (up) and are drawn in other cells (middle). Triggering detailed inspection information with finger touch (down).

175

deployed applications, get the human-readable name of the data he or she has refined or extracted after exploration.

Implementation ScatterDice Mobile was original developed and tested on recent Apple iOS 4.x/5.x devices with built-in gyroscopes (such as iPhone 4/4s, iPod Touch 4th Gen and iPad 2/3). It has also been ported and tested on a Google Nexus S smart phone running Android 2.3. Pilot User Study We have conducted a 12-user pilot user study to explore whether the idea of SDM is intuitive to understand and whether users can learn and use SDM efficiently to complete real world tasks (figure 4). After finishing a brief tutorial (less than ten minutes), participants were asked to complete a total of 14 data exploration tasks involving two data sets (digital camera dataset, 12 dimensions, 1038 samples; and automobiles dataset, 8 dimensions, 398 samples; 7 tasks for each dataset) through ScatterDice Mobile. Preliminary results show that SDM are easy to learn and intuitive to use despite participants’ initial unfamiliarity with orientation aware interfaces.

Conclusion We present ScatterDice Mobile (SDM), an interactive visualization system that leverages embodied motion and orientation gestures for intuitive and effective

exploration of multi-dimensional data on mobile devices. SDM uses the gyroscope sensor recently available on mobile devices to establish an orientation aware “dice rolling” metaphor for browsing scatterplot matrix visualizations mapped to a cube on mainstream mobile devices without hardware modification. Experimental results show that SDM are easy to learn and intuitive to use despite participants’ initial unfamiliarity with orientation aware interfaces.

References [1] Elmqvist, N., Dragicevic, P., Fekete, J.-D., Rolling the Dice: Multidimensional Visual Exploration using Scatterplot Matrix Navigation. In Proc. InfoVis 2008. [2] Fraden, J., Handbook of Modern Sensors: Physics, Designs and Applications, 4th ed, Springer; ISBN: 1441964657; 2010 [3] Hinckley, K., Pierce, J., Sinclair, M., Horvitz, E., Sensing Techniques for Mobile Interaction. In Proc UIST 2000. [4] Li, F., Dearman, D., Truong, K., Virtual Shelves: Interactions with Orientation-Aware Devices. In Proc UIST 2009. [5] Stavness, I., Lam, B., Fels, S., pCubee: a perspective-corrected handheld cubic display. In Proc CHI 2010. [6] Wang, J, Zhai, S., Canny, J., Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study. In Proc of UIST 2006. [7] Yee, K. P., Peephole Displays: Pen Interaction on Spatially Aware Handheld Computers. In Proc CHI 2003.

Figure 4. Sample pictures taken from the user study.

176