A Case Study on the Development of 3D User Interfaces for Mobile Platforms

João Marcelo Teixeira¹ Daliton da Silva¹ Guilherme Moura¹

Luiz Henrique Costa¹ Nacha Costa¹ Veronica Teichrieb¹ Judith Kelner¹

¹Centro de Informática � Universidade Federal de Pernambuco (UFPE)

Av. Prof. Moraes Rego S/N, Prédio da Positiva, 1º Andar,

Cidade Universitária, 50670-901, Recife, Pernambuco

{jmxnt, ds2, gsm, lhcbc, ncb, vt, jk} @cin.ufpe.br ABSTRACT This paper presents mivaDesk, a mobile, interactive, virtual and autonomous 3D desktop running on a mobile platform, named miva. Mobility is achieved through the use of miniaturized devices. The graphical interface model implements different desktop metaphors as well as advanced interaction methods, including the support for non-traditional devices. Further, users may interact with both the virtual and the real environments, enhancing not only his/her interaction, but also action and perception. The autonomy of the system allows users to rely on its long term availability for performing their tasks uninterruptedly. A number of usability studies are followed by some enhancement suggestions for future versions of mivaDesk.

Keywords Wearable computer, virtual desktop, mobility, interactivity.

1. INTRODUCTION Given the growing demand for mobility and the increase of computational power even for miniature processing devices, this work draws its strength from two different areas: mobile computing and sophisticated 3D desktop technology. The authors studied the use of diverse emerging technologies in the development of a mobile platform that takes into consideration device size, weight, energy consumption for long term continuous and mobile utilization. Such platforms are also known as wearable computers. As a result, this work resulted in the development of a wearable computer-like platform, namely miva. To acknowledge the importance of having elaborate interfaces, advanced 3D desktop technology has been considered as part of this project. Similar recent works include XGL and the Looking Glass projects, which have considerably inspired the design of the mobile, interactive, virtual and autonomous desktop (mivaDesk). This paper introduces mivaDesk and discusses its architectural components and the choices made during the design phase.

The main initial goal was to give users the opportunity to explore, in the context of their daily activities, the use of new interaction concepts that are driving the evolution of computing. Mobility

support, also by-product of the miniaturization of devices, has become a common requirement. Through the use of non-traditional devices, users achieve much more power of expression and consequently more ease, pleasure and satisfaction when using a system. An example of such an advanced device is the HMD (Head-Mounted Display) with an attached camera while using Augmented Reality (AR). Long term energy availability is paramount for the successful deployment of such use scenarios.

The mivaDesk platform allows a mobile user to work on his/her desktop in an intuitive manner while manipulating 3D objects. To reach such a platform, careful design planning and important choices had to be made. The authors wish to share some of that with the interested reader.

After releasing the first version of the mivaDesk prototype, a usability evaluation was performed, in order to identify both its strengths and weaknesses. This usability evaluation was based on a commonly used methodology that identifies interface problems. Note that the usability issues were somehow initially set aside during the design of mivaDesk�s interfaces as the authors were more interested in verifying project feasibility. In such a context, the initial approach was to build a small and lightweight wearable computer but yet capable of efficiently running graphical applications.

Section 2 presents some related work to mivaDesk and other 3D desktop applications. Sections 3 and 4 describe in details the miva platform and the mivaDesk application, respectively. The visual interfaces designed are also examined. Section 5 presents the usability evaluations performed on the interfaces developed, suggesting some enhancements and modifications. Section 6 concludes the presented work, pointing out to the importance of usability analysis as a tool for better understanding user interaction issues.

2. RELATED WORK One of the major problems that affect today�s society is

information overload. Most of us have received daily e-mail, digital newsletters, unsolicited and other information that often exceeds our capacity of processing or digging through it. Further, with the growing tendency to use ubiquitous computing nowadays (laptops and personal digital assistants - PDAs), it is only natural that we see a seamless convergence between the digital world and the daily activities. Today�s workers must be able to leave their

desktops and continue working on their daily tasks, while

id9859609 pdfMachine by Broadgun Software - a great PDF writer! - a great PDF creator! - http://www.pdfmachine.com http://www.broadgun.com

remaining connected to computing and communications resources. Consequently, the use of laptops and PDAs has increased considerably in the last years. Such devices, however, are still too intrusive to be considered natural enhancements of human�s capacities, as they require undivided and full user attention while being manipulated.

Fortunately, a great deal of research on wearable computing has emerged in order to address such problems [1], [2]. Miniaturization has been able to develop small-sized devices yet with powerful computing power, such as Xybernaut [3], VIA EPIA Boards [4] and Microsoft Origami [5]. This allowed the construction of important applications in the medical [6], industrial [7] and military areas [8], to name only a few.

However, a robust infrastructure is necessary to provide such devices with effective mobility and autonomy, such as the one proposed by the MobiDesk Project [9]. Basically, it virtualizes a user�s computing session by abstracting underlying system

resources such as display, operating system (OS), and network, so that users can move through different locations.

In addition, Multimedia and Virtual Reality (VR) research has been investigating the development and use of non-conventional interactive devices in order to facilitate user interaction. Conventional devices, though spread out all over the world, do not always use the best interaction technique to perform all tasks. Studies about interaction interfaces reveal how far human-machine interaction can be improved.

Some current solutions try to virtualize existing devices, without modifying significantly their characteristics, as proposed in ARKB [10]. They suggested the use of devices in a natural way, since it keeps the aspects which users are used to. The ARWin project [11] proposes to augment the real environment with virtual elements through the use of markers, positioning this way a desktop application in user's field of vision, as shown in Figure 1. The limitation of this solution is its lack of mobility present in the system, since it depends on markers with predefined positions.

Figure 1. The ARWin application uses real markers to position virtual objects.

"The office of the future" project [12] offers a collaborative virtual environment in which people and real objects are mixed with virtual projections that represent avatars of remote users, as shown in Figure 2.

NaviCam [13] is a device proposal focused on users interacting with the real environment and not only the human-machine interaction. By the use of a see-through video device, messages are placed over the real environment's image captured by a digital camera. One noticeable disadvantage of NaviCam is that the user occupies one of his/her hands holding the device, while using it, as illustrated in Figure 3.

Figure 2. The office of the future: sketch (top image); what exists today (bottom image).

Figure 3. User holding NaviCam device.

Many aspects of human-computer interaction must be considered before the development of new 3D user interfaces. The current processing power of embedded hardware solutions may stimulate the development of rich, 3D user interfaces for mobile applications. However, in order to allow true mobility, such interfaces must have minimal interference in a users' eye vision.

Figure 4. Looking Glass screenshot.

Figure 5. 3DNA Desktop example.

For this reason, typical 3D desktop managers present in Desktop Personal Computers (PC), such as the Looking Glass [14] (shown in ) or 3DNA Desktop [15] (shown in ) cannot be used, as they

block the visualization of the real world. �Cleaner� solutions such

as the ARWin Desktop [11], as well as mivaDesk, introduced in this work, are in our view more appropriate.

3. MIVA PLATFORM Conventional computational desktop platforms offer by default support to VR based and interactive applications. Both input and output interfaces found in desktop PCs can be considered at least sufficient to fill the needs of this type of application. In order to allow the execution of applications similar to the ones mentioned in Section 2, a small and lightweight platform has been defined and built, also known as miva (mobile, interactive, virtual and autonomous). This platform, besides providing the same default support found on desktop PCs, is still capable of presenting mobile and autonomy characteristics.

The miva platform is shown in Figure 6. Its basic components are a miniaturized processing board, a HMD with a connected webcam, network adapters (bluetooth and WiFi), a data glove, and a tracker. The user wears the back pack with an acrylic box (see right side of Figure 6), a HMD with a connected webcam, and a data glove with an attached tracker.

The miva platform has as one of its main objectives being self-sufficient and self-contained, in other words, applications running on it must not have the need for additional hardware/software. In scenarios where requirement can not be reached, the inclusion of new elements must be made simple. Depending on the application running, a customized OS can be generated (Windows XP Embedded) and additional hardware components may be used, like a cell phone in mivaDesk�s case (see Section 4). This way, all infrastructure is used in a optimized form and the quantity of peripherals coupled to the platform vary depending on what is needed by the currently running application.

Figure 6. On the left: A) mini-board with processor; B) board power supply; C) HMD and webcam; D) cell phone and Bluetooth USB adapter; E) USB four-port hub; F) battery; G) wireless USB adapter; H) tracker and data glove. On the right: user wearing

miva prototype.

Grouping selected hardware and software components (mainly shown in Figure 6) has given origin to a high-performance multimedia platform, capable of running graphics applications and sometimes presenting better results than common computers, besides giving a whole new level of experience to the user due to mobility.

User's mobility is related directly to the weight and size of the device used. The miva solution introduced here uses an ADL855PC board in the assembled platform (this board can be seen in Figure 7, compared in size to a desktop computer motherboard). The ADL855PC allows having significant power of processing in a relatively small area. For instance, the prototype�s

current version has about 31x36 square centimeters of area, weighting approximately 3 kg.

Figure 7. Computer boards: a) common desktop computer motherboard; b) ADL855PC, used by miva.

The need for network connections is supplied by the existing WiFi and Bluetooth connectors. Two batteries are used to supply the platform's energy (both board and HMD). Some platform tests have indicated a lifetime of four full hours using the two Lithium Ions batteries (LiOn), each one with a total capacity of 6.5 Ampere Hour.

In addition too the traditional mouse, keyboard and audio connectors, the miva platform has 7 USB connectors for generic use. Other devices can be connected through these connectors, like trackers, data gloves and webcams, which will serve as support for some type of specific interaction that may be defined for an application.

The "virtual" concept is achieved by the miva platform using DirectX latest version, together with a specific Windows XP Embedded image created for multimedia dedicated applications. This way, the most processing running over the OS is directed to multimedia applications. The OS generated image makes the performance of these applications higher than that of desktop computers, even without having an offboard video card. The HMD is used for virtual objects and application interface exhibition.

4. MIVADESK VISUAL INTERFACES mivaDesk is an application running on the miva platform and provides a user with a virtual work table, containing typical resources of a real one, integrated with a mobile phone. User�s

experience may be enhanced even more through the use of a webcam attached to a HMD, since this brings real world perception into mivaDesk. Nowadays, the virtual desktop offers the following user services: Phonebook, Agenda, Calendar, File Transfer and Phone service. This application has been developed in order to validate the miva platform, and jointly represents a case study to provide universal access in human-computer interaction.

For this application, the developed interfaces were divided according to their construction paradigm, i.e. 2D or 3D. This division was based on the different use characteristics provided by each one of these structures.

A significant aspect of 2D interfaces is the fact that daily users are already used to them, aiding interface operation comprehension. Furthermore, the layout simplicity makes this kind of interface the most indicated for complex tasks, especially data control. The last feature to be considered is their rigid visual aspect, which generally transmits environment stability and control [16].

On mivaDesk�s 2D interfaces, the use of transparency is tied to

the fact that the user needs to see the real surrounding environment and the use of a grayscale window background showed better visualization results due to its neutrality. The use of separated windows provides such interfaces with a clean and organized environment, making user�s comprehension easier.

Figure 8 illustrates this features implemented in the Agenda tool of mivaDesk.

Figure 8. Agenda 2D interface.

Since the control accuracy supplied by the support platform (tracker) is low, the use of large colored icons was preferred in order to facilitate interface navigation. This lack of control accuracy is due to the fact that the data glove used in the experience did not have an embedded tracker attached to it. Since the tracker is a 3DOF one, it worked as a common mouse responding to a user moving his/her pulse (axis direction is mapped to a screen position).

The system currently has three 2D interfaces, implemented in the Phonebook, Agenda and File Transfer tools. It is relevant to say that these interfaces will make frequent use of the virtual keyboard tool, shown in Figure 9, since they demand textual and numeric input and for that reason some new suggestions related to the icons� position were made during the evaluation phase to avoid overlapping of information and control buttons. The iconography used was chosen in order to favor familiar interface utilization, based on most used icons related to computer platforms. As an example of a familiar interface implemented, Figure 12 illustrates the File Transfer tool, which was created based on Windows Explorer views. However, some icons presented comprehension issues (e.g., the Agenda interface shown in Figure 8) on the usability tests (see Section 5).

Figure 9. Virtual keyboard interface.

On the other hand, 3D interfaces are less rigid than 2D ones, allowing manipulation in more DOF. In addition, building a specific 3D object as base for an interface brings a singular and new interaction way for manipulating such object. Consequently, this new interaction generally is more attractive than usual interaction ways, requiring, however, an longer adaptation time.

When building 3D objects for mivaDesk, a set of technical restrictions generated by the support hardware was considered. Among these there were reduced data storage capacity, and restrictions related to the video card. Because of such limitations, the use of simplified meshes for the 3D objects was the solution for obtaining better performance results, to the detriment of visual aspects.

Three 3D interfaces were implemented: the Main Menu, the Phone and the Calendar tools. These interfaces present associated characteristics; each one was created to attend a specific functionality. As an example, the 3D object representing the calendar service is a visual representation of a calendar, shown in Figure 10.

Figure 10. Calendar 3D interface.

The application menu consists of a 3D cylinder (shown in Figure 11) capable of rotating according to pointer position, while the Calendar and Phone objects are more likely ellipsoidal shapes populated by function buttons. The rounded shape was chosen for the last two tools because it provides cleaner visualization, since it hides the background less than a simpler box shape.

Figure 11. mivaDesk menu and a formal reference example.

5. USABILITY EVALUTION For the usability evaluation of mivaDesk interfaces, a sequence of procedures was established based on known methods of data survey and organization related to performance and functionality of systems.

At a first stage, a heuristic evaluation based on Mayhew�s

principles was used in order to allow the construction of a general overview of the interfaces and to determine which aspects should guide the remaining evaluation procedures [17]. Familiarity (how much common is the interface to the user?), compatibility (how adequate is the solution for the problem? is there a better choice?), simplicity (how easy is to the user to work with this interface?), technology (level of technology used in the system) and

robustness (is the interface easy to recover and error-safe?) were some of the principles adopted in this analysis.

As result of this first stage a series of questions were established, some of them presented in Table 1, and they gave origin to an adequate checklist for verification of the interfaces. This checklist was defined having as its focus the interaction and use aspects of the interfaces, and considering the control devices supplied by the miva platform (tracker and data glove). The main issues noticed when applying the checklist were: functions associated to the commands of the glove, appropriate use of icons, presence of confirmation windows, which interfaces (3D or 2D ones) really need to use the virtual keyboard.

One by one, mivaDesk interfaces were checked and verified. For example, in the File Transfer interface, shown in Figure 12, there is a problem regarding duplicity of buttons in the interface. Since just one window list is used at a time, there is no need for two icons representing the same action (above both window lists, there are three icons representing deletion, renaming and new folder functions, respectively).

Table 1. Question table based on heuristic evaluation.

Questions for checklist construction

1. All movement commands are used?

2. Is there any input data on this interface?

3. Is there an easy application exit?

4. Are the icons understandable?

5. Is it possible to do all daily tasks(running, taking a bus, or reading, for example) with this interface?

Figure 12. Screenshot showing a duplicate icon problem.

With the data provided by the checklist, it was possible to define a table of priorities related to the next step of the evaluations, which is the flowcharts construction. First, an investigation was performed through the construction of flowcharts. This way, each interface functionality could be analyzed and after that an evaluation of severity based on the checklist and flowcharts results was made.

The flowcharts, some of them illustrated in Figure 13, have followed the models of construction of Iida and Baxter, given some few adaptations for this particular case [18], [19]. They allowed better understanding of all interfaces and redoing them, turning the interfaces closer to their specific functionalities.

Figure 13. One of mivaDesk interfaces flowchart.

Since all the investigation and problem identification procedures were performed, a severity evaluation based on Nielsen was done to guide the procedure of correcting the problems found [20]. A sequential relation of activities was constructed for the resolution of the problems and interface optimization, based on the flowcharts was created.

This severity evaluation was based on the nature of the problems, development process, problem frequency and also the amount of interference that the necessary change would bring to the system [20]. This evaluation identified some required changes and the problems that were related to each change. The frequent problems that appeared were also perceived. The most important items in the severity list are showed in Table 2.

Table 2. First five elements in severity list.

List of most significant problems

1. Day note bug (Agenda interface)

2. Duplicate buttons (File Transfer interface)

3. Year and month displays (Calendar interface)

4. File selection in file transfer panels (File Transfer interface)

5. Lack of confirmation dialogs in all deletion and inclusion operations

Based on that severity list, the development team was able to begin the corrections and improvements of mivaDesk, directing the tasks performed on its interfaces to a more optimized flow.

6. CONCLUSION AND FUTURE WORKS mivaDesk, as an application constructed to run on the miva platform, served to validate the use of the resources provided by the platform. The working structure and the communication between platform and devices to support mobility and autonomy were tested.

The results accomplished with the mivaDesk application were satisfactory from the point of view of performance, since the program was executed with at least the same performance than on a common desktop PC, despite platform hardware limitations.

Considering the designed architecture, an interdependency between physical interface (miva platform) and the application

implemented to run on it was observed. The miva solution provides the basic structure for creating systems and applications that require mobility, autonomy and interactivity while performing specific tasks.

Since the development process was not made simultaneously for usability evaluations, related to system�s use and its functionality, there is a need for some reviews of the created interfaces. However, it�s possible to say that system�s construction viability

was not affected by this late usability and functionality studies.

As future works, a mivaDesk application review may be made based on the evaluation results. Next, an evaluation of the physical interface (miva solution) is required in order to fill the gaps from the development process related to functionality and usability.

Taking as basis the miva platform, a second application will be designed with the objective of proving the physical interface independence. The development of mivaTherm project, an AR based application for general hardware maintenance support, including a thermal camera to the infrastructure, is currently in progress. It will use the same underlying platform and services, such as connectivity, modularity and flexibility to new hardware components utilization.

7. REFERENCES [1] Rhodes, B. �The Wearable Remembrance Agent: a System

for Augmented Memory�, International Symposium on

Wearable Computing, pp. 123-128, 1997.

[2] Starner, T., Mann, S., Rhodes, B. J., Levine, J., Healey, J., Kirsch, D., Picard, R., and Pentland, A. �Augmented Reality

through Wearable Computing�, Presence: Teleoperators and

Virtual Environments, MIT Press, Massachusetts, USA, pp. 386-398, 1997.

[3] Xybernaut Mobile Products. Available: Xybernaut site. URL: http://www.xybernaut.com, visited on January 2006.

[4] VIA EPIA Boards. Available: VIA site. URL: http://www.via.com.tw, visited on January 2006.

[5] Microsoft�s Origami Project. Available: Origami Project site.

URL: http://origamiproject.com, visited on December 2006.

[6] Raskovic, D., Martin, T., and Jovanov, E. �Medical

Monitoring Applications for Wearable Computing�, The

Computer Journal, Oxford University Press, Oxford, UK, pp. 495-503, 2004.

[7] Kaefer, G., Prochart, G., and Weiss, R. �Wearable Alertness

Monitoring for Industrial Applications�, International

Symposium on Wearable Computers, pp. 254-255, 2003.

[8] Combat-ready Computers and Displays. Available: Argon site. URL: http://www.argoncorp.com, visited on January 2006.

[9] Baratto, R., Potter, S., Su, G., and Nieh, J. �MobiDesk:

Mobile Virtual Desktop Computing�, ACM International

Conference on Mobile Computing and Networking, 2004.

[10] Lee, M., and Woo, W. �ARKB: 3D Vision-Based Augmented Reality Keyboard�, International Conference on

Artificial Reality and Telexistence, pp. 54-57, 2003.

[11] DiVerdi, S., Nurmi, D., and Höllerer, T. �ARWin - A Desktop Augmented Reality Window Manager�, International Symposium on Mixed and Augmented Reality, pp. 298-299, 2003.

[12] Raskar, R., Welch G., Cutts, M., Lake, A., Stesin, L., and Fuchs, H. �The Office of the Future: A Unified Approach to

Image-Based Modeling and Spatially Immersive Displays�, Conference on Computer Graphics and Interactive Techniques, pp. 179-188, 1998.

[13] Rekimoto, J., and Nagao, K., �The World through the

Computer: Computer Augmented Interaction with Real World Environments�, Symposium on User Interface

Software and Technology, pp. 29-36, 1995.

[14] Looking Glass. Available: Project Looking Glass site. URL: http://www.sun.com/software/looking_glass, visited on May 2006.

[15] 3DNA Desktop. Available: 3DNA Home Page. URL: http://www.3dna.net, visited on May 2006.

[16] Poupyrev, I., Tan, D., Billinghurst, M., Kato, H., Regenbrecht, H., and Tetsutani, N. Developing a Generic Augmented-Reality Interface, IEEE Computer, 2002.

[17] Mayhew, D. Principles and Guidelines in Software User Interface Design, Prentice Hall, New Jersey, 1992.

[18] Baxter, M. Projeto de Produto, Ed. Edgard Blücher, São

Paulo, 2000.

[19] Iida I. Ergonomia - Projeto e Produção, Ed. Edgard Blücher,

São Paulo, 1993.

[20] Nielsen, J., Severity Ratings for Usability Problems. Available: Use-it.com: Jakob Nielsen´s Web site. URL:

http://www.useit.com, visited on December 2006.