Rapid Authoring for VR-Based Simulations of Pervasive Computing Applications

B.J.C. Jackson, P.L. Watten, P.F. Newbury, P.F. Lister The Centre for VLSI and Computer Graphics, Department of Informatics, University of

Sussex, BN1 9QH, UK, p.newbury@sussex.ac.uk

Abstract. A new paradigm in computing has arrived [1]. The pervasive computer provides a new application layer and presents new challenges for application designers. The combination of computing devices provides a unique environment-specific platform for the development of applications. Applications not only rely on the pervasive computing devices and their interconnections, but also the location of these devices with respect to each other and the application user. This paper presents an overview of a design-flow, tool-chain and runtime environment which enables the rapid authoring of VR-based simulation components to facilitate the development of pervasive computing applications. A home automation example is used to evaluate the systems and to demonstrate the approach.

1. Introduction

As computing devices increase in power and decrease in visibility, technology is becoming ever more pervasive. Embedding computing devices into the fabric [2] of our environment forms new and unique application platforms. Designing applications for such distributed heterogeneous computing platforms requires not just the modelling of software and hardware, but the modelling of the environments where the devices operate: our homes, cars and workplaces. However, this conceptualisation and modelling of pervasive computing devices and applications can be costly and time-consuming when tangible environments and devices need to be created. This has been illustrated by projects such as Cooltown[3], which have required substantial effort and financial backing. A virtual reality approach provides the necessary environmental conditions to satisfy the contextual nature of pervasive computing applications, but allows for a shorter design cycle and rapid development.

Pervasive computing encompasses a variety of elements from a wide range of areas [4] and much work has been completed on providing frameworks for producing pervasive computing applications [5][6][7]. However, none of these projects address the task of modelling applications within a virtual environment. This paper describes an approach to the rapid design of VR-based simulations of pervasive computing applications. This not only enables the simulation of pervasive devices and applications, but also the environments in which they run. A design-flow and accompanying tool-chain are presented for the rapid creation of interactive 3D worlds with the addition of a runtime environment to enable pervasive computing simulation and visualisation in real-time 3D. As an addition

to the traditional VR and haptic mix a new VRAPTIC (see section 3.3) is proposed to support the simulation of pervasive computing interfaces.

The design-flow attempts to strike a balance between time consuming game production and VR techniques to provide an end result which can be rapidly created with levels of interactivity which support pervasive computing simulations. The real-time 3D environment comprises numerous applications to offer navigable 3D with interactive pervasive computing simulation capabilities, VRAPTIC interfaces plus the traditional audio, video, haptic feedback and head tracking. Inter-application communication facilities provide bidirectional message transfers between the runtime components.

An example simulation is presented as an evaluation of the approach using a navigable scaled model of a known real-world environment. Real-time rendering and network support of the model is provided by conversion to X3D [8], which is ISO compliant and the W3C [9] recommended format for 3D scene description. X3D feature and interaction authoring has been carried out to provide control of the model and to represent the physical aspects of its devices. The available real-time performance and number of media features provides a system that is capable of rendering the multiple device and user interactions that are needed to represent a pervasive computing environment. Figure 1 shows an overview of the runtime platform that supports this environment.

2. Content Creation

A content creation design-flow has been developed to provide a trade-off between resource intensive game development techniques for fast real-time 3D and rapid development processes for experimental VR. The use of automation and “off the shelf” solutions is favoured over manual development techniques, while pre-calculations and optical illusion are favoured over highly accurate image generation. The design-flow has enabled the development of a tool-chain to support the generation of the interactive scene-graph. Figure 2 shows the content creation design-flow.

Low Cost Haptics/Head-Tracking/Stereoscopy

Language IndependentSystem/Hardware

Simulations

VRAPTIC Clients e.g.PDAs

ExampleInterface SimulationsMessage

Services

Multi-user VR Environment

Figure 1 Overview of the runtime platform

The following sections describe the design-flow in more detail and highlight some of the applications that form the tool-chain. Each application has been selected for the feature-set, availability and developer familiarity. At every stage of the tool-chain applications can be replaced, with minimal impact to the overall design-flow. The process of selecting technologies to form the tool-chain has indicated the importance of pre-existing user skills as an asset in development. Familiarity with development tools directly affects the ability of a developer to implement project requirements in a timely manner. The benefits of language/application neutral concepts include better re-use of developer effort and reduced training times. The evaluation of alternative applications is beyond the scope of this paper.

2.1. Environment Capture

Environment capture aims to provide data to enable the production of an efficient model that represents a real-world example. To provide geometry data, real-world dimensions are captured using simple measuring devices such as tape measures and optical measuring devices. Texture data can be captured with a domestic digital camera, similarly audio can be captured with basic sound recording equipment. The use of commonly available tools reduces the need for specialist capture skills and facilities and enables rapid production and iteration of models.

2.2. Asset Assembly

The data generated from the capturing exercise is used to generate and assemble the necessary assets for a high-quality offline model. Low polygon modelling techniques are used to construct a mesh which provides sufficient detail with low runtime processing requirements, while limiting editing complexity and facilitating rapid model construction. Texturing and lighting target offline model rendering and use computationally expensive techniques to produce high quality source content.

3DS Max [10] is used for mesh generation by entering geometry data captured in the previous step. Flattened UVW layouts are passed to Adobe[11] Photoshop for texture processing and positioning. Video textures are processed to set start and end points, frame

Measurement Capture

Texture Capture

Audio Capture

Mesh Construction

Texture Layout Animation

Audio Process

Lighting

Texture Baking

Texture Processing

VRML Authoring Scripting

XML-encoded X3D

EnvironmentCapture

Asset Assembly

Collationand Deployment

Design Flow

Figure 2 Content creation design-flow

rates, dimensions and codecs as well as to composite extra lighting information. Adobe Premiere is used for general video editing with Adobe After Effects used for more advanced processing such as texture layout for movie textures. Audio is manipulated using Adobe Audition.

Once the offline model is complete a pre-calculation exercise produces online versions of the assets. The radiosity solution provided by 3DS Max is used for the lighting simulation. This improves the visual quality of the scene, although is computationally expensive when large numbers of textures are generated. Preserving the workflow is achieved by distributing the task to remote render nodes thus taking the process offline. The resulting calculations are burnt into textures using the render-to-texture feature of 3DS Max. This handles the task and provides automated generation of new UVW co-ordinates rendering multiple sources to a single texture.

The final texture stage can comprise format conversions, compositing and shader development. Shader support in X3D is still in its early stages, however the X3D specification (amendment 1) includes bindings for the OpenGL shader language GLSL, the Microsoft HLSL and the nVidia Cg shading language. Shaders are becoming major components of real-time 3D and support for modern shader languages offers an opportunity to re-use a large portion of the development effort currently occurring in the entertainment industry [12].

Figure 3 shows a baked texture with automatically generated layout. The black areas are as a result of the inefficient placement algorithms, but are acceptable tradeoffs in this design-flow. Despite the drawback in implicit control, the automation provides a rapid solution and sufficient control to maintain an acceptable level of visual quality. Development speed is favoured over design efficiency.

2.3. Asset Collation and Deployment

The final stage of the content creation design-flow results in real-time X3D data. The present tool-chain requires a conversion from VRML97 to XML-encoded X3D. At the time of writing X3D is still a relatively new format and therefore the feature sets of a number of different tools are used to produce the final output with the minimum amount of development time. Also there is currently no single browser that implements the entire specification. As a result authoring requirements differ with each browser, however the

Figure 3 Baked texture (automatic UVW layout) and scenes showing real-time rendered output

variations are greater for browsers using classic encodings. A section of the VRML97 specification is implemented in 3DS Max, which allows most of the content to be exported directly to VRML97.

X3DEdit [13] is a scene-graph editing tool which uses the IBM Xeena editor [14] to provide a novel interface for authoring XML encoded VRML. The features of this editor greatly assist in authoring and comprehension of the scene-graph [15]. X3DEdit uses the Vrml97ToX3dNist [16] translator from the National Institute of Standards and Technology (NIST). Direct editing of the XML-encoded source is carried out using PSPad [17] file editor as this has the capacity to deal with the large line counts and lengths common with VRML scene-graph descriptions, as well as features such as tree views and syntax highlighting.

3. Runtime Environment

The runtime environment offers an interactive virtual simulation of a world rich with pervasive computing devices. The world is rendered in real-time with a visual quality that approaches current gaming technology and is presented simultaneously to multiple users. Simulation of pervasive computing devices and applications are visualised in the scene and haptic interfaces provide interaction. A new haptic interface, the VRAPTIC, offers pervasive computing specific interaction.

The runtime environment comprises a number of applications distributed across several workstations a messaging system is used to synchronise the visualisation, interaction and simulation activity of the environment. The following sections describe the components of the runtime environment.

3.1. High Quality Real-time 3D

To allow high levels of visual quality with rapid development times standard game authoring techniques, such as low polygon count geometry and pre-calculation of lighting and reflections are used (see section 2.2). These techniques successfully restrict the runtime processing requirements of the high-quality visual representations to the capabilities of the target graphics processing hardware and software.

The choice of rendering technology is important as there are variations in the feature-sets supported by X3D browsers. It is desirable that XML encoded X3D is supported along with an interface to enable external interaction with the scene-graph. MPEG4 [18] is also required to enable advanced media services. Finally fast, low-latency rendering with low visual anomalies is required to present a comfortable viewing experience of potentially complex models. For these reasons the Octaga Professional [19] browser has been selected. Additional preparatory and experimental work also uses the XJ3D [20] browser for standards compliance testing and early implementations of example nodes.

3.2. Displays and Haptics

The media features of the X3D browser are augmented with low cost hardware devices that provide additional interfaces for tactile feedback, head tracking and immersive stereo visualisation. Implementation of these additional features takes advantage of existing applications and services, however a custom haptic approach is implemented for interface based pervasive computing interaction.

3.3. The VRAPTIC

The Virtual Reality Annexed Pervasive Technology Interface Client (VRAPTIC) combines a rendered virtual interface with a tangible handheld device. This enables the virtual manifestation of devices such as remote controls to be presented in a more realistic form rather than a floating rendering in the 3D scene. Handheld devices display virtual representation of buttons, switches and screens as an external annexe to the rendered world. Users can navigate around the virtual environment whilst holding a physical simulation of a pervasive device under test, thus presenting a more realistic interaction scenario to the user [21]. Figure 4 shows a home-automation remote control visualised on the VRAPTIC device with the ability to interact with the scene.

Many different applications can be used to provide the interface model on the VRAPTIC device. In particular Web pages and Java [22] applications are technologies that enable the rapid design and evaluation of user interfaces. Additionally a 2D graphical modelling tool, developed in the IST VIPERS project [23] can provide interface modelling and design specifically for connection to hardware simulators.

Real-time distribution of the virtual interfaces to a VRAPTIC device is via tightVNC [24][25], a remote frame buffer system. The tightVNC server resides on a Linux-based X-Window platform which supports multiple desktops with geometry set to sizes that are useful for mobile displays. A VRAPTIC interface is implemented on a handheld computer with a VNC [26] client providing connection to a remote desktop. The very thin client [27] enables computationally expensive simulations to be visualised on low power handheld devices. The benefit VNC provides is that it reduces the development concerns to handling

Figure 4 The VRAPTIC interface modelling a domestic remote control in Java implemented on a PDA and connecting to the simulation environment via Bluetooth and VNC

the screen area. The graphics compositing engine is visualised via tightVNC on a handheld computer. This enables the result of complex real-time modelling to appear on the handheld device.

Multiple VRAPTICS can be used in a simulation providing un-tethered interaction with the virtual environment via devices that provide tangible interfaces for the simulation users. Additional satellite devices such as tablet PC’s can also be similarly included though often these devices will have sufficient resources not to require the abstraction of a remote frame buffer connection. In these cases the Remote frame buffer connection remains an option available to assist in the managing the distribution of processing requirements.

3.4. Messaging and String Handling

The communication mechanism provides message transport facilities between the scene-graph and contributing simulation components. The distribution of simulation services to both local and remote hosts allows features such as multi-user 3D environments and incorporation of hardware/system simulators. At present messages are passed as socket-based string transactions (see section 5) with each application processing and responding to messages as valid instructions arrive. This technique provides uncomplicated generic access to the scene-graph and supporting services.

The Scene Access Interface (SAI) component of the X3D specification defines both Java and JavaScript [28] bindings to allow external access to the scene-graph; these are specified in ISO/IEC FDIS 19777:2005. The SAI is an updated version of the VRML97 External Authoring Interface (EAI), specified in ISO/IEC 14772. No support for scripting languages is required by ISO/IEC 14772; the scripting requirements are specified in annex B (Java) and annex C (JavaScript). Both the EAI and SAI components of the specifications are still currently limited by the levels of support offered by VRML Browsers. To minimise repetition of developer effort and maximise flexibility, an approach to external access has been adopted which provides centralised communication points within the scene-graph. This approach minimises the number of separate connections required to perform simulation activities and takes advantage of the standard compliant support for internal message routing.

The simplest example of a supporting simulation component developed for the project uses a web page. The web page is equipped with a Java Applet to handle dynamic interaction with the socket connection, and the Netscape Live Connect [29] package enables communication between the applet and JavaScript embedded in the web page. Consequently any available technologies that can be embedded in a web page are available for use. Modern implementations of Java and JavaScript have support for regular expressions which allows rapid development of the components message handling section. Other benefits include the use of web-based media services and the large web development skills-base. The security restrictions [30] imposed to prevent the misuse of applets can present problems; the solutions are described on the Sun Microsystems Java web site.

The security restrictions for applets are not imposed on Java applications which increase the implementation flexibility. The use of a standard language like Java assists in providing connections to external 3rd party applications and services e.g. Mathworks Matlab. There is also a large Java development community to provide support. Device modelling and simulation technology, implemented in C++, also easily integrates with the runtime environment. This enables a home automation remote control to be implemented and visualised on a VRAPTIC interface.

3.5. A Home Automation Example

To evaluate the design-flow, tool-chain and runtime environment a pervasive computing based home automation scenario was used. A model of a known environment was rapidly built in 3DS Max. Figure 5 shows a screenshot of an external view of the virtual environment created by following the design flow and rendered in the run-time environment. Each virtual room in the environment is equipped with sensors to capture user activity with every surface in the virtual environment configured to allow some level of interactivity. This ranges from texture swapping to visualise variations in lighting, to more complex options such as media playback and device control. Interaction and monitoring devices are implemented using various web and application approaches. The home automation system comprises numerous remote controllable devices, for example lamps, media systems, cooking appliances etc. In order to demonstrate the simplicity and flexibility of the system example control, interaction and monitoring simulations have been prepared.

4. Conclusion

The design-flow and tool-chain enables rapid development of navigable 3D environments in a timely manner due to trade-offs between game development approaches and tooling automation. The use of common technologies ensures a large skills base which reduces the need for specialised resource intensive facilities and expertise.

The increasing use of 3D technology in entertainment has encouraged the development of tools which provides the ability to rapidly generate 3D assets. Recent changes to the X3D specification have bought the standard inline with the state of modern 3D graphics and other XML technologies making the open standard a desirable and extensible choice. The performance increases of modern systems are not restricted to graphics and the improvements mean that behavioural device simulations can be rapidly written using high level tools. Traditional design and simulation tools can be incorporated into the system providing additional simulation [23]. The combination of this virtual simulation of environment, applications and devices with the concept of external VRAPTICs provides a powerful simulation, development and testing environment. The sum of these developments has lead to the system demonstrated in this paper that is capable of rapidly providing the

Figure 5 Virtual Environment rendered in real-time

multiple visualisations and interactions necessary to simulate both pervasive computing environments and applications.

5. Further Work

At the time of writing a simulation architecture is under development to provide an XML/SOAP-based framework for underlying pervasive computing simulations. This system will integrate with the design-flow, tool-chain and runtime environment discussed in this paper.

The Distributed Interactive Simulation (DIS) standard [31] IEEE 1278 defines the binary layout of distributed simulation messages for network communications [32]. Example DIS applications are freely available from the DIS-Java-VRML working group [33] and may provide greater re-use possibilities for the runtime system.

Incorporating load sensing nodes in the X3D environment offers to ability to respond to simulator response times. Using these approaches it becomes possible to change the behaviour of the simulation components according to computational requirements. This provides an indication for the modification of the distribution of simulation components as network or processing constraints reduce the quality of service to inhibitive levels.

Low cost forced feedback hardware commonly uses audio signals to trigger hardware response. Independent control of the driving audio channels may allow a greater number of these devices to be incorporated into a simulation with greater levels of control. Distribution of the audio playback tasks could provide high quality multi-channel surround sound with the ability to incorporate position an orientation data from the scene.

6. References

[1] Ubiquitous Computing, http://www.ubiq.com/hypertext/weiser/UbiHome.html, July 2005

[2] Newman, M. W., Sedivy, J. Z., Neuwirth, C. M., Edwards, W. K., Hong, J. I., Izadi, S., Marcelo, K., Smith, T. F., Sedivy, J., and Newman, M. 2002. Designing for serendipity: supporting end-user configuration of ubiquitous computing environments. In Proceedings of the Conference on Designing interactive Systems: Processes, Practices, Methods, and Techniques (London, England, June 25 - 28, 2002). DIS '02. ACM Press, New York, NY, 147-156. DOI= http://doi.acm.org/10.1145/778712.778736

[3] T. Kindberg and J. Barton, "A web-based nomadic computing system," Computer Networks, vol. 25, pp. 443 - 456, March, 2001.

[4] A. Ranganathan, J. Al-Muhtadi, J. Biehl, B. Ziebart, R. H. Campbell, and B. Bailey, "Towards a pervasive computing benchmark," in Proceedings of the 3rd IEEE Pervasive Computing and Communications Workshop. Kauai, Hawaii, March, 2005, pp. 194 - 198.

[5] Ulbrich, A., Weis, T., Geihs, K. A Modeling Language for Applications in Pervasive Computing Environments, In 2nd International Workshop on Model Based Methodologies for Pervasive and Embedded Software, Rennes, France, 2005

[6] Becker, C., Handte, M., Schiele, G., and Rothermel, K. 2004. PCOM - A Component System for Pervasive Computing. In Proceedings of the Second IEEE international Conference on Pervasive Computing and Communications (Percom'04) (March 14 - 17, 2004). PERCOM. IEEE Computer Society, Washington, DC, 67.

[7] M. Roman, C. Hess, R. Cerqueira, A. Ranganathan, R. H. Campbell, and K. Nahrstedt, "A middleware infrastructure for active spaces," IEEE Pervasive Computing, vol. 01, pp. 74-83, October 2002.

[8] Web3D Consortium, http://www.web3d.org/, July 2005

[9] The World Wide Web Consortium (W3C), http://www.w3.org/, July2005

[10] Discreet, 3DSMax, http://www4.discreet.com/3dsmax/, July 2005

[11] Adobe, http://www.adobe.com/products/, July 2005

[12] de Carvalho, G. N., Gill, T., and Parisi, T. 2004. X3D programmable shaders. In Proceedings of the Ninth international Conference on 3D Web Technology (Monterey, California, April 05 - 08, 2004). Web3D '04. ACM Press, New York, NY, 99-108. ISBN:1-58113-845-8

[13] X3DEdit, http://www.web3d.org/x3d/content/README.X3D-Edit.html, July 2005

[14] IBM, Xeena, http://www.alphaworks.ibm.com/tech/xeena, July 2005

[15] Brutzman, D. 2003. X3D-edit authoring for extensible 3D (X3D) graphics. In Educators Program From the 30th Annual Conference on Computer Graphics and interactive Techniques (San Diego, California, July 27 - 31, 2003). GRAPH '03. ACM Press, New York, NY, 1-1. DOI= http://doi.acm.org/10.1145/965106.965137

[16] NIST, Vrml97ToX3dNist, http://ovrt.nist.gov/v2_x3d.html, July 2005

[17] PSPad, http://www.pspad.com/, July 2005

[18] Cha, K. and Kim, S. 2005. MPEG-4 STUDIO: An Object-Based Authoring System for MPEG-4 Contents. Multimedia Tools Appl. 25, 1 (Jan. 2005), 111-131., ISSN:1380-7501

[19] Octaga, Octaga Professional, http://www.octaga.com/, July 2005

[20] Xj3D, Web3D Consortium, Xj3D, http://www.xj3d.org/, July 2005

[21] Halttunen, V. and Tuikka, T. 2000. Augmenting virtual prototyping with physical objects. In Proceedings of the Working Conference on Advanced Visual interfaces (Palermo, Italy). AVI '00. ACM Press, New York, NY, 305-306. DOI= http://doi.acm.org/10.1145/345513.345363

[22] Sun Microsystems, Java, http://java.sun.com/, July 2005

[23] Lister, P.F. Trignano, V. Bassett, M.C. Watten, P.L. and Jackson, B.J.C. ‘Applying ViPERS: A Virtual Prototyping Methodology for hand-held electronic device’ The RF-Remote Control Device Case Study, The Embedded Systems Show 2004 (ESS 2004), In Proceedings of UK Embedded PhD Forum, NEC Birmingham,13-14 October 2004

[24] tightVNC, http://www.tightvnc.com/, July2005

[25] Li, S. F., Spiteri, M., Bates, J., and Hopper, A. 2000. Capturing and indexing computer-based activities with virtual network computing. In Proceedings of the 2000 ACM Symposium on Applied Computing - Volume 2 (Como, Italy). J. Carroll, E. Daminani, H. Haddad, and D. Oppenheim, Eds. SAC '00. ACM Press, New York, NY, 601-603. ISBN:1-58113-240-9

[26] PocketPC VNCViewer, M Midgley, Client, http://www.cs.utah.edu/~midgley/wince/vnc.html, December 2004

[27] Tristan Richardson, Quentin Stafford-Fraser, Kenneth R. Wood, Andy Hopper. "Virtual Network Computing," IEEE Internet Computing, vol. 02, no. 1, pp. 33-38, January/February 1998.

[28] JavaScript, http://wp.netscape.com/eng/mozilla/3.0/handbook/javascript/, July 2005

[29] Netscape Live Connect, http://wp.netscape.com/eng/mozilla/3.0/handbook/plugins/, July 2005

[30] Sun Microsystems, Java, http://java.sun.com/developer/technicalArticles/Security/applets/, July 2005

[31] X3D DIS-XML Working Group, DIS-XML , http://www.web3d.org/x3d/workgroups/dis.html, July 2005

[32] Huo, C. 1993. An activity model for standards process for the Distributed Interactive Simulation (DIS). In Proceedings of the 25th Conference on Winter Simulation (Los Angeles, California, United States, December 12 - 15, 1993). G. W. Evans, M. Mollaghasemi, E. C. Russell, and W. E. Biles, Eds. WSC '93. ACM Press, New York, NY, 1013-1020. ISBN:0-7803-1381-X

[33] Distributed Interactive Simulation DIS-Java-VRML Working Group, http://web.nps.navy.mil/~brutzman/vrtp/dis-java-vrml/, July 2005