what you sketch is what you get: quick and easy augmented...

What You Sketch Is What You Get:Quick and Easy Augmented RealityPrototyping with PintAR

Danilo GasquesUniversity of California San DiegoLa Jolla, CA 92093, [email protected]

Janet G. JohnsonUniversity of California San DiegoLa Jolla, CA 92093, [email protected]

Thomas SharkeyUniversity of California San DiegoLa Jolla, CA 92093, [email protected]

Nadir WeibelUniversity of California San DiegoLa Jolla, CA 92093, [email protected]

ABSTRACTAugmented Reality(AR) tools are currently primarily targeted at programmers, making designing forAR challenging and time-consuming. We developed an interactive prototype, PintAR, that enables theauthoring and rapid-prototyping of situated experiences by allowing designers to bring their ideas to

Figure 1: PintAR in use. The AR designeris visualizing a sketch he recently drew onthe tablet.

life using a digital pen for sketching and a head-mounted display for visualizing and interacting withvirtual content. In this paper, we explore the versatility such a tool could provide through case studiesof a researcher, an artist, a ballerina, and a clinician.

INTRODUCTIONKEYWORDSAugmented Reality, Sketching, Prototyping,HoloLens

Over the last few years, we have seen a growth of developer-oriented tools for Augmented Reality(AR) such as Microsoft’s Mixed Reality toolkit as well as Application Programming Interfaces (APIs)

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without feeprovided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and thefull citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contactthe owner/author(s).CHI’19 Extended Abstracts, May 4–9, 2019, Glasgow, Scotland Uk© 2019 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-5971-9/19/05.https://doi.org/10.1145/3290607.3312847

CHI 2019 Late-Breaking Work CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK

LBW1416,

https://doi.org/10.1145/3290607.3312847

exposed by game engines like Unity3D and Unreal Engine that attempt to ease the entry to ARauthoring. However, they still require a high degree of programming proficiency, presenting multiplebarriers [4] and a steep learning curve for designers or non-programmers [6]. This lack of dedicatedtools for authoring interactive AR experiences results in slow iteration cycles that impede creativityand leave designers fatigued.

Table 1: PintAR features Across theHoloLens Application and the Tablet

PintAR Features Tablet HoloLensDraw (Create,Edit,Update) Sketch XSend Sketch to HoloLens XMove, Scale, Rotate Sketch XAdd Interactions to Sketch XCreate Frames XChange Current Frame XAssign Sketch to Frames XSketch Library XTop-Down View (WIM) XDelete Sketch X XRecord Video X X

This is especially true for immersive AR experiences like those afforded by Microsoft’s HoloLensand Magic Leap One. These experiences allow digital content to both blend with the real world andbe spatially dispersed in 3D space. While low-fidelity prototyping methods like sketching and paperprototyping allow designers to envision their ideas without deciding on specific details, the processproves to be demanding and limiting [6] for spatial interfaces because of their 3D nature.

Low-fidelity prototyping methods, especially sketching, facilitate ideation and experimentation [3]and are natural to users from different backgrounds. PintAR – ‘To paint’ in Portuguese – combines theadvantages of sketching with the unique affordances of head-mounted AR displays by enabling directmanipulation of digital content in AR while relying on a digital pen user interface for sketching on atablet. PintAR also allows users to specify interactive behaviors without writing code and provides animplicit state machine to express different stages of an experiences. Furthermore, it provides users withan aerial overview of digital content to facilitate debugging through a world-in-miniature view [9]. Inthis paper we explore how PintAR facilitates rapid-prototyping of AR experiences using 2 case studieswith an anesthesiologist, and a researcher. We also explore the versatility of using AR as a creativemedium with 2 case studies involving a visual artist and a ballerina.

RELATEDWORKPrevious work exploring content authoring in AR through sketches range from using finger ges-tures [1] and tracked wands [10], to using commands drawn on paper to generate digital content [5].SymbiosisSketch [2] combines a hand-held device with a tracked pen to ease switching from 2D to3D sketching, and Langlotz et al. [7] uses mobile phones as an authoring and visualization device,allowing users to sketch lines, 3D primitives, and copy the geometry of real objects. However, thesetools focus on static content creation and cannot mock interactive experiences. DART [8], on the otherhand, is an authoring tool that allows users to prototype interactive AR experience, but it confinesauthoring to a desktop computer and requires users to write code to express complex behaviors.

PINTARPintAR is an interactive lightweight prototyping tool aimed to address the demand for an easy andquick way of prototyping in AR. It does so by allowing users to express interactive experiences bycreating sketches and their corresponding interactions using a digital pen and tablet and transferthem into the real world where they can be viewed, manipulated, and interacted with in AR.


LBW1416,

PintAR is implemented as twomain components: AHoloLens application, and aweb interface viewedthrough a tablet. The HoloLens application has been developed using Unity3D and Microsoft’s MixedReality Toolkit. The tablet’s web app responds to touch events to enable sketching and communicateswith the HoloLens through a NodeJS server. We chose a web interface over an iOS application tofacilitate running it from any mobile device regardless of the operating system.

PintAR separates tasks that users can tackle effectively on a 2D screen from tasks that are physical innature and require a 3D interaction. Thus, sketching and animating is delegated to the tablet. Moving,Scaling, and Interacting with content is only available through gestures, gaze, and head-position onthe AR headset (see Table 1).

PintAR: Tablet Features

Figure 2: Drawing Pad and Tablet Inter-face

(1) Draw Sketch and Send To HoloLens: The drawing pad (Fig. 2) can be used to create a sketchthat can be brought into the user’s augmented space using a ’Send To HoloLens’ button. Asingle sketch can be sent to the HoloLens multiple times to create multiple instances. Users canupdate sketches, with the updates propagating to all instances of that sketch in AR.

(2) Add Interactions:Users can add implicit interactions to their sketches allowing them to appear,disappear, rotate, and move based on the direction of the user’s gaze and/or the user’s positionrelative to the sketch. Any combination of these interactions can be added to a sketch to allowfor more complex behaviors.

Figure 3: Library of all sketches authoredby the user

(3) Sketch Library: All sketches sent to the HoloLens are saved in the Library (Fig. 3) where theycan be accessed later, deleted, or edited.

(4) Frames: Users can organize different stages of their experience using frames. Sketches andtheir corresponding interactions can be placed into a specific frame, or they can be modifiedto be shown/hidden in various frames. Any number of frames can be added and they can becycled through to show sequences of progressions of the experience.

Figure 4: TopView (WIM–World InMinia-ture) showing all sketches the user placedin their environment

(5) Top View (WIM, or World In Miniature): Sketches in the user’s augmented space can beviewed in an aerial or top-down view (Fig. 4). The positions of the sketches are scaled to thetablet screen allowing easy identification of sketches by its relative position to other sketches.

(6) Record: Users can use the ’Record’ button to record a video of the HoloLens view that showsboth the real world around the user as well as their sketches in the augmented space and anyinteractions triggered as they interact with it.

PintAR: HoloLens Features(1) Grabbing and Placing Sketches:When a sketch is sent from the tablet to the HoloLens, it

appears in front of the user. Users can then gaze at the sketch and then air-tap1 to grab it and1air taps are analogous to a mid-air clicks,where the index finger is pinched toward thethumb


LBW1416,

place it in their desired location in the augmented space (Fig. 5, left). They can also walk aroundthe space while dragging it along.

(2) Rotating and Scaling Sketches:Users can use two hands to select and scale or rotate sketches(Fig. 5, Right).

(3) Deleting Sketches: Sketches can be deleted by scaling them down to a very small size. A redbox then appears around it, indicating it will be deleted if released.

Figure 5: (Left) Grabbing and movingsketches. (Right) Scaling and rotatingsketches with two-handed Air tap CASE STUDIES

To observe whether PintAR could facilitate rapid-prototyping of AR experiences, we invited two partic-ipants that demonstrated an interest in using AR in their respective fields of work: an anesthesiologistand a cross-device development researcher. Both of them had previously tried a head-mounted display(HoloLens) and had a basic understanding of the affordances of AR.

Because being able to add interactive digital content to ones environment allows users to mockmore than just AR experiences, we also invited a visual artist and a professional ballerina to explorehow PintAR enables using AR as a medium that supports the creative process.

Figure 6: Anesthesiologist prototypingusing PintAR. Top: first prototype, thesketched lines serve to guide needle-entrypoints. Middle: second prototype show-ing where to move an ultrasound probeto scan specific locations of the heart.Bottom: point-of-view from the head-mounted display.

For all the four participants, we spent approximately 10 minutes demonstrating PintAR’s features.We spent an additional 10 minutes introducing gestures to the visual artist and the ballerina sincethey had never used a HoloLens before. Participants then had about 30-45 min to use the tool. Finally,we spent approximately 10 minutes gathering their feedback and impressions of PintAR.

Prototyping Augmented Reality ExperiencesHere we report on the use of PintAR to prototype AR Experiences by the anesthesiologist and thecross-device development researcher .

Anesthesiologist: The anesthesiologist wanted to explore different ways to help train novices toperform ultrasound-guided procedures. He had previously envisioned an AR tutoring application toguide his students but struggled to evaluate the usefulness of his idea as he could not find developersto help prototype it. He used PintAR to prototype for the two major challenges he faces in his teachingactivities: (1) using AR to show the different angles students could use to insert a needle inside atarget under ultrasound (Figure 6, Top); (2) guiding students by showing the correct location andorientation of an ultrasound probe to scan the different valves of a human heart (Figure 6, Middleand Bottom).

Cross-device development researcher: The researcher intended to create video prototypes fortwo separate AR applications so that he could share those with his team. His first prototype (Figure 7)reflected an application that highlighted electronic devices’ “technical abilities”. Users could themdrag those to create connections between different devices. For instance, dragging the “play” capability


LBW1416,

of a TV to a smart door would make the video on the TV play when the door is opened. His secondprototype used PintAR’s gaze interaction to demonstrate a system that highlights parts of robotsthat can be modified to inform the creation of a new one (Figure 8).

AR as a medium for creative expression

Figure 7: Researchers’s first prototype.Top: looking at a TV, the AR interfaceshows two affordances, play and seek.Middle: looking at the smart door, the ARinterface shows that it is available for pro-gramming. Bottom: the researcher dragsthe Play button from the TV to the smartdoor so that the video starts playing whenthe door opens

Here we report on how the visual artist and ballerina (Figure 9) used PintAR during their creativeprocess.

Visual Artist: The Visual Artist was preparing for an upcoming interactive art installation but wasnot sure where the many heavy hardware components (computer and projectors) should go. She usedPintAR to created sketches to represent the different hardware components, scaled them to the sizeof the real objects, and moved them around to try different room layouts.

Ballerina: The ballerina used PintAR as a way of creating objects for her to dance around. Shecreated squiggly lines, scaled andmoved around in her space, and added interactions to her sketches sothey would appear or disappear based on her location or when she looked at them. She then improviseddance moves as she moved around and under the sketches that she placed in the environment aroundher, while still wearing the headset. Another ballerina passing by then used the same setup to createher dance moves as she moved around and reacted the existing interactive sketches.

DISCUSSION & FUTUREWORKOverall, the feedback for PintAR was highly positive. Sketching served as an easy-to-use interface,allowing users from different backgrounds to quickly explore ideas in AR. From our early deployment,we collected feedback on both usability problems (ranging from bugs in our application to limita-tions of HoloLens’s gesture tracking and field-of-view), and possible directions for further investigation.

Coarser vs Finer Input: PintAR was designed with a clear separation of what each technology(tablet vs head-mounted display) would afford (see Table 1). For instance, PintAR only allowed movingand scaling virtual objects in space through gestures on the headset. Similarly, sketching was onlyavailable through the tablet as a digital pen gives users fine control over it. Some users (Ballerina,Visual Artist) where willing to forgo quality for faster interactions. For instance, they wanted to usethe World-in-miniature view to quickly but coarsely move objects in the real world. They only usedthe gesture interface later to refine objects’ locations. Similarly, they wanted the ability to producerougher sketches in AR when finer drawings were not necessary.

Beyond Interaction Mimicry: PintAR’s gesture interface was deemed intuitive because movinga virtual object works in a similar way as moving a real object. However, “moving things can take along time, and it is quite tedious” (Researcher). This is especially true when trying to create and moveseveral objects in space.


LBW1416,

Environment Eliciting New Ideas: From our observations, prototyping immersed in the environ-ment not only helps users envision content and interactions better, it can also foster idea generation.“Being in AR, it just happened that there was a TV there and I was able to prototype an interface and thinkof it in a way I would not have in a development space...[on creativity] I think it is really just synthesis ofsurrounding information, with AR you have the whole world to synthesize with” (Researcher).

Figure 8: Researcher’s prototype. Top: noinformation is displayed when not gazingat a specific object. Bottom: a text saying"editable" shows up when the user gazesat a specific object

Figure 9: (Top) Artist scaling a computerstand using her hands. (Bottom) Ballerinadancing around a sketch (sketch not visi-ble in the picture).

Tablet as a Wizard-Of-Oz Control Station (WoZ): While in our deployment a single personused both the HMD and the tablet, in a user test scenario a second person could use the tablet tocontrol the HMD’s wearer experience. This approach would turn the tablet interface into a WoZcontrol station, where a “wizard” changes frames to react to actions of the person wearing a HMD.

All in all, while we presented only a few small case studies which could not evaluate the systemthoroughly, insights from our observations provided an indication of the breadth of experiences thatcan be prototyped using a lightweight tool such as PintAR. In the future we plan to conduct a longerstudy to evaluate how the next iteration of PintAR can aid different design phases.

REFERENCES[1] J. Amores and J. Lanier. Holoart: Painting with holograms in mixed reality. In Proc. CHI 2017, pages 421–424. ACM, 2017.[2] R. Arora, R. Habib Kazi, T. Grossman, G. Fitzmaurice, and K. Singh. Symbiosissketch: Combining 2d & 3d sketching for

designing detailed 3d objects in situ. In Proc. CHI 2018, page 185. ACM, 2018.[3] B. Buxton. Sketching user experiences: getting the design right and the right design. Morgan Kaufmann, 2010.[4] M. Gandy and B. MacIntyre. Designer’s augmented reality toolkit, ten years later: implications for new media authoring

tools. In Proc. UIST 2014, pages 627–636. ACM, 2014.[5] N. Hagbi, R. Grasset, O. Bergig, M. Billinghurst, and J. El-Sana. In-place sketching for content authoring in augmented

reality games. 2010.[6] A. J. Hunsucker, K. McClinton, J. Wang, and E. Stolterman. Augmented reality prototyping for interaction design students.

In Proc. CHI 2017, pages 1018–1023. ACM, 2017.[7] T. Langlotz, S. Mooslechner, S. Zollmann, C. Degendorfer, G. Reitmayr, and D. Schmalstieg. Sketching up the world: in

situ authoring for mobile augmented reality. Personal and ubiquitous computing, 16(6):623–630, 2012.[8] B. MacIntyre, M. Gandy, J. Bolter, S. Dow, and B. Hannigan. Dart: The designer’s augmented reality toolkit. In Proc.

Symposium on Mixed and Augmented Reality, pages 329–330. IEEE, 2003.[9] R. Stoakley, M. J. Conway, and R. Pausch. Virtual reality on a wim: interactive worlds in miniature. In Proc. CHI 1995,

pages 265–272. ACM Press/Addison-Wesley Publishing Co., 1995.[10] B. Yee, Y. Ning, and H. Lipson. Augmented reality in-situ 3d sketching of physical objects. In Intelligent UI workshop on

sketch recognition, volume 1. Citeseer, 2009.


LBW1416,

what you sketch is what you get: quick and easy augmented...

Documents