Cluster ComputDOI 10.1007/s10586-014-0354-3

Designing interactive narratives for mobile augmented reality

Yanghee Nam

Received: 15 November 2013 / Revised: 22 January 2014 / Accepted: 31 January 2014© Springer Science+Business Media New York 2014

Abstract An increasing number of personal mobile devicessuch as smartphone are playing an important role in our dailylives. Among constituing technologies for such pervasivecomputing environment, mobile augmented reality (mobileAR) is a technique that extends physical world with virtualobjects or information in truly mobile settings. That is, awayfrom the carefully conditioned environments of research lab-oratories and special purpose work areas, general people canengage in location aware or physical object related contentusing their portable devices. Deploying attractive mobile ARservices, however, has been regarded quite difficult becausecomputer-vision based relevant techniques are very com-plex. Because of that, discussions on the practical possi-bilities of AR have tended to stay in location-aware infor-mation delivery service or as a consumption platform ofdeveloper-supplied content. Against those popular researchtrends pursuing technical advances of AR, this paper attemptsto explore the potential of AR as a general peoples’ cre-ative medium. It can provide a good storytelling environmentbecause a specific location or real-world objects can easilybecome story subject matters and stimulate people’s imagi-nation. There exist, however, some barriers preventing usersto actively participate in onsite creative activity construct-ing their own story. In mobile situation, it could be cumber-some to create a narrative at site, requiring to author somesequences of events. Therefore, the careful design regard-ing such situational characteristics of mobile AR is essentialfor realizing real-time mobile interactive AR narrative. Thispaper thus presented design factors for interactive mobile ARstorytelling systems, and applied narrative theory to design

Y. Nam (B)Division of Digital Media, Ewha Womans University,SK-Telecom Building #402, Seoul, Koreae-mail: yanghee@ewha.ac.kr

and explore actual possibilities of interactivity levels achiev-able using mobile AR medium. Technical difficulties areruled out as possible so that the design could be focused onreal-time onsite story creation activity. Proposed design con-cepts are developed as three kinds of prototypes, each of themreflecting different level of narrative interactivity. They arealso deployed in public exhibition and the suggested designfactors are evaluated in terms of user experience and inter-views. The result of evaluation shows that even simple ARinteractive narrative setup can have strong power to allowusers playful experience of in-situ narrative creation.

Keywords Interactive narrative · Digital storytelling ·Augmented reality · Mobile media

1 Introduction

Augmented reality (AR) is a technique that extends physi-cal world with virtual objects or information. It can providea good storytelling environment because a specific locationor real-world objects can stimulate people’s imagination andcontribute to formation of a story. In particular, when weregard user’s participatory generation of AR content, it hasgreat potential to produce user-created stories associated withreal things or locations. As the supporting technology, more-over, there has been recent growth on mobile platform includ-ing network infrastructure [1], location-aware mechanisms[2], and storage platforms for pervasive computing [3]. Theseadvances in mobile network, storage and positioning mech-anisms also greatly encourage practical and commercial useof AR, potentially as user’s content generation and sharingplatform [4].

However, previous works on AR have mainly stayed indiscussions on development of computer vision or graph-

123

Cluster Comput

ics techniques for visual composition of real and virtualscenes and their correct alignment. There also have beena few researches having regarded AR in relation to sto-rytelling, most of them, however, have actually presentedAR as an interactive story simulation environment with pre-determined stories created by developer or designers of thesystem in advance.

Comparatively, this article pays attention to AR as an inter-active storytelling medium, that is, a creative space for gen-erating a user’s own narrative. Based on a literary theoryon digital narrative texts, the design aspects enabling user’sin-situ augmented storytelling is discussed in this paper.Accordingly, prototype AR systems are implemented reflect-ing various interactivity levels of narrative and deployed in aninternational art festival for a month and let visitors interactwith them during the exhibition period. Based on the evalu-ation lasted for a month and also thereafter have shown thateven simple interactive AR setup for user-generated narrativecould have strong power to allow users playful experience ofin-situ storytelling.

The rest of this paper is organized as follows. Section 2summarizes the previous research on AR content authoringwith comparison to the proposed goal here. Section 3 is themain part introducing design considerations and the catego-rization of narrative interactivity, and based on those aspectsthree kinds of design goals are suggested. Section 4 presentsthe actual implementation of the proposed types of interactiveAR narrative and the results of empirical studies are brieflymentioned. Finally, conclusions are given in Sect. 5.

2 Related work

Existing works on AR content authoring have mainly beenclassified into one of the three categories. The lowest levelapproaches among those categories provide complex toolk-its, libraries or scripting frameworks for application devel-opers. Software toolkits such as ARToolkit [5], and morerecent libraries such as Layar SDK [6] or Vuforia [7], andmoreover HTML/KML based AR scripting [8] are examplesin this category. On the other hand, the higher level authoringframeworks have mainly included graphical user interfacesthat enable content designers to build AR applications with-out coding or scripting. Non-developers can build AR sceneswithout knowing how to code using, for instance, BuildARtool [9]. The aforementioned authoring systems in both cate-gories, however, have been built for the usage on the desktopor the web.

More recently, the third type authoring approaches haveconcerned allowing the user in-situ authoring of AR con-tent. In-situ modeling or authoring means a creation processwhereby content or scenes are constructed using AR systemwithin which they will be used [10].

Since the purpose of our research is to provide interac-tive creation of user’s own narrative in AR environments, wefurther investigated existing works on real-time in-situ con-tent creation which are in real time applicable to end userscarrying their mobile devices.

So far, exiting research on in-situ AR authoring havemostly focused on letting users define visual shape of vir-tual objects or mixed scene in a simple manner. Hengel’swork [10], for instance, introduced in-situ geometric model-ing from real time video and inserting it back into the samelive video stream. Hagbi et al. [11] presented in-place sketch-ing method for AR content that allows user’s sketching gamearena using visual rules. Lee et al. [12] attempted tangibleauthoring of 3D virtual scenes using marker, RFID, cameraand tangible miniatures, while Langlotz et al. [13] suggestedreal time composition of video content in mobile augmentedreality. All these in-situ modeling or sketching approachesare targeting intermediate role of users creating visual aspectof AR content to serve it for other end users. Also, user inter-actions are not relevant with creating narrative but relatedonly to visual augmentation of content with regard to physi-cal site.

Only few works have concerned end-users involvement ofin-situ AR content creation. Zhu et al. [14] attempted pro-viding an authorable AR system to assist the maintenancetechnicians by enabling them to superimpose the mainte-nance instructions virtually on the real equipment. Wetzel etal. [15] presented Tidy City AR game where users can add in-situ content in addition to web-based pre-built content. Thisallowed user’s scavenger hunt type mission creation basedon simple rules to construct place-oriented riddles. In Wet-zel’s work, it seemed that users could have augmented nar-rative experience while having power to create some riddlesin-situ. While both Zhu and Wetzel’s approaches were thesame in the aspect that they all utilized partly pre-developedAR environment and users could add some variation to that.However, either work has only remained a case study with-out further discussion on the types and possibilities of user’sin-situ augmented narrative creation in broader sense.

When we ponder on why existing works on AR authoringhardly considered user’s narrative-level creativity, we couldfind several reasons. First, real time object tracking and reg-istration techniques for AR have remained so far most urgentand difficult problems in AR scene rendering. In-situ author-ing of 3D objects is still also a challenge. However, user’sexperience and motivation towards the use of digital sys-tems are not encouraged only by visual aspects but also bynarrative and interactive experience given to the users [16].Secondly, in-situ content creation process should be simpleenough for mobile users to carry out creative task while stop-ping at or hovering around some physical location, moreoverin standing or walking situation. It is therefore not feasibleto ask users to do complex composition of content in-situ.

123

Cluster Comput

Table 1 Comparison ofproposed work against relatedworks

Comparable aspects−→ Related works

Onsite narrativeauthoring capability

Onsite authoringcomplexity

Offline authoringeffort

Support for gen-eral guidelines

[10,13] Out of concern(focus: visualshape modeling)

Complex(segmentingvideo, etc.)

Required General (forshape creation)

[14] Limited (totechnician’sillustration)

Simple (text +trajectorysketch)

Required General (fortechnicalillustration)

[15] Limited (byfilling-up riddletemplate)

Simple Required A case study

Proposed work Full narrative Simple Mostly none General (fornarrativestructure)

Table 1 summarizes a comparison of the proposed workagainst the aforementioned related works. Qualitative nov-elty is shown in the aspect that while most previous workshaving focused on visual content authoring, this article aimsto support user’s in-situ narrative creativity. Our distinctiveapproach is that the proposed designs of augmented narra-tive systems are based on narratology. Accordingly, severaltypes of user’s story creation involvement are differentiatedand prototyped in terms of user-narrative interaction levels.

3 System design

This section describes our consideration on design factorsand goals, and then the properties of resulting designs foruser’s participatory narrative in AR.

3.1 Design factors

User activities for in-situ narrative creation are prone to tire-some, in particular in users’ standing and moving situationwhile carrying their smartphones. Also, composing a storyinstantly might cause cognitive pressure of users and theymight thus avoid using the system. With the above in mind,we set up a set of design factors that guide our design processfor interactive AR narrative systems.

3.1.1 Factor 1: Level of interactive narratives

Many literary scholars have mentioned there exist several lev-els of user’s narrative involvement [16–18]. From branchingnarratives such as hypertext to highly interactive computergames, user’s influence levels on resulting digital story canvary diversely. Exploring such diversity will be consideredin our designs of AR narrative interactivity.

3.1.2 Factor 2: Affordable interface

Interface affordance also matters to effectively induce user’svoluntary participation for plot creation. While the number

of meaningless or subsidiary actions such as menu selectionshould be minimized or reduced, users should be able to jumpdirectly into the storytelling process. The degree of freedomgiven to the users should be carefully be constrained whilestill allowing user experience of real time plot generation.

3.1.3 Factor 3: Flow between physical and digital world

The design is driven by the aim to support a fluid interactionacross the physical and the digital domains, which constitutethe digitally augmented space. A fluid interaction perspectivetreats the digitally augmented space as a whole and aims ata continuous experience. To reduce possible discontinuitiesthat can arise from cross-space interaction, the approach heretries to choose adequate interaction metaphors that supportcross-space flow and also integrated narrative.

3.1.4 Factor 4: Joy and playfulness

Besides exerting physical and intellectual activity, it wouldbe nice if the system could meet people’s desire for joy andplayfulness. Joy, fun and gameplay is inherent in both chil-dren and adults. The designs proposed here try to considera discovery by chance and game-like playful events. Sen-suous modalities for interaction is also considered to helpemotional and joyful AR experience.

3.2 Layers of narrative interactivity

As mentioned in our design factor 1, several layers of possi-ble user interactivity in digital narrative text have been dis-cussed in literary context. This paper mainly refers a promi-nent literary scholar M. L. Ryan’s digital narrative theory[17] that adapted Aarseth’s user functions in cybertext [18].As if peeling the onion, there exist the outer shallow layerwhich allows only of user’s exploratory functions and theinner layer for ontology-level user involvement. That is, onthe outer layer, interactivity affects only the presentation ofthe story, and the story itself pre-exists to the running of

123

Cluster Comput

the system. On the inner layer, however, the story is createddynamically through the interaction between the user and thesystem.

This article takes into account these kinds of user’s mul-tiple level engagement through different layers of interac-tivity in augmented narratives. To do that, this paper beginswith characterizing both layers with regard to mobile ARcontext.

3.2.1 Outer layer: exploratory AR narrative

First, in the outer layer of user interactivity in digital text, theuser is able to navigate the database of the pre-determinedstory segments deciding the order to take, but she has noimpact on the virtual story world itself. Therefore, the sim-ple interaction mechanism is enough for users only to controlnarrative discourse and the presentation of the story. While‘story’ refers to the actual chronology of events in a narra-tive, ‘discourse’ refers to the manipulation of that story inits presentation. Figure 1a shows the concept of this layer’snarrative. The important thing in designing this layer’s inter-activity is to provide users some way of controlling the orderor part of the pre-defined story, or determining whether aug-menting it or not in real space according to mobile condition.

In this case, physical objects’ role is to make the AR systemdecide which part of the story to pick out, for instance, bycorrelating some of the real object to some specific part ofthe story or more naturally by using semantic relationshipbetween objects and story content.

Against such a fixed story permutation type, however,probabilistic branching narratives can also be regarded as(b) in Fig. 1. While (a) corresponds to the typical hypertextconcept in literary context, where hypertext means text con-taining links to other texts, (b) loosens the regularity of totallength of the story and constitution of story segments. That is,episodes to explore next are constrained by pre-defined set ofstory segments, but the actual selection of the next story partto explore is determined stochastically and the length of thestory can also vary every execution. Whether it is probabilis-tic or not, user’s main interaction will be choosing links todetermine which part or story to retrieve next to weave a newstory. In this type of probabilistic branching narratives, theAR system’s role is mediating in user’s probabilistic retrievalof some story segment associated in physical objects. A phys-ical object, however, does not have to correspond to certainspecific segment of the story this time, but can have varia-tions so that the user can select one of those variations byprobability.

Fig. 1 Several interactivitytypes of digital narratives

123

Cluster Comput

3.2.2 Inner layer: ontological AR narrative

On the other hand, in the inner layer of AR narrative interac-tivity, ontological form of the story happens when the usershave the power to change the story world. Story world is notfixed in advance, but created on-the-fly based on real-timeuser interaction. Every run of the AR system should result ina different story in this case, and the program should there-fore be re-playable for the fresh new content every time. Inthis layer, it is important to allow user own plot generationonsite in real-time. The problem is that whole story creationcan require quite a time and cognitive effort of users andthis can bring up a question that such onsite real-time storygeneration activity by users is reasonable and feasible. Thedesign and configuration of AR system is thus very importantto support such possibly cumbersome process of onsite cre-ation. Also, the resulting story types allowing easy formationof ontological narrative and the short story length should beconsidered. Therefore, relatively simple story types such aspoem or comic strips are preferred as ontology of narrativein this layer of AR interactivity.

3.3 Design of interactive AR narrative system

Considering suggested design factors in Sect. 3.1 and layersof narrative interactivity described in Sect. 3.2, three typesof AR narrative systems are elaborated in design. Figure 2

shows a generic model of interactive mobile AR narrativesystem where AR app interprets user action into some mean-ingful story manipulation or generation activity and trans-forms or displays the story world based on that interpre-tation. Also, it recognizes physical context associated withstory world and generates feedback of current story contextthrough mobile interface.

Although there might not be one to one correspondencebetween design factors and the components of interactivemobile AR narrative system, each factor can be describedin relation to the design of elements in Fig. 2. The vari-ous levels of interactivity in AR narratives (the design factor1) are related to the variation or generation types indicatedas (a), while affordable interface (the design factor 2) canbe achieved by providing adequate metaphor and interfaceas symbolized by (b) in Fig. 2. Also, the way of associa-tion between physical and story world denoted by (c) con-tributes to the flow between the physical and digital worlds(the design factor 3) and their integrated perception. Finally,a user’s overall on-site authoring activity referred as (d) andaugmented multimodal feedback are likely to give users joyand playful experience (the design factor 4).

Three types of interactive AR narrative designs are elabo-rated according to the interactivity layers introduced in Sect.3.2, and Table 2 summarizes the characteristics of the pro-posed three interactivity layer designs in terms of a–d inFig. 2.

Fig. 2 Generic interactivemobile AR narrative system

Table 2 Three types of designfor AR interactive narrative

Designs −→ Design factors Type A Type B Type C

(a) Layer of narrativeinteractivity (Variation orgeneration)

Exploratory, outerlayer ofinteractivity

Medium layer ofinteractivity

Inner, ontologicallayer ofinteractivity

(b) Metaphor or affordance Uncovering Visiting, catching Comic stripmetaphor

(c) Association of physicaland story world

Fixed mapping Loosely coupled Dynamic association

(d) User’s on-site activity Permutation(Discourse-levelordermanipulation)

Probabilisticselection &arrangement

Dynamic creation

123

Cluster Comput

3.3.1 Type 1 interaction design: ‘uncovering’ metaphor

As a way of doing permutation of content or controlling orderof the presentation, this article proposes ‘uncovering’ con-cept as the key element for this outer-most layer of narrativeinteractivity. Users can explore and uncover a hidden partof the story which is pre-determined but not discovered yetso that the content appears and augments real world in anypersonalized order. User’s smartphone can be utilized as if itis a window to the physical world or object which is alreadylinked to some specific part of the story world as in Fig. 3.In this way, presentation of the story or the concrete expo-sure of discourse will be controlled by users. By graduallyuncovering the parts of the story, the user can complete thewhole narrative.

Actual design of uncovering framework can vary acrossthe applications. In an experiment, for instance, the userdrags reading glass icon appeared in the mobile display asin Fig. 4. When the glass approaches some part of the phys-ical objects, pre-designated corresponding part of the storycontent appears in visually augmented form relatively to realworld objects. Since the user can expose only some part of

Fig. 3 Uncovering concept of interactive AR narrative

the whole story at a moment in this case, she has to assemblethe pieces of puzzles as a whole in her brain. This uncoveringmethod could thus be a way of giving users cognitive fun andan enjoyable surprise.

3.3.2 Type 2 interaction design: ‘visiting’ or ‘catching’metaphor

In a mobile AR environment, content augmentation is asso-ciated with real world context including physical objects orlocations. Rather than literally clicking links between texts,user’s actions thus better have to be re-designed for spa-tial or in-situ interaction with the augmentation of content.Our design decision for this medium layer interactivity typeemploys ‘loosely coupled hypertext’ concept using ‘visiting’,or ‘catching’ metaphor as the spatial and physical mech-anism for selection of the next story segment. Differenti-ated from the type 1 interactivity, loosely coupled hypertextcontains probabilistic links between story elements and thephysical object. By visiting different physical locations orcatching virtual objects from the real space by chance, userscan manipulate variations and generate different versions ofa story, while physically embedded set of story segments arepre-defined. Those virtual objects can be fishes, balls, and soon that are related to some sentences or story parts waitingin the theme pool to be selected by the user.

Figure 5 shows the conceptual structure of this interac-tivity layer. The user can catch some contents or sentencesusing their AR app from each category of themes or contentbox. While the content category would be known to the user,actually retrieved item from each theme box would only leanon a chance. Using her smartphone the user can repeat thisprobabilistic selection as much as she want and also she canmove toward other location or physical object for retrievingcontent from other theme collection. The actual content ofthe theme or category boxes and the level of detail and consti-tution of content can vary according to the application. Theymight be sentence or word collections categorized by themesor part of speech, and otherwise they might be collections ofvisual representations of characters, behaviors, or objects. Asif catching story content at each theme collection pools, userscan collect content and can thus discover unexpected com-

Fig. 4 Type 1 example of‘uncovering’ by controllingreading glass

123

Cluster Comput

Fig. 5 Type 2 structure of probabilistic catching content from themepools

position of content by chance. In addition to that, multipleendings can be generated from one execution to another byuser’s probabilistic path selection. Such an accidental storyweaving would give the user a pleasure of discovery.

3.3.3 Type 3 interaction design: dynamic ‘comic strip’creation

Interactivity design for the inner layer of AR narrative con-siders genuine ontology-level story generation by users. Thatis, stories are not pre-determined, but rather this final typedesign attempts to provide users dynamic creation of a com-plete story on-the-fly. As aforementioned in section 3.2, at aglance, it might be burdensome for mobile users to create awhole story themselves because a plot or a story requires atleast a sequence of events. Onsite story construction mightincrease user’s cognitive load a lot and make them feeluncomfortable to use AR system. For this dynamic story cre-ation layer, therefore, the simplicity of the interface and theintuitive interaction process design is very important. Inter-face affordance and adequate metaphor thus matters to induceuser’s voluntary participation for plot creation.

The design choice here for ontological level interactivityemploys ‘comic strip’ metaphor because comic strip do notseem lengthy and look simple enough to create onsite. Acomic strip is a sequence of drawings arranged in interrelatedpanels to display brief humor or formulate a narrative, oftenserialized, with text in balloons and captions. ‘Word bubbles’in a comic-strip, for instance, can function as an interfaceaffordance that encourages users to fill in them to complete astory panel. Comic strip metaphor can effectively constrainthe degree of freedom given to the users while still allowingexperience of real-time plot generation. Figure 6 shows thislayer’s story creation process through the interaction between

Fig. 6 Type 3 structure of dynamic AR comic-strip creation

the user and the system. A story can come in part from thesystem, and in part from the user. Every run of the programresults in a different story.

There could be many ways to realize such comic stripmetaphor in augmented reality narrative systems. Figure 7shows two examples of the realization of comic strip author-ing concept. As (a) in Fig. 7, the AR system can presentempty word bubbles associated with pre-designated phys-ical objects. In this realization scenario, the user can userher inspiration from the related real-world things to fill inthose bubbles. Some other time, visitors to the location canread them, add to, or modify them by their own inspiration.Figure 7b shows another conceptual realization of the pro-posed comic strip authoring metaphor. In this case, the emptybubbles do not have any pre-designated correlation with realworld objects. Users have more freedom to drag empty bub-bles and assign them to physical objects in the real worldscene. The message or text, of course, should also be filledin by the user. Because of comic strip metaphor, users caneasily understand where to put their text and which virtualtext is related to which of the physical objects, without com-plicating the AR interface.

4 Prototype implementation and empirical study

The prototypes corresponding to the proposed three designsare built on the iOS platform for a smartphone AR app basedinteractive narrative. The purpose of this study is not solv-ing technical difficulties of 3D object tracking and regis-tration which have been typical topics of AR research, butinvestigating different layers of interactive narrative based onAR technique. Therefore, the technical complexity of objectdetection and target tracking is minimized in building theprototypes. That is, markers or objects with well-classifiablefeatures in terms of colors or shapes are selected as physical

123

Cluster Comput

Fig. 7 Type 3 design exampleof dynamic AR comic-stripcreation

Fig. 8 Implementation of interactive uncovering AR narrative prototype

objects. Also, video overlay method is employed for aug-mented scene synthesis while the change of user’s view istracked through iPhone sensors and camera image analysis.Actual implementation of user interaction is also simplifiedusing touch-based interaction on a smartphone AR display.

For the first layer prototype using uncovering metaphor,‘scrubbing dirt in the body’ interaction scenario is adoptedfor fun and pleasure. This is an example of cultural design ofuncovering method applied to AR content discovery becausepeople in Korea have traditions of having their body cleanedby scrubbing using, typically, rough green flannels. Employ-ing such cultural convention as uncovering method, it isexpected for users to be able to enjoy more partial discov-ery of the story by scrubbing dirt out and revealing hiddencontent to be overlaid in the physical space. Since the fun

factor of the augmenting content is also regarded as impor-tant, an actual scene of a famous cartoon or a body tattoocreated by Korean tattoo artist Jun Kim are employed. Asfor more playful interaction, the implemented system givessmartphone vibration feedback while scrubbing so that theuser can feel as if they are actually scouring some rough sur-face while scrubbing the smartphone surface. Smartphoneshaking interaction is further supplemented to allow usersto dust piled dirt out of the scene towards the shaking slope(Fig. 8).

The second prototype implemented ‘catching’ metaphorfor probabilistic narrative creation from given set of sen-tences or content. Several physical marks in round shape areinstalled on the floor in the experiment. They are meant to rep-resent ‘ponds’ where fishes are swimming. The ponds have

123

Cluster Comput

Fig. 9 Implementation of interactive uncovering AR narrative prototype

different colors each other for easy detection and discrimina-tion by an AR app. Different types of fishes are designed toinhabit different ponds, and again, several kinds of fishes areliving in each pond. Among fishes living in a pond, some kindof fish appears very often while some other kind of fish showup very rarely. That is, fishes show up with different proba-bility. User’s act of catching fishes from a pond is detected bysmartphone app’s monitoring the angular velocity from gyrosensors. And the meaning of catching activity is that the userselects a specific pond and catches a fish stochastically amongseveral candidate fishes living in the pond. A pond here cor-responds to a pool of words or sentences, while some specificfish in the pond contains a specific sentence in thematic cat-egory or so. To give users fun and playful experience, theimplemented prototype app displays the score achieved bycatching virtual fishes from the ponds. At the same time, thecaught fishes are sent to the common hyperspace projectedon the wall or ceiling as in Fig. 9. The types and the num-bers of fishes caught on user’s smartphone app are sent tothe web server system and displayed on the webpage. Thewebpage shows the common pool of caught fishes associated

with sentences. When the fishes on the page are projected onthe large screen, the associated sentences are also crossingeach other at the same time on the display.

On the other hand, the final prototype using comic stripmetaphor is built with physical artefact installation in thecomic panel shape. The artefact shape is intended to provideinterface affordance and at the same time to enable usersto have final result of story creation as the form of four-cutcomic strip. Figure 10 shows the four panels installed on thewall in real world. When the user approaches one of the pan-els and sees it through a mobile AR app, a picture appearsand augments the physical corresponding panel. Since eachpanel overlays different picture or drawing, the user can fillin the empty word bubbles with her thought or imaginationinspired by the given picture. This implementation using con-tent that came in part from the system and in part from theuser lessens user’s cognitive load and psychological burdenof creating a whole story.

The initial versions of the three prototypes are developedfor INcheon Digital Art Festival (INDAF) and exhibited inpublic during a month in September 2010. Thereafter they

123

Cluster Comput

Fig. 10 Implementation of interactive uncovering AR narrative prototype

Table 3 Post activity questionnaire

No Question

1 The basic idea of interactive storytelling was easy tounderstand.

2 It was easy to learn how to perform interaction with the app.

3 I understand the relation between real world objects andwhat is overlaid in the mobile display.

4 I enjoyed using the app and feel like creating a story withthe app.

are further revised with more focusing on interactive narra-tives. Empirical study and user observation was carried outin part with visitors during the INDAF period and in partwith other participants afterwards. Total 58 participants wereasked to fill in a questionnaire relating to the mobile AR nar-rative experience for each of three prototypes from the view-points regarding the aforementioned four design factors (asin Table 3). After that, the participants were debriefed in asemi-structured interview. Figure 11 shows the simple statis-tics of participants’ response. Users regarded the prototype 3as most interactive one in the viewpoint of narrative creationexperience, while the narrative interactivity of prototype 2was less understood seemingly because there provided indi-rect mapping between catching fishes and narrative genera-tion with text. It seems that the similar reason also loweredthe interface affordance of the prototype 2, while both proto-type 1 and 3 were regarded as supporting highly affordableinterfaces. On the other hand, the prototype 1 had weak flowbetween the physical object and the virtual augmented objectbecause the participants only focused on dirt-scrubbing activ-ity once the virtual body appeared on the mobile screen. Onthe contrary, as for the aspect of joy and playful experience,the prototype 1 got the highest score. It turned out that the

participants, even middle-aged users enjoyed it most becausethe concept of scrubbing dirt in the body and of dusting thedirt out by shaking mobile phone also with vibrating feed-back gave them quite a playful feeling. Most participantsresponded the first prototype was the most playful in spite ofweak power of narrative creation.

5 Conclusions

Mobile AR applies the concept of AR, which means extend-ing real world with virtual information, in truly mobile set-tings. Most of the existing mobile AR systems, however, havetended to give pre-built or pre-defined content or story to theusers, and we considered that this is partly due to the lackof design framework for making users engage in onsite storycreation activity. Against those unilateral authoring and pro-vision of content, this article thus attempted to explore vari-ous level designs of interactivity in mobile augmented realitysystems. Novel designs supporting onsite interactive AR nar-rative were suggested to make mobile users not only consumepre-defined story but also contribute to real-time discoveryor generation of an AR narrative. Three prototypes, each ofthem representing different level of interactivity in digitaltext, were built and exhibited in public for a month. Theprototype with simplest interactivity offers interactive dis-covery of story segments using uncovering metaphor. Andthe medium level interactive AR system employed catchingmetaphor that allowed users catch words or sentences so thatthey constituted a new narrative leaning on coincidence. Thefinal prototype allowed plot-level composition of a user nar-rative in terms of comic strip metaphor.

Based on the empirical study and user observation,we showed proposed design approach for AR storytellingenabled users effectively involve onsite storytelling experi-

123

Cluster Comput

Fig. 11 Post activity response (scale 1–5, 1 being fully disagree, 5 fully agree)

ence with embracing physical objects as their AR story ele-ments. The proposed interactive AR narrative designs provedto be a joyful and engaging experience for the participantsboth in children and adults. All of three prototypes are fairlysatisfied with regard to four design factors that are narrativeinteractivity, affordable interface, flow between the physicaland digital world, and the experience of joy and playful-ness. Even the third prototype design adopting the form ofcomic strip metaphor required quite a cognitive effort whileusers are standing in front of the physical artefacts, the par-ticipants still enjoyed to generate their own story and triedseveral times for different versions. The result of the empiri-cal study showed that an AR system could be better than justgiving some information overlaid to the physical world. It isthus encouraging that mobile AR systems allow user’s activeinvolvement in narrative creativity with regard to the physicalthings or locations. Provided good interface affordance andnatural flow between the physical and digital worlds, interac-tive AR narrative experience would be useful in many waysin both entertainment and educational contexts.

The limitation of this study is that mobile interaction areatends to be bound to a small locational spot. Our future chal-lenge is expanding the proposed designs for mobile AR nar-ratives so as to allow users intuitive creation and weaving ofa story encompassing scattered multiple scenes and objects.

References

1. Lee, D., Lee, H., Park, D., Jeong, Y.S.: Proxy based seamless con-nection management method in mobile cloud computing. ClusterComput , Online Issue (2013)

2. Capkun, S., Hamdi, M., Hubaux, J.P.: GPS-free positioning inmobile ad hoc networks. Cluster Comput. 5(2), 157–167 (2002)

3. Zheng, W., Xu, P., Huang, X., Wu, N.: Design a Cloud storageplatform for pervasive computing environments. Cluster Comput.13(2), 141–151 (2010)

4. Hasswa, A., Hassanein, H.: A smart spaces architecture based onheterogeneous contexts, particularly social contexts. Cluster Com-put. 15(4), 373–390 (2012)

5. Hirokazu, K., Kihl, M.: ARToolKit: library for vision-based aug-mented reality. IEIC Tech. Rep. 101(652), 79–86 (2002)

6. Layar Augmented Reality Browser, Available: http://www.layar.com/

7. Qualcomm Vuforia Platform, Available: http://www.vuforia.com/8. Hill, A., MacIntyre, B., Gandy, M., Davidson, B., Rouzati, H.:

KHARMA: An open KML/HTML Architecture for mobile aug-mented reality applications, IEEE ISMAR, pp. 233–234, 2010.

9. BuildAR, Available: http://www.buildar.com/10. Hengel, A.V.D.: Image-based Modeling for Augmenting Reality,

Int’l symposium on ubiquitous virtual reality (Invited Paper), pp.1–4, 2010.

11. Hagbi, N., Grasset, R., Bergig, O., Billinghurst, M., El-Sana, J.: In-place sketching for content authoring in augmented reality games,IEEE, virtual reality, pp. 91–94, 2010.

12. Lee, J.Y., Seo, D.W., Rhee, G.W.: Tangible authoring of 3D virtualscenes in dynamic augmented reality environment, computers inindustry, pp. 107–119, 2011.

123

Cluster Comput

13. Langlotz, T., Zingerle, M., Grasset, R., Kaufmann, H., Reitmayr,G.: AR record&replay: situated compositing of video content inmobile augmented reality, ACM OZCHI’12, pp. 318–326, 2012.

14. Zhu, J., Ong, S.K., Nee, A.Y.C.: An authorable context-aware aug-mented reality system to assist the maintenance technicians. Int. J.Adv. Manuf. Technol. 66, 1699–1714 (2013)

15. Wetzel, R., Blum, L., Oppermann, L.: Tidy city–a location basedgame supported by in-situ and web-based authoring tools to enableuser-created content, ACM Int’l Conference on the foundations ofdigital games, pp. 238–241 (2012)

16. Murray, J.: Hamlet on the holodeck: the future of narrative incyberspace, free press. A Division of Simon and Shuster Inc., NY(1997)

17. Ryan, M.L.: The interactive onion: layers of user participation indigital narrative texts. In: Page, R., Thomas, B. (eds.) New narra-tives: stories and storytelling in the digital age, pp. 35–62. Univer-sity of nebraska press Location, Lincoln (2011)

18. Aarseth, J.E.: Cybertext, perspectives on ergodic literacture. TheJohns Hopkins Press, Baltimore (1997)

123