484

AUGMENTED CITIES: REVEALED HISTORIESAndrzej ZarzyckiNew Jersey Institute of Technology, Newark, NJ, USA

Introduction

The human relationship with the environment, history, and culture is framed by our already embedded experiences and mental images—we see what we expect to see. In turn, cultural and technological context (the technological tool kit) continuously redefines self-identity and forms new mental images, which then feed back into the aforementioned conformational biases. This continuous feedback loop is pointedly expressed by the recent closing statement to Microsoft’s HoloLens device release video: “when you change the way you see the world, you can change the world you see.”1 This exact paradigm of mutually dependent perception and reality is at the core of the contemporary discourse on immersive media environments. While this is not a new phenomenon, but one which has occurred multiple times throughout history, current implementations seem to be more disruptive, transformative, and at the same time promising in the way they engage everyday lives and connect emotionally with users. This paper looks into interactive augmented reality (AR) environments as places where people engage with the past and experience the future; environments that subscribe to broader cultural history, evoking emotional connections with the physical world and promoting new social interactions.

As during the Renaissance, when the discovery of visual perspective led to new concepts and organization of spaces2, today emerging digital technologies redefine our expectations for the outside reality. This has already been evident in early digital visualizations such as the unbuilt monuments (Fig. 1) that virtually recreated unbuilt designs or no-longer-existing structures to provide new insights about these projects. This new way of visually representing spatial environments has also triggered new technologies that allow us to analyze, dissect, and conceptualize space, often forming unexpected relationships between the virtual and physical worlds. This congruency between the physical and the digital is apparent in projects such as Digitarama3 and Deskrama4 (Fig. 2) that employ the established tradition of an apparatus to experience visual space as discussed by Nagakura and Oshi.

1 http://www.youtube.com/watch?v=aThCr0PsyuA (quote is at time 1m:41s) (accessed 26.03. 2015).2 As discussed by White, John in the book: The birth and rebirth of pictorial space.3 http://cat2.mit.edu/arc/research/digitarama/paper_e_index.html (accessed 26.03. 2015).4 http://cat2.mit.edu/deskrama/ (accessed 26.03. 2015).

485

Augmented Cities: Revealed Histories

Similar to past inventions, current developments do not merely provide a new look into unknown space, but allow for new forms of spatial relations and conceptual connections to be made. AR technology, including projection mapping, is at the forefront of spatial visualization. It is technology that not only considers users but also understands their geo-location and their broader cultural context. Furthermore, AR provides for the multiplicity of personalized perspectives, evoking rich visual and mental narrative collages that involve the participant in active content creation and interpretation.

Fig. 1.Tatlin’s Tower-Unbuilt MonumentsSource: Nagakura, Zarzycki, Brick, and Sich, 1999.

Fig. 2. Deskrama as a space browser designed for three dimensional architectural designsSource: Takehiko Nagakura, and Jun Oshi, 2006.

PART 3. Communication of Heritage

486

New Modes of Seeing

While some of the current AR research focuses on the photographic and near-original virtual recreations that are situated within geographic settings (Fig. 3). This allows for combining three-dimensional photogrammetric models with their corresponding geo-locations and historical data. A parallel lineage treats AR representations as a knowledge building and mapping exercise by Nagakura and Sung. This approach follows the nineteenth-century analytique tradition where multiple drawing fragments together create a visually new and conceptually resolved composition. In this implementation, an analytique becomes a knowledge recording device where element proportions and tectonic relationships are made explicit. Nagakura’s and Sung’s Ramalytique project (Fig. 4) follows this tradition by merging multiple forms of media in a mutually informing relationship. However, the advantage of AR technology lies in the ability to combine the conceptual and diagrammatic representation with photorealistic imagery. The former allows for compositional and tectonic relationships while the latter situates design within an actual environment and provides a commonly-accepted path to understanding the project.

The ability to interact with three-dimensional virtual models by turning on and off various building components provides better understanding of the overall structures as well as facilitates progressive knowledge-building. Furthermore, it takes advantage of the just-in-place and just-in-time learning philosophy based on the contextualized knowledge achieved through virtual, yet situated, context. It also reestablishes a cohesive and unified form of the representation, wherein multiple forms of data and media are tightly connected. This extends the earlier-discussed arguments of Nagakura and Sung (regarding traditional methods of representation) from table-top AR installations to full fledge urban environments.

Fig. 3. Photogrammetric 3D models of the tomb and tombstone at the Mt. Auburn Cemetery, Cambridge, MA. This form of visualization allows for greater user engagements

Source: Andrzej Zarzycki, Mystery Spaces App.

487

Augmented Cities: Revealed Histories

The Mystery Spaces app5 with embedded small photogrammetric virtual models provides a similar layering and registration of multiple media layers, as discussed earlier in the tabletop examples.

The Emergence of Context Awareness

While originally virtual worlds were often developed as dreamscapes or design projections responding to “what if” scenarios, more recently an interest in physically and contextually grounded virtual designs has gained significance. This can also be seen in consideration of materiality and physics-based behavior in architectural digital design6 or the use of physics in gaming engines. The inclusion of the context is an essential element that situates the virtual and serves as the reference scale. It provides a greater relevance of design for everyday users since they can go seamlessly back-and-forth between the physical and the virtual. In this aspect, the situated virtual worlds extend physical experiences and the perception of reality.

5 http://mysteryspaces.com/ (accessed 20.03.2015).6 As discussed by Zarzycki, in: Integrating Physical and Digital Assemblies.

Fig. 4. Deskrama as a space browser designed for three dimensional architectural designsSource: Takehiko Nagakura, and Woong-ki Sung, 2014.

PART 3. Communication of Heritage

488

Look into the Past

The context awareness deployed with AR technology has multiple meanings and methods of implementation. It can refer to a geo-location with an understanding of spatial arrangement in relation to other artifacts. The context can also be understood as visual, cultural or historical. For example, it organizes experience according to chronological events. AR technology is also used to visualize intangible aspects of physical environments. This is the case in a number of mobile AR projects that reveal histories and facilitate urban explorations7. The Niedmermair’s application recognizes, tracks, and augments the existing urban setting while providing 3D model overlays in the historic context by superimposing a virtual model over a camera video feed. The sophistication of the project lies in the combination of image recognition (image targets with CV) and GPS tracking. The image recognition in the context of architecture is particularly challenging, since the visual appearance of buildings (facades) changes depending on the time of the day, lighting conditions, and seasonal variations that include greenery and street-level activities.

TimeWarp8, another mobile edutainment application designed as an AR game situated in Cologne, Germany, focuses on virtual reconstruction of historic buildings by superimposing virtual imagery over currently existing structures. The application not only shows no-longer-existing buildings as they originally appeared but also visualizes design changes that occurred over time to still-present structures. Along the same lines and more recently, ARMedia introduced Coliseum AR app9 that presents the digital reconstruction of the missing portion of the structure (Fig. 5).

7 A project by Stefan Niedmermair (see bibilography).8 A project by A. Herbst (see bibilography).9 http://www.youtube.com/watch?v=WOVjISxlhpU (accessed 25.03.2015).

Fig. 5. Augmented Coliseum by AR-mediaSource: AR-media, 2013.

489

Augmented Cities: Revealed Histories

It also introduces an element of gamification by building in a 3D puzzle that can be played directly on the real Coliseum.

The Immersive Experience of Cultural Heritage10 project uses an AR tour approach to provide tourists with a more realistic experience by placing virtual characters within historical structures. Visitors to the heritage sites of Sajeongjeon and Gangnyeongjeon in Korea can use their mobile devices to access additional facts associated with the showcased physical content. While a similar approach is routinely used by many museums, this particular project does not rely on AR markers such as QR codes. It implements visual camera tracking of the rectangular display space to position its virtual actors without a need for visually intrusive markers.

Look into the Future

AR technologies not only provide an opportunity to contextualize unbuilt designs in their proper urban setting, as is the case with the AR environments showcasing winning entries for the SHIFTboston 2009 Ideas Competition in the Future City Tour app11, but also provide a broader solution for reconnecting semantic layers of multiple histories and traditions. This can be seen in the High Line AR app12 that provides High Line Park visitors with information on historical, current, and future developments of this landmark elevated railroad structure.

The location-aware functionalities allow for positioning and filtering relevant data based on the user geolocation – it knows where the user is and which direction the phone is pointing. Through the app, users can track year-around activities with photographs of various plant species and seasonal foliage. They can visit a particular section of a project and freely navigate through historic photographs and future design proposals related to this locality. In many aspects the app functions as a time capsule that combines multiple layers of information into a single geo-location. These multiple layers can be individually accessed and combined to provide an individualized perspective into the project. To some extent this media overlay provides a third alternative to “renovate and lose the charm of the past” versus “keep the past untouched and do not adapt to new uses or current needs”. The AR component, at least virtually, preserves to a certain extent the original conditions and memories of the past.

Gamified Cites

This is also the case with the AR environment called Mystery Spaces developed by the Tremont Underground Theater Space (TUTS) initiative. This AR app is using gamified virtual tourism media not only to popularize ideas of the adaptive reuse

10 A project by A. Herbst (see bibilography).11 http://www.shiftboston.org/exhibits/future_city_tour.php (accessed 25.03.2015).12 http://www.layar.com/layers/highlinefinal/ (accessed 25.03.2015).

PART 3. Communication of Heritage

490

of the abandoned public infrastructure but also to build social constituency and connect with the general public. The Mystery Spaces AR app encourages users to “[b]ecome an urban explorer and discover the mystery of abandoned public infrastructure, secret tunnels, bridges, architectural landmarks and rooftops. Navigate through the unknown, forgotten, and underappreciated public spaces. Document their daily lives and share them with the rest of the world. Discover what architecture guides can’t tell you!”13

Mystery Spaces is an interactive, locative (context-aware) app for urban adventures. It engages players in mystery-solving-like pursuit with the goal of slowly revealing the secrets hidden within a city. As a game, it gives a new edge to traditional scavenger hunts or geocaching activities by providing engaging ways to unlock hidden clues and visit forgotten urban jewels. Unlike traditional offline games, Mystery Spaces allows players to contribute the knowledge about the city and new Points of Interest (POIs) into an AR environment. The game authoring as well as the social media sharing components are critical new additions to mobile social gaming.

Another important new development allows for following multiple routes while pursuing urban discoveries. This is particularly effective with the multiplayer option, where different players follow routes that adapt to their own knowledge of the city and the ability to solve required clues. The Mystery Spaces app can be used as a multiplayer, location-based online urban game or just as an interactive guide for discovering mysteries of the city. This dual functionality allows for stepping up and down from different levels of participatory engagement.

The AR spatial mechanics are organized around (1) individual POIs as mystery events hidden from players/participants, (2) portals that provide an opportunity to enter the environment/game-space, and (3) keys that unlock the clues for further explorations. Portals are located among POIs and are always visible from a distance when viewed on a mobile device. When players approach a portal, they hear an announcement informing them that they are entering the portal zone. At that point keys with clues become visible, or at least able to be spotted.

Keys are harder to find: they are only visible when a player is within the portal zone and points a mobile device at the right spot – when the camera view aligns with the key’s location, the key appears on your screen with menu buttons and instructions. Finding a key opens two or three mystery spaces POIs. This starts the exploration route - once you find one, other hidden locations will appear. The spatial structure of the environment follows the logic of the gallery and museum design, allowing for multiple and exclusive routes.

According to the app directions, “[p]layers can proceed in any sequence, passing through a different mystery spaces every time. This means that they can come back and play again – the game will be new every time!” (Fig. 6).

13 http://mysteryspaces.com/ (accessed 25.03.2015).

491

Augmented Cities: Revealed Histories

Fig. 6

. Mys

tery S

pace

s, a m

ap w

ith P

OIs a

rrang

ed in

the f

orm

of the

game

play

. Mult

iple r

outes

lead

thro

ugh t

he A

R en

viron

ment.

(Left

) Ind

ividu

al PO

Is (R

ight)

Sour

ce: A

ndrz

ej Za

rzyc

ki, M

yste

ry S

pace

s App

.

PART 3. Communication of Heritage

492

Situated Histories

From individual AR apps overlaying the physical world with context-aware information (High Line AR app) to broader repositories of geo-located data and Web sites/apps, such as the Museum Without Wall14 by cultureNOW, the established physical edifices such as museums and galleries start losing their exclusive role as cultural disseminators. While there is a significant appeal in seeing an original artwork in the controlled environment of a museum, there is also a sense of loss when the artwork is uprooted from its original cultural context and presented as a context-less object.

Situating cultural artifacts in their proper context could lead to a greater appreciation of their value by the public as well as to forming new conceptual and semantical links with new interpretations. The Museum Without Walls is a Web-based datascape that collocates physical environs with data overlays. While not an AR app, it provides contextualized information on art, architecture, and history with broad multimedia content. Its content and the way it is formatted fit naturally into a context-aware model of AR technology.

Final Thoughts

AR has brought the virtual and the physical world closer and made them highly interconnected and interdependent through location awareness, enhanced data overlays, and user-focused content. It also finds its applications in a diverse range of disciplines. AR-based applications increasing occupy an important place in tourism, education, and preserving historical and cultural heritage.

AR technology increases the synergy between the virtual and the physical worlds, making them more interconnected and interdependent through location awareness and user-focused content. It also advances an idea of “learning anytime, anywhere”, which builds on Weiser’s vision for the role of computers in the 21st century.

This new role synergizes key characteristics of AR environments that include location awareness, always-connected networks, and the ability to superimpose digital data over the physical world with interactive and user-friendly graphics. Participants in these AR environments not only visually experience static information but also interact with data and author it in highly dynamic and synthetic ways. These interactions promote an environment of increased user participation with the benefits of experiential learning and the authorship of the public realm. The authorship of the public realm that is at the core of the role architecture plays in culture and society.

14 http://www.culturenow.org/ (accessed 25.03.2015).