Hearing voicesAUGMENTED REALITY AUDIO CAN CHANGE THE WAY PEOPLE EXPERIENCE THE WORLD, BOTH AT HOME AND IN TRAVEL

Tim HaynesPrincipal StorytellerSabre Labs

What is augmented reality (AR) audio?

• A layer of digital audio information that enriches the physical world

• Usually associated with playback of location-based (geotagged) digital audio information

• Includes the hardware and software necessary to play layers of geotagged content in a meaningful way

• Can also include digital augmentation of aural senses to affect a user’s perception of an environment

Partial graveyard of failed AR audio platforms

Causes for the graveyardDiscoverability — Lack of standard name or buzzword• 2011, “dynamic spatial audio” & “ambient spatial audio”

were buzzwords for the next big thing with smartphones• Also: location-based audio, geotagged audio, locative

audio, augmented reality audio, location-aware audio…Failure of the “why” — Startups failed to find a broad and compelling use case for AR audio, focusing on nichesLack of partnership — Most AR audio apps are startups and not integrated into, partnered with or promoted by an existing, successful platform and user base

Continuing challenge: poor mobile interfacesContent playback• Many apps (both past and present) were never

optimized for mobile, even in-app• If it’s hard to find or to play content, an app will failContent creation• Most apps have very poor interfaces for adding

location-specific content, especially mobile creation• The best creation interfaces have tended to be focused

on art and documentary uses in niche locations

Usability premise:Audio makes the real world more interesting

This is Massachusetts Ave. in Cambridge, MA.On the next slide, try different clips of music and see how it changes the way the scene feels.I recommend trying Yann Tiersen, Amelie Soundtrack, “La valse d’Amelie, version original,” followed by some classic jazz and then something hip hop or alt-country.

Presenter

Presentation Notes

This is Massachusetts Ave. in Cambridge, MA.�Try several different clips of music and see how it changes the way this scene feels. I recommend trying Yann Tiersen, Amelie Soundtrack, “La valse d’Amelie, version original,” followed by some classic jazz and then something hip hop or alt-country.

How is AR audio being used?

• Art — theater and conceptual art installations• Documentary— sharing and cataloguing stories• Gaming — AR and MR possibilities are endless• Navigation — associating sounds with places• Notifications — directions, alerts, messages, etc.• Social — sharing experiences, both real and virtual• Tours — cities, historical sites, museums

Placing sounds in space as interactive artArt

2010 – Audio Graffitiproject, Connecticut

2010 – Scapes project*, Decordova Sculpture Park, Lincoln, MA*created w/Roundware

Blended audio tours of art spacesArt

Voices: FAMSF* (2015) An “immersive soundscape” with multiple voices and music centered on 21 sculptures; users can personalize with a mix of curators, art experts and fellow patrons*created w/Roundware

Ghosts of poets in Harvard YardArt

re~verse (2015)commissioned for Harvard Litfest, “consists of a continuous layer of location-sensitive music punctuated with recorded voices of poets who have visited Harvard in the past and given readings of their work.”

*created w/Roundware

High Street StoriesChristchurch, NZ, aftermath of 2011 earthquakes

Docu

men

tary

Tributaries*Tyneside, Northern England, WWI recordings

Docu

men

tary

*created w/Roundware

Listening to the voices I forgot where I was and was taken somewhere else.”

—KERRY BOSSONSPLAY STORE REVIEW OF TRIBUTARIES

“

Docu

men

tary

Location-based gaming is on the rise

Gam

ing

Geocaching Zombies, Run! Pokemon GO

Incorporating AR audio into gaming

Gam

ing

• Several ambitious / experimental smartphone games over the past few years have used AR audio as their main interface:• Inception and Dimensions produced location-based sound and music

and use the phone’s microphone to generate a unique soundscape• Papa Sangre allowed players to navigate a virtual horror soundscape,

attempting to gather lost souls and escape enemies• Meltdown never made it out of prototype, but was an Alien-esque

game in which you hunted aliens on foot using only your ears

• None of these AR audio games are still around (AFAIK) • Immersion in AR games can benefit tremendously from

location-based audio

Using AR audio to navigate

Nav

igat

ion

• Turn-by-turn directions are a common feature of virtually all GPS-based mapping apps, but that’s the most basic use

• Just as the sounds of a city help tell you where you are, an AR audio layer of sound effects and music can convey additional spatial or location information

• Art installations are examples—sound effects and music are tied to places in a exhibition and volume and directionality are directly affected by your location

MIT Media Labs, Loco-Radio project

Nav

igat

ion

Music for navigation and discovery

Using AR audio to give people important context

Not

ifica

tions

• At it’s most basic, having earbuds in can provide private notification of messages or alerts

• More importantly, location-aware alerts can inform people of:• Points of interest• Transportation options• Potential hazards

• On the commerce side—location alerts can market promotions, sales, and/or experience availability

Discoverable content for the visually impaired

Not

ifica

tions

Ariadne GPS (2012)Featured as part of Apple Keynote 2012; create personalized maps with alerts and access audio information through touch about maps in any area. Allows users to understand an area they are unable to see with their eyes

Social elements of AR audio overlap other uses

Soci

al

• Startups have attempted to use location-based audio with chat, friend proximity and other social services

• Art and documentary installations connect visitors with other participants over time

• Gaming uses have the potential to be deeply social• Synchronized tour groups are inherently social as

shared experiences

People want to feel an authentic connection with... what makes a place different from all the other places they could visit in the world.”

—ANDREW MASONDETOUR FOUNDER AND CEO

“Tour

s

Audio tours are increasingly important to travelTour

s

• Plethora of apps for “traditional” audio tours: Izi, Pocket Guide, Rick Steves, UniGuide, etc…

• High adoption by Millennial and Z generations• Detour is the current AR audio darling, one-upping

traditional tour approaches and incorporating limited use of AR audio elements

+

DetourNPR-style tours of San Francisco +9 other cities

• Current tours are professionally produced, taking 3-6 months each

• Participants need to follow tour directions

• Music and speech run separately to account for different movement rates

• Groups can sync tours to share in the experience

• “Descript” turns editing voice to word processing

Tour

s

Audio tours can be cinematic experiences that get you closer to the feeling of walking in another person’s shoes than any other medium can.”

—ANDREW MASONDETOUR FOUNDER AND CEO

“Tour

s

Touring doesn’t have to be proscribedTour

s

• I experimented with various platforms for AR audio content:• Audio Walk — Assign audio files to geographic locations; all

features currently appear broken• Echoes — Limited content (mostly European), significant login

issues and lack of support for content creation• GeoTourist — Basic tours, decent discoverability; open-to-public,

Web-based content creation tool (no mobile creation)• HearUsHere — Handful of tours, poor discoverability of nearby

content, and content creation done only by company• Roundware – really excellent open-source, but it would take a

team effort to create a viable experience• SonicMaps — Terrible interface; lots of broken links; counter-

intuitive content creation

What are the advantages of AR audio?

• Constant context• Adds information without monopolizing attention• Fully adaptive to individual pace and schedule• No fiddling with a device – you interface by moving• Aural illusion of directionality and distance• The brain is really good at filtering sounds, allowing

audio AR to target different layers of your attention

How does AR audio apply to travel?

AR audio opportunities for travel: Art• Theater performances could overlap w/games, potentially

allowing users to take on roles asynchronous w/other users or create new kinds of improve performances

• Art installations enrich content and add a participatory element to art cultural experiences – either standalone AR audio or integrated w/other art – potentially driving additional traffic to specific locations

AR audio opportunities for travel: Documentary• Needs a shared social platform for documentary content for

viability• If not a standardized creation app, at least a site or app that

aggregates, curates and maps niche documentary apps to increase discoverability

• Documentary AR audio allows people to experience locations from new and diverse perspectives, enriching the value of traveling—and spending additional time—in a particular place

AR audio opportunities for travel: Gaming• Huge opportunities for AR audio gaming experiences, both

standalone and with AR visuals (more common), as well as integrated into deeper merged reality experiences

• Location-based digital souvenirs are likely to be a huge part of future travel, largely driven by younger generations; if not the impetus for travel, definitely part of the anticipated travel experience

• When AR audio and visuals combine, “gamification” of everyday life is likely to become more common; lessons learned now by experimenting will help shape future uses

AR audio opportunities for travel: Navigation• AR audio is increasingly essential to travel, both for direction

and for discoverability• Layering a location-based soundtrack could be a powerful

wayfinding method for large buildings or city areas, tying locations or areas to certain music or soundscapes

• Layered with gaming elements, audio navigation can potentially drive people to visit new places

• In the near future, may start to include “superpowers” like near-real time translation of voices and music

AR audio opportunities for travel: Notifications• Being able to layer directions with other sounds makes travel

more seamless (vs. current method of most audio apps/streams demanding exclusivity (e.g. Google Maps stops your music to give you directions)

• Audio alerts can help maintain hands-free, eyeballs-free travel through both indoor and outdoor spaces

• Listening to messages (and advertisements) can be layered as part of background audio, helping with multitasking while moving

AR audio opportunities for travel: Social• To date, efforts to make AR audio social have largely failed, in part

due to the challenge of most AR audio apps doing too little to be worth using; none have been better than existing social options

• Airbnb Tours may be the app that creates critical mass to encourage AR audio experiences, and the leisure nature of much of Airbnb’s business may help give AR audio an inherent social utility of experiencing AR audio in the real world together

• A simpler route that may emerge is for a photo-first platform like Instagram to make it easy to integrate audio clips with geotaggedimages, changing how people catalogue both everyday and travel experiences

AR audio opportunities for travel: Tours• AR audio elements need to be integrated into existing city-

discoverability platforms, from mainstream services like TripAdvisor & Rough Guides to startups like Dojo and Peek

• Tour needs to take guidance from both Detour and from prototypes like Loco-radio, taking users and usage possibilities into account in soundscape design

• When Detour opens its platform and Descript audio editing, it may radically accelerate tour content development (could take off the same way podcasts have)

• Integration of AR audio content into Google Maps (or similar) would be a game changer, normalizing ad hoc tour design

Recommendations to experiment w/ AR audio

Considerations for implementation• Rate of user’s movement (walking, biking, driving, etc.)• Sensory immersion level (e.g. walking=high, driving=low)• Scaling for distance and directionality of sound• Foreground vs. background layers (speech vs. music or noise)• Overlapping layers of sound• Appropriate duration of audio elements (affected by rate of

movement and density of audio elements)

Reco

mm

enda

tions

GeotouristTop option for basic, location-based tour creation

Reco

mm

enda

tions

RoundwareTop development platform for sophisticated AR audio• Roundtable is an open-source platform being used for many of the

most creative examples of AR audio • Ideal for creating and experimenting with “niche” experiences• Allows real-time, geotagged collection of audio from smartphones• Filters and plays back collective, continuous audio streams• Creates seamless, non-linear, location-sensitive layer of audio

mixed on the fly based on user input• Uses Python for server, Swift for iOS app, Java for Android app• “Soon” to be expanded to include video, photos and text

Reco

mm

enda

tions

Possible inflection points for AR audio taking off• Airbnb promoting Detour, then Detour making both creation

and distribution tools open to the public• Apple, Google or Microsoft making AR audio an integral part

of their existing map software• Shift to wireless headphones may be a minor, but notable step

towards always-on AR audio• Visual AR (Hololens, etc.) becoming widespread, and bringing

AR audio along with it (several years out)

Reco

mm

enda

tions

Thank you for reading.I HOPE THIS HAS BEEN HELPFUL.IF YOU HAVE QUESTIONS OR WISH TO DISCUSS TRAVEL IMPLICATIONS OF AUGMENTED REALITY OR OF OTHER TECHS, PLEASE REACH OUT.timothy.haynes@sabre.com