digital media insight · design and production: nakono and sportbusiness group sportbusiness group...

February 2017 www.sportbusiness.com

February 2017

Prepared by:

http://www.sportbusiness.com/

Published February 2017

© 2017 SportBusiness Group

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without the permission of the publisher.

The information contained in this publication is believed to be correct at the time of going to press. While care has been taken to ensure that the information is accurate, the publishers can accept no responsibility for any errors or omissions or for changes to the details given.

Readers are cautioned that forward-looking statements including forecasts are not guarantees of future performance or results and involve risks and uncertainties that cannot be predicted or quantified and, consequently, the actual performance of companies mentioned in this report and the industry as a whole may differ materially from those expressed or implied by such forward-looking statements.

Author: Nakono Ltd

Design and production: Nakono and SportBusiness Group

SportBusiness Group is a trading name of SBG Companies Ltd a wholly-owned subsidiary of Electric Word plc

Registered office:

133 Whitechapel High Street, London, E1 7QA

Tel. +44 (0)207 265 4100

Fax. +44 (0)207 265 4220

Registered number: 3934419


This report was written by Nakono, and is presented by SportBusiness International. It is available exclusively to SportBusiness International and Nakono subscribers.

Nakono produces industry research focused on the digital economy. We explain in words and numbers how the various segments comprising the digital economy work, what is driving their development and how they will evolve in the future.

For more information, go to www.nakono.com

SportBusiness International provides news, analysis, best practice and forecasts for sports industry professionals. The service supports smarter, faster business decision-making with intelligent analysis of critical trends and news, detailed case studies of industry-leading projects, and insight from the industry’s brightest minds.

For more information, go to www.sportbusiness.com/sportbusiness-international


http://www.nakono.com/

http://www.sportbusiness.com/sportbusiness-international

FULL CONTENTS ........................................................................................................................... 4

EXECUTIVE SUMMARY ............................................................................................................... 6

1. ACTIVELY EXPERIMENT WITH AR HEADWEAR ......................................................................... 8

2. INTEGRATE AR INTO STUDIO GRAPHICS TOOLS ........................................................................ 8

3. DISTRIBUTE STUDIO AR FEATURES USING MOBILE APPS ......................................................... 8

4. COLLECT AND MONETISE REAL-TIME CONTEXTUAL DATA ...................................................... 9

INTRODUCTION ........................................................................................................................... 10

DEFINITIONS: VR AND AR/MR ................................................................................................ 12

VIRTUAL REALITY (VR) ................................................................................................................ 12

AUGMENTED REALITY (AR) / MIXED REALITY (MR) ................................................................... 14

INSIGHT: AR/MR WILL BE MUCH BIGGER MARKET THAN VR ....................................................... 15

HOW AR IS BEING USED IN SPORT TODAY ......................................................................... 17

AR STRATEGY #1: CONVENTIONAL MOBILE APPS ......................................................................... 17

Example 1: Seahawks ‘Raise the flag’ ....................................................................................... 17

Example 2: Chelsea FC ‘Kicker’ app ........................................................................................ 19

AR STRATEGY #2: PREMIUM-PRICED AR HEADWEAR .................................................................. 21

AR STRATEGY #3: SPATIALLY-AWARE MOBILE DEVICES.............................................................. 23

GOOGLE TANGO ............................................................................................................................. 23

Intel RealSense ........................................................................................................................... 25

FUTURE USE OF AR IN SPORT ................................................................................................. 27

CONTINUE TO INTEGRATE AR INTO STUDIO GRAPHICS TOOLS ...................................................... 27

FAN ACCESS TO STUDIO AR FEATURES USING MOBILE APPS ......................................................... 30

REAL TIME CONTEXTUAL DATA OVERLAYS ................................................................................... 31

LONG TERM: NEAR VR (NVR) ...................................................................................................... 36

Does a full VR experience make sense in these situations? ....................................................... 37

Lightweight AR glasses will enable a ‘pseudo VR’ experience .................................................. 38


AR HEADWEAR: TODAY AND IN THE FUTURE .................................................................. 40

KEY FUNCTIONAL ELEMENTS ......................................................................................................... 42

Digital object insertion .............................................................................................................. 42

Gaze tracking ............................................................................................................................. 46

Positional tracking ..................................................................................................................... 48

Spatial mapping ......................................................................................................................... 49

Software...................................................................................................................................... 50

CONCLUSION .................................................................................................................................. 50

RECOMMENDATIONS ................................................................................................................ 53


Augmented Reality (AR) is where text, pictures and animations are digitally inserted into someone’s field of view to enhance their experience of the real world.

These digital augmentations can be inserted using AR headsets/glasses but they can also be inserted into mobile apps and integrated into conventional TV broadcasts:

User opens AR app and points their smartphone at a building. A flag is added to the video footage

captured by smartphone’s camera. User can raise/lower the flag.

A studio commentator accesses and ‘displays’ a heat map which shows where a given competitor has spent

the most time during a match. Viewers can see the heat map although this is not visible in the studio.

Realtime data is captured using a sensor

mounted on a snowboard. Data is included in live TV broadcast

User views a digital representation of a real football match which is projected on a table top.

Figure 1: Examples of Augmented Reality

Augmented reality (AR) will grow into a huge, global market in the coming decades for two main reasons:

AR will eventually deliver major incremental value benefits to all web users: The use cases that will be enabled by lightweight, stylish, high performance and low cost AR headwear are utterly compelling and will be valued by anyone who has a smartphone today.

In the end, we will carry a pair of AR glasses with us just as we carry a smartphone today and those glasses will be the primary way we access the web. While the smartphone will live on, it will be reimagined, perhaps in the form of a wristband with a much-reduced display that will be mainly used for text alerts.

When the user wants to access the web, watch a soccer match or join a video conference they will use their AR glasses.

This vision will not be realised for several decades, but when it is then everyone will want AR.


Delivering consumer AR will be hard, but it is technically feasible: The main technical challenge lies in miniaturising the complex optics and electronics contained in advanced AR headsets like Microsoft HoloLens (below) so that it can be integrated into a pair of compact, lightweight glasses.

Figure 2: Components contained in Microsoft HoloLens (courtesy The Verge)

But because AR glasses represent a completely new category of technology – where the rate of improvement in size, weight, performance cost will be governed by a new instance of ‘Moore’s Law’ – we can expect to see improvements similar to those which allowed mobile phones to be reduced in size from a shoulder bag to a matchbox – while the cost dropped and performance increased.

We foresee a time in the future when users will carry their AR glasses with them in a similar way to how they carry smartphones today.

2017 Future: 10-20 years

Microsoft HoloLens AR headset

Lightweight, integrated model (concept)

Figure 3: Evolution of AR Headwear

But while AR holds great promise in the long run, AR glasses/headsets currently cost $2,750 to $5,000, have miniscule market penetration, and are too big and heavy to be interesting to ordinary users.


This means that a mass market for glasses will not exist for at least 10 years (see Figure 3), and even when mass market-ready AR glasses finally start arriving it will take a further decade or more before the technology has been widely adopted.

Sports brands should therefore view ‘headwear-based AR’ mainly as an experimental category for now: there will be no chance of using AR-based headsets or glasses to enable substantial, incremental financial upsides for many years.

On a practical level, sports industry executives have four ways of getting involved in AR today:

Given its massive long-run potential it is important that sports brands set aside at least some level of budget in order to understand how the technology can be applied to their specific situations. At the very least, sports brands should closely monitor how related sports markets are using AR.

AR experimentation programmes should include top-end headsets, such as Microsoft HoloLens, new technologies such as Google Tango (which requires a special smartphone) and also nearer-term, more practical options, such as finding ways to introduce ‘AR-like’ features into existing digital assets, such as mobile apps, an internet website or even conventional TV coverage.

A guiding principle during the AR experimentation phase should be to focus on AR features that can deliver incremental benefits to fans – which ultimately means bringing fans closer to the action and creating a more immersive experience.

Experimentation should not include trying random ideas that require fans to change their behaviour for benefits that are unclear: remember that more than 90 per cent of the entire sports rights market is focused on live coverage and that fans are generally interested in the action on the field of play, not what is going on behind them.

Companies who provide studio-grade sports graphics packages like PIERO will continue integrate AR features into their existing software: Ericsson announced an AR package for PIERO in April 2016 and we think that the company and its rivals are very likely to continue investing in this area (see the ‘heatmap’ shown in Figure 1 for an example).

This approach allows studio commentators to use AR features which are visible to fans watching around the globe – without those fans needing a mobile phone or special glasses.

This can be an excellent way of adding value into sports TV coverage while also educating fans about AR – without them needing to do anything different.

Sports graphics packages like Ericsson PIERO allow broadcasters to deliver ‘more immersive and engaging experiences’ to their audiences. But impressive as they are, PIERO-type features are currently accessible to studio commentators only - not fans.

But there is no technical reason why fans should not have access to these features directly, say using a mobile app and this could become a revenue driver.

http://www.ericsson.com/broadcastandmedia/what-we-do/piero/

https://www.ericsson.com/news/2004174


For instance, a broadcaster who had exclusive live rights for a given sport could offer an ‘AR app’ as part of a premium sports subscription package. Or, alternatively, the app could be positioned as a key retention feature, aimed at building value into a sports package subscription.

It seems likely that pay-TV operators, for whom the furtherance of their live sport franchises is critical, would be interested in such an idea.

The approaches outlined in the previous two sections depend on the vendors of sports graphics products adding AR features to their software.

This means that sports teams, leagues, venue operators and distributors are unable to bring such AR features to the market without the help of sports media software vendors.

But recent technological developments are now providing a way for broadcasters and sports event operators to directly add ‘AR-like’ features to conventional TV broadcasts (see ‘snowboard’ example in Figure 1).

For example, announced at CES in January 2016 and costing a claimed $10.00, Intel’s Curie Module is a coin-sized computer that contains a 6-axis accelerometer and gyroscope which allows the following types of sports data to be captured in real time:

The height of a jump

Ramp take off speed

Number of rotations in air

Time spent in air

In 2016, Intel announced partnerships with New Balance, Red Bull (X-Fighters series) and ESPN in order to deploy Curie on sneakers, snowboards and motocross bikes.

While adding such data to a regular TV broadcast might not appear to be augmented reality per se, it is important to remember that the idea of augmenting reality by adding digital information is not limited to situations where users wear special glasses in order to experience the augmentations.

Advanced data-collection technology like Intel’s Curie module has clear applications in the field of sports performance, both for elite and amateur athletes.

For instance, with Curie sensors attached to the backs of the hands, head and feet etc. it would be possible to carry out a very detailed analysis of a swimmer’s stroke. The same would be true for other skill sports such as golf and skiing.

The data captured would allow coaches to identify and address problems. Meanwhile, amateur athletes would have a way to accelerate the development of their ability.

While wearable sensors have been used to optimise the performance of elite athletes since at least 2011, developments like Curie mean that such technology will eventually work its way down to everyday sports practitioners, as has been the case with heart rate monitors and carbon fibre bicycle frames.


AR currently requires the user to wear a special headset, like Microsoft HoloLens, or to view a mobile app which programmatically combines video captured using the device's camera:

Microsoft – HoloLens Price: $3,000 to $5,000

(aimed at enterprise applications & consumers)

Osterhout Design Group (ODG) – R7 Price: $2,750

(aimed at enterprise applications)

More advanced digital augmentations involve the insertion of complex images and animations.

The more limited capabilities of AR glasses currently restrict digital augmentations to

textual and simple graphical overlays.

Figure 4: Examples of existing AR headwear options and examples of augmentation

Apple CEO Tim Cook said in October 2016 that AR has higher potential than VR. This might seem surprising considering the paucity of AR applications available today: within the sports sector use cases for AR are still at the experimental stage and the industry is far from the point where it is possible to create AR propositions that would be compelling to TV-scale audiences.

So what is the basis for Cook's enthusiasm?

And what convinced companies like Google, Qualcomm and Warner Brothers to collectively invest over $1.4bn of risk capital in the secretive AR startup Magic Leap? The company’s mysterious 'photonic lightfield chip' technology became even more mysterious in December 2016 when an impressive-looking press demonstration was discovered to be an illustration


of what Magic Leap aspires to deliver, rather than an example of what the company can actually deliver.

With a fair amount of fog hanging over key parts of the AR landscape, how should sports industry professionals regard AR?

Is AR another example of an over-hyped technology that lacks a solid foundation, or does AR genuinely have the potential to go all the way – as Tim Cook and others appear to believe?

We prepared this report to answer this question.

It begins by explaining the fundamental differences between AR, VR and MR (mixed reality).

The report then reviews and discusses the pros and cons of the three different product approaches to AR today: AR headwear, conventional mobile apps, and Tango-based mobile apps (which require special Tango-based mobile device hardware).

Next, in order to better understand the potential of AR in sport in the long term, the report describes four AR use cases, each of which has the potential to be interesting to sports audiences worldwide.

Given that the full potential of AR cannot be realised without the ready availability of low-cost, lightweight, stylish and affordable glasses it is essential to get a proper grip on when the glasses will arrive.

By taking a careful look at how AR glasses work, the report clearly explains why affordable AR glasses are at least 10 years away. In this section we can see where Magic Leap might have run into problems, why in 2014 Google pivoted its AR project, Google Glass, to focus on specific applications in the enterprise market and why Apple has not been able to deliver an AR product yet.

The report concludes with eight recommendations that will help executives in the sports industry develop a realistic plan that will deliver some tangible, short-term benefits while laying the foundations for a future where AR will become part of a far larger market – the ‘visual web’.


The objective of VR is for the user to believe they are somewhere else, which may or may not exist in reality:

Inside the cell (See here)

Dive on a coral reef (See: here)

Sistine Chapel

(See here)

First-person-shooter games

(see here)

Figure 5: Remote viewing situations where a ‘full VR’ experience is clearly relevant

With VR, the user wears a close-fitting headset which completely blocks light coming into the eyes from the real world.

The user’s eyes are instead illuminated using a digital display that is part of a VR headset. The digital display can take the form of a smartphone which is clipped into a ‘viewer’ housing or it can be built into a dedicated display.

VR works best when the target reality naturally invites visual exploration in all three dimensions – which means that the user would naturally want to look up/down, left/right and forward/behind.

This implies that VR would probably not make sense for conveying what it would be like to be at a football match, because when at a football match a fan’s natural visual focus is on the action – not on what is going on elsewhere in the stadium, on the ground or whether a plane might be passing overhead.

https://www.youtube.com/watch?v=wJyUtbn0O5Y

https://www.youtube.com/watch?v=fIpJCMxnGeY

http://www.telegraph.co.uk/news/worldnews/the-pope/11493221/Pope-gives-homeless-a-Sistine-Chapel-treat.html

https://www.youtube.com/watch?v=8nS6b9jEXzU


For those sporting situations where VR does make sense such as scuba diving or possibly climbing, then footage will need to be captured using a special VR camera (see Figure 6).

The requirement for a special camera, special video editing tools and changes to media workflows means that VR is a not simply a new video technology. Instead, VR represents a new creative format which demands a new creative approach.

GoPro Odyssey VR Rig Price: $15,000

The camera was developed in conjunction with Google,

which offers a VR content editing and production toolset called ‘Jump’.

Nokia OZO Price: $45,000 (accessories extra)

The price includes access to software tools that are needed to capture and edit the VR content.

Figure 6: Examples of VR camera systems


https://vr.google.com/jump/

Smartphone Display: Budget

(Low-cost solutions that use a smartphone as a display)

Smartphone Display: Premium

(Premium-priced solutions that use a smartphone as a display)

Integrated Display: Premium

(This solution works with a tethered PC)

Homindo ‘Mini’ (Cardboard)

$16.70

Facebook/Oculus Rift

$599 (add $199 for ear

pieces)

HTC Vive

$799

POWIS ViewR (Cardboard)

$29.99

Samsung Gear VR

$99 (only works with Samsung

Galaxy smartphones)

Figure 7: VR headset options

With VR, the user has to make a conscious decision to wear VR headwear in order to disengage from the real world as it exists around them and go somewhere else, which might or might not exist in reality (see Figure 5).

In contrast, with AR/MR, the user wants to extract more information, enjoyment or insight from the real world as it exists around them.

VR headsets fit tightly to the user’s face to prevent light from the real world entering the eyes.

But with AR/MR the goal is to allow the user to see everything that exists around them.

AR/MR involves adding digital information and objects to the user’s field of view. This can be accomplished using headsets or glasses, by adding digital content to a mobile app or even by digitally augmenting a conventional TV broadcast (see Figure 1).

https://vr.google.com/cardboard/get-cardboard/

https://vr.google.com/cardboard/get-cardboard/


The term ‘mixed reality’ or MR is being used by certain technology vendors, such as Microsoft and Magic Leap, mainly to differentiate their offerings from augmentation products that are less advanced than their own.

MR is best viewed as an advanced form of AR where the visual augmentations are far more sophisticated than the simple text overlays and planar graphics which are commonly associated with AR.

MR instead involves inserting complex digital objects into the user’s field of view which seem to fit naturally into the user’s real physical environment.

Achieving the sort of advanced visual augmentations shown on the right hand side of Figure 8 requires dramatically more complex AR headwear and software than that needed to render the relatively simple images shown on the left of the figure.

In the long term we believe the capabilities of AR/MR headwear and the associated software will become so advanced that service providers will be able to create applications where it will become hard to tell what is real, and what is not:

MR Example 1: A user has a conversation with a virtual person who is sitting across the room where the interaction is so realistic it is hard for the user to know whether the virtual person is real, or not.

MR Example 2: A user pays for a kitchen design service which allows him to see what a new kitchen would look like in his home.

The level of detail and accuracy is such that, when looking from a distance, it is hard to tell the difference between the real kitchen (which would have been digitally removed from the user’s vision) and the new kitchen (which would have been digitally added).

Having clarified the differences between VR and AR/MR we can infer that AR/MR has fundamentally higher potential than VR.

If we imagine a point in the far future where the consumer hardware has been perfected for both the VR and AR/MR markets, then it seems clear that VR would still only be interesting for some people for some of the time: with VR, a user needs to make a conscious decision to stop what they are doing and enter a VR session which is of a finite length. VR is an intense, single-task, time-bounded experience where the user chooses to isolate himself from the real world for a defined period of time. While within a VR experience the user cannot do anything else.

In contrast, because AR/MR aims to augment how the user experiences their current reality then it would seem to have the potential to be interesting to a larger proportion of the population for a larger proportion of the day.


Augmented Reality (AR) Simple visual augmentations

Mixed Reality (MR) Advanced visual augmentations

(MR is best regarded as an advanced form of AR)

Company/Product: Google/Glass

Company/Product: Microsoft/HoloLens

Company/Product: Augment/Mobile app

Company/Product: Magic Leap/Visualisation

Figure 8: Difference between AR and MR


In November 2016, NFL team the Seattle Seahawks developed an AR app for iOS and Android devices that allows fans to ‘raise the flag’ of their favourite team.

The app works as follows:

The user first places a real Seahawks flag or Seahawks rally card on the ground, on a table, on a handrail, or anywhere else they choose. The user then opens the ‘Raise the flag’ app and points the smartphone’s camera at the flag or rally card they have placed in front of them.

The app then analyses the captured video footage and, using image recognition software, identifies the location of the flag or rally card. The app then uses this knowledge to insert a digital representation of a flagpole which the user can then control, say by raising/lowering the flag using the phone’s volume control buttons.

The end result is that the footage that has been captured by the smartphone camera has been ‘augmented’ by a virtual flagpole:

Figure 9: Seahawks ‘Raise the flag’ AR app for iOS and Android

The Seahawks app is a relatively simple example of a widely-established trend in which marketers are encouraging their audiences to use smartphone apps to access rich visual information and interactivity features – simply by opening an app and pointing their phone at a physical object, such as a brochure.


Figure 10: How AR can be used to bring a print campaign to life (Image: Augment)

Before scanning with Layar After scanning with Layar

User scans a carton of milk

User can view a video explaining how the product is sourced

User sees an ad for a new movie, user scans the ad

User can then watch a trailer for the movie

Figure 11: Examples of how the Layar app adds digital interactivity to print media

It is easy to see ways in which this technology could be used in sports situations:

A fan sitting in a stadium could use an AR app to scan their seat ticket to access a full event programme.

Collectable athlete cards or stickers could be used to invoke a hologram of the athlete.


In motor racing, fans could view 3D representations of each driver’s car by pointing the smartphone at an event brochure.

First announced in December 2016 at a trade event in China, this mobile app for Android smartphones allows fans to engage with their favourite players, who appear superimposed on video footage captured by the smartphone’s camera.

In Figure 12 the top image shows a Chelsea player which has been digitally added on top of real-world video footage captured by a smartphone (the user must open the ‘Kicker’ app to see this augmentation).

The same applies for the two righthand-most images in Figure 12: in both of these cases digital representations of Chelsea players are shown performing tricks against the background of a stylised football stadium.

However, when using the app, the background the user sees in these two cases would be not be a stylised football stadium, but whatever their smartphone camera is capturing at that time - for instance a real football field, the sky or their living room:

User can choose a player User chooses casual or match attire for their

player

User can test their ball control skills (uses phone accelerometer)

Figure 12: Chelsea Kicker mobile app

This application is an interesting combination of a conventional mobile-app based game, where the user has to use precise hand gestures to perform some task, and AR-based games like Pokemon Go, where a mobile app inserts virtual characters into the video footage captured by a smartphone camera:


http://www.chelseafc.com/news/latest-news/2016/12/blues-launch-ar-game-in-front-of-china-delegation.html

Figure 13: Pokemon Go (Niantic/Nintendo/the Pokemon Company)


The AR applications outlined in the prior section are based on mobile apps which means they can be actively pursued without the need for complex and expensive AR head gear. This section is focused on AR applications which require the user to wear an AR headset.

In September 2016 at IBC Ericsson announced a new feature for the company’s studio-grade data visualisation system, PIERO, which allowed Microsoft HoloLens users to view a graphical representation of the football match between Iceland and England at Euro 2016 – on a table top.

As indicated below in Figure 14, having ‘anchored’ the virtual football field to a suitable physical object (e.g. coffee table) the user could then walk around the match as the action took place.

Figure 14: Football match rendered in miniature on a home tabletop

Ericsson’s demo included a limited amount of interactivity – users could reach out and ‘touch’ a player which would then open a video clip of the actual match at that point.

Most if not all of the AR implementations we have see so far and which have relevance in sport have involved Microsoft HoloLens which, as indicated in Figure 4, presently costs $3,000 (developer edition) and $5,000 (aimed at organisations).

Another headgear option is the ODG R-7, although this more compact product does not have the performance of HoloLens and requires a cable connection to a nearby computer (the computer that is needed for HoloLens is integrated into the actual headset).

Neither of these products are aimed at consumers and each requires its own development approach, which means that an app developed for HoloLens will not work with any other AR headset.

There is currently no AR equivalent of a ‘Google Cardboard’ project which allows users to have some fun and enjoy a VR experience at very low cost (see Figure 7).


https://www.ericsson.com/research-blog/media-coding/future-sports-visualization/

The reason for this will become clear later in this report when we take a closer look at the technology that needs to be integrated into an AR headset - which is far more complex than that needed for VR.

It is for this reason that Facebook has been very public in declaring that the VR will come before AR. Mark Zuckerberg commented in June 2016 that “in five to 10 years, AR will be where VR is today”.

This observation was also made in October 2015 by Oculus Chief Scientist Michael Abrash who said “I think AR will be here, but it’s a long road to get there”.

In April 2016, The Verge reported that Zuckerberg had defined augmented reality glasses as being “what we’re trying to get to.” However, he was also careful to highlight the initial role of smartphones: “the phone is probably going to be the mainstream consumer platform [where] a lot of these AR features first become mainstream, rather than a form factor (glasses) that people will wear on their face.”

In order for headwear-based AR use cases to become a mass market reality, the pricing of products like HoloLens will need to come down to around $200 mark, while the size and weight will need to be reduced by at least half.

Our analysis is that it will take at least 10 years before AR headwear reaches this ‘critical mass’ threshold.

To be clear, these remarks apply to AR applications that require AR headwear – they do not apply to AR applications that use mobile apps, like those outlined in the prior section.

In summary, use of premium-priced AR headwear for sports purposes today will only make sense if viewed in terms of intangible benefits, such as the learning and educational aspects needed to develop a new market, and in terms of managing brand value, for example by generating buzz amongst fans and attracting media interest.

http://www.theverge.com/2016/4/12/11415366/mark-zuckerberg-facebook-f8-virtual-augmented-reality-glasses


A ‘spatially-aware’ mobile device incorporates optical and positional sensors that allow the user to create a 3D digital model of the surrounding environment. The user opens an app and then walks around ‘pointing’ their phone at their environment, mapping its spatial details as they go. This is not possible using a conventional smartphone.

As we will see later, spatial mapping is one of the key capabilities of state-of-the-art AR headsets such as Microsoft HoloLens: spatial mapping makes it possible to position virtual objects within the user’s environment – rather than simply having those objects appear to ‘float’ in space:

Standard AR Device App has no understanding of the background

Spatially-aware AR Device App understands user’s spatial environment

Example: Pokemon Go

Simple object insertion: the AR app has no programmatic understanding of what the device’s

camera is looking at. This is a basic AR feature

Magic Leap Demo

Advanced object insertion: the inserted object is positioned partially behind the leg of the table and

casts a shadow. This is an advanced AR feature.

Figure 15: Difference between spatially-aware and non-spatially-aware AR apps

Before we look at what is possible with spatially-aware AR apps, we want to point out that the explosive success of Pokemon Go proves it is possible to create a very successful AR app using a low-tech approach.

This strongly suggests that sports media professionals should be on the look-out for simple AR features that can resonate with fans, while also experimenting with the top-end products.

For instance, it might be viable to develop a Pokemon Go-inspired mobile app where fans were given clues in advance as to where specific players might be in a town. If fans could then find those players, they might receive an entry into a prize draw, an invitation to a team training session, or special seats at an upcoming match.

There are two main development environments that can be used to build spatially-aware AR mobile apps: Google Tango and Intel RealSense:

First announced in June 2014 as Project Tango, but now called Tango, this is a Google-defined AR development environment that allows developers to create ‘HoloLens-like’ AR applications which users can enjoy using only their smartphone. But the user will need to invest in a special Tango-enabled smartphone, of which there is currently just one model available worldwide; the $500 Lenovo Phab 2 Pro:


https://get.google.com/tango/

http://shop.lenovo.com/us/en/tango/

Figure 16: Tango smartphone – Lenovo Phab2 Pro (introduced September 2016)

Tango devices are presently realised in the form of a highly customised smartphones that include three cameras, one of which is actually a pulsed IR laser, plus the embedded software and silicon needed to create 3D spatial models of the user’s surrounding environment.

One of the two cameras fitted in the Tango device is emits pulses of laser light which – similar to Microsoft Kinect and HoloLens – allows the device to build a 3D model of the surrounding environment.

Having ‘scanned’ their room using their Tango smartphone the user can open a merchant’s app to see what a given furniture item would look like in their home.

Announced in January 2017, BMW and Accenture jointly developed a Tango app that allows visitors to BMW showrooms to use a Tango device to ‘see’ a range of cars that are not actually in the showroom. The sales person hands the Tango device to the customer who can then explore various cars by walking around a specially-assigned area in the showroom where customers can ‘see’ virtual cars. An obvious application of this concept in the sports sector would be to allow motor racing fans to ‘place’ 3D virtual models of Indy F1 cars on the ground. Users would then be able to walk around the car while it remained in a fixed position. This would enable a far richer and more immersive experience than what would be possible by viewing the sort of 3D models that are common at online shopping sites or even with the sort of AR apps outlined in the previous section.

Figure 17: Example AR applications using Google Tango smartphone


We would once again reiterate that there is little point in sports brands trying to imagine applications for Tango-based applications that are aimed at fans because of the paucity of compatible Tango devices.

Given the (current) restriction to one smartphone device, whose market penetration is negligible, sports brands should mainly view Tango as a vehicle for experimentation or for use in situations such as trade events and stadium exhibitions where the limited device availability can be overcome.

A user holding a Tango device has walked down a set of stairs and this has resulted in a 3D model of the staircase being captured.

User is walking inside an office and the Tango device is recording in real time a 3D model of the office (see Google video here).

Figure 18: Examples of 3D spatial mapping using Google Tango

Originally positioned as Intel Perceptual Computing, RealSense is mainly an electro-optic hardware program driven by Intel which aims to equip a broad range of mobile devices with what the company positions as ‘human-like sight’ capability. Intel says this is possible because the Intel-supplied hardware components can measure the depth and relative position of objects within the field of view.

In contrast to Google Tango, Intel’s project is focused on encouraging widespread adoption of the company’s camera sensors and the associated chips that are needed to capture the raw digital data.

The company has a range of camera concepts that can be integrated into and clipped onto smartphones (see below):


https://www.youtube.com/watch?v=Qe10ExwzCqk

http://www.intel.com/content/www/us/en/architecture-and-technology/realsense-overview.html

Intel announced the first RealSense-enabled prototype smartphone in August 2015 at the Intel Developer’s Forum.

This smartphone became available for public purchase in January 2016 but has now been withdrawn, following the comapny’s wider decision to withdraw from the smartphone microprocessor market, where Intel’s share never climbed above the low single digits.

The Intel RealSense 400 miniature camera system was announced in August 2016 at the Intel Developer’s Forum.

This device can be integrated into any suitable device, including smartphones tablets, drones, bicycles, robots, helmets, etc.

The purpose of the device is to allow developers to add spatial awareness functions to arbitrary devices.

Figure 19: Intel RealSense Camera Systems

In this example application of RealSense 400, a drone can fly within a forest without bumping into the surrounding trees or other obstacles.

An example of a sporting application for this technology would be capturing close video footage in situations – such as a rally – where it is not possible to use conventional camera techniques.

Figure 20: Applications of Intel RealSense Technology


The very high price of advanced AR headsets and their extremely low market penetration means that sports brands can forget about developing ‘mass market’ headwear-based AR apps for now – and probably for the next 10 years.

Even if a company like Apple or Magic Leap surprised the market by suddenly announcing a pair of ‘breakthrough’ AR glasses in the next year or so – meaning HoloLens-like performance in a compact package for, say, $200 – then while the AR market would be transformed in terms of developer interest it would still take another two to five years before consumer penetration rose to a meaningful level.

The other option today is to use a Tango-based device, but this is not a mass market solution either because there is presently only one Tango-enabled smartphone, and it has miniscule market penetration. This means that the Tango market can be disregarded for now as well.

We should point out, however, that while sports brands cannot realistically consider developing headwear-based AR apps for the consumer market right now, it is still important to actively experiment with the technology, for instance by demonstrating AR service concepts at trade shows, in stadium, and elsewhere in order to learn what the technology can do and to generate buzz and attract media coverage.

In addition, as noted in the prior section, there remains the possibility to develop AR apps which do not require special AR headwear or special mobile devices.

Given these limitations, what is the next phase in bringing AR experiences to sports fans?

Companies who provide studio-grade sports graphics packages like Ericsson who offers PIERO are likely to continue integrating AR features into their existing software: Ericsson announced an AR package for PIERO in April 2016 and we think that the company and its rivals are very likely to continue investing in this area.

This approach allows studio commentators to use AR features which are visible to fans watching around the globe – without those fans needing a mobile phone or special glasses. This is an excellent way of adding value into sports TV coverage while also educating fans about AR – without them needing to do anything different.

Commentator can analyse goals of specific

players in current and prior matches. Commentator can access and display a ‘heat map’

of a specific player.

Figure 21: Ericsson PIERO – AR features


http://www.ericsson.com/broadcastandmedia/what-we-do/piero/

https://www.ericsson.com/news/2004174

This idea of ‘indirect AR’ is broadly similar to that used by BMW, which has executed a slick-looking Tango-based AR app for use at BMW showrooms – where the cost and limited availability of compatible mobile devices is not a problem.

Golf In this case TV coverage is offered in split screen mode with a digital representation of the shot rendered on the right hand side. The yellow line represents the trajectory taken by the ball.

Cricket Here we see that a range of shots and data relating to part of a cricket match between Australia and New Zealand has been overlaid on top of a digital representation of the stadium. In this image we see that the landing positions of the throws of various bowlers from the New Zealand team have been recorded and overlaid onto the pitch so viewers can compare the accuracy of different players.


Figure 22: Examples of what is currently possible with 3D sports graphics software (Images: VirtualEye)

In this case, the viewing device – a Tango tablet – is owned by the showroom and is temporarily handed to prospective customers who can use the device to ‘walk around’ a virtual car while being able to see on a 1:1 scale what a given car would look like in a different colour or with a different interior specification.

The performance of sports graphics products like Ericsson PIERO and VirtualEye is already quite advanced, as can be seen in Figure 22 and Figure 21.

We can imagine how these already-powerful graphics tools could be further developed.

For example, current state-of-the-art 3D studio graphics allow a complete motor race to be digitised (see Figure 23).

While the quality of the simulated footage is presently very far below the level needed for it to replace conventional TV coverage, improvements in rendition quality and more advanced data integration and analysis features suggest some interesting possibilities:

New camera angles: A digital representation of a complete race circuit means that a camera could be positioned anywhere and capture any perspective. Users would potentially be able to select their own camera position and angle or follow a favourite driver. If they wanted to look behind a given car, to see how close a rival was, then that could be easily accomplished;

Data integration: Users could potentially access on a realtime basis key data on given cars to create their own ‘virtual dashboard’ which could contain data such as g-force, rev counter, speed, fuel etc.

Overtaking windows: Software could be used to calculate, in real time, potential overtaking opportunities for a given driver and these ‘overtaking windows’ could be displayed on the live footage for a given driver.

Knowing the performance of each car and the historic lap times & cornering performance of given drivers it would be possible for fans to see how many passing opportunities a given driver has missed (which could be regarded in a similar way to ‘percentage of first serves in’ during a tennis match).

While carefully noting the sport-specific nature of such features it nevertheless seems clear that when the quality of simulation and appeal of data-driven features reaches a certain point then some fans at least might choose to view a live race in ‘live-simulated’ format, rather than in ‘live-real’ format:


http://virtualeye.tv/

Figure 23: Digital representation of Monaco Formula One Grand Prix – see http://virtualeye.tv/the-galleries/motorsport-videos.

The features illustrated in Figure 21 and proposed above suggest how AR features can be integrated into to a wide range of sports coverage situations.

Ericsson positions its PIERO platform as a way for broadcasters to deliver ‘more immersive and engaging experiences’ to their audiences. But impressive as they are, PIERO features are accessible to studio commentators only - not fans.

For instance, a given virtual object is selected by a studio commentator and then added digitally to the video feed coming from the studio camera rig. The combined feed is then uplinked for broadcast on network TV or sent to a media hub for distribution online.

But there is no technical reason why fans should not have access to these features directly – say using a mobile app.

For example, let’s say that the ‘heat map’ shown on the right hand side of Figure 21 represents the position of a football player. But what say a given fan wants to compare the player’s heat map with that of a teammate? Or the equivalent player on the opposing team? Or what say the fan wants to compare the aggregate heat maps of different positions on the team, say midfield vs. forwards?

Given that all this data probably exists – but that only a very small proportion of it can be exposed within the context of a live television broadcast (pre-/post-match discussion and at half time) then there seems to be far more potential demand for this data that can be accommodated within a standard broadcast media format. It seems plain that fans who follow specific athletes or teams could find value in exploring data about those athletes or teams.

Another example would be in a motor racing situation where some 3D sports graphic packages, like VirtualEye, already offer studio presenters the ability to ‘call up’ a given car, which then appears in the studio, as shown below:

http://virtualeye.tv/the-galleries/motorsport-videos


Figure 24: Studio commentator ‘calls up’ a virtual F1 car (Image: VirtualEye)

But again, this sort of content is currently limited to being inserted at the studio end and viewers can only enjoy the imagery if they view it on a regular TV broadcast.

There is no technical reason why the exact same features could not be accessed by fans on an ‘on-demand’ basis using a mobile app, or any other interactive digital asset.

Therefore we see a possibility for, say, Ericsson to develop a mobile app that could be branded for specific broadcasters (perhaps who owned the rights for event-related data). The app could be downloaded by fans for use during live broadcasts, or potentially at any time.

Alternatively, the raw data could be made available for integration into existing apps.

Depending on the type of data available there is a possibility that such an app could be a direct revenue driver. For instance, the broadcaster who had the live rights for the sport could offer an ‘AR app’ as part of a premium sports subscription package. Or, alternatively, the app could be positioned as a key retention feature aimed at building value into a subscription.

It seems likely that pay-TV operators, for whom the furtherance of their live sport franchises is critical, would be interested in such an idea.

The envisaged app would have a fan-facing UI that would, for example, allow the fan to create comparative data profiles, create alerts and access recorded video footage.

The AR applications discussed in the previous two sections depend on sports media software vendors adding AR features to their software suites.

This means that sports teams, leagues, venue operators and distributors are unable to take action to bring such AR features to market without the help of the vendors.


http://virtualeye.tv/

But recent technological developments are providing a way for broadcasters and sports event operators to use digital technology to ‘augment’ the TV coverage they are providing to their audiences.

Announced at CES in January 2016 and costing a claimed $10, the Curie Module is a coin-sized computer that contains a 6-axis accelerometer and gyroscope (see Figure 26).

This module can be integrated into a very small wireless package which can accurately measure speed and motion in six axes. Intel has announced partnerships with New Balance, Red Bull X-Fighters series, and ESPN in order to deploy Curie models on sneakers, snowboards and motocross bikes.


Application #1

RedBull X-Fighters

A technician holds an Intel Curie sensor prior to addition to a motocross motocycle (Image: Intel)

Application #2 ESPN – Snowboarding

Curie module attached to a snowboard – see white package in foreground (Image: Intel)

Figure 25: Intel Curie system on a chip, realised a complete sub-assembly including battery.

So far, the technology is at an early stage with a customised approach being needed for each sport. In the case of Intel’s deal with Red Bull Media House, for the June 2016 Madrid event the data capture by the Curie module fitted to competitor motorcycles allows the following data to be logged:

The height of a jump

Ramp take off speed

Total time spent off the ground

Time spend off the bike when in mid-air

This data is reported in real time for processing, after which it is made available in a form that can be integrated into live television broadcasts, say as text overlays or graphics objects.

For the example where Intel worked with ESPN to equip the snowboards of X-Games competitors with Curie modules, the following data types were recorded in real time:


https://newsroom.intel.com/chip-shots/intel-curie-red-bull-x-fighters/

Figure 26: Examples of how data captured by the Intel Curie module can be inserted into live TV coverage

Maximum G-force on take-off and landing

Number of rotations

Time spent in air

Jump height

The above examples strongly suggest application in other sports such as cycling, motor racing, and also some team sports such as football where players could potentially wear a Curie module during a competition. This would allow a level of detail that is far greater than what can be inferred from analysing video footage, which is how systems like Ericsson PIERO currently work.

Taking football as an example, it would be easy to compare the total distance covered by a player, peak acceleration, maximum sprint speed and total activity level. And if the motions of a whole team were analysed then it would be possible to programmatically identify the commencement of a specific set play.

It seems that once the technology and back-end analytics software has been productised for one sport then it could be rolled out globally.

While adding such data to a regular TV broadcast might not appear to be augmented reality per se, it is important to remember that the idea of augmenting reality by adding digital information is not tied exclusively to situations where users wear special glasses in order to experience the augmentations.

Therefore we think that this sort of digital augmentation does indeed fall within our definition of AR.

Curie-type technology has clear applications in the field of sports performance – both for elite and amateur athletes.

For instance, with Curie sensors attached to the backs of the hands, head, feet, etc. it would be possible to carry out a very detailed analysis of a swimmer’s stroke. The same would be true for other sports such as golf, skiing, running, etc.


The data captured would allow coaches to identify and address problems. Meanwhile, amateur athletes would have a way to accelerate their progress.

While the use of sensors for sports performance improvement is already well advanced – with examples in baseball (motion sensors located in bats), football (StatSport’s ViperPod) and others, the arrival of devices like Intel Curie represents a step forward in terms of cost and, therefore, market potential.

Rather than being limited to optimising the performance of elite athletes, the likelihood is that such technology will eventually work its way down to amateur sports practitioners, as has been the case with heart rate monitors and carbon fibre bicycle frames.

BMX bikes equipped with two Intel Curie sensors allows the competitor’s tricks to be recorded and categorised in real time. In this case, the software analysing the competitor’s trick detects that part of the trick was performed with some over rotation of the front wheel. It seems that there are clear applications in related sports, such as gymnastics and racket sports.

This shows a screenshot taken of a dynamically-generated computer model of the BMX bike which is based on the motion of the actual bike.

Figure 27: Intel Curie sensors mounted on a BMX bike


http://statsports.com/technology/viper-pod/

In this section we will look at how the advent of affordable, high-performance AR glasses could enable a completely new form of filmed content which is not VR, not AR and not MR.

We have called this new, envisaged type of filmed content ‘near VR’ or NVR.

The best way to see how NVR could arise is to look at a category of highly visual sports action that is being targeted by VR filmmakers who aim to provide fans with a sense of what is like in the thick of the action:

Extreme Night Ski

GoPro Snow: Leo Taillefer, Val d’Isere (See here)

Proximity Wingsuit Flight

GoPro: Le Brevent, Chamonix (See: here)

Formula One

GoPro: Champions Race Mercedes Formula One (See here)

Supercross

GoPro: Cole Seely Heat Race 2017 (See here)

Figure 28: Sports where VR initially seems to make sense may not work in practice

In all of the above situations we can imagine how a suitably-compact VR camera rig could be used to capture footage which would then be viewed using a VR headset.

However, if we think about this a little more carefully we can identify two problems:

https://www.youtube.com/watch?v=5PoL6Vp8GEw

https://www.youtube.com/watch?v=RbcbjMhvjEs

https://www.youtube.com/watch?v=10iDgR05DSE

https://www.youtube.com/watch?v=zZSZPeqkDzs


If we review the definition of VR presented earlier in this report, we said that VR works best when the target reality contains relevant, interesting visual information in all three dimensions – which means that the user would naturally want to look up/down, left/right and forward/behind. But that is not the case in any of the examples shown in Figure 28 – in all of these situations the visual action is coming mainly from one direction: up ahead.

For instance, few fans viewing the footage captured by the camera mounted on the F1 car in the bottom left of Figure 28 would be naturally interested in what lies behind.

Whereas in situations where VR truly makes sense – diving on a coral reef, for instance – then the user would be interested in what lies behind, as well as to the side, and above, and below. See Figure 5 for other examples.

If we now think about the 54 sports identified below in Table 1, we see that many – or most – are in this category, which is that viewers are interested in the action which is mainly in one spatial direction only.

In Table 1 we have divided the 54 sports into six segments, each of which has similar visual coverage requirements. For instance, the approach taken to capture and present live coverage of rugby is broadly similar to that needed for water polo (both ‘Set Team Sports’).

We see that for the purposes of this report we can simplify the coverage requirements of 54 sports by thinking about just six general coverage situations, and we see that for each of these six segments:

Viewers are mostly interested in what is happening where the action is: at the front of the race, around the ball, or the performance of individual competitors as they compete.

Viewers are also far less interested in accessing camera angles that do not show interesting action.

We conclude that VR is far less applicable in most live sports coverage situations than might at first be assumed.

Segment Example sports

Individual sports Golf, Rally Driving, Gymnastics, Downhill Skiing, Horse Racing, Shooting, Indoor Climbing, Archery, Dressage, Show Jumping, Canoeing/Kayaking, Air Racing, Diving, Snooker, Weightlifting, Cycling (individual time trial), Surfing, Freediving

Close rivalry sports

IndyCar, Formula One, Supercross, Road Cycling, Swimming, Triathlon, Cross-Country Skiing, Biathlon, Speed Skating, Rowing

Combat sports

Boxing, Mixed Martial Arts, Judo, Karate, Wrestling, Fencing

Racket sports Tennis, Badminton, Squash

Set team sports

American football, Soccer, Rugby, Ice Hockey, Field Hockey, Basketball, Beach Volleyball, Water Polo, Handball, Polo, Cycling, Indoor Volleyball

Set field sports

Baseball, Cricket, Softball

Table 1: Segmentation of sports market for the purpose of understanding applicability of VR


If we now fast forward to the point when suitable low-cost, high-performance AR glasses exist then we can assume that they will be capable of rendering full motion, high resolution video.

This capability will allow filmmakers to create a more immersive viewing experience, simply because their material will occupy a higher proportion of the user’s field of view than what is possible using any practical physical display screen, as indicated below:

Figure 29: ‘Near-VR (NVR) will become possible when suitable AR glasses arrive

Viewing the footage as indicated on the right hand side of Figure 29 would be like having the footage of the aircraft occupying practically all of the user’s field of vision.

Currently, all sports video is intended to be rendered on a rectangular, planar display screen whereas the type of ‘full view’ rending suggested on the right of Figure 29 would mean that video captured at a live sports event would need to be programmatically adjusted so that when it was viewed using a pair of advanced AR glasses it was not simply like looking at a huge display screen.

But assuming that this correction were possible then the sort of ‘full view’ rendering suggested in Figure 29 would be far more immersive than what is possible with any conventional planar display used for viewing TV content.

Given that for most sports there seems to be a questionable need for a full VR experience anyway (see prior section) then an NVR experience might prove to be good enough for the majority of fans.

One potentially helpful analogy is the case with digital music, where MP3-type sound quality is ‘good enough’ for the vast majority of users: while digital technology allows music to be streamed at studio quality, the incremental improvement in sound quality is simply not worth the downsides such as the need for a fast internet connection, reduced device storage capability, and increased download times.

Thus, we predict that for many sports fans an NVR experience might be good enough.

User views RedBull Air Race on a laptopAction occupies little of the field of view

Weak sense of presence

Users’ Field of view Users’ Field of view

User views RedBull Air Race using AR glasses (future)Action occupies most of the field of view

Stronger sense of presence


Another advantage of NVR is that it could be delivered using conventional cameras which means that sports producers would not need to invest in expensive VR camera equipment or think about the creative questions that stem from how a given sport would best be covered in VR.

Finally, the NVR concept would be applicable to all sports – not just those which seem at first to be good candidates for VR, of which there seem to be few strong candidates.

What we are referring to here is also the partial or, in some cases, complete replacement of a traditional viewing screen with a pair of AR glasses which, by this stage, would probably not be called ‘AR glasses’ – because AR would simply be one usage. Other applications for the same glasses would be VR, visual search and visual computing.


In this section we delve into the technical details of how AR headsets/glasses work, for three reasons:

Why will it take 10+ years for AR glasses to be widely deployed? We would like to explain our rationale for asserting that AR glasses will not reach the mass-market stage for at least 10 years.

Is shrinking HoloLens to a near-conventional pair of glasses feasible? We want to explain why the technology contained in products like Microsoft HoloLens will, in time, be miniaturised to the point where it can be integrated into a pair of stylish glasses that the average user would be happy to wear when they want to access the web.

Why AR is going to be huge? It is not possible to properly understand the long-term potential of a new technology like AR simply by looking at products and services that exist in the market’s formative phase. Instead, it is important to appreciate broadly what AR hardware might evolve into in 20 years.

If our analysis of the AR market were to be solely based on the current form factor of products like Microsoft HoloLens and ODG R-7 then – regardless of the compelling nature of AR applications – our conclusion would be that AR will forever remain a niche market because the headwear is too bulky and too expensive for mass adoption.

Our analysis is that the AR market would be utterly transformed if cost effective, high-performance AR glasses existed. Indeed, the transformation would be so great that AR would simply become one part of a far larger market which could be described as ‘visual computing’.

Therefore it is important to clearly understand where AR hardware is headed.

2016 Selected AR Headwear Systems

Future: 10-20 years Integrated Design

Microsoft HoloLens

Lightweight, integrated model (concept)

Figure 30: Envisaged transformation in the size, weight and performance of AR headwear

It might seem impossible to imagine how a HoloLens headset could be shrunk to fit within the form factor of a conventional pair of glasses, as suggested above in Figure 30, but we already know that a similar scale of improvement is possible: the first GSM transportable appeared around 1985, but a far higher level of performance and functionality is now available at a fraction of the price and also in a tiny form factor (see Figure 31).


Clearly, dramatic improvements have been made in four dimensions: mass, size, performance and price.

To provide readers with some confidence that this scale of improvement is possible with AR headsets, as suggested in Figure 30, we would make two points:

Today, we have access to a far broader and deeper knowledge base so things are going to go faster: The time needed to effect a similar-scale improvement for AR will be less than that needed for mobile phones because, today, engineers have access to a body of knowledge and technical expertise which did not exist in 1985.

For instance, the engineers working on Magic Leap’s so-called ‘photonic light field chip’ can, in principle, draw up the IP developed to design and manufacture the miniature structures used in semiconductors. None of this IP existed in 1985.

AR glasses represent the start of a new instance of Moore’s law which is an easier problem than, say, trying to reduce the Zanco Fly (below) to the size of a fingernail: The well-known geometric rate of improvement in the price-performance point of a given technology – most widely seen in the field of semiconductors, but also observed elsewhere, say in gene sequencing – is just beginning now with AR glasses.

The various electro-optic components, sensors and electronic sub-assemblies that are part of HoloLens have hardly experienced any integration or rate of improvement at all and it is very likely that we can look forward to several orders of magnitude in improvement in the coming decades.

On balance, we think that there is a very good chance that the scale of improvement indicated above in Figure 30 is possible and that a timescale of between 10 and 20 years is feasible, rather then 20 to 30 years as has proven to be the case with mobile phones.

1985 Vodafone VT1 Transportable (4.7 kg, $5,900 in 2016 terms)

2016 Zanco Fly GSM Micro Phone

(20g, $32)

Figure 31: Dramatic improvement in size, weight and performance of mobile phones.


In the remainder of this section we will review the technology contained in a product such as Microsoft HoloLens in order to explain the scale of the challenge that lies ahead. We will show that – unlike with mobile phones – the problem is not so much one of compacting a computer into a tiny package (that job has already been done) but to achieve a level of electro-optic integration that has never been attempted before.

The AR glasses problem amounts to making micro-scale electro-optic and mechanical structures, which is a very different challenge to that which has applied in mobile phones and semiconductors. It is actually this fundamental difference in the nature of the technical challenge that gives us a high degree of confidence that vendors really will deliver the goods, although serious challenges lie ahead.

In the following sections we will review the main technological functions that are included within advanced AR headsets like HoloLens.

We will use the term ‘headset’ throughout, but keep in mind that we are envisioning that all of the functionalities will eventually be miniaturised so that they can be integrated into a pair of lightweight, affordable and stylish glasses.

By the end of this section it will become clear that the headset technology needed to realise a ‘full’ AR experience is far more complex that that needed for VR.

An AR headset needs a way of inserting a digital object into the user’s field of view – without blocking light coming from the surrounding environment.

The rather crude approach used by Google Glass (which actually first appeared on the market in 1996 courtesy of IBM/Olympus) is illustrated below in Figure 32Figure 37:


Figure 32: Google Glass optical projection system

We can see that the inserted objects reside in a small plastic block that is positioned above the wearer’s left eye.

An alternative approach is used on the Meta-2 headset as shown below in Figure 33:

Overlay image coming from internal microdisplay is reflected off the prism surface

Light coming from background passes through.

Composite light field


Announced early in 2016, the Meta-2 AR headset seemed to rely on advanced image projection technologies in order to render impressive-looking images such as the wired ball illustrated above…

...until we learnt that the device is actually using a smartphone display in reverse – in order to project an image onto a plastic screen mounted at 45-degree angle which reflects part of the projected light into the user’s field of vision.

Figure 33: Meta-2 AR headset

The approach used by Google has the advantage that can be accommodated into a pair of glasses, but the disadvantage is that it offers very limited capabilities for digital object insertion – simply because the display is so small and only covers a small percentage of the wearer’s vision.

In contrast, the approach used by Meta-2 and HoloLens allows digital objects to be inserted over a far higher proportion of the wearer’s field of view. But the downside is that a bulky headset is needed.

Techniques have been developed in research projects that promise to allow digital objects to be projected into the edge of a tapered glass lens, whereafter repeated reflections recreate the image in front of the user’s eyeball.

The advantage of this system is that digital objects can be inserted over a far larger proportion of the user’s field of view than what is possible with Google Glass, and a bulky headset is not needed (see Figure 34).


This system uses a specially-formed two-part lens to combine incoming light with digitally added light (left hand diagram) while also being able to measure gaze (right hand diagram)

Figure 34: Technique developed by Dr Hong Hua and Chunyu Gao in 2012/13

The disadvantage of this system is that the resulting lens is very thick – about 15mm, which means that the system would not be viable for mass adoption.

This display allows light to pass through, which allows it to be used as a projector (see Figure 34)

Figure 35: Microdisplay

As suggested below, in Figure 36, Magic Leap has stated that a special ‘photonic lightfield chip’ is in development and also that a manufacturing facility is being built in Florida to produce the device.

If we assume that the image released by Magic Leap of the referenced device is representative of the final product then it seems unlikely that the company is planning to use a micro display, as indicated in Figure 36.

There has been some speculation that the company is instead using a scanning fibre optic tip, which would require very precisely-controlled mechanical driver to control the direction of the fibre to ‘write’ the digital objects onto the lens. Perhaps the fibres, or micro projector system would be located in the frame of the glasses with the images being written directly onto a special lens.


One of Magic Leap’s patent applications includes reference to the term ‘micro projectors’. However, this amounts to little more than speculation.

Today

Lens used by Dr Hong Hua in 2012/13 that simultaneously performs light transmittal, ‘any depth’

digital image addition and eye/gaze tracking

The performance of the 4-year old technology was impressive at the time, but the form factor is still far

from that needed to catalyse a mass market.

2017/18?

This is the lens that Magic Leap has said will be used in its AR glasses system.

If such an apparently thin lens works as claimed then this would represent a major step forward

(Image Magic Leap)

This is the lens assembly that is used on Microsoft HoloLens. In principle, this does not look too dissimilar to the lens that Magic Leap has used in media communications – see above (Image: The Verge).

Figure 36: Is Magic Leap’s ‘photonic lightfield chip’ lens really going to be a elegant as suggested here?

It should be clear that the above technology is already quite complex, but we are just getting started in our description of what is included in an AR headset.

When a user is looking at a particular point, an object in the foreground is automatically defocused by the eye. Whereas if the user shifts their gaze onto the object in the foreground then it immediately comes into focus, while what they were looking at in the background goes out of focus.

This means that if a virtual object is inserted into the user’s vision and the user then looks away from that object then the headset software will need to adjust the object’s focus


accordingly, otherwise the user will find the scene unnatural and prolonged usage might lead to headaches or nausea.

The job of automatically bringing objects into and out of focus as the user’s gaze changes is handled by software. The software needs to know where the user is looking and this means that sensors need to be added to the headset:

PC Realisation: Gaze tracking product is clipped onto the bottom of a PC display.

Smart Glasses realisation: IR light source and camera that captures light reflected

from eyeball are integrated in the frames.

(Image: SMI)

Figure 37: Eye tracking - all AR/VR headwear must be able to measure gaze direction

One way of accomplishing eye tracking in a smart or AR glasses application is to install infrared light transmitters in the frames of the glasses in order to illuminate the eyeballs with (non-visible) infrared (IR) light. IR sensors mounted on the frames of the glasses then analyse the reflected light in order to calculate the direction of gaze (see here).

Eye tracking products for PCs are available from vendors such as Tobii, which has developed a product that can be clipped onto the bottom of a PC (see Figure 37).

Eye tracking can also be used as part of the user interface, for instance when a user wants to select a virtual menu object that is floating in his vision.

While eye tracking glasses like those produced by SMI (see Figure 37) have found application in certain situations, for instance by market research companies who want to measure what


https://www.youtube.com/watch?v=ojGBo9sLqYk

shoppers are looking at while in a supermarket, even these glasses are impractical to be used as AR glasses by a general user.

In an effort to address this issue, the glasses that were developed in 2013 by Dr Hua contained a secondary feature which allows the same lens to be used to measure gaze direction (see Figure 38).

This was achieved by directing the light from a small IR LED into the side of the lens which, following repeated reflections, bathed the eyeball in IR light. The reflected light was then partially captured by the same lens which then directed it towards an IR detector which, with software, allowed the direction of gaze to be measured.

Figure 38: System developed by Dr Hong Hua to achieve eye/gaze tracking by using the same lens that is needed to combine projected images with background images.

This additional functionality also needs to be built into the AR headwear system.

An AR headset must not only know what direction the user is looking in, but also where their head is within the surrounding environment. Essentially, there is a difference between direction of gaze (which is what an eyeball tracking system can measure) and source of gaze (where the eyeball is in terms of x, y, z, relative to the surroundings).

If the position of the user’s head is known then this will allow inserted digital objects to appear in a fixed position, or otherwise to move in a way that would make sense to the user. Alternatively, as the user walks about or moves their gaze, a given inserted object will need to appear in a fixed location if it is not intended to appear to be moving.

Other important effects, such as adding shadows, would also become possible if we knew the position of the user’s head.

Positional tracking is also needed during the time when the user is mapping their surrounding environment: as the user walks about mapping their location the software needs to know their position at all times so it can calculate the form, size and position of surrounding physical objects.


This function can be accomplished using a combination of accelerometers and software.

The next major area of functionality that needs to be integrated into an AR headset is the ability to create and digitally model the user’s local physical environment.

The device must have an ability to create a digital model of a room (for instance) while taking into account the optical properties of certain objects, for example doors windows and chairs – which will allow digital objects to be partially occluded.

It is this ability that makes effects such as that illustrated below in Figure 39 possible:

Spatially-aware AR Device App understands the spatial environment being

viewed by the user

Magic Leap Demo

Advanced object insertion: the inserted object is positioned behind the leg of the table – this is a very

advanced feature.

Figure 39: Advanced virtual object insertion: object opacity and partial occlusion

The system used by Microsoft on the Kinect product which was introduced in November 2010 (see Figure 40) has essentially been compressed into the format shown below in Figure 41 in order for it to be integrated into HoloLens:

By using an infrared laser and knowing the direction of aim, the time taken for the light to reach an object in the room and come back and also the separation between adjacent dots, the system can build a model of the surrounding physical environment.

An infrared laser/camera assembly mounted on front of the Kinect device emits/analyses the light reflected off nearby objects. Software can then calculate a 3D model of the room.

Figure 40: Spatial mapping – as implemented on Microsoft Xbox 360 Kinect


Figure 41: Spatial mapping: Relevant spatial mapping sub-assembly as implemented on Microsoft HoloLens

The final functional element needed in an AR headset is the software that actually performs the image insertion, gaze tracking, positional tracking and spatial mapping functions outlined above – as well as a range of more conventional computational tasks.

In the Microsoft HoloLens this software runs on a stripped-down Windows 10 computing platform that is integrated into the headset.

In this section we have outlined the key functional elements that form part of advanced AR headsets like Microsoft HoloLens.

Five main functionalities needed are:

Digital object insertion

Gaze tracking

Positional tracking

Spatial mapping

Software

That such a range of functions have already been compressed into a headset-sized form factor is already quite impressive but, as we explained at the beginning of this section, we are at the start of a new instance of ‘Moore’s Law’ for this particular category of technology. We can therefore expect to see dramatic improvements in both size, weight, cost and performance in the coming years and decades.

Having studied AR technology in some detail and satisfied ourselves that major, incremental value benefits will be delivered on a generic basis when the headset technology reaches maturation, we are sure that vendors like Microsoft, Apple and others will continue to invest heavily in the technology.


Indeed, some researchers are thinking beyond the form factor of headwear or even glasses and in terms of providing AR functions within the form factor of a contact lens, although the market will not reach this stage for many decades.

This concept would involve creating a transparent, planar structure which would incorporate an electro-optic film comprising individually-addressable 'cells' that could be set to any colour, or set to 'transparent' mode. Other than allowing digital information to be added onto the user's field of view, this concept would also allow for the display of full-motion colour video.

For more information, see Jelle De Smet’s February 2014 PhD thesis ‘The smart contact lens: from an artificial iris to a contact lens display’ (see TED talk here).

User wears smart contact lens during the day Smart contact lens contains electronics, RF antenna and micro battery

Key functional components of smart contact lens Smart contact lenses are charged overnight

Figure 42: Future smart contact lens concept outlined by Dr Jelle De Smet in September 2013


https://www.youtube.com/watch?v=WnjVDzYRv30

Figure 43: Contact lens-type structure which can display images

We note that Sony filed a patent application in April 2016 for a contact lens that can display and record colour video and also that Apple and Samsung have filed similar patent applications for various designs of smart contact lens.


In this report we have explained why a mass market for affordable, attractive, high-performance AR glasses will not exist for at least 10 years.

Sports brands should therefore view headwear-based AR mainly as an experimental category for now: there will be no possibility of using headwear-based AR to generate meaningful financial benefits for some years.

However, because all web users will eventually want it, AR will grow into a huge, global market – similar in size to today’s smartphone market, and equally as successful.

Given the high potential of AR in the long run it is therefore important that sports brands set aside at least some level of budget in order to understand how headwear-based AR technology can be applied to their specific situations.

At the very least, sports brands should closely monitor how related sports markets are using AR.

In order to maximise the rate of learning, AR experimentation programmes should include a range of leading-edge technologies, such as top-end headsets like Microsoft HoloLens, and new smartphone-based technologies like Google Tango.

Sports brands should develop their AR experimentation projects in ways that achieve maximum marketing and brand benefits.

For instance, sports brands should court media coverage of sports-specific AR executions, encourage the use of specially-developed AR applications by celebrity competitors, and forge partnerships with technology vendors who are actively looking for opportunities to deeply embed their AR technology into specific sports.

While experimentation with the latest AR technologies is important, so is the need to find creative ways to introduce ‘AR-like’ features into existing assets, such as mobile apps, internet websites, print collateral, sports merchandise, sports venues and stadiums, and live TV coverage.

In order to gain short-term commercial benefits from AR, sports brands should think creatively about how AR features such as text overlay can be deployed on existing digital assets, such as within the context of a commentary studio or a conventional mobile app.

The reason for this is that if they are found to be successful, such features could be relatively easily repurposed and rendered using AR glasses, when they finally arrive.

AR is not a single technology – such as 4K – which can be rolled out in the same way across the entire sports landscape.

Instead, AR is a far more loosely-defined technology that enables an ‘enhanced visual experience’ with the details of how that enhancement is realised being highly dependent on the sport under consideration.

Therefore, sports brands should take a customised approach to their AR programmes. Clearly, as with existing forms of visual coverage, what works for golf, say, is unlikely to work for downhill skiing.


In searching for the right way to bring AR to a given sporting situation, a guiding principle should be to focus on AR features that can deliver incremental benefits to fans – which ultimately means bringing fans closer to the action and creating a more immersive experience.

Experimentation should not focus on random ideas that require fans to change their behaviour for benefits that are unclear or illusory: remember that more than 90 per cent of the entire sports rights market is focused on live coverage and that fans are interested in the action, not on whether a plane is passing overhead or on checking email.

In summary, headwear-based AR is at least 10 years from the point where it could be legitimately considered ‘mass market ready’.

In the meantime, sports brands will need to think creatively in order to find ways to integrate AR features or even ‘AR-like’ features into their existing digital and physical assets – without users needing to buy special headsets, glasses or mobile devices.

While the commercial rewards for such endeavours will be modest for some time – mainly limited to intangible marketing benefits, although there may be some exceptions – it is important to forge ahead with the experimentation phase and start gently educating fans.

The reason for this is that AR is going to become a huge, global market as big as smartphones and far, far bigger than VR: the use cases are utterly compelling.

Sports brands would be wise to allocate a limited amount of capital to get involved with an emerging technology that, for once, is likely to live up to the hype.

digital media insight · design and production: nakono and sportbusiness group sportbusiness group...

Documents