AUGMENTED REALITY
Gerald Nielson
Overview
What is Augmented Reality? Types of AR Computer Vision overview Future of Augmented Reality AR development
Augmented Virtuality - the merging of real world objects into virtual worlds Virtual Reality - computer-simulated environment that can simulate physical presence in places that are in the real world
What is Augmented Reality?
Augmented- meaning:AmplifiedImprovedEnhanced
What is AR?
Augmented Reality is a live, direct or indirect, view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data.
AR examples On TV
• NFL - First down marker• Team of 4 to run
• NHL – hockey puck tracer• NASCAR – Racef/x
Examples on the internet
• Virtual Shopper – tracks movements and portrays virtual model of clothing
• Ray Ban sunglasses -virtual Mirror• USPS mailbox Size
Useful Applications
Many useful applications have been developed – a lot of room for improvements
Museum “guide-less” Tours BMW augmented Reality - http://
www.youtube.com/watch?v=P9KPJlA5yds
Mobile AR
Many mobile phones have everything needed to create augmented RealityCamera – to determine what is being viewedGPS data – locationCompass – what direction you are facingAccelerometer - orientationInternet connection – provide relevant data
And many people have them - 1/3 of American adults own smartphones
Mobile AR Layar Browser
The world’s leading mobile augmented reality platform Has thousands of “layers” with different applications Location based layers can help the user find nearby
locations Example-BWW Google Goggles Wikitude
Stella Artois Le Bar Guide Displays the location
and information of all bars that serve Stella Artois
Many more apps like this one
Yelp –designed to help people connect with local businesses
Wikipedia – Displays information from Wikipedia entries near your location
Types of AR
Marker Based GPS based Currently, most are based on marketing and
entertainment applications As technology develops, augmented reality
will make its way into other areas
Types of AR Marker based – using a camera these
applications recognize a marker in the real world, and calculate its position & orientation to augment the reality. In simple words they overlay the marker/image with some content or information.
USPS Priority Mail Simulator Advertisements
Types of AR GPS based – these applications take advantage of
Global Positioning System [GPS] tools in your phone. The applications use the ‘position’ of your phone to find landmarks and any other point of interests [POI] near you.
• Stella Artois Le Bar Guide (iPhone only)
• Yelp! – local businesses• Many Restaurants
Types - GPS based Once the POI or landmark is revealed the user
can get additional information about it or get directions to reach there.
Things like ratings of a restaurant, distances, and menus or specials can be displayed.
Overview of Computer Vision
Computer vision has great potential for Augmented Reality
Instead of relying on specific markers to register the camera, natural features can be used
To Register the camera – i.e. enable the camera to recognize certain images
Requires the process of feature detection and matching from specific points in the images
Overview of Computer Vision - Terms Marker - used to specify where/what to
place the information or content Natural features – points/parts of an
object being viewed Detector – used to search images for
points that are repeatable Descriptor – used to analyze the image
for points to be used in the matching step- they characterize the point/region
What is a feature A feature is A specific location in the image Unique point/edge/corner in an image Used to find a small set of
corresponding locations in different images
Points
RegionsStraight Lines
Edges
Feature could be – points or
Feature Detection and Matching
Many methods exist to describe, detect, and match images
Points, Edges, regions, and straight lines of images can all be used
Four steps are involved in the feature detection and matching process
What is a Detector?
Detectors are used to create the descriptor needs to be repeatable meaning, the same feature needs to be
detected in two or more different images of the same scene (Lighting/viewpoint changes)
Taken From the image being viewed
Accounting for viewpoint changes
• Occlusion• Lighting
Changes• Viewpoint
changes• Changes in
contrast/clarity
What is a Descriptor?
A description of the distinctive point From the image stored in the database/application/service
Contains information on the change of color or intensity of an image – 2-Dimensional measurementsUsed for matching
Feature Detection and Matching
Stable detectors are selected in the image
Each interest point is represented by a feature vector- a description of the point Descriptor needs to be distinctive -
distinguishing for this particular point
Feature Detection and Matching
Four stepsFeature DetectionFeature DescriptionFeature MatchingFeature Tracking
Many techniques are used SIFT – Scale Invariant Feature Transform SURF- Speeded Up Robust Features Many More
Feature detection What makes a good feature? some patches can be localized or
matched with higher accuracy than others.
Problems for different image points
The two images (yellow and red) are overlaid. Red arrow is the distance between the centers of the points (a) stable (“corner-like”) flow – easily matched (b) classic aperture problem (barber-pole
illusion) (c) texture-less region
simplest matching method
It is unknown which other image location(s) the feature will end up being matched against.
Therefore, it can only be computed how stable/repeatable it is by comparing the image point against itself
Using the auto-correlation function or surface
Compare the image patch against itself
The original image (a) is marked with three red crosses to mark where these surfaces were computed.
Shows how unique/repeatable a certain point is
Auto Correlation Three different auto-correlation surfaces shown as
both grayscale images and surface plots. Patch (b) is from the flower bed (good unique
minimum), Patch (c) is from the roof edge (one-dimensional
aperture problem), Patch (d) is from the cloud (no good peak)
Obtaining Image Information
Image gradients - a directional change in the intensity or color in an image. Image gradients may be used to extract information from images.
using wavelet responses - SURF
To build the descriptor, a grid is laid over the interest point.
For each square, the wavelet responses are computed - Which are the sums dx, |dx|, dy, and |dy|(amount of change)
Using Image Gradients
• The descriptor entries represent the intensity pattern. • Left: A uniform region, all values are relatively low. • Middle: In presence of frequencies in x direction, the value of |dx| is
high, but all others remain low.• If the intensity is gradually increasing in x direction, both values dx
and|dx| are high.
GAFD
Gravity Aligned Feature Descriptors Recently researched by Metaio More and more mobile devices are being
equipped with inertial sensors (accelerometer) Using the gravity vector to help describe the
orientation of a feature speeds up descriptor computation and the matching process
GAFD – Approaches to take advantage of the gravity
Regular feature descriptors with relative gravity orientation
Gravity aligned feature descriptors Gravity aligned feature descriptors with
relative local orientation local orientation - computed from
neighboring pixels, it is usually computed so that it provides the same region at any viewpoint and view direction.
GAFD
Og – Global Orientation (Blue)
Ol - Local Orientation (Red)
Ol - usually computed so that it will provide the same normalized region at any viewpoint.
• Using the local orientation for descriptor alignment (b) leaves the relative global orientation as part of the descriptor.
• Relative global orientation: Ogl = Og − Ol
• Alignment with the global orientation (c) allows to enrich the descriptor with the relative local orientation
• GAFD in Action
Last step - Feature Matching Once features and their descriptors have been
found and calculated from two or more images, the next step is to establish some feature matches between these images.
Method used to match Depends on the applicationCould be trying to match Images that are known to
overlap – known as image stitching or Images with no correspondence – matching against
images in a database First, select a matching strategy, then devise
efficient ways to perform matching as quickly as possible
The matching algorithm can be evaluated by using the number of true and false matches and match failures, using these definitions:
TP: true positives - the number of correct matches FN: false negatives - matches that were not correctly
detected FP: false positives - estimated matches that are incorrect TN: true negatives - non-matches that were correctly rejected
1
1
1
3
4
2
Simplest Strategy: Set a threshold
Simplest Strategy: Set a threshold
The black digits 1 and 2 are the features being matched against a database of features in other images.
At the current threshold setting (white circles), the green 1 is a true positive (good match), the blue 1 is a false negative (failure to match), and the red 3 is a false positive (incorrect match).
If we set the threshold higher (red circle), the blue 1 becomes a true positive, but the brown 4 becomes an additional false positive.
1
1
1
3
4
2
Feature Tracking An alternative to the last step - Matching
find a set of likely feature locations in a first image then search for their corresponding locations in subsequent
images. This kind of detect then track approach is more used
for a video tracking application Rather than search through all possible images for a
match, keep track of likely features from previously matched images and match accordingly to those
Like the BMW application
The Future of Augmented Reality
”a pair of Google-made glasses that will be able to stream information to the wearer’s eyeballs in real time.”
Where will it be in 10 years? 20 years?Google to Sell Heads-Up Display Glasses by Year’s End - http://bits.blogs.nytimes.com/2012/02/21/google-to-sell-terminator-style-glasses-by-years-end/
Contact lenses are also being developed with electronic displays
The future of Augmented Reality AR is an area of advertising that hasn’t
been fully tapped into yetSome things are not defined yetA lot of potential for marketing to find new
ways of attracting people
Many useful applicationsHeads Up DisplaysPhone applications
Facial Recognition
Privacy Issues Other apps could pull up
other information using facial recognition software
All that is required is a face
Virtual Air Rights
It is a copyright infringement, but it is your phone
An area where the law is not defined yet…
Virtual Advertising Space Who owns the
virtual advertising space?
App creator could charge Starbucks
Someone could make an app that displays a Dunkin Donuts ad over the Starbucks ad
Developing GPS Based Applications Layar offers an open platform to publish on their site
Sign up to be a developer Define and edit a layer on the publishing site Prepare the database
○ stores the POI informationGather POIs information
○ Google Maps is used to get the GPS coordinates – others can be used
Build a web service ○ to fetch the POI information – needs to be formatted in
JSON(JavaScript Object Notation)Test the layer Publish the layer
AR Development
Many tools to build your own Augmented Reality Application
FLARToolkitFLash based Augmented RealityCreated my project using ActionScript in
FlashDevelop Helper classes are included in the library
- Papervision3D(2.0) flash 3D engine to display a 3D object
Sources Bay, Herbert, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. "Speeded-Up Robust
Features (SURF)." Web. 2012. <http://www.cs.jhu.edu/~misha/ReadingSeminar/Papers/Bay08.pdf>.
Frommelt, Daniel M. "Augmented Reality." Augmented Reality. 2009. Web. 21 Jan. 2012. <http://www.uwplatt.edu/web/presentations/PennState/ar/index.html>.
Kurz, Daniel, and Selim Ben Himan. "Inertial Sensor-aligned Visual Feature Descriptors." Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2011. 161-66. Http://da.nielkurz.de/. Web. 4 Feb. 2012. <http://da.nielkurz.de/data/CVPR2011_Kurz_BenHimane_preprint.pdf>.
Lepetit, Vincent. On Computer Vision for Augmented Reality. Tech. Web. 4 Feb. 2012. <http://cvlab.epfl.ch/~lepetit/papers/lepetit_isuvr08.pdf>.
"Re: FLARTOOLKIT/FLASH AUGMENTED REALITY GETTING STARTED." Web log comment. Mikko Haapoja's Blog. 2008. Web. 2012. <http://www.mikkoh.com/blog/2008/12/flartoolkitflash-augmented-realitygetting-started/>.
Szeliski, Richard. "Feature Detection and Matching." Computer Vision: Algorithms and Applications. 205-35. Http://szeliski.org/Book/. 3 Sept. 2010. Web. 11 Feb. 2012. <http://szeliski.org/Book/drafts/SzeliskiBook_20100903_draft.pdf>.
"Developers." Layar. Web. 11 Feb. 2012. <http://www.layar.com/development/>.
Questions?
Demo
FLARToolKit
http://www.uwplatt.edu/web/ar/engineering_ar/engineering_ar.swf
http://www.ray-ban.com/international/science/virtual-mirror/intro
https://www.prioritymail.com/simulator.asp
http://www.webcamsocialshopper.com/