planarwaveguideapparatus with publication · pdf fileapplicant: magic leap, inc., dania beach,...

(19) United States (12) Patent Application Publication (10) Pub. No.: US 2015/0016777 A1

Abovitz et al.

US 20150016777A1

(43) Pub. Date: Jan. 15, 2015

(54)

(71)

(72)

(73)

(21)

(22)

(60)

PLANARWAVEGUIDEAPPARATUS WITH DIFFRACTION ELEMENT(S) AND SYSTEM EMPLOYING SAME

Applicant: Magic Leap, Inc., Dania Beach, FL (US)

Inventors: Rony Abovitz, Hollywood, FL (US); Brian T. Schowengerdt, Seattle, WA (US); Matthew D. Watson, Bellevue, WA (US)

Assignee: MAGIC LEAP, INC., Dania Beach, FL (US)

Appl. No.: 14/331.218

Filed: Jul. 14, 2014

Related U.S. Application Data Provisional application No. 61/845,907, filed on Jul. 12, 2013, provisional application No. 62/012,273, filed on Jun. 14, 2014, provisional application No. 61/950,001, filed on Mar. 7, 2014.

1

808

810

Publication Classification

(51) Int. Cl. GO2B 6/34 (2006.01)

(52) U.S. Cl. CPC ........................................ G02B 6/34 (2013.01)

- - - - - - - - - - - - - - - - - - - - - - - - - - - 385/37

(57) ABSTRACT A waveguide apparatus includes a planar waveguide and at least one optical diffraction element (DOE) that provides a plurality of optical paths between an exterior and interior of the planar waveguide. A phase profile of the DOE may com bine a linear diffraction grating with a circular lens, to shape a wave front and produce beams with desired focus. Waveguide apparati may be assembled to create multiple focal planes. The DOE may have a low diffraction efficiency, and planar waveguides may be transparent when viewed nor mally, allowing passage of light from an ambient environ ment (e.g., real world) useful in AR systems. Light may be returned for temporally sequentially passes through the pla nar waveguide. The DOE(s) may be fixed or may have dynamically adjustable characteristics. An optical coupler system may couple images to the waveguide apparatus from a projector, for instance a biaxially scanning cantilevered optical fiber tip.

Patent Application Publication Jan. 15, 2015 Sheet 1 of 119 US 2015/0016777 A1


3OO 310

32O

S. S Š & & Š S & 8 & &

-20 - 10 O 10 2O

FIG. 3C


FIG. 4B

FIG. 4C


Y1 NA

REE DRIVE SOURCE ||ELECTRONICS

2 11 1

FIG. 5A


1O

YN4.

REEE| DRIVE SOURCE ELECTRONICS

2 11 1

FIG. 5B


10

\ NC

RE DRIVE SOURCE ||ELECTRONICS

2 11 1

FIG 5C


602a

> --" >4-1

N 6 O O

x FIG. 6


700 1 112b 118 M

2a

2C

702 112a

FIG. 7


rts 810 FIG. 8


1102a 1114 1 2 1102b

4NQ-4 Sté Y(N(4)K Šk trarietarrifts


S. S Š

S

& S S

& & &

x (mm)

FIG. 12


4002 -400

FIG. 13a


4004 -400

In3):=

FIG. 13b


Out2=

FIG. 13C


-4008

Out3=

FIG. 13C


Out4=

FIG. 13e

US 2015/0016777 A1 Jan. 15, 2015 Sheet 17 of 119 Patent Application Publication

L-HET GJET

~0001 yoed equOuy (OL

XOed equO (OL

US 2015/0016777 A1

99

plº

Patent Application Publication


S &


56

58

60

62

64

FIG. 18

US 2015/0016777 A1 Jan. 15, 2015 Sheet 22 of 119


~ozov


2100 1.

Detect location of the user 21 O2

Draw a radius denoting area around user that 2104 communicates position and intended direction

Of user

Retrieve piece of passable world based on 2106 drawn radius around the position of the user

Upload information obtained from the user's 2108 environment to passable world model.

Render passable world associated with the 2110 position of the user

FIG 21


22OO 4.

Capture pose-tagged images of the user's 22O2 Surroundings

2204 Extract a set of sparse 3D points from images

Use 2D segmentation methods to find regions that correlate 2D segmented image features with sparse 3D points to derive, using 2D/3D 22O6

fusion, object structure and properties. Run image processing algorithm to segment

Out a particular image.

Parameterize geometry of object 2208

221 O Synchronize parametric geometry to cloud

2212 Re-insert geometric and parametric information

FIG. 22


2400 1.

24O2 Capture a picture of the user's surroundings

Generate a color histogram of the picture

Run a search against database of Color histograms

DOes ldentify location Color histogram No Create new node based on known match an existing in topological map

histogram histogram?

24O6

FIG. 24


() CD S c -

> CD Y


Topological map layered on top of geometric map

v keyframe

FIG. 26


2700 1.

ldentify point of stress based on topological 2702 map

Retrieve set of keyframes associated with 2704 node of topological map

Initiate a bundle adjust on that point of loop 27O6 CloSure

Propagate stress to n=1 around the point of 2708 StreSS, and then to n=2, n=3.

FIG. 27


O

s

2 2 2. se S2

2 s AS S

SSXS(2 N e &Yek Nae %x& SAAké ) RZX SeaSSN-XXXs e V

SSS

W

N & eS ASN

Jan. 15, 2015 Sheet 31 of 119

; 2s 52 2 2 e é es

N & Sy. s S N &A N

W

US 2015/0016777 A1


-2900

Retrieve M key frames with no prior known map 2902 points

Take PCA of normals of the keyframes 2904

Select principal that is most orthogonal to all the keyframes 12906

Place a virtural Camera on the axis of Selected principal 2908

Render a feature buffer 2910

Identify bright spots as candidate locations of N features 2912

Create a label buffer to stack lines and Save M, N labels 2914

Draw a mask radius around each bright spot to eliminate poorly conditioned rays 2916

Feed collection of masked rays to bundle adjuster to estimate location of map points 2918

FIG. 29


Z uu??SKS J?ST)

90ZZ


Use Cameras and various input 23O2 elements to Collect images, points and data

Determine sparse points based on input elements 2304

Crawl through points to recognize 2306 One or more objects using the map database

Transmit the mapping information 2308 and recognized objects to the local AR system

Display a desired virtual Scene to 2310 the user in relation to recognized objects

FIG. 31


-3200 Totem Eye Gesture

Game Tracking Tracking Tracking

Hands/gestures; Cinematic totem Reality

Eye Cameras (x2)

(Stereo Cameras) ImageS (X2) Depth Map 1

IR active Fusion projectors

(Wide FOW Cameras)

Images (X2) MU

FIG. 32


-3300

Identify sparse point

Perform point normalization

Generate sparse point descriptor

ldentify dense point

Perform point normalization

Generate dense point descriptor

2403b

2405b

Store as map data

FIG. 33


34OO M

Structured Of Patterned O Textured Light Light Light

Y, *

Camera 1 Fiber-based Camera 2 Projector

Projection data

Texture Extraction Module

FIG. 34

US 2015/0016777 A1

SJOJJE80G8 Jan. 15, 2015 Sheet 38 of 119

Z099



-3600 Image Data

Yes

NO

Yes

NO

Yes

NO EtC.

FIG. 36A


ldentify candidates 36O2

Perform analysis 36O4

3606

Remove Candidates that fall 3608 below designated threshold(s)

FIG. 36B


?peoseO

oggg-^


-3660

FIG. 36D


7 -36 O

Normalize Orientation

FIG. 36E


-3680

Subtract background

R>

FIG. 36F


-3690

FIG. 36G


-3698

2

FIG. 36


?O?IÐS

KdOO/QeJ?)


Detect and capture user's interaction 2702 With a totem

Detect a position, orientation and/or movement of totem with respect to 2704

reference frame

Map user's interaction with position/ 27O6 Orientation and/or movement Of totem

Determine user input based on the 2708 mapping

FIG. 39


4024 4.

FIG. 42C


S. FIG. 43b

FIG. 43C

FIG. 43C


4040


4040 M

FIG. 45b.

4044-N ES

FIG. 45C

Jan. 15, 2015 Sheet 57 of 119 US 2015/0016777 A1 Patent Application Publication

FIG. 46a

FIG. 46b


-4100

Identify virtual Ul 4102

Generate data for Virtual Ul

Send data to user device

Display UI based on Sent data

Wait for Command from USer

Recognize gesture or

Other COmmand?

Generate virtual content/perform action based

On COmmand

FIG. 47


4832 y


N

FIG. 48g

FIG. 48h


N

Y L

e D CD O CD L C H n O C

CD


N


43OO M 30

Detect movement Of 43O2 user's fingers/hands

Recognize a valid gesture 4304 based On Cetected movement

Retrieve map points corresponding 4306 to position of fingers/hands

Create Umirroring user's 4308 fingers/ hands

Display UI in real-time at position 4310 selected by the user

Detect another movement signifying 4312 end of user interface Creation

Display UI at map Coordinates 4314 desired by user

FIG. 49


5002

FIG.50a


FIG. 5Ob


5006 M


FIG 52a

FIG. 52b


52O6 N

3. V2 2 ZJ

FIG. 52C

US 2015/0016777 A1 Jan. 15, 2015 Sheet 77 of 119

807€” J?Aues pnORO 0099_^


US 2015/0016777 A1 Jan. 15, 2015 Sheet 78 of 119

oorvº



4200 1. Receive data of real-WOrld 42O2

activity

Identify real-world 4204 activity

Load knowledge base associated with 42O6 real-world activity

Receive audio/visual/sensory input from 42O8 USC

Determine specific activity and other related 421 O information based on received user input

Check knowledge base to determine relevant data based On determined

specific activity

Display relevant information? virtual content 4214

4216 4218

Recognize other yes Perform action based real-world activity On user feedback

4212

Receive user feedback?

FIG. 55


N

US 2015/0016777 A1 Jan. 15, 2015 Sheet 87 of 119

CIENEdO WOON W ENWH ÅCIWERHTV7 TIOÅ



St. iii.)

a 1- y -

X a - 1 a O

1. As Y ear O

72Y ... -- M --- s

^: , / - Maa 2

O

Ne: O 27. - - -

s'Y Wr

Y M a Y 2

S. O N ea Yss

A. y N

planarwaveguideapparatus with publication · pdf fileapplicant: magic leap, inc., dania beach,...

Documents