augmented reality for mobile personal computing · • 증강현실기술은향후it...

2009.06.25 코엑스 인터컨티넨탈호텔

Augmented Realityfor Mobile Personal Computing

가톨릭대학교 디지털미디어학부교수 이상국

[email protected]

The 17th Korea Internet Conference

가톨릭대학교 디지털미디어학부교수 이상국 [email protected] KRnet 2009

Prologue

02/54

What will be the role of mobile devices for AR in the year

2015?

What are the technological and conceptual challenges?

What are the future directions of hardware manufactures and

service providers?

What will be potential applications, and who will benefit from

such approaches?


Prologue

03/54

Workflow of AR:

Tracking a picture

Tracking a camera’s position and orientation

Rendering virtual object


Contents

• What’s AR ?

• Hype cycle for AR

• Core technologies for AR

• Applications

• Positioning

• Technology Roadmap for AR

• Technology Tree for AR

• Opportunity Search for Mobile AR

• Conclusion

04/54


What’s AR

05/54

Realenvironment

Augmentedvirtuality

Virtualenvironment

Augmentedreality

Mixed reality

Ref.> Milgram’s reality-virtuality continuum, 1994


Applications

Assembly Medical Game

Technologies

Core technologies for AR

Identificationof real object

Tracking

Computer-generated

data

Rendering

S/W to create& package

contents

Authoring

Simple & tangibleInterface

AR Interaction

06/54


Hype cycle for AR (I)

• 증강현실 기술은 향후 IT 기술 전분야의 발전과 변화에 미칠 파급효

과가 매우 큰 ‘주목해야 할 기술’

07/54

Ref.> 2008 Gartner 기술 분석 보고서


Market Driver

08/54

What is the biggest success story in mobile industry

during past years?

Camera phones

Require better LCD’s

Larger RAM, larger memory cards, DSP

GPU’s (e.g. video processing)

Cameras drive hardware evolution

3D Games benefit

AR will benefit too


What has changed?

09/54

Cameras

Displays

Physical size remains limited

Computation

http://cvlab.epfl.ch/research/augm/fig/tracking/facetrackingBIG.mpg


GPU

10/54

ATI

NVIDIA

Bitboys

PowerVR / Imagination

Falanx

Many others: Mitsubishi, Toshiba, Qualcomm, …


How far is AR from mass-market?

11/54

Hardware evolution makes Augmented Reality possible

Display, camera, processing storage – becoming good

enough

Trackers/sensors

• Possible: magnetic, ultrasonic, inertial… Not all

devices (cost)

• More likely: GPS, RFID, compass, depth camera,

auto-stereoscopic display

• Assume only vision based tracking



12/54

Advantage: Simpler than HMD based AR

Small latencies and registration errors do not matter

Less obtrusiveness

Problems to be addressed for early mass-market Mobile AR

Fast algorithms for tracking (2D & 3D)

Visualization on small display

Integration with OpenGL ES/OpenMAX, easy to use cross-

platform APIs



13/54

Vision Infrastructure

Current camera feature – optimized just picture taking and video capture

Still significant barriers to create new applications(no support for camera/imaging in mobile SDKs)

Require computer vision expertise, image/signal processing experience

From a platform perspective, vision infrastructure is first step:

• Easy-to-use software infrastructure is key: high-level tools are necessary to allow non-expert developers to build AR applications.


Applications

14/54

User interfacesScrolling large documents / web pages / zooming photo etc.Feature-based tracking of incoming video to estimate motion direction and magnitude

Games2D/3D game interaction, shake detection etc.

NavigationLocation-based services,Friend finder etc.


Applications

15/54

http://www.ai.mit.edu/projects/medical-vision/surgery/Images/girl_overlay_large.gif


Applications

16/54

http://webloria.loria.fr/~gsimon/Cga/campusLIN2.mpg


Applications

17/54


Applications

18/54


Applications

19/54

Ref.> ISMAR 2006, 2007 Ref.> SIGGRAPH 2007

http://www.handcircus.com/wp-content/uploads/2006/08/jegghan.jpg


Applications

20/54

Ref.> International Symposium on Wearable Computer 2008


User Evaluation of See-Through Vision for Mobile Outdoor Augmented Reality

21/54

Provides users with see-through vision, allowing visualization ofoccluded objects textured with real-time video information.a user study that evaluates the user's ability to view this information

Ref.> ISMAR 2008


OutlinAR: an assisted interactive model building system

22/54

Allows online building of 3D wireframe models through a combinationof user interaction and automated methods from a handheld camera-mouse

Ref.> ISMAR 2008


Optical Free-Form Surfaces in Off-Axis Head-Worn Display Design

23/54

Single-element eyeglass display design and the fabricated mirrorprototype. Dual-element eyeglass display design with the revisedopto-mechanical packaging. Studies on the field of view limit with an8mm pupil for the dual-element design.

Ref.> ISMAR 2008


A Differential GPS Carrier Phase Technique forPrecision Outdoor AR Tracking

24/54

The position from an initial point is tracked by accumulating thedisplacement in each time step, which is determined using DifferentialSingle Difference. (position error is 10cm, and the drift is 0.001ms-1)

Ref.> ISMAR 2008


The Design of a Mixed-Reality Book

25/54

Semantics of a mixed reality book, the design space and the user experience with MR book type of interface.

Ref.> ISMAR 2008


Relative Pose Calibration of a Spherical Camera and an IMU

26/54

Algorithm for estimating the relative translation and orientation, which does not require any additional hardware, except a piece of paper with a checkerboard pattern on it.

Ref.> ISMAR 2008


An Optical See-Through Head Mounted Display with Addressable Focal Planes

27/54

See-through HMD with addressable focal planes utilizing a novelactive optical element - a liquid lens. Subjective and objectivemeasurements of the accommodation responses of the viewer to thefocal distances.

Ref.> ISMAR 2008


In-Place Augmented Reality:

28/54

The image of the embedded virtual models and their behaviors areunderstandable to a human without using an AR system. No databaseor network communication is required to retrieve the models.

Ref.> ISMAR 2008


Realistic integration of virtual and real imageryin video see-through AR

29/54

Modeling and simulating each step of the imaging process, includingdistortions, chromatic aberrations, blur, bayer masking, noise andcolor-space compression, all while requiring only an RGBA image andan estimate of camera velocity as inputs.

Ref.> ISMAR 2008


Photometric Registration by Adaptive High Dynamic RangeImage Generation for AR

30/54

Virtual objects are overlaid in real-time by adjusting the dynamicrange of the rendered image with tone mapping according to theexposure time of the camera. The HDR image is generated frommultiple images captured with various exposure times.

Ref.> ISMAR 2008


Dynamic Gyroscope Fusion in Ubiquitous Tracking Environments

31/54

Sensor fusion in order to increase robustness and accuracy. Inparticular, the dynamic combination of mobile and stationary trackersenables the creation of wide-area tracking concepts.

Ref.> ISMAR 2008


Collocated AAR: Augmenting After Action Review with Mixed Reality

32/54

Allows users to review past training experiences in situ with the user’scurrent, real-world experience and enables a user controlledegocentric viewpoint, a visual overlay of virtual information, andplayback of recorded training experiences collocated with the user’scurrent experience.

Ref.> ISMAR 2008


Pose Tracking from Natural Features on Mobile Phones

33/54

Present two techniques (SIFT and Ferns Classification) for naturalfeature tracking in real-time on mobile phones. Interactive frame ratesof up to 20Hz for natural feature tracking from textured planar targetson current-generation phones.

Ref.> ISMAR 2008


Virtual Redlining for Civil Engineering in Real Environments

34/54

Annotating either printed paper maps or a 2D geographic informationsystem on a notebook computer taken to the field. finding thephysical location to be annotated on the physical or digital map.

Ref.> ISMAR 2008


Supporting Order Picking with Augmented Reality

35/54

Progress in the iterative process of exploring, evaluating and refiningAugmented Reality-based methods to support the order pickingprocess.

Ref.> ISMAR 2008


Robust and Unobtrusive Marker Tracking on Mobile Phones

36/54

Marker techniques, which are less obtrusive than the usual black andwhite squares. Tracking beyond the visibility of these markers further improving robustness. Implemented in a single tracking library, are highly efficient in their memory and CPU usage and run at interactive frame rates on mobile phones.

Ref.> ISMAR 2008


Fast Annotation and Modeling with a Single-PointLaser Range Finder

37/54

Creating object-aligned annotations with very little user effort.Techniques to segment and pop-up foreground objects. Method usingthe laser range finder to incrementally build 3D panoramas from afixed observer’s location.

Ref.> ISMAR 2008


모바일 혼합현실 기반 체험투어 기술 개발

38/54

모바일 단말기 상에서 카메라에 의한 실 영상 위에 부가적인 정보를 제공하는 가상 객체를 투영하여 혼합현실 상황을 반영한 퍼스널 가이드 및문화/관광 정보를 제공하는데 필요한 핵심 기술 및 시스템

복원된 문화재

디지로그 문화/관광 가이드

황룡사 9층목탑 : 신라 삼보(三寶)의하나로, 선덕여왕 15년(646년)에 완공. 황룡사 찰주본기가 심초석 안에서

발굴됨

*디지로그 퍼스널 가이드

좌측 전방2km 앞 남산

목적지XX 빌딩

웹 링크

콘텐츠 3D 모델 DB

관광/지형/건물/웹 정보

모바일 단말기

혼합현실 콘텐츠 가시화 엔진

위치 및 자세 트래킹 시스템

콘텐츠 서버

모바일 혼합현실 기술 개념도


Non-Visual AR

39/54

• Acoustic augmentation

– Navigation for the blind

– Car parking Aids

– Acoustic Ambient UIs

• Tactile augmentation

– Tactile pen interfaces

– Wearable tactile devices

– Tactile augmentation in cars

• Olfactory augmentation


Non-Visual AR

40/54

• Acoustic augmentation

– Navigation for the blind

– Car parking Aids

– Acoustic Ambient UIs

• Tactile augmentation

– Tactile pen interfaces

– Wearable tactile devices

– Tactile augmentation in cars

• Olfactory augmentation


Positioning

41/54


Current

Status

시나리오기반AR 기술 로드맵

발표자료 형식:Needs, Approach,

Benefit, Competitors

Scenario-based Technology Roadmap for Mobile AR

• AR 기술 및 서비스 응용의 다양한 불확실성을 고려한 복수의 시나리오別로 로드맵을

전개하고, 미래를 대비하여 상황변화의 징후(E.W.S.)를 감지하는 전략수립 기법 적용

Ref.> 이상국, 전략 통합형 4세대 R&BD를 위한 과학적 연구방법론, KIRD 연구개발 인력교육원 연구기획 중급과정, 2008.5.27~28, 대전

42/54


S-based Technology Roadmap for MAR

4343/54

.

.

.

.

.

Ex.> 3년 후 Mobile AR 주요 service application은 무엇으로 할 것인가 ?


Vision based T. SIFT

Ferns

Epipolar …

:GIS based T.

Sensor T.

구성요소

혹은

해결 방법

Tracking

Rendering

Authoring

M. AR

사업 Domain 기술 = Seed 제품, 제법

문제와 해결의 연쇄

(원리에 근거한 기능전개)

Technology Tree for Mobile AR

• Mobile AR기술에 대한 종합적인 관점 하에 R&D과제의 Domain 확인

과제의 기본원리부터 세부기술까지 체계적 전개

선택과 집중해야 할 핵심기술의 도출, PERT 수립, Outsourcing기술 선정

기술DB 구축: 종합적인 특허, 논문 조사분석


44/54


Opportunity Search for Mobile AR


45/54


Opportunity Search for Mobile AR

[1] Idea 발굴(Idea 도출/정의)

[2] Idea Clustering(Idea를 Grouping)

[3] Clusters 선정

<Examples>- Realistic Navigator- Mobile Input Device- Edu./Comm. Devices- Comfortable NED

[4] Application profiling(주요 성능/기술 규명)

46/54


Key Issues (I)

47/54

Industry standards – easy to use software infrastructure (vision, multi-sensor framework, …) for non-expert developers

Scalability – robust tracking solution on any CPU / memory / resolution

Markerless 2D/3D tracking – balance btw. speed and accuracy

Use of DSP/GPUs – for image processing/vision/rendering

Integration with OpenGL ES, OpenMAX platforms


Key Issues (II)

48/54

User interaction - UI metaphors for AR interaction

Power saving solutions

‘Longer’ Term

Projective displays

Tangible UIs

3D off-the shelf cameras (multi-view n x 2D and depth

camera systems 2D+Z


What should R&D focus on? (I)

49/54

Mobile phones are ideal candidates for handy AR platformNeeds for general framework

• Multiple platforms• High-level authoring tools• Multiple sensors

To Mobile manufacturers: APIs and SDK for AR functionsFast matrix computing librariesCompact database systems

To researchers of the tracking mechanism: Robust tracking algorithm for outdoor use

Hybrid method of inertial sensor and vision-based algorithm


What should R&D focus on? (II)

50/54

Massive multiuser applications

Now that everybody has an AR gadget, how can we make

them work together?

Meaningful location based applications

If you can take your AR gadget everywhere, how to turn the

world into interactive application?

Non-Visual AR

Killer Application


Conclusion

Industry standards :

S/W infrastructure, vision and multi-sensor for non-expert developers

Scalability :

Robust tracking solution regardless CPU, Memory size and resolution

Markerless 2D-3D tracking :

Trade-off btw. speed and accuracy

Use of DSP/GPUs :

for image processing (vision and rendering)

User interaction :

UI metaphor for AR interaction

Non-visual AR

Power saving

51/54


Conclusion : Hype cycle for AR (II)

• 증강현실 기술은 초기 과다 비용 지출과 터무니 없는 사업 계획을

초래 할 ‘속 빈 강정’ ?

52/54


Conclusion : Hype cycle for AR (III)

Ref.> 이상국, 혼합현실 기술 연구개발 동향 및 전망, 한국그래픽스학회 논문지, Vol.13, No.2, pp.1~15, 2007

53/54

2009.06.25

Thank you

가톨릭대학교 디지털미디어학부교수 이상국

[email protected]

The 17th Korea Internet Conference

augmented reality for mobile personal computing · • 증강현실기술은향후it...

Documents