"how computer vision is accelerating the future of virtual reality," a presentation from...
TRANSCRIPT
![Page 1: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/1.jpg)
Copyright © 2016 AMD 1
How Computer Vision is Accelerating
the Future of Virtual Reality Allen Rush
May 2, 2016
![Page 2: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/2.jpg)
Copyright © 2016 AMD 2
• Virtual Reality — current view
• Computer Vision support in VR
• Future of VR with Advanced CV
• Challenges — Technical, Business
Agenda
![Page 3: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/3.jpg)
Copyright © 2016 AMD 3
VR Has Come a Long way
• Experience primarily thru close
display and optics
• Immersion thru control of local view
& positional info
• Factors: resolution, optics
accommodation, vergence, frame
rate, dynamic range…
• 2016: ~2M users, 200,000 developers
• >260 startups + key players
![Page 4: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/4.jpg)
Copyright © 2016 AMD 4
VR Experiences — Imagination to Immersion
Training and Simulation
Gaming
Big Data Visualization
Entertainment Virtual Social World Remote Presence
Medicine Education
![Page 5: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/5.jpg)
Photorealism
in VR
Full Presence
Progression of REAL-TIME Visual Experiences
Basic 2D
Rendering
Physically
Based
Rendering
Basic 3D
Rendering
Immersive
2D Displays
Shaders VR
Level of
Presence
Today
![Page 6: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/6.jpg)
Copyright © 2016 AMD 6
VR + Camera, CV Basic Functions
Viewport Display vs Human FOV
![Page 7: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/7.jpg)
Copyright © 2016 AMD 7
Functions and processes to produce VR:
Latency challenges
![Page 8: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/8.jpg)
Copyright © 2016 AMD 8
• Elementary: Feedback for eye tracking, positional information
(X,Y,Z, pitch, roll, yaw, x,y,z motion- 9 DOF), extremity detection
(hands, feet, etc.)
• Fundamental: IR tracking, positioning, calibration
• Complex: deep learning training for user’s eyes expressions,
motion prediction
• Sensory fusion: combining visual, tactile, accelerometers to
model dynamic environment
Computer Vision in VR
![Page 9: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/9.jpg)
Copyright © 2016 AMD 9
• “Rendering of a virtual human that can purposefully interact with a real person — for
example, through speech recognition, the generation of meaningful sentences, facial
expression, emotion, skin color and tone, and muscle and joint movements — is still beyond
the capabilities of real-time computer graphics and artificial intelligence,” write
neuroscientist Maria V. Sanchez-Vives and computer scientist Mel Slater
• Key improvements in: latency, resolution, FOV, color, dynamic range, focus management
• Key additions: eye tracking, surround vision capture, position accuracy, object detection and
tracking, 3D view angle
• Key Assumption: Speeds and feeds will improve, optics will improve, additions such as eye
tracking and immersive capture will develop as natural components of VR platform
• Emergence of VR/CV development community — SDKs, app development, user groups,
metrics, reviewers, etc.
Virtual Reality — Quickly evolving and
improving, but has a way to go…
![Page 10: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/10.jpg)
Copyright © 2016 AMD 10
Example: Leap Motion “Virtual Hands”
• Start of Visual Projection + Visual Feedback
• Next Step: Complex rendering, recognition, depth, tracking, etc.
![Page 11: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/11.jpg)
Copyright © 2016 AMD 11
• Application developers: rich collection of topics to choose from
• Military (long history), medical, games (obvious), cinema (new), social media,
education, immersive work environment
• Content creation: CGI, cameras, video composite
VR Application Space — Diverse and Innovative
VR Touch UI HTC Vive Prototype Demo VR Hockey
![Page 12: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/12.jpg)
Copyright © 2016 AMD 12
• “The seat at the 50-yard line in the first or second
row, that might go for around $1,000 a ticket, but
what you could also do is sell that same seat for
$10 a ticket to 100,000 people.”
– Max Cohen, Oculus
• Seeing vs. experiencing: immersive and sensory
• Collaborative: Immersive telepresence
• CV: eye tracking, face reconstruction,
expression projection, etc.
Advanced VR Applications: Extending Basic
Ideas to Make Compelling Experiences
Virtual conferencing
![Page 13: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/13.jpg)
Copyright © 2016 AMD 13
What Does Advanced VR Look Like?
Gaze detection, learning and prediction
Automotive safety: obstacles,
cues, dictionary detection
Once baseline VR is acceptable,
many new apps, features, ideas,
and functions can be developed
![Page 14: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/14.jpg)
Copyright © 2016 AMD 14
• New solutions to old ideas
• Low latency rendering: creative
management of pipeline rendering
(ex: LiquidVR)
• Foveated rendering
• Sync of sensor, detection, overlay
• New Tools
• Stitching, mixing content,
workflow management
What Does it Take?
Videostitch panoramic workflow
Mix in: CV detection, recognition,
learning, tracking and tagging
![Page 15: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/15.jpg)
VR Vision
To achieve the vision of
full presence you need
Scalable
CPUs, GPUs,
Accelerators
Sound Other Senses
Sight
Sensory
Integration
![Page 16: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/16.jpg)
Copyright © 2016 AMD 16
Business Opportunities and Ecosystem
![Page 17: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/17.jpg)
Copyright © 2016 AMD 17
• Content Developers
• Obvious: game developers. 3D games natural transition to 3D VR
• VR cinema. 360o capture, reformat, render, projection to VR display
• UI: mouse/keyboard replacement
• Captured content mixed with graphics content
• CV overlays: detection, object tagging, tracking
• Platform SDKs: standard methods for development
• Testing, metrics, distribution, support
• Optimization: compilers, libraries, porting to new platforms
Ecosystem
![Page 18: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/18.jpg)
Copyright © 2016 AMD 18
• Game titles: sell vs. rent vs. service?
• Collaborative: time on station? Server access
• 1+1+1=4 (cardboard + smartphone + camera = VR/CV)
• Embedded: system integration + services
Business Models
![Page 19: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/19.jpg)
Copyright © 2016 AMD 19
Challenges Going Forward
![Page 20: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/20.jpg)
Copyright © 2016 AMD 20
• Improve render and display experience
• Limit of human vision, based on foveal
cone spacing
• 2x210degx150@ 1pix/arc-min-> ~226Mpix
• At 90FPS ->~20Gpix/sec rendering speed:
settle for much less
• Need good optics, lens shading, CA, etc.
• Render complexity
• 1k-10k+(!) Ops/Pix ->100’s to1000’s GOPs
on GPU
Technical and Design Challenges —
Display and Graphics
![Page 21: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/21.jpg)
Copyright © 2016 AMD 21
Technical Challenges — Camera and CV
• Example: Hand and Eye Tracking
• Detail capture, frame rate,
tracking, position, prediction
• New research
• Deep learning to customize
and train
![Page 22: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/22.jpg)
Copyright © 2016 AMD 22
• Content Development: titles, stories, 360/180 surround, volume audio…
• Distribution: Gb of content- how to efficiently transport
• How to test, evaluate, use feedback in content creation
• Who can make money?
Challenges — Ecosystem and Business
Environment
![Page 23: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/23.jpg)
Copyright © 2016 AMD 23
• VR — picking up as a business!
• Improvements in historical limitations
• Cost effective solutions
• Improvements in performance
• CV — accelerates emergence of wide variety of applications
• Common attributes — detection, tracking
• Unique attributes — accuracy, categories (medical vs. games, etc.)
• Ecosystem — SDK, user groups, content creators, reviewers,
• Business — new paradigms in entertainment, efficiency, capabilities
Summary
![Page 24: "How Computer Vision Is Accelerating the Future of Virtual Reality," a Presentation from AMD](https://reader031.vdocuments.site/reader031/viewer/2022030316/58728cbf1a28ab36118b55b1/html5/thumbnails/24.jpg)
Copyright © 2016 AMD 24
• Alger, Mike, “Visual Design Methods for Virtual Reality” Sept 2015
• MacVey, Matthew “Exploring Future Reality”, NYC Media Lab, 2016
References