nvidia vrworksimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 vrworks 杨...nvidia confidential. do...
TRANSCRIPT
![Page 1: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/1.jpg)
NVIDIA VRWORKS 加速及提升VR体验
GTC China 2016
杨雪青 内容技术开发工程师
英伟达(NVIDIA)
![Page 2: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/2.jpg)
2 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
在模拟真实环境时对计算能力的挑战
图形 (GRAPHICS)
物理模拟(PHYSICAL
SIMULATION)
声音 (AUDIO)
触碰 (TOUCH)
![Page 3: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/3.jpg)
3 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
传统情况 = 60 MP/S (1920 X 1080 @ 30 FPS)
虚拟现实 = 450 MP/S (3024 X 1680* @ 90 FPS)
1920
1080
1512 1680
1512
VR 对性能的要求 极高的分辨率及帧率
![Page 4: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/4.jpg)
4 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
Motion to Photon: ≤ 20 ms
VR 对性能的要求 极低的延迟
![Page 5: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/5.jpg)
5 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
图形渲染简介
创建3D几何体 投影至2D视图 对像素进行计算
![Page 6: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/6.jpg)
6 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
VR渲染简介
渲染的图像 变形后的图像
![Page 7: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/7.jpg)
7 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
渲染的图像 变形后的图像
GPU渲染了许多最终不会被显示的像素
VR渲染简介
![Page 8: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/8.jpg)
8 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
透镜匹配着色(LENS MATCHED SHADING) 渲染到一个经过透镜矫正的表面
![Page 9: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/9.jpg)
9 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
传统的立体渲染 需要渲染两遍几何体
左眼 (Pass 1) 右眼 (Pass 2)
![Page 10: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/10.jpg)
10 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
单遍立体(SINGLE PASS STEREO) 通过一遍几何体渲染来完成左右眼的渲染
左眼
右眼
![Page 11: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/11.jpg)
11 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
VRWORKS GRAPHICS 为VR带来现代的视觉体验
单遍立体(SPS)允许更高的几何体精度
透镜匹配着色(LMS)带来更快的像素着色
优先占用(PREEMPTION)带来更高效的TIMEWARP
![Page 12: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/12.jpg)
12 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
方向 传播
VR中的声音效果模拟
综合各种音源
创建各种音源
传入的声音的位置 声音如何在空间中传播
![Page 13: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/13.jpg)
13 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
定向音频(DIRECTIONAL AUDIO) ‘HRTF’ 对大脑用于判断声音方位的声音特征进行了建模
Level L
Level R
耳间声音强度差(Level Difference)
耳间时间差
Time L
Time R
频谱特征(Spectral Cues)
![Page 14: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/14.jpg)
14 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
使用光线追踪对声音的定向及传播进行建模
![Page 15: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/15.jpg)
15 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
使用光线追踪对声音的定向及传播进行建模
![Page 16: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/16.jpg)
16 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
VR TOUCH 为触觉反馈提供真实的碰撞检测
力 (FORCE)
摩擦 (FRICTION)
PhysX API
PhysX Constraint Solver
Haptics Layer
触面复原能力 (RESTITUTION)
![Page 17: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/17.jpg)
17 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
PHYSX FLEX
HAIRWORKS FLOW
DESTRUCTION
CLOTH
NVIDIA PHYSX 在VR中模拟物理规律
![Page 18: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/18.jpg)
18 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
全新的 VRWORKS 迈向全面表现的一大步
VRWORKS GRAPHICS
PHYSX
MULTI-PROJECTION
OPTIX
API
ENGINE
SIGHT BEHAVIOR SOUND TOUCH
PHYSX
VRWORKS AUDIO
![Page 19: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/19.jpg)
19 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
NVIDIA VR FUNHOUSE 在Github上有完整的源代码以供下载
![Page 20: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/20.jpg)
20 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
创建VR的两个途径
渲染 图像捕获
![Page 21: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/21.jpg)
21 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
制作360度全景视频涉及巨大的计算量
图像捕获 拼接(Stitch) 显示
解码校准均衡化拼接编码 4k相机 单个360视频
![Page 22: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/22.jpg)
22 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频
• 实时和离线拼接4k相机组的数据
• GPU加速的视频解码, 校准, 均衡化, 拼接, 及编码
• 对立方图(cube-map)和等距圆柱全景图(equi-rectangular panorama)做360度投影
• 与GPUDirect视频技术一同工作可获取低延迟的视频摄取
“捕获和拼接360度视频十分耗时且有大量的计算需求. NVIDIA的VRWorks 360 Video SDK有助于加速 STRIVR的工作流程, 提供实时的,
高质量的360视频.”
— Masaki Miyanohara, CTO, STRIVR
![Page 23: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/23.jpg)
23 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
针对VR开发者的综合性SDK
GRAPHICS HEADSET AUDIO TOUCH & PHYSICS
NVIDIA VRWORKS
PROFESSIONAL
VIDEO
![Page 24: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/24.jpg)
24 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
VRWORKS已经对游戏引擎做了支持!
在Github上有集成了VRWorks Graphics的版本!
在NVIDIA有支持VRWorks Graphics的版本!
![Page 25: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/25.jpg)
25 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
VR
![Page 26: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/26.jpg)
26 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
理解不同的VR类型
VR 快照
预渲染, 静态的结果
你可以四处看但不能移动
通过NVIDIA “Pro VR Viewer”可以转变成任何数量的视图
Iray VR
预渲染, 可探索的结果
你可以移动到任何合适的地点进行观察(在你的viewing volume内)
融合了Iray的精确性及VR的自由度
渲染时间 – 类似于一个转盘动画(turntable animation) 渲染时间与分辨率相关
![Page 27: NVIDIA VRWORKSimages.nvidia.com/cn/gtc/downloads/pdf/vr/508 VRWorks 杨...NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 22 VRWORKS 360 VIDEO 介绍 实时捕获,拼接及流化360º全景视频](https://reader034.vdocuments.site/reader034/viewer/2022042310/5ed813abcba89e334c673380/html5/thumbnails/27.jpg)
谢谢大家 提问 ?