![Page 1: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/1.jpg)
Nucleus: Eight GPU Platform for Visual Simulation
David Morgan
Principal Engineer
Aechelon Technology
S9224
![Page 2: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/2.jpg)
Session Trajectory
• Visual Simulation Background
• Monsters, Clusters, and Moore’s Law
• Nucleus Architecture
• Challenges
• Demo
![Page 3: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/3.jpg)
Aechelon Technology
![Page 4: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/4.jpg)
Image Generation
![Page 5: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/5.jpg)
2000: RealityMonster
• 5 Racks• 8 Graphics “Pipelines” (GPUs)• 24 CPUs• 9GB RAM (NUMA)• 140GB Storage• Single IRIX OS• 15kW• $2.8M
![Page 6: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/6.jpg)
Scalability Matters
• 128 CPUs• 256GB RAM• 16 GPUs
![Page 7: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/7.jpg)
2001: GeForce 3
![Page 8: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/8.jpg)
2002-Today: PC Clusters
• 1-3 Racks per IG• 1 GPU per node• 1U Diskless Renderers• 3U Pager w/88TB Storage• Windows OS Per Node• Ethernet Interconnect• 7000W (8ch)• Unlimited Scalability
![Page 9: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/9.jpg)
Moore’s Law is Dead
Cluster
Nucleus
2
2.2
2.4
2.6
2.8
3
3.2
3.4
3.6
3.8
4 6 8 10 12 14 16 18 20 22 24 26 28
GHz
Cores
Skylake 2017
Broadwell 2016
Haswell 2014
Ivy Bridge 2013
Sandy Bridge 2012
![Page 10: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/10.jpg)
2016: 8-GPU Support
![Page 11: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/11.jpg)
Multi-GPU is Hard
![Page 12: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/12.jpg)
Nucleus
• 4U• 8 Quadro GPUs• One display per GPU• 36 CPU Cores• 192GB RAM• 36TB Storage• One Windows OS• $100-200K• Operates up to 35C• 2000W• Limited Scalability
![Page 13: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/13.jpg)
Dual Root Complex
![Page 14: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/14.jpg)
Single Root Complex
![Page 15: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/15.jpg)
GPU Affinity
• Exposed in OpenGL through WGL_NV_gpu_affinity extension
• Quadro feature necessary to address individual GPUs on Windows
• pC-Nova Maps GPU device handles to screens in the Windows
virtual desktop
• Beware driver crashes enumerating more than 4 screens per GPU!
![Page 16: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/16.jpg)
EDID Management
http://johnsciacca.webs.com/apps/blog/show/16852621-installation-nightmares-9-professional-horror-stories
![Page 17: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/17.jpg)
DWM Is…
• Independent GPUs’ video timings phase shift.
• Normally correctable by tracking the phase
• “Full-Screen Exclusive Mode” is gone.
• DWM intermediates all drawing on multi-display systems.
• One display is Primary.
![Page 18: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/18.jpg)
DWM Is Evil
https://www.pandza.xyz/article/16/dwm,-dxgi,-swap-chains,-latency,-throughput-and-you
GPU 2
GPU 1
![Page 19: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/19.jpg)
Workaround: Framelock
• Quadro Sync II supports 8 GPUs per system
• Shared oscillator ensures displays remain in phase with Primary
• DWM placated!• Downside: Video timings must
all match• Downside: Wiring is delicate
![Page 20: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/20.jpg)
Future Work
• GPU Multicast
• Or Dual Root Complex?
• VR Direct?
• Clusters of Nuclei
![Page 21: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/21.jpg)
Thanks
• Doug Traill
• John Chaney
• Tim Woodard
• Steve Nash
• Ian Williams
![Page 22: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/22.jpg)
Demo
![Page 23: Nucleus: Eight GPU Platform for Visual Simulation · Nucleus: Eight GPU Platform for Visual Simulation David Morgan Principal Engineer Aechelon Technology S9224. Session Trajectory](https://reader036.vdocuments.site/reader036/viewer/2022062505/5ee0ba14ad6a402d666bdb73/html5/thumbnails/23.jpg)
Questions?