optical sims using gpus - nersc · 2020. 8. 18. · 3 motivation to gpus •replace existing cpu...
TRANSCRIPT
![Page 1: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/1.jpg)
1
Optical sims using GPUs
Nersc Users Group Lightning Talk Oisín Creaner1, Maria Elena Monazi2, Lisa Gerhardt1, Quentin Riffard3, Sam Eriksen4
1LBNL/NERSC, 2Stanford University/LZ, 3LBNL/LZ, 4University of Bristol/LZ
17th August 2020
![Page 2: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/2.jpg)
2
LZ
•LUX-ZEPPLIN (Large Underground Xenon ZonEd Proportional scintillation in LIquid Noble gases)
•Next-generation Dark Matter detector
•Particle interactions with the Xenon detection medium
•Compare with simulated interactions.
•Photon propagation: time- and resource-intense
![Page 3: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/3.jpg)
3
Motivation to GPUs
•Replace existing CPU module with GPU solution
• Photon propagation is simulation bottleneck
• Faster simulations
• More frequent simulations
• Better use of available hardware
•Opticks: A GPU Accelerated Optical Photon Simulation using NVIDIA OptiX
Blythe, S (2019) Renders of the chimney region of JUNO detector as
an Opticks Geometry
![Page 4: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/4.jpg)
4
LZ at NERSC
•LZ is major NERSC user
•Simulations: CPU basedData storageData analysis:
•Mock Data ChallengeScience Run 1 starting soon!
•Existing Optics implementation Local/CloudMove to CoriGPU Prepare for Perlmutter
•
NERSC, 2019
![Page 5: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/5.jpg)
5
Containerisation: Docker and Shifter
•Shifter: o Security requirement to run on CoriGPU (and Perlmutter)o Build a docker image, export to Shifter to runo READ ONLY and NO ROOT ACCESS
•Docker:o Write Dockerfile with all commands required to buildo Create image with all depencencies and specific
versionso Should be very portable: solution should continue to
work when upgraded•
![Page 6: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/6.jpg)
6
Challenges in creating an image
•Many undocumented assumptions
•Hard-coded or semi-hard-coded (e.g. $HOME) paths
•Hardware awareness of compiled libraries
•Security restrictions on writing
•Unexpected write commands buried deep in layers
•NVidia Runtime mismatch with drivers
•Specific version and/or commit dependencies
![Page 7: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/7.jpg)
7
Solutions
•Containerisation & Portability
o Versions locked in when image created
•Fork development from stable point
• Identify assumptions and hard-coding and make soft links
•Mount data in writable partitions (e.g. $SCRATCH)
•Hardware-aware compilation has to happen on the correct host
![Page 8: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/8.jpg)
Optical sims using GPUsOisín Creaner, Maria Elena Monazi, Lisa Gerhardt,
Quentin Riffard, Sam Eriksen
![Page 9: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/9.jpg)
LZ LUX-ZEPPLIN (Large Underground Xenon ZonEd Proportional scintillation in LIquid Noble gases)
Next-generation Dark Matter detector
Particle interactions with the Xenon detection medium
Compare with simulated interactions.
Photon propagation: time- and resource-intense
![Page 10: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/10.jpg)
Motivation to GPUsReplace existing CPU module with GPU solution
Photon propagation is simulation bottleneck
Faster simulations
More frequent simulations
Better use of available hardware
Opticks: A GPU Accelerated Optical Photon Simulation using NVIDIA OptiX
![Page 11: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/11.jpg)
LZ at NERSCLZ is major NERSC user
Simulations: CPU basedData storageData analysis:
Mock Data ChallengeScience Run 1 starting soon!
Existing Optics implementation Local/CloudMove to CoriGPU Prepare for Perlmutter
![Page 12: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/12.jpg)
Containerisation: Docker and ShifterShifter:
Security requirement to run on CoriGPU (and Perlmutter)
Build a docker image, export to Shifter to run
READ ONLY and NO ROOT ACCESS
Docker:
Write Dockerfile with all commands required to build
Create image with all depencencies and specific versions
Should be very portable: solution should continue to work when upgraded
![Page 13: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/13.jpg)
Challenges in creating an imageMany undocumented assumptions
Hard-coded or semi-hard-coded (e.g. $HOME) paths
Hardware awareness of compiled libraries
Security restrictions on writing
Unexpected write commands buried deep in layers
NVidia Runtime mismatch with drivers
Specific version and/or commit dependencies
![Page 14: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/14.jpg)
SolutionsContainerisation & Portability
Versions locked in when image created
Fork development from stable point
Identify assumptions and hard-coding and make soft links
Mount data in writable partitions (e.g. $SCRATCH)
Hardware-aware compilation has to happen on the correct host
![Page 15: Optical sims using GPUs - NERSC · 2020. 8. 18. · 3 Motivation to GPUs •Replace existing CPU module with GPU solution • Photon propagation is simulation bottleneck • Faster](https://reader035.vdocuments.site/reader035/viewer/2022070213/610c27dc4765ca7c78546568/html5/thumbnails/15.jpg)
Image ReferencesBlythe, S, 2019 Opticks : GPU Optical Photon Simulation for Particle Physics using NVIDIA® OptiXTM. EPJ Web of Conferences 214, 02027
NERSC 2019, Cori Supercomputer