a vision for exascale, simulation, and deep learning
TRANSCRIPT
![Page 1: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/1.jpg)
“AVisionforExascale:Simulation,DataandLearning”
RickStevensArgonneNationalLaboratory
TheUniversityofChicago
Crescatscientia;vitaexcolatur
![Page 2: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/2.jpg)
Data-DrivenScienceExamplesFormanyproblemsthereisadeepcouplingof observation(measurement)andcomputation(simulation)
Cosmology:Thestudyoftheuniverseasadynamicalsystem
SampleExperimentalscattering
Material composition
Simulated structure
Simulatedscattering
La60%Sr 40%
Materialsscience:Diffusescatteringtounderstanddisorderedstructures
ImagesfromSalmanHabibetal.(HEP,MCS,etc.)andRayOsborneetal.(MSD,APS,etc.)
![Page 3: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/3.jpg)
HowManyProjects?
![Page 4: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/4.jpg)
By 2020, the market for machine learning will reach $40
billion, according to market research firm IDC.
Deep Learning market is projected to be ~$5B by 2020
![Page 5: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/5.jpg)
![Page 6: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/6.jpg)
![Page 7: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/7.jpg)
![Page 8: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/8.jpg)
![Page 9: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/9.jpg)
MarketsareDevelopingatDifferentRates~2020
• HPC(Simulation)à [email protected]%• DataAnalysisà [email protected]%• DeepLearningà ~$5B@65%
• DL>HPCin2024• DL>DAin2030
![Page 10: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/10.jpg)
BigPicture
• Mixofapplicationsischanging• HPC“Simulation”,“Big”DataAnalytics,MachineLearning“AI”
• Manyprojectsarecombiningallthreemodalities– Cancer– Cosmology– MaterialsDesign– Climate– DrugDesign
![Page 11: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/11.jpg)
DeepLearning inClimateScience
• StatisticalDownscaling• Subgrid ScalePhysics• DirectEstimateofClimate
Statistics• EnsembleSelection• Dipole/AntipodeDetection
![Page 12: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/12.jpg)
![Page 13: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/13.jpg)
DeepLearninginGenomics
![Page 14: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/14.jpg)
PredictingMicrobialPhenotypes
![Page 15: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/15.jpg)
ClassificationofTumors
Usingdeeplearningtoenhancecancerdiagnosisandclassification,ICML2013
![Page 16: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/16.jpg)
HighThroughputDrugScreening
DeepLearningasanOpportunityinVirtualScreening,NIPS2014
![Page 17: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/17.jpg)
DeepNetworksScreenDrugs
![Page 18: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/18.jpg)
DeepLearningandDrugDiscovery
![Page 19: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/19.jpg)
DeepLearningInDiseasePrediction
![Page 20: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/20.jpg)
LearningClimateDisease
EnvironmentAssociations
BigDataOpportunitiesforGlobalInfectiousDiseaseSurveillanceSimonI.Hay,DylanB.George,CatherineL.Moyes,JohnS.Brownstein
![Page 21: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/21.jpg)
NeuralNetworksin
Materialsscience
• EstimateMaterialsPropertiesfromCompositionParameters
• EstimateProcessingParametersforSynthesis
• MaterialsGenome
![Page 22: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/22.jpg)
SearchingForLensedGalaxies
![Page 23: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/23.jpg)
15TB/NightUseCNNtofindGravitationalLenses
![Page 24: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/24.jpg)
DeepLearningisbecomingamajorelementofscientificcomputingapplications
• AcrosstheDOElabsystemhundredsofexamplesareemerging– Fromfusionenergytoprecisionmedicine– Materialsdesign– Fluiddynamics– Genomics– Structuralengineering– Intelligentsensing– Etc.
![Page 25: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/25.jpg)
WEESTIMATEBY2021ONETHIRDOFTHESUPERCOMPUTINGJOBSONOURMACHINES
WILLBEMACHINELEARNINGAPPLICATIONS
SHOULDWECONSIDERARCHITECTURESTHATAREOPTIMIZEDFORTHISTYPEOFWORK?
HOWTOLEVERAGEEXASCALE?
![Page 26: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/26.jpg)
TheNewHPC“Paradigm”
SIMULATION
DATAANALYSIS
LEARNING
VISUALIZATION
![Page 27: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/27.jpg)
TheNewHPC“Paradigm”
SIMULATION
DATAANALYSIS
LEARNING
VISUALIZATION
![Page 28: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/28.jpg)
TheCriticalConnectionsI
• EmbeddingSimulationintoDeepLearning– Leveragingsimulationtoprovide“hints”viatheTeacher-StudentparadigmforDNN
– DNNinvokes“SimulationTraining”toaugmenttrainingdataortoprovidesupervised“labels”forgenerallyunlabeleddata
– Simulationscouldbeinvokedmillionsoftimesduringtrainingruns
– Trainingratelimitedbysimulationrates– Ex.CancerDrugResistance
![Page 29: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/29.jpg)
HybridModelsinCancer
![Page 30: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/30.jpg)
Teacher-StudentNetworkModel
![Page 31: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/31.jpg)
Teacher-StudentNetworkModelSimulationBasedPredictions
![Page 32: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/32.jpg)
IntegratingMLandSimulation
![Page 33: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/33.jpg)
TheCriticalConnectionsII
• EmbeddingMachineLearningintoSimulations– Replacingexplicitfirstprinciplesmodelswithlearnedfunctions
– Faster,LowerPower,LowerAccuracy(?)– FunctionsinsimulationsaccessingMLmodelsathighthroughput
– Onnodeinvocationofdozensorhundredsofmodelsmillionsoftimespersecond?
– Ex.Nowcasting inWeather
![Page 34: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/34.jpg)
AlgorithmApproximation
NeuralAccelerationforGeneral-PurposeApproximateProgramsHadi Esmaeilzadeh AdrianSampsonLuisCeze DougBurger∗UniversityofWashington∗MicrosoftResearch
![Page 35: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/35.jpg)
ReplacingImperativeCodewithNNComputedApproximations
NeuralAccelerationforGeneral-PurposeApproximateProgramsHadi Esmaeilzadeh AdrianSampsonLuisCeze DougBurger∗UniversityofWashington∗MicrosoftResearch
![Page 36: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/36.jpg)
2.3xSpeedup,3xPowerReduction,~7%Error
NeuralAccelerationforGeneral-PurposeApproximateProgramsHadi Esmaeilzadeh AdrianSampsonLuisCeze DougBurger∗UniversityofWashington∗MicrosoftResearch
![Page 37: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/37.jpg)
JointDesignofAdvancedComputingSolutionsforCancerDOE-NCIpartnershiptoadvancecancerresearchandhighperformancecomputingintheU.S.
NCINationalCancerInstituteDOE
DepartmentofEnergy
Cancerdrivingcomputingadvances
Computingdrivingcanceradvances
DOESecretaryofEnergy
DirectoroftheNationalCancerInstitute
![Page 38: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/38.jpg)
ScalableDataAnalytics
DeepLearning
Large-ScaleNumericalSimulation
DOEObjective:DirveIntegrationofSimulation,DataAnalyticsandMachineLearning
CORALSupercomputersandExascaleSystems
TraditionalHPC
Systems
![Page 39: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/39.jpg)
Exascale Node ConceptSpace
AbstractMachineModelsandProxyArchitecturesforExascale ComputingRev1.1SandiaNationalLaboratoryandLawrenceBerkeleyNationalLaboratory
![Page 40: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/40.jpg)
LeverageResourcesontheDie,inPackageorontheNode
• Localhigh-bandwidthmemorystacks• Nodebasednon-volitile memory• High-BandwidthLowLatencyFabric• GeneralPurposeCores• DynamicPowerManagement
![Page 41: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/41.jpg)
WhatKindofAccelerator(s)toAdd?
• VectorProcessors• DataFlowEngines• PatchesofFPGA• Many“Nano”Cores(<5MTr each?)
![Page 42: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/42.jpg)
Hardwareandsystemsarchitecturesareemergingforsupportingdeeplearning
• CPUs– AVX,VNNI,KNL,KNM,KNH,…
• GPUs– Nvidia P100,V100,AMDInstinct,BaiduGPU,…
• ASICs– Nervana,DianNao,Eyeriss,GraphCore,TPU,DLU,…
• FPGA– Arria10,Stratix10,FalconMesa,…
• Neuromorphic– TrueNorth,Zeroth,N1,…
![Page 43: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/43.jpg)
Aurora21
• Argonne’sExascale System• Balancedarchitecturetosupportthreepillars
– Large-scaleSimulation(PDEs,traditionalHPC)– DataIntensiveApplications(sciencepipelines)– DeepLearningandEmergingScienceAI
• Enableintegrationandembeddingofpillars• Integratedcomputing,acceleration,storage• Towardsacommonsoftwarestack
![Page 44: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/44.jpg)
DeepLearningApplications• DrugResponsePrediction• ScientificImage
Classification• ScientificText
Understanding• MaterialsPropertyDesign• GravitationalLens
Detection• FeatureDetectionin3D• StreetSceneAnalysis• OrganismDesign• StateSpacePrediction• PersistentLearning• HyperspectralPatterns
ArgonneTargetsforExascaleSimulationApplications• MaterialsScience• Cosmology• MolecularDynamics• NuclearReactorModeling• Combustion• QuantumComputer
Simulation• ClimateModeling• PowerGrid• DiscreteEventSimulation• FusionReactorSimulation• BrainSimulation• TransportationNetworks
BigDataApplications
• APSDataAnalysis• HEPDataAnalysis• LSSTDataAnalysis• SKADataAnalysis• MetagenomeAnalysis• BatteryDesignSearch• GraphAnalysis• VirtualCompound
Library• NeuroscienceData
Analysis• GenomePipelines
44
![Page 45: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/45.jpg)
DeepLearningApplications
• LowerPrecision(fp32,fp16)• FMAC@32and16okay• Inferencingcanbe8bit(TPU)• Scaledintegerpossible• Trainingdominatesdev• Inferencedominatespro• Reuseoftrainingdata• Datapipelinesneeded• DenseFPtypicalSGEMM• SmallDFT,CNN• EnsemblesandSearch• SingleModelsSmall• ImoreimportantthanO• Outputismodels
DifferingRequirements?SimulationApplications
• 64bitfloatingpoint• MemoryBandwith• RandomAccesstoMemory• SparseMatrices• DistributedMemoryjobs• SynchronousI/Omultinode• ScalabilityLimitedComm• LowLatencyHighBandwidth• LargeCoherencyDomains
helpsometimes• OtypicallygreaterthanI• Orarelyread• Outputisdata
BigDataApplications
• 64bitandIntegerimportant• DataanalysisPipelines• DBincludingNoSQL• MapReduce/SPARK• Millionsofjobs• I/Obandwidthlimited• Datamanagementlimited• Manytaskparallelism• Large-datainandLarge-data
out• IandObothimportant• Oisreadandused• Outputisdata
45
![Page 46: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/46.jpg)
Aurora21Exascale Software
• SingleUnifiedstackwithresourceallocationandschedulingacrossallpillarsandabilityforframeworksandlibrariestoseamlesslycompose
• Minimizedatamovement:keeppermanentdatainthemachineviadistributedpersistentmemorywhilemaintainingavailabilityrequirements
• SupportstandardfileI/OandpathtomemorycoupledmodelforSim,DataandLearning
• Isolationandreliabilityformulti-tenancyandcombiningworkflows
![Page 47: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/47.jpg)
TowardsanIntegratedStack
![Page 48: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/48.jpg)
TheNewHPC“Paradigm”
SIMULATION
DATAANALYSIS
LEARNING
VISUALIZATION
![Page 49: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/49.jpg)
Acknowledgements
ManythankstoDOE,NSF,NIH,DOD,ANL,UC,MooreFoundation,SloanFoundation,Apple,Microsoft,Cray,Intel,NVIDIAandIBMforsupportingourresearchgroupovertheyears
![Page 50: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/50.jpg)
End!
![Page 51: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/51.jpg)
![Page 52: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/52.jpg)
OurVisionAutomateandAccelerate
![Page 53: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/53.jpg)
![Page 54: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/54.jpg)
TheCANDLEExascaleProject
![Page 55: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/55.jpg)
![Page 56: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/56.jpg)
56
DrugResponse CANDLEGeneralWorkflow
56
![Page 57: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/57.jpg)
ECP-CANDLE :CANcerDistributedLearningEnvironmentCANDLEGoals
Developanexascaledeeplearningenvironmentforcancer
BuildingonopensourceDeeplearningframeworks
OptimizationforCORALandexascaleplatforms
Supportallthreepilotprojectneedsfordeep
CollaboratewithDOEcomputingcenters,HPCvendorsandECPco-designandsoftwaretechnologyprojects
57
![Page 58: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/58.jpg)
CANDLESoftwareStack
HyperparameterSweeps,DataManagement(e.g.DIGITS,Swift,etc.)
ArchitectureSpecificOptimizationLayer(e.g.cuDNN,MKL-DNN,etc.)
Tensor/GraphExecutionEngine(e.g.Theano,TensorFlow,LBANN-LL,etc.)
Networkdescription,ExecutionscriptingAPI(e.g.Keras,Mocha)
Workflow
Scripting
Engine
Optimization
58
![Page 59: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/59.jpg)
DLFrameworks“TensorEngines”• TensorFlow(c++,symbolicdiff+)• Theano(c++,symbolicdiff+)• Neon (integrated)(python+GPU,symbolicdiff+)• Mxnet (integrated)(c++)• LBANN (c++,aimedatscalablehardware)• pyTorch7THTensor(clayer,symbolicdiff-,pgks)• Caffe (integrated)(c++,symbolicdiff-)• Mocha backend(julia+GPU)• CNTKbackend(microsoft)(c++)• PaddlePaddle(Baidu)(python,c++,GPU)
![Page 60: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/60.jpg)
• Variational AutoEncoder– Learning(non-linear)featuresofcoredatatypes
• AutoEncoder– Moleculardynamicstrajectorystatedetection
• MLP+LCNNClassification– Cancertypefromgeneexpression/SNPs
• MLP+CNNRegression– Drugresponse(geneexp,descriptors)
• CNN– Cancerpathologyreporttermextraction
• RNN-LSTM– Cancerpathologyreporttextanalysis
• RNN-LSTM– Moleculardynamicssimulationcontrol
CANDLEBenchmarks..Representativeproblems
![Page 61: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/61.jpg)
![Page 62: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/62.jpg)
ProgressinDeepLearningforCancer• AutoEncoders – learningdatarepresentationsforclassificaitonandpredictionofdrugresponse,moleculartrajectories
• VAEsandGANs– generatingdatatosupportmethodsdevelopment,dataaugmentationandfeaturespacealgebra,drugcandidategeneration
• CNNs – typeclassification,drugresponse,outcomesprediction,drugresistance
• RNNs– sequence,textandmoleculartrajectoriesanalysis
• Multi-TaskLearning– terms(fromtext)andfeatureextraction(data),datatranslation(RNAseq<->uArray)
![Page 63: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/63.jpg)
CANDLE- FOM– RateofTraining• “Numberofnetworkstrainedperday”
– sizeandtypeofnetwork,amountoftrainingdata,batchsize,numberofepochs,typeofhardware
• “Numberof‘weight’updates/second”– ForwardPass+BackwardPass
• TrainingRate=∑ni=1 aiRi whereRi istherateforourbenchmarki andaiisaweight
![Page 64: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/64.jpg)
7 CANDLEBenchmarks
Benchmark Type Data ID OD SampleSize
SizeofNetwork
Additional(activation,layer
types,etc.)1.P1:B1Autoencoder MLP RNA-Seq 105 105 15K 5layers Log2(x+1)à [0,1]
KPRM-UQ2.P1:B2Classifier MLP SNPà
Type106 40 15K 5layers TrainingSetBalance
issues3.P1:B3Regression MLP+LCN expression;
drug descs105 1 3M 8layers DrugResponse
[-100,100]
4.P2:B1Autoencoder MLP MDK-RAS 105 102 106-108 5-8layers StateCompression
5.P2:B2RNN-LSTM RNN-LSTM MDK-RAS 105 3 106 4layers StatetoAction
6.P3:B1RNN-LSTM RNN-LSTM Pathreports
103 5 5K 1-2layers Dictionary12K+30K
7.P3:B2Classification CNN Pathreports
104 102 105 5layers Biomarkers
BenchmarkOwners:• P1:FangfangXia(ANL)• P2:BrianVanEssen(LLNL)• P3:ArvindRamanathan(ORNL)
64
https://github.com/ECP-CANDLE
![Page 65: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/65.jpg)
TypicalPerformanceExperienceCANDLE- Predictingdrugresponseoftumorsamples• MLP/CNNonKeras• 7layers,30M- 500Mparameters• 200GBinputsize• 1hour/epochonDGX-1;200epochstake8days(200GPU
hrs)• Hyperparametersearch~200,000GPUhrsor8MCPUhrs
Proteinfunctionclassificationingenomeannotation• DeepresidualconvolutionnetworkonKeras• 50layers• 1GBinputsize• 20minutes/epochonDGX-1;200epochstake3days(72
GPUhrs)• Hyperparametersearch~72,000GPUhrsor2.8MCPUhrs
![Page 66: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/66.jpg)
GithubandFTP
• ECP-CANDLEGitHubOrganization:• https://github.com/ECP-CANDLE
• ECP-CANDLEFTPSite:• TheFTPsitehostsallthepublicdatasetsfor thebenchmarksfromthreepilots.
• http://ftp.mcs.anl.gov/pub/candle/public/
![Page 67: A Vision for Exascale, Simulation, and Deep Learning](https://reader034.vdocuments.site/reader034/viewer/2022052318/5a64a1897f8b9a31568b5bdd/html5/thumbnails/67.jpg)
ThingsWeNeed• DeepLearningWorkflowTools• DataManagementforTrainingDataandModels• PerformanceMeasurement,ModelingandMonitoringofTrainingRuns
• DeepNetworkModelVisualization• Low-levelSolvers,OptimizationandDataEncoding
• ProgrammingModels/RuntimestosupportnextgenerationParallelDeepLearningwithsparsity
• OSSupportforHigh-ThroughputTraining