![Page 1: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/1.jpg)
Deep Learning System Stack
Pooyan JamshidiUofSC
![Page 2: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/2.jpg)
The Deep Learning Systems Juggle
![Page 3: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/3.jpg)
Typical Deep Learning System Stack
![Page 4: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/4.jpg)
Typical Deep Learning System Stack
![Page 5: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/5.jpg)
Example: Logistic Regression
![Page 6: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/6.jpg)
Logistic Regression in Numpy
![Page 7: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/7.jpg)
Logistic Regression in Numpy
![Page 8: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/8.jpg)
Logistic Regression in TinyFlow (TensorFlow like API)
![Page 9: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/9.jpg)
Logistic Regression in TinyFlow
![Page 10: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/10.jpg)
Logistic Regression in TinyFlow
![Page 11: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/11.jpg)
Logistic Regression in TinyFlow
![Page 12: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/12.jpg)
Logistic Regression in TinyFlow
![Page 13: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/13.jpg)
The Declarative Language: Computation Graph
![Page 14: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/14.jpg)
Computational Graph Construction by Step
![Page 15: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/15.jpg)
Computational Graph Construction by Step
![Page 16: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/16.jpg)
Computational Graph Construction by Step
![Page 17: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/17.jpg)
Computational Graph Construction by Step
![Page 18: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/18.jpg)
Execution only Touches the Needed Subgraph
![Page 19: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/19.jpg)
Discussion: Computational Graph
![Page 20: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/20.jpg)
Discussion: Numpy vs TF Program
![Page 21: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/21.jpg)
Typical Deep Learning System Stack
![Page 22: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/22.jpg)
Computation Graph Optimization
![Page 23: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/23.jpg)
Parallel Scheduling
![Page 24: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/24.jpg)
Typical Deep Learning System Stack
![Page 25: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/25.jpg)
GPU Acceleration
![Page 26: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/26.jpg)
Typical Deep Learning System Stack
![Page 27: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/27.jpg)
Supporting More Hardware backends
![Page 28: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/28.jpg)
Each Hardware backend requires a software stack
![Page 29: Deep Learning System Stack - Pooyan Jamshidi · Discussion: Numpy vs TF Program. Typical Deep Learning System Stack. Computation Graph Optimization. Parallel Scheduling. Typical Deep](https://reader034.vdocuments.site/reader034/viewer/2022042108/5e876a5f0011d919ac0b827a/html5/thumbnails/29.jpg)
New Trend: Compiler based Approach