the data science technology stack - nitrd · the data science technology stack contrasting critical...
TRANSCRIPT
![Page 1: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/1.jpg)
The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors
Andrew W. Moore [email protected]
![Page 2: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/2.jpg)
This talk
• Examples from the largest scale commercial big data systems.
•My personal top five recommendations for critical technology investments for large data systems
![Page 3: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/3.jpg)
![Page 4: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/4.jpg)
![Page 5: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/5.jpg)
Decorated Entities
Ingested Unstructured Facts
![Page 6: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/6.jpg)
Images
Decorated Entities
Ingest Unstructured Facts
Normalize
Human-in-the-loop
![Page 7: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/7.jpg)
BA
CK
GR
OU
ND
SER
VIN
G
Images
Decorated Entities
Ingest Unstructured Facts
Normalize
Human-in-the-loop
Query
Delivery
Model Click Streams
Context
Result Page
Inventory
ConOps
![Page 8: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/8.jpg)
FLEET B
AC
KG
RO
UN
D
SERV
ING
Images
Decorated Entities
Ingest Unstructured Facts
Normalize
Human-in-the-loop
Query
Delivery
Model Click Streams
Context
Result Page
Inventory
Telemetry Weather Map Hot Swap
HwOps
ConOps
![Page 9: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/9.jpg)
FLEET B
AC
KG
RO
UN
D
SERV
ING
TR
UST
Images
Decorated Entities
Ingest Unstructured Facts
Normalize
Human-in-the-loop
Query
Delivery
Model Click Streams
Context
Result Page
Inventory
Telemetry Weather Map Hot Swap
HwOps
ConOps Recommender
Opinions
Mystery Shopping Anti Fraud
![Page 10: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/10.jpg)
FLEET B
AC
KG
RO
UN
D
SERV
ING
TR
UST
Knowledge Data Action
Images
Decorated Entities
Ingest Unstructured Facts
Normalize
Human-in-the-loop
Query
Delivery
Model Click Streams
Context
Result Page
Inventory
Telemetry Weather Map Hot Swap
HwOps
ConOps Recommender
Opinions
Mystery Shopping Anti Fraud
![Page 11: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/11.jpg)
My personal top five recommendations
1 The Top of The Stack
2 Entities
3 Data Intensive Computing Architectures
4 Delineation of the Data Science Stack
5 Human-in-the-loop
![Page 12: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/12.jpg)
My personal top five recommendations
1 The Top of The Stack
2 Entities
3 Data Intensive Computing Architectures
4 Delineation of the Data Science Stack
5 Human-in-the-loop
![Page 13: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/13.jpg)
My personal top five recommendations
1 The Top of The Stack
2 Entities
3 Data Intensive Computing Architectures
4 Delineation of the Data Science Stack
5 Human-in-the-loop
![Page 14: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/14.jpg)
My personal top five recommendations
1 The Top of The Stack
2 Entities
3 Data Intensive Computing Architectures
4 Delineation of the Data Science Stack
5 Human-in-the-loop
Decision Support Visualization, Consulting Workflow, Human-in-loop systems
Modeling Prediction, Clustering, Structure Discovery
ML Components Spatial Join, Fuzzy Join, MLE, Sampling
Data Science Kernel Layer Blobstore, KeyVal, Redundancy Management
Device Layer Multicore, GPU, Sensors
![Page 15: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/15.jpg)
My personal top five recommendations
1 The Top of The Stack
2 Entities
3 Data Intensive Computing Architectures
4 Delineation of the Data Science Stack
5 Human-in-the-loop Panstarr telescope image (Kaiser et al)
![Page 16: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/16.jpg)
My personal top five recommendations
1 The Top of The Stack
2 Entities
3 Data Intensive Computing Architectures
4 Delineation of the Data Science Stack
5 Human-in-the-loop
![Page 17: The Data Science Technology Stack - NITRD · The Data Science Technology Stack Contrasting critical issues in the public, scientific and commerce sectors Andrew W. Moore awm@cs.cmu.edu](https://reader031.vdocuments.site/reader031/viewer/2022021823/5b410f287f8b9af6438dc4e6/html5/thumbnails/17.jpg)
My personal top five recommendations
1 The Top of The Stack
2 Entities
3 Data Intensive Computing Architectures
4 Delineation of the Data Science Stack
5 Human-in-the-loop
Autonomy
Cognitive Assistance
Decision Support
Modeling
ML Components
Data Science Kernel Layer
Device Layer