dat 5 minute lightning talk
TRANSCRIPT
![Page 1: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/1.jpg)
Dat: version and share your data
Karissa McKelveySoftware Developer and Project Manager and Science Evangelist and Designer (I wear a lot of hats) U.S. Open Data
@karissamck
![Page 2: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/2.jpg)
karissa $ ~
![Page 3: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/3.jpg)
dat is a non profit
![Page 4: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/4.jpg)
![Page 5: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/5.jpg)
Reproducible Research
![Page 6: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/6.jpg)
“A rule of thumb … is that half of published research cannot be replicated”
![Page 7: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/7.jpg)
How do we replicate research today?
![Page 8: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/8.jpg)
How do we replicate research today?collaborate on
![Page 9: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/9.jpg)
How do we replicate research today?collaborate on
data analysis
![Page 10: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/10.jpg)
How do we collaborate today?
![Page 11: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/11.jpg)
How do we collaborate today?
![Page 12: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/12.jpg)
How do we collaborate today?
![Page 13: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/13.jpg)
How do we collaborate today?
![Page 14: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/14.jpg)
????????
![Page 15: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/15.jpg)
![Page 16: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/16.jpg)
![Page 17: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/17.jpg)
![Page 18: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/18.jpg)
![Page 19: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/19.jpg)
![Page 20: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/20.jpg)
![Page 21: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/21.jpg)
![Page 22: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/22.jpg)
How do we replicate research today?
![Page 23: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/23.jpg)
![Page 24: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/24.jpg)
me@home $ dat push me@campus $ dat pull
you@work $ dat clone
![Page 25: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/25.jpg)
dat workflow• import
• version
• publish
• replicate
![Page 26: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/26.jpg)
.csv.csvdata
you
![Page 27: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/27.jpg)
.csv.csvdata
you
![Page 28: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/28.jpg)
.csv.csvdata
you
![Page 29: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/29.jpg)
.csv.csvdata
import
you
![Page 30: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/30.jpg)
$ dat init
$ dat add dataset cities
$ dat add rows cities cities.csv
$ dat add files cities city_model.gz
import
![Page 32: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/32.jpg)
$ dat listen
![Page 35: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/35.jpg)
$ dat clone
![Page 37: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/37.jpg)
Versioning
$ dat add files cities us_cities_viz.pngThis will override us_cities_viz.png at c2342. OK?
$ dat cities add rows updated_data.csvThis will update 3,434,245 rows. OK?
$ dat push
![Page 39: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/39.jpg)
http://my-data.bids.edu
publish
.csv.csvdata
http://my-data.indiana.edu
![Page 40: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/40.jpg)
.csv
.png.png.png
.csv.csv
.csv.csv.R
.csv.csv.pyINTEROPERABILITY in Python and R
![Page 41: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/41.jpg)
.csv
.png.png.png
.csv.csv
.csv.csv.R
.csv.csv.pyECOSYSTEM
![Page 42: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/42.jpg)
• Goal: manipulate datasets with scripting
• Supported keywords: run, pipe, map, reduce, fork, keyword
• Bash-like
• Platform-independent
• Uses node.js streams (fast!)
Datscript
![Page 43: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/43.jpg)
Top: Datscript “pipe” command Bottom: Equivalent command in bash
Datscript: pipeline example
![Page 44: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/44.jpg)
Datscript: example commands
background - executes command, but doesn’t wait for it to finish map- pipes first argument into rest of arguments
run- a serial command (executes and finishes command)
![Page 45: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/45.jpg)
![Page 46: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/46.jpg)
![Page 47: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/47.jpg)
Karissa McKelvey - @karissamck
Melanie Cebula - @melaniecebula
http://dat-data.com
![Page 48: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/48.jpg)
.csv
.png.png.png
.csv.csv
.csv.csv.R
.csv.csv.pyINTEROPERABILITY in Python and R
![Page 49: Dat 5 minute Lightning Talk](https://reader034.vdocuments.site/reader034/viewer/2022042701/55a50fa91a28abf6228b456c/html5/thumbnails/49.jpg)
.csv
.png.png.png
.csv.csv
.csv.csv.R
.csv.csv.pyECOSYSTEM