pydata dc 2016: a doc conundrum
TRANSCRIPT
![Page 1: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/1.jpg)
a DOC Conundrum
Star Ying, Data Scientist at Department of Commerce
![Page 2: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/2.jpg)
first, some background
![Page 3: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/3.jpg)
![Page 4: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/4.jpg)
grow the economy
![Page 5: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/5.jpg)
⅓ of all federal public data
![Page 6: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/6.jpg)
new data released constantly
![Page 7: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/7.jpg)
![Page 8: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/8.jpg)
$
![Page 9: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/9.jpg)
a DOC conundrum
![Page 10: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/10.jpg)
how to impart better understanding of our data
![Page 11: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/11.jpg)
of any data
![Page 12: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/12.jpg)
so really a data conundrum
![Page 13: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/13.jpg)
a different perspective
![Page 14: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/14.jpg)
def wdtd()……
![Page 15: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/15.jpg)
def wdtd()……
#wdtd: what did this
#do?
![Page 16: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/16.jpg)
is inherited
![Page 17: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/17.jpg)
a quick quiz
![Page 18: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/18.jpg)
american community survey
![Page 19: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/19.jpg)
how many erratas have been issued for 2016?
![Page 20: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/20.jpg)
how many erratas have been issued for 2016?
http://www.census.gov/programs-surveys/acs/technical-documentation/errata.html
![Page 21: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/21.jpg)
viirs nighttime lights
![Page 22: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/22.jpg)
which pixels are really blank?
![Page 23: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/23.jpg)
which pixels are really blank?
http://ngdc.noaa.gov/eog/viirs/download_monthly.html
![Page 24: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/24.jpg)
survey of income and program participation
![Page 25: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/25.jpg)
which weights do I use?
![Page 26: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/26.jpg)
which weights do I use?
http://www.census.gov/programs-surveys/sipp/methodology/weighting.html
![Page 27: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/27.jpg)
how do we convey the necessary information to use our product
![Page 28: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/28.jpg)
now, an anecdote
![Page 29: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/29.jpg)
can we tie satellite images to economic
activity?
![Page 30: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/30.jpg)
accounted for cloud coverage,
population, etc...
![Page 31: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/31.jpg)
forgot the earth is a
sphere
![Page 32: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/32.jpg)
tendency to silo ourselves
![Page 33: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/33.jpg)
real insights and outcomes can only be derived from true synthesis of
knowledge of the data and mechanics in processing it
![Page 34: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/34.jpg)
so what are we doing about it?
![Page 35: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/35.jpg)
it is a communication problem
![Page 36: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/36.jpg)
Data Usabilitycommerce.gov/datausability
handcrafted tutorials with working open code
![Page 37: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/37.jpg)
![Page 38: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/38.jpg)
I’d love to hear your ideas
![Page 39: PyData DC 2016: A DOC Conundrum](https://reader031.vdocuments.site/reader031/viewer/2022030305/5872f26b1a28ab8c718b4d61/html5/thumbnails/39.jpg)
no really