Slide 1

The Universe. As it Happens. Fragmentation: Astronomys Data Problem Dexter Jagula

Slide 2

2 Overview Introduction Fragmentation

Slide 3

3 Fin Designing a Solution Our Vision Our Roadmap

Slide 4

4 A little about me

Slide 5

5 Perceived Problems FRAGMENTATION Heterogeneity Data is available from a multitude of sources Siloed Data No reuse of data for other science objectives Storage Inconsistent methods of how data is stored Archival Data preservation is usually an afterthought Sharing Inefficient methods are used to share data Collaboration Third-party tools dont cater to astronomers

Slide 6

6

Slide 7

7 Heterogeneity Siloed DataStorageArchivalSharing Collaboration

Slide 8

8 NASA Space Apps Win Created a platform for public access to astronomy data. SaaS Solve single challenges to create a suite of services to be offered to the community. New Surveys Projects like LSST and SKA will provide a firehose of data for the community to manage. Data Aggregator Collecting catalogued data and providing a single-point access. How did we get here?

Slide 9

9 Built-in collaboration tools throughout all services One-click sharing capability on any datasets, files, or feeds Full and complete control of data SkyWatch Services Intelligent Filtering Aggregating data from multiple sources Filtering false positives Curating data for specific science objectives Utilizing machine learning for real-time classification of incoming data Raw File Archival Data Processing Tools Collaboration Tools Utilizing Hadoop for distributed file storage Ability to tag data Query on metadata extracted from FITS files Utilizing Google Cloud Uploading/importing datasets Leveraging Apache Spark for ad-hoc analysis and data reduction Implementing machine learning techniques on data Utilizing Google Cloud

Slide 10

10 Principal Investigator Unprecedented access to curated data Collaborator Annotate and share data easily and efficiently Follow-up Observatory Direct alerts for easy prioritization Author Perform analysis using modern technologies Data Ensure preservation Survey Effective brokering of data Advances for

Slide 11

11 Our Vision ? ? ? ?

Slide 12

12 01 Build-up the R/T Feed Add search capability Mobile-friendly APR 2015 - JUN 2015 02 Observatories can publish to the R/T Feed Socket connection to the Feed Roles and permissions Image processing JUL 2015 - SEP 2015 03 Collaboration tools built-into the R/T Feed Machine learning classifier used for image data OCT 2015 - DEC 2015 04 Improvements to the R/T Feed and image processing Enhanced roles and permissions Tabular data storage and processing JAN 2016 - MAR 2016 05 Continuous improvement to all facets Increased workbench capabilities Ability to integrate workbench and intelligent filtering Tools for theorists APR 2016 -

Slide 13

13

Slide 14

14

Slide 15

15 Acknowledgements Iair Arcavi Scott Barthelmy Guillaume Belanger Eric Bellm Federica Bianco Todd Boroson Peter Brown Yi Cao Hsin-Yu Chen Eric Christensen Diana Dragomir Simon Hodgkin Andy Howell Tim Jenness Kyler Kuehn Andrej Prsa Sumner Starrfield Rachel Street Jonathan Swift John Swinbank Brad Tucker Stefano Valenti Giacomo Vianello LCOGT NASA-GSFC ESA-ESAC Caltech NYU LCOGT Mitchell Institute / Texas A&M Caltech University of Chicago University of Arizona LCOGT Institute for Astronomy, Cambridge University LCOGT LSST Australian Astronomical Society Villanova University ASU/Earth and Space Exploration LCOGT The Thacher School Princeton Mt. Stromlo Observatory, ANU / UC Berkeley LCOGT Stanford university

Slide 16

dexter@skywatch.co +1 647-966-4621 skywatch.co @SkyWatchApps THANK YOU!