big data task force legacy from nac it …...big data task force legacy from nac it infrastructure...
TRANSCRIPT
-
Big Data Task ForceLegacy from
NAC IT Infrastructure Committee
Charles P HolmesChair, BDTF
Formerly Vice-Chair, ITICFebruary 16, 2016
-
Mandatory Flow Chart
NASA Advisory Council
Human Ops Aeronautics
Planetary Science
Heliophysics
Science
Astrophysics
Earth Science
Planetary Protection
IT Infrastructure Etc.
NAC Structure 2010 - 2013
2/16/16 NAC Big Data Task Force 2
-
NAC Committee on IT Infrastructure Recommendation #1 July 31, 2013
• Recommendation: The NASA NAC ITIC & Science Committees should collaboratively explore the existing and planned evolution of NASA’s science data cyberinfrastructure that supports broad access to data repositories for NASA SMD missions. This exploration should be undertaken in the context of effective practices within NASA, other Federal agencies, as well as industry and research institutions.
Wording Agreed to by Both ITIC and Science CommitteesJuly 31, 2013
Work Will Continue as Big Data Taskforce Under Science Committee
NAC Big Data Task Force2/16/16 3
-
NAC Committee on IT Infrastructure Recommendation #1
• Recommendation: To enable NASA to gain experience on emerging leading-edge IT technologies such as:
• Data-Intensive Cyberinfrastructure, 100 Gbps Networking, GPU Clusters, and Hybrid HPC Architectures,
•••
we recommend that NASA aggressively pursue partnerships with other Federal agencies, specifically NSF and DOE, as well as public/private opportunities. We believe joint agency program calls for end users to develop innovative applications will help keep NASA at the leading edge of capabilities and enable training of NASA staff to support NASA researchers as these technologies become mainstream.
2/16/16 NAC Big Data Task Force 4
-
NAC Committee on IT Infrastructure DRAFT* Recommendation #2
• Recommendation: NASA should formally review the existing national data cyberinfrastructure supporting access to data repositories for NASA SMD missions. A comparison with best-of-breed practices within NASA and at other Federal agencies should be made. We request a briefing on this review to a joint meeting of the NAC IT Infrastructure, Science, and Education committees within one year of this recommendation. The briefing should contain recommendations for a NASA data-intensive cyberinfrastructure to support science discovery by both mission teams, remote researchers, and for education and public outreach appropriate to the growth driven by current and future SMD missions.
•
* To be completed after a joint meeting of ITIC, Science, and Education Committees in July 2012 and the final recommendation submitted to July 2012 NAC meeting
2/16/16 NAC Big Data Task Force 5
-
NAC Committee on IT Infrastructure Recommendation #2 (continued)
• Major Reasons for the Recommendation: NASA data repository and analysis facilities for SMD missions are distributed across NASA centers and throughout U.S. universities and research facilities. – There is considerable variation in the sophistication of the integrated
cyberinfrastructure supporting scientific discovery, educational reuse, and public outreach across SMD subdivisions. The rapid rise in the last decade of “mining data archives” by groups other than those funded by specific missions implies a need for a national-scale cyberinfrastructure architecture that can allow for free-flow of data to where it is needed. Other agencies, specifically NSF’s Ocean Observatories Initiative Cyberinfrastructure program, should be used as a benchmark for NASA’s data-intensive architecture.
–
–
• Consequences of No Action on the Recommendation: The science , education, and public outreach potential of NASA’s investment in SMD space missions will not be realized .
2/16/16 NAC Big Data Task Force 6
-
ITIC Finding• SMD Data Resides in a Highly Distributed Servers
– Many Data Storage and Analysis Sites Are Outside NASA CentersAccess to Entire Research Community Essential–
• Over Half Science Publications are From Using Data ArchivesSecondary Storage Needed in Cloud with High Bandwidth and User Portal
•
– Education and Public Outreach of Data Rapidly Expanding• Images/Videos for Public Relations
Apps for Smart PhonesCrowd Sourcing
••
PI
Research
Community
Education
Public Outreach
-
Partnering Opportunities with DOE: ARRA Stimulus Investment for DOE ESnet
National-Scale 100Gbps Network Backbone
2/16/16 NAC Big Data Task Force
Source: Presentation to ESnet Policy Board 8
-
Global Partnering Opportunities:The Global Lambda Integrated Facility
Research Innovation Labs Linked by 10Gps Dedicated Networks
www.glif.is/publications/maps/GLIF_5-11_World_2k.jpg
-
SMD is a Growing NASA HPC User Community
p r o
j e
c t e
d
Source: Tsengdar Lee, Mike Little, NASA
-
EOS-DIS Data Products DistributionApproaching ½ Billion/Year!
-
NAC – Information Technology Infrastructure Committee
-
NAC – Information Technology Infrastructure Committee
-
Solar Dynamics Observatory 4096x4096 AIA Camera – 57, 600 Images/Day
JSOC is Archiving ~5TB/day From 6 CamerasLeads to over 1 Petabyte per year!
March 6, 2012 X5.4 Flare from Sunspot AR1429 Captured by
the Solar Dynamics Observatory (SDO)
in the 171 Angstrom Wavelength
Credit: NASA/SDO/AIA
-
0
200
400
600
800
1000
1994 1997 2000 2003 2006 2009 2012 2015 2018
Tera
byte
s
Projected
Multi-Mission Data Archives at STSIWill Continue to Grow - Doubling by 2018
Cumulative Petabyte Over 20 Years
JWST
JWSTS&ITOther
JWST
Chart3
Projected
HST199419951996199719981999200020012002200320042005200620072008200920102011201220132014201520162017201820191.351.92.53.75.26.16.87.2916.8722.226.7430.6631.5236.5444.8452.4960.5268.5576.5884.6192.64100.67108.7116.73124.76HLA199419951996199719981999200020012002200320042005200620072008200920102011201220132014201520162017201820193.51054.276470126.8175145.27225999999999164.82028185.28201.34217.4233.46249.52GALEX199419951996199719981999200020012002200320042005200620072008200920102011201220132014201520162017201820190.110.458000000000000023.666.9816.816.67000000000000224.3524.298080808080808080KEPLER199419951996199719981999200020012002200320042005200620072008200920102011201220132014201520162017201820190.314.019999999999999610.614.618.60000000000000122.626.630.634.638.642.6Other199419951996199719981999200020012002200320042005200620072008200920102011201220132014201520162017201820190.010.61.23.19999999999999974.31000000000000054.825.4235.9847.5710.7914.20914.4214.71600000000000115.36700000000000115.85699999999999915.9416.0916.5917.2918.3419.91499999999999922.277525.82124999999999931.13687499999999639.11031249999999951.070468749999996JWST S&IT199419951996199719981999200020012002200320042005200620072008200920102011201220132014201520162017201820194.014.335812162024282828JWST1994199519961997199819992000200120022003200420052006200720082009201020112012201320142015201620172018201950200
Terabytes
Sheet1
HLAHSTOtherIUEFUSEVLA FIRSTOTHERDSSGSC I+IIGALEXKEPLERMAST+HSTJWST S&ITJWSTEUVEEPOChSmallXMM-OMHLSPFudge
19941.350.010.011.36
19951.90.60.62.5
19962.51.21.23.7
19973.73.20.60.10.10.51.96.9
19985.24.310.60.10.1112.59.51
19996.14.820.60.10.121.52.510.92
20006.85.4230.60.0930.10.1322.512.223
20017.25.9840.60.1440.10.142.52.513.184
200297.570.60.220.10.1542.516.57
200316.8710.790.60.30.150.2472.50.127.66
200422.214.2090.60.6050.1840.32102.5136.4090.10.060.16
200526.7414.420.60.7360.1840.4102.50.45841.160.10.060.24
200630.6614.7160.60.9520.1840.48102.53.6645.3760.10.060.140.18
20073.531.5215.3670.61.0330.1841.05102.56.9846.8870.10.060.390.380.12
20081036.5415.8570.61.20.2071.35102.516.852.3970.10.060.470.660.06
200954.2744.8415.940.61.2030.2071.43102.516.670.3160.780.10.060.470.80
20106452.4916.090.61.2030.2071.58102.524.354.0268.584.010.10.050.060.470.90
20117060.5216.590.61.2030.2072.08102.524.2910.677.114.3350.10.050.060.471.40
2012126.817568.5517.290.61.2030.2072.78102.58014.685.8480.10.050.060.472.10
2013145.2722676.5818.340.61.2030.2073.83102.58018.694.92120.10.050.060.473.150
2014164.8202884.6119.9150.61.2030.2075.405102.58022.6104.525160.10.050.060.474.7250
2015185.2892.6422.27750.61.2030.2077.7675102.58026.6114.9175200.10.050.060.477.08750
2016201.34100.6725.821250.61.2030.20711.31125102.58030.6126.49125240.10.050.060.4710.631250
2017217.4108.731.1368750.61.2030.20716.626875102.58034.6139.836875280.10.050.060.4715.946875
2018233.46116.7339.11031250.61.2030.20724.6003125102.58038.6155.840312528500.10.050.060.4723.9203125
2019249.52124.7651.070468750.61.2030.20736.56046875102.58042.6175.83046875282000.10.050.060.4735.88046875
HSTGSC I&IIDSSIUEFUSEVLA-FIRSTGALEXEUVESmall ProjectsXMM-OMHLSP
22954.582500100002.4605.430.82896.866406255012.536222.60440625Mar-0562.5
23370.222500100004.8639.781.6563439622.536977.956Apr-05118.5
23730250010000600659.86183.9863439662.138174.946May-05158.1
23996250010000600673.53183.986458.59662.138570.116Jun-05158.1
24302.78250010000600697.22183.986458.59662.138900.586Jul-05158.1
24625250010000600716.34183.986458.59662.139241.926Aug-05158.1
24867250010000600722.76183.986458.59662.139490.346Sep-05158.1
25758250010000600722.76183.986458.59662.140381.346Oct-05158.1
25434250010000600725.56183.986458.59662.140060.146Nov-05158.1
26379.21250010000600726.07183.986458.59662.141005.866Dec-05158.1
26737.54250010000600736.38183.986458.59662.141374.506Jan-06158.1
27038.44250010000600752.92183.986458.59662.114.22641706.172Feb-06172.326
27421.47250010000600769.25183.986458.59662.114.22642105.532Mar-06172.326
27763.69250010000600827.02183.986720.59662.114.22642767.522Apr-06172.326
28107.86250010000600871.34183.9861469.369662.114.22643904.872May-06172.326
28313.99250010000600890.49183.9861469.369662.114.22644130.152Jun-06172.326
28590.13250010000600899.813183.9861469.369662.114.22644415.615Jul-06172.326
28891.24250010000600917.4183.9861469.369662.114.22644734.312Aug-06172.326
29132.29714250010000600923.2183.9861469.369662.214.22644981.26914Sep-06172.426
29443.14250010000600928.7183.9861469.369662.214.22645297.612Oct-06172.426
29784.57250010000600935.43183.9861469.369662.214.22645645.772Nov-06172.426
30281.41250010000600939.63183.98619199662.214.22646596.452Dec-06172.426
30659.06250010000600951.82183.9863659.89662.214.22648727.092Jan-07172.426
30807.39250010000600967.33183.9863681.89662.214.22648912.932Feb-07172.426
30957.59250010000600970.73183.9863681.89662.214.22649066.532Mar-07172.426
31049.49250010000600970.73183.9863681.89662.214.22649158.432Apr-07172.426
31123.99250010000600985.8183.9863681.89662.214.22649248.002May-07172.426
31162.432500100006001000.87183.98645599662.214.22650178.712Jun-07172.426
31189.722500100006001004.84183.98646599662.214.22650309.972Jul-07172.426
31128.3115432500100006001011.96183.98647559662.238.6350376.087543Aug-07196.83
31031.0543262500100006001027.10178183.98647929662.238.6350330.972106Sep-07196.83
30738.2946482500100006001032.80369183.98652269662.238.6350477.914338Oct-07196.83
30738.3952500100006001032.806183.9865255.4499662.238.6350507.466Nov-07196.83
31446.21067820492500100006001035.4884657143183.9865273.4499662.238.63379.4769687551615.4411126692Dec-07576.30696875
31522.132500100006001032.686183.98669809662.238.63384.2753399.902Jan-08581.1
31951.42215763682500100006001053.1477916921183.98674249662.238.63384.2754293.6559493289Feb-08581.1
322302500100006001053.1477916921183.98674249662.238.63384.2754572.2337916921Mar-08581.1
New HLA
201064
2011706
2012126.817556.8175
2013145.2722618.45476
2014164.8202819.54802
2015185.2820.45972
Chart2
HLA19941995199619971998199920002001200220032004200520062007200820092010201120123.51054.276470126.8175HST19941995199619971998199920002001200220032004200520062007200820092010201120121.351.92.53.75.26.16.87.2916.8722.226.7430.6631.5236.5444.8452.4960.5268.55Other19941995199619971998199920002001200220032004200520062007200820092010201120120.010.61.23.19999999999999974.31000000000000054.825.4235.9847.5710.7914.20914.4214.71600000000000115.36700000000000115.85699999999999915.9416.0916.5917.29
Calendar Year
Terabytes
Chart1
HST19941995199619971998199920002001200220032004200520062007200820092010201120121.351.92.53.75.26.16.87.2916.8722.226.7430.6631.5236.5444.8452.4960.5268.55Other19941995199619971998199920002001200220032004200520062007200820092010201120120.010.61.23.19999999999999974.31000000000000054.825.4235.9847.5710.7914.20914.4214.71600000000000115.36700000000000115.85699999999999915.9416.0916.5917.29MAST+HST19941995199619971998199920002001200220032004200520062007200820092010201120121.362.53.76.99.510000000000001610.9212.22299999999999913.18400000000000116.5727.6636.40899999999999941.1645.37600000000000546.88752.39699999999999860.7868.5877.1185.84
Calendar Year
Terabytes
3D Chart
MAST DATA VOLUME GROWTH
MAST (with HST SM4)1994199519961997199819992000200120022003200420052006200720081.362.53.76.99.510000000000001610.9212.22299999999999913.18400000000000116.5727.6636.40899999999999941.1645.37600000000000546.88752.396999999999998
Year
Terabytes
Terabytes
ChartX
FUSEHSTIUEVLADSSGSCOther0.22110.40.60.11520.16
Sheet2
FUSE0.221
HST10.4
IUE0.6
VLA0.11
DSS5
GSC2
Other0.16
Sheet3
-
32 of the 200+ Apps in the Apple iStore that Return from a Search on “NASA”
-
Crowdsourcing Science: Galaxy Zoo and Moon Zoo Bring the Public into Scientific Discovery
More than 250,000 people have taken part in Galaxy Zoo so far.In the 14 months the site was up Galaxy Zoo 2 users helped us make over 60,000,000
classifications. Over the past year, volunteers from the original Galaxy Zoo project created the world's largest database of galaxy shapes.
www.galaxyzoo.org
-
Finding #1
• The U.S. government has issued several new guidance and directives on open data:
– OSTP February 22, 2013 Increasing Access to the Results of Federally Funded Scientific Research
OSTP March 29, 2013 Big Data is a Big Deal
Presidential Exec Order May 9, 2013 Open Data Policy-Managing Information as an Asset
–
–
-
White House Big Data Initiative
• National Science Foundation National Institutes of HealthDepartment of DefenseDepartment of EnergyU.S. Geological Survey
••••
NAC – Information Technology Infrastructure Committee20
-
NAC Information Technology Infrastructure Committee
-
Re-organization of the NASA Advisory Council –(Memo signed April 28, 2014)
The NASA Administrator shall establish the following Council committees, subcommittees, and task forces:
– Aeronautics Committee. Human Exploration and Operations Committee. Science Committee.
––
• Astrophysics Subcommittee. Earth Science Subcommittee. Heliophysics Subcommittee. Planetary Protection Subcommittee. Planetary Science Subcommittee. Ad Hoc Task Force on Big Data.
•••••
– Technology, Innovation, and Engineering Committee. Institutional Committee. Ad Hoc Task Force on Science, Technology, Engineering, and Mathematics (STEM) Education.
––
2/16/16 NAC Big Data Task Force 22
-
Timeline
• ITIC in existence – April 2010 – Dec 2013NAC reorganized – April 2014 •– Science Committee to have a Big Data Task Force
• BDTF Terms of Reference signed Jan 8, 2015BDTF members appointed Dec 2015SMD appoints Exec. Sec. who solicits feed back from the Committee members and subcommittees1st meeting of BDTF – Feb 16, 2016
••
•
2/16/16 NAC Big Data Task Force 23
-
NAC Committee on IT Infrastructure Recommendation #1 July 31, 2013
• Recommendation: The NASA NAC ITIC & Science Committees should collaboratively explore the existing and planned evolution of NASA’s science data cyberinfrastructure that supports broad access to data repositories for NASA SMD missions. This exploration should be undertaken in the context of effective practices within NASA, other Federal agencies, as well as industry and research institutions.
Wording Agreed to by Both ITIC and Science CommitteesJuly 31, 2013
Work Will Continue as Big Data Taskforce Under Science Committee
NAC Big Data Task Force2/16/16 24
-
Need I say more?
2/16/16 NAC Big Data Task Force 25
Big Data Task Force�Legacy from �NAC IT Infrastructure Committee Mandatory Flow ChartNAC Committee on IT Infrastructure �Recommendation #1 July 31, 2013NAC Committee on IT Infrastructure �Recommendation #1NAC Committee on IT Infrastructure �DRAFT* Recommendation #2NAC Committee on IT Infrastructure �Recommendation #2 (continued)ITIC FindingPartnering Opportunities with DOE: �ARRA Stimulus Investment for DOE ESnetGlobal Partnering Opportunities:�The Global Lambda Integrated FacilitySMD is a Growing NASA HPC User Community EOS-DIS Data Products Distribution�Approaching ½ Billion/Year!Slide Number 12Slide Number 13Solar Dynamics Observatory �4096x4096 AIA Camera – 57, 600 Images/DayMulti-Mission Data Archives at STSI�Will Continue to Grow - Doubling by 2018Slide Number 1632 of the 200+ Apps in the Apple iStore that Return from a Search on “NASA”Crowdsourcing Science: Galaxy Zoo and Moon Zoo Bring the Public into Scientific DiscoveryFinding #1White House Big Data InitiativeSlide Number 21Re-organization of the NASA Advisory Council – (Memo signed April 28, 2014)TimelineNAC Committee on IT Infrastructure �Recommendation #1 July 31, 2013Need I say more?