big data and cloud computing expert panelbig data and cloud computing expert panel assessing...
Post on 06-Jul-2020
8 Views
Preview:
TRANSCRIPT
SESIP-0716-DP
Big Data and Cloud Computing
Expert Panel
Assessing Cloud-Native Architectures for EOSDIS
Applications
Dan Pilone - dan@element84.com
The material is based upon work supported by the National Aeronautics and Space
Administration under Contract Number NNG15HZ39C
https://ntrs.nasa.gov/search.jsp?R=20160009335 2020-07-31T01:59:30+00:00Z
SESIP-0716-DP
2
15+ years of Earth Science Data
Deletion of
MODIS
Collection 4
Products
Landsat-7 Data
Migrated from
LPDAAC to USGS
Addition of
AQUARIUS and
ICEBRIDGE
products
Addition of
ALOS PALSAR
& UAVSAR
products
ICESat MEASURES 2006 MEASURES 2012TERRA AQUA AURA
Addition of SNPP
CERES&OMPS
Total EOSDIS Accumulated Data Archive Volume
(Petabytes)
2
SESIP-0716-DP
3
EOSDIS Archive Growth Estimate
(Prime + Extended)
2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025
Cumulative Archive Size (PB) 13.8 20.0 27.0 34.8 42.7 65.0 118.0 170.5 223.1 275.6 328.2
Archive Growth Rate (PB) 4.9 6.2 7.0 7.9 7.9 22.4 52.9 52.6 52.6 52.6 52.6
0.0
50.0
100.0
150.0
200.0
250.0
300.0
350.0
400.0
PB
Archive Growth Rate (PB) Cumulative Archive Size (PB)
SESIP-0716-DP
EOSDIS Vision 2020 (cloud relevant)
- Data Analysis at Scale- Users are able to analyze the entire data record for any data variable
over any arbitrarily defined area.
- Data and Processing Mobility- Data and processing move transparently as necessary to achieve optimal
performance.
- Dataset Upgrading- High-value datasets are upgraded as necessary to fully support in the
rich capabilities available in the data systems.
- Virtual Collections...- ...can be organized/oriented around a science problem.
- Combining Data, Combining Tools- NASA data can be combined* with data from other agencies, nations and
other entities
- Tools and services within the community are easy to combine.
*combined = compared / merged / fused / assimilated
SESIP-0716-DP
6
So what does this impact?
SESIP-0716-DP
7
Infrastructure, Scaling, Monitoring,
Security, …
SESIP-0716-DP
8
Development model, Compliant
Deployment, Operations, …
SESIP-0716-DP
9
Application Architecture
SESIP-0716-DP
10
CMR in the Cloud
SESIP-0716-DP
11
GIBS in the Cloud?
SESIP-0716-DP
12
Systems Architecture
SESIP-0716-DP
13
Cumulus I&A Prototype
Data
Pipeline
Cloud Based Storage
Data
Validation
Reformattin
g
Ingest Archive Access
Metadata
Generation
CMR
Submission
MEaSUREs
Projects
HTTP
Distribution
OPeNDAP
Reprojection
Access
Projects
ICEBridge
Flight Path Val
FRB
Generation
Daily
Composites
SMAP
Unique…
DAAC 1
DAAC 2
Cloud based
Services
Cloud based
Services
Cloud based
ServicesYou?
SESIP-0716-DP
14
Exploiting cloud computing sets the
stage for EOSDIS as a platform,
enabling an AppStore model for
services and micro-services
meeting end-user needs.
SESIP-0716-DP
What does this mean?
• Need widespread knowledge of cloud
architectures and patterns (inc. NGAP,
serverless, 12 factor applications)
• Rapid pace of change
• Potentially dramatic change in data usage
patterns
• Need for Cloud Native data formats
• Need for Cloud Native services
SESIP-0716-DP
Outstanding Questions…
• Existing prototyping efforts have been restructured to systematically and realistically investigate and test commercial cloud environments for essential EOSDIS and DAAC operations.– Isolated prototypes have demonstrated unrestricted cloud
environments offer performance and cost benefits under specific situations, but can operational NASA science data systems function effectively in a commercial cloud environment?
– Do commercial clouds offer increased efficiency within a NASA environment and/or provide additional functionality? How do costs compare to opportunities and risks?
– How can NASA efficiently partner with commercial cloud providers to improve services and usage of data and services?
16
SESIP-0716-DP
18
This material is based upon work
supported by the National
Aeronautics and Space
Administration under Contract
Number NNG15HZ39C.
top related