gsaw 2011 - matsu- an elastic cloud connected to a ... · rf link cmd/ telemetry ... sample display...

Matsu: An Elastic Cloud Connected to a SensorWeb for Disaster

ResponseResponse (Session 12F Working Group: Cloud Computing for Spacecraft Operations)

Daniel Mandl NASA/GSFC 3/2/11Daniel Mandl - NASA/GSFC 3/2/11

1

https://ntrs.nasa.gov/search.jsp?R=20110008678 2018-06-24T02:55:33+00:00Z

SensorWeb High Level Architecturefl d fi

SensorML

SensorMLCapabilities Documents

DataProcessing Node

floods, fires, volcanoes etc

SensorMLCapabilities

ode

e

SensorMLWebProcessing

Service(WPS)

WebCoverageService(WCS)

WebCoordinate

TransformationService(WCTS)

SASSAS

Sensor Planning

Web FeatureService (WFS)

DocumentsEO-1Satellite

nsor

Dat

a N

o

sor D

ata

Nod

e

ctRSS Feeds

L1G

SOS

WFS

SOS

WFS

Sensor PlanningService (SPS)

Sensor AlertService (SAS)In

-situ

Sen

UAV

Sen

s

Sate

llite

se

nsor

data

pro

duc

InternetSensorData

RSS Feeds

SPSSPSSensor

ObservationService (SOS)

Satellite Data Node

ProductsOpenID 2.0

W kfl

Web Coverage Processing Service (WCPS)

GeoBPMSWorkflowsService (WCPS)

Le el 1 & higher

USGS target requests

Science Validation Team targets

Technology Validation Team activities

User interfaceOld Way of Doing Things with EO-1

Level 0

Level 1 & higher processed science data products

Level 0 processed science data Ops engineeringrequests

Planning CommitteeDeputy Mission Scientist

equests

USGSLevel 0

Processing at GSFC

White SandsScheduling

contact times

De-conflicted,

JPL Flight Ops

Mission Planner

GSFC Mission Sys EngineerMission Planner

USGS RepresentativeScheduling

group

stationin-views times overflight

rawsciencedatavia

manually selected weekly schedule

Mission Ops Planning &

Daily plan

ASIST Telemetry &

Flight Dynamics Support Sys

times timesviaX-band

Planning & Sched Sys

Daily activity plan

Non-GSFCUser

Command Sys

tracking data

commandstelemetry

Alaska,Norway,

RF Link cmd/ telemetry

y,WallopsGround Stations Phase 1 Standard Ops Architecture

November 2000

Le el 1 & higherUSGS target requests

Science Validation Team targets

Technology Validation Team activities

User interface

JPL

Step 1 Improvement for EO-1

USGS

Level 0

Level 1 & higher processed science data products

Level 0 processed science data Ops engineeringrequests

Planning CommitteeDeputy Mission ScientistMi i S E i

requestsJPL users

Level 0Processing

at GSFC

USGSLevel 0

Processing at GSFC

White SandsScheduling

contact times

JPL Flight Ops

Mission Planner

targets

GSFC Mission Sys EngineerMission Planner

USGS RepresentativeJPL RepresentationDe-conflicted, Scheduling

group


rawsciencedatavia

Dailyplan


pmanually selected weekly schedule(backup approach &maneuvers)

De-conflicted, manually generated replacement record

ASIST Telemetry &


times

ASPEN Ground Planner

timesviaX-band


Daily activity plan

file

goals

Non-GSFCUser

Sciencegoals

Command Sys with Web Interface

tracking data

Commands, goals telemetry Onboard EO-1

GSFC OpenID Provider (OP) Server

CASPERSCL-Meta-

d

cmdsscience data

ScienceProcessing

Alaska,Norway,

RF Link cmd, goals/ telemetry

CASPEROnboardPlanner

command controlleractivities

Phase 2 Add Onboard Autonomy October 2003

y,WallopsGround Stations

Disaster target requests

USGS target requests

Technology Validation activities

Mission

Step 2 Improvement for EO-1

USGSUSGS

Level 0

Level 0 science data

GSFC Sensor Observation Service(SOS) for ALI

L1R, L1G

L1R, L1G

L2 ProductsSystems Engineer

JPL Sensor Observation Service(SOS) for Hyperion

JPL Web Processing Service(WPS) for Hyperion

E t l dLevel 0Processing

at GSFC

Level 0Processing at

GSFCWhite SandsScheduling

contact times

Mission Planner

( )GSFC Web Processing Service(WPS) for ALI L2 Products

JPL Sensor Planning Service (SPS)

External and Internal User targets

FOTScheduling

group


rawsciencedatavia

Dailyplan


Mission Science Office

backup JPL userstargets

Service (SPS)

GSFCGeoBPMS(Secure Web Auto grnd

ASIST Telemetry &


times

ASPEN Ground Planner

timesviaX-band


Daily activity plangoals

NASA Investigator targets

(Interface)sensor

triggersMisc targets

Non-GSFCUser

Sciencegoals

Command Sys with Web Interface

tracking data

Commands, goals telemetry Onboard EO-1

GSFC OpenID Provider (OP) Server

CASPERSCL-Meta-

d

cmdsscience data

ScienceProcessing

Alaska,Norway,

RF Link cmd, goals/ telemetry

CASPEROnboardPlanner

command controlleractivities

Phase 3 Add Web Services October 2008

y,WallopsGround Stations

Open Science l dData Cloud

Astronomical dataBiological data (Bionimbus)(Bionimbus)

NSF‐PIRE OSDC Data ChallengeEarth science data (& disaster relief)

Focus of OCC Large Data Cloud Working Group

App App App App App

Table‐based Data Services

Relational‐like Data Services

App App

Cloud Compute Services (MapReduce, UDF, VM‐based data parallelism & other parallel programming frameworks)

App App

Cloud Storage Services

p p g g )

• Developing APIs for this framework.

7

p g

Hyperion and ALI External users, NASA Investigators

Step 3 Improvement for EO-1 - OverviewHyperion and ALILevel 0 Processed data from GSFC, building 3 server

,especially international (e.g. disaster workers)

NASA Investigators

Technologistsg

Starlight 100 Level 1R and Level 1G Processing for ALI & Hyperion

Atmospheric Correction for ALI & Hyperion

Web Coverage Processing Service (WCPS) to enable users to customize Level 2 products

Gigabit Ethernet Exchange

10 Gbps users to customize Level 2 products

Eucalyptus-based Elastic Cloud SW300+ core processors40 x 2 Tbytes of storage10 Gb ti t GSFC

Nambia Flood Dashboard

2 year data product 10 Gbps connection to GSFC

- being upgraded to 80 Gbps (Part of OCC)At Univ of Illinois at ChicagoSupplied by Open Cloud ConsortiumO S i D t Cl d Vi t l M hi &

y parchive

Open Science Data Cloud Virtual Machines & http server to VM’s

Phase 3 Add Elastic Cloud Ongoing Feb 2011OCC = Open Cloud Consortium

Transformation to On-Demand Product Cloud Part 1EO-1 Data Product Pipeline

EO-1 Level 0 Processor

Server Storage – 1 yearHyperion & ALI Level 1R

p

EO-1 Level 1 Processor

Service

EO-1 Hyperion Atmospheric Correction -

FLAASH Service

yp

Storage – Available Algorithms

Service FLAASH Service

EO-1 Hyperion Atmospheric Correction –

ATREM ServiceEO-1 Level 1

WCPS AlgorithmGeneration Service

Hyperion Level 1RALI Level 1R

algorithms

ATREM Service EO 1 Level 1 Geospatially

Corrected Service Storage – 1 year

Hyperion & ALI Level 1R

WCPS RuntimeService

Hyperion Level 1G

EO-1 ALI Atmospheric Correction – FLAASH

Service

and Level 1G AC

Storage – 1 yearHyperion & ALI Level 1G

Hyperion Level 1GALI Level 1G Storage – 1 year

User Defined L2Products

Generate a new product with this new algorithm

Select algorithm & d t tdata to run against

Phase 3 Add Elastic Cloud Ongoing Feb 2011

CREST Hydrological

On-Demand Product Cloud Part 2Flood Dashboard (Matsu)y g

Model( )

6 Namibian River Gauge Stations -Daily MeasurementsTRMM based Global

Rainfall Estimates

MODIS Daily Flood E t t M Namibia River Gauge

Radarsat Images

Extent Map Storage – 1 year

Hyperion & ALI Level 1R

Namibia River GaugeData base

Global Disaster and Alert and Coordination

Storage – 1 yearsHyperion & ALI Level 1R

Storage – 1 yearHyperion & ALI Level 1G Flood Dashboard

Display Service- Mashup- Google Maps Inset

System (GDACS)

Hyperion & ALI Level 1Rand Level 1G AC

Storage – 1 yearUser Defined L2

- Plot Package

Productse.g. EO-1 Flood Mask http server


Detail of Processing Image Data in OCC Open Science Data CloudOpen Science Data Cloud

Hadoop / HBase

HBase storage of multiple

missions overStorage – 1 year

Storage – 1 yearHyperion & ALI Level 1R

Hadoop / HBasePartition into Cloud Cache

Suitable forGoogle Earth / Open Layers

missions over multiple days

Storage – 1 yearsHyperion & ALI Level 1R

Storage – 1 yearHyperion & ALI Level 1G

Web map Service

and Level 1G AC

Storage – 1 yearUser Defined L2

Products Service (WMS)

Productse.g. EO-1 Flood Mask


Top Level Flood SensorWeb Concept

Customized plan of needed satellite images

Manual or automated triggered requests for satellite imagery in

SPS

SPS

GeoBPMS – Web

g yarea of interest SPS

SPSGeoBPMS Web based satellite tasking tool

SPS

Flood di i

Flood alerts

conditionsGround flood measurements to validate modelto users

Compare to history

to validate model

history

*SPS = Sensor

Improved flood prediction model

Planning Service

AngolaPortion of 2011 Namibian Flood SensorWeb Early Warning Pilot

Namibia

y g

ShanalumonoRiver Gauge Station

TRMM based rain estimates=

Global Disaster and Coordination

Early user alert

Shanalumono

Monitor rains upper basin

System‐ (Based on AMSR‐E)

River Gauge Station

MODIS Daily Flood MaskGeoBPMS

Follow flood wave down basin

Auto trigger Auto triggers

Flood

Daily flood gauge levels & predicted river

Auto‐triggerHi‐res Satellite images

Auto triggers

FloodDashboard(mashup)High resolution

satellite imagery

levels plots

(e.g. EO‐1)

Portion of 2011 Namibia Flood SensorWeb Early Warning Pilot:

Experimental Namibian Flood SensorWeb WebpageNote blue bars indicatinga surge of rainfall upstream

Then a flood wave appearsdownstream at Rundu river

Flood Dashboard

gauge days later

Namibia Short Term Pilot for 2010• Colored areas represent catchments where rainfall collects and drains to riverwhere rainfall collects and drains to river basins • River gauges displayed as small circles• Detailed measurements are available on the display by clicking on the river gaugethe display by clicking on the river gauge stations.• This display can be viewed and manipulated at: http://geobpms geobliki com/namibiahttp://geobpms.geobliki.com/namibiaandhttp://geobpms.geobliki.com/namibia2

15

Zambezi basin consisting ofupper, middle and lower catchments

Shanalumono River Gauge Station and Part of CommunityProne to Flooding Downstreamg

Experimental Namibian Flood SensorWeb WebpageView of Available Envisat & EO-1 Overlay Images

Experimental TRMM-based Flood Forecast Products

Flood Dashboard

Envisat SAR and EO-1 Optical Image Overlays

Estimated Rainfall Webpage Based on TRMM Data

i d i h iExperimented with various hydrometeorolgical information for flood forecasting models

remote sensing

rainfall estimates

24 February 2010

NASA Servir Africa

red is > 35 mmred is > 35 mm

Experimental Global TRMM Based Flood Forecast

Experimental Flood Extent Data Product Derived from MODIS

First product out of automated MODIS flood extent map pipeline prototype Used data from March 2009 when large floodsprototype. Used data from March 2009 when large floods occurred to test.

Recent MODIS Daily Flood Extent

Area toured on January 2011 trip

Recentflooded area

http://oas.gsfc.nasa.gov/SERVIR_Africa/calendar.html?latlong=010E010S

Sample of Planned Addition of Higher Resolution Flood Product Overlay Using EO-1

Mashup of Satellite Data and River Gauge Data UsingNamibia2 (Google Earth Version) Webpage Tool

Zambia water linesfrom old database

Lower Zambezi catchment

M lti i

NORMAL, 2009 AND 2010 WATER LEVELS AT KATIMA MULILO ‐ UPDATE 23 FEBRUARY

Multiyear river gaugemeasurements

6.500

7.500

normal

2009

2010

4.500

5.5002010

2.500

3.500

Envisat swath Radarsat DataEO-1 Data

0.500

1.500

1‐Jan 21‐Jan 10‐Feb 2‐Mar 22‐Mar 11‐Apr

Envisat swathMarch 25, 2009March 2009

Envisat DataMarch 2009

Mock up of Revised River Gauge Plot Page

Various flood models such asCREST model (Univ. of Oklahoma)

Rainfall predictionpFrom GEOS-5

TRMM based daily rainfall estimates

Sample Display of Multi-year Satellite Measurements (in month of March) of Katima Mulilo Linked to JRC Via Namibia Flood Mashup

Based on Terra AMSR-E Microwave InstrumentBased on Terra AMSR-E Microwave Instrument

Sample Alert During PilotSample Alert During Pilot

N ibi d il fl d b ll ti 03 M h 2010Namibia daily flood bulletin 03 March 2010: There have again been heavy rains in parts of the Zambezicatchment See attached NASA map The waterlevels atcatchment. See attached NASA map. The waterlevels atChavuma started rising again. See attached graph. Ourforecast remains that the Katima Mulilo waterlevels areheading for 7 m by mid‐March 2010. For perspective, theflood would be:

similar to 2007similar to 2007higher than 2008lower than 2009

h ll d d h d h hBut much will depend on the rains and the catchmentresponse in the coming weeks.

Sample Time Sequence Flood Map Generated by Unosat, Derived from Multiple Satellite Data Setsp

Vision is to generate similar product automatically when floods predicted andpair them with river gauge measurements

Conclusion

• Combining Sensorwebs with an elastic computation cloud enables surge• Combining Sensorwebs with an elastic computation cloud enables surge capacity for disasters by enabling parallel processing of various algorithms and other processes within the cloud

• Elastic cloud provides work space for user to customize their experience instead• Elastic cloud provides work space for user to customize their experience instead of a preset outputs

28