grid technologies in disaster management
TRANSCRIPT
Grid Technologies in Disaster Management & Infrastructure
Protection Geoffrey Fox
Computer Science, Informatics, PhysicsPervasive Technology Laboratories
Indiana University Bloomington IN 47404
[email protected]://grids.ucs.indiana.edu/ptliupages/presentations/CIGridGDINmar28-04.ppt
http://www.infomall.orghttp://www.grid2002.org
Introduction• Grid is NOT a mesh in this talk!!!!• The Grid represents “Internet Scale Distributed Computing”
or the “Managed Internet”• Major commercial and research Initiative – IBM HP Oracle
Sun SGI Platform• We describe how one can use familiar system of systems
(Grid of Grids) language for critical infrastructure and emergency response (command and control) Grids
• We give infrastructure examples from flood and earthquake cases with HPC simulations linked to other Grid resources
• Discuss GIS and Collaboration Grids
e-Business e-Science and the Grid e-Business captures an emerging view of corporations as
dynamic virtual organizations linking employees, customers and stakeholders across the world.
e-Science is the similar vision for scientific research with international participation in large accelerators, satellites or distributed gene analyses.
The Grid or CyberInfrastructure integrates the best of the Web, Agents, traditional enterprise software, high performance computing and Peer-to-peer systems to provide the information technology e-infrastructure for e-moreorlessanything.
A deluge of data of unprecedented and inevitable size must be managed and understood.
People, computers, data and instruments must be linked. On demand assignment of HPC resources, experts,
computers, networks and storage resources
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
IMAGING INSTRUMENTS
COMPUTATIONALRESOURCES
LARGE-SCALE DATABASES
DATA ACQUISITION ,ANALYSIS
ADVANCEDVISUALIZATION
e-Defense and e-Crisis Grids support Command and Control and provide Global
Situational Awareness • Link commanders and frontline troops to themselves and to archival and
real-time data; link to what-if simulations
• Dynamic heterogeneous wired and wireless networks
• Security and fault tolerance essential
System of Systems; Grid of Grids• The command and information infrastructure of each ship is a Grid; each
fleet is linked together by a Grid; the President is informed by and informs the national defense Grid
Crisis Management and Response enabled by a Grid linking sensors, disaster managers, and first responders with decision support• Grid supports On-demand HPC Simulations
Define and Build DoD relevant Services – Collaboration, Sensors, GIS, Database etc.
Raw (HPC) Resources
Middleware
Database
PortalServices
SystemServices
SystemServices
SystemServices
Application Service
Libraries
UserServices
“Core”Grid
Typical Grid Architecture
Application Service
Application Service
Re-use
Re-use
ApplicationCustomization
Each service should beable to run independently on separate machines
Service OrientedArchitectures scalebetter than traditionalDistributed objects
A typical Web or Grid Service In principle, services can be in any language (Fortran .. Java ..
Perl .. Python) and the interfaces can be method calls, Java RMI Messages, CGI Web invocations, totally compiled away (inlining)
The simplest implementations involve XML messages (SOAP) and programs written in net friendly languages like Java and Python
PaymentCredit Card
WarehouseShippingcontrol
WSDL interfaces
WSDL interfaces
Security CatalogPortalService
Web Services
Web Services
Composing Systems• We are familiar with a hierarchy
Lines of Code Methods Objects Programs Packages
• Grids extend the software hierarchy to larger distributed systems and link it to a corresponding resource hierarchy
Overlayand ComposeGrids of Grids
Methods Services Functional Grids
CPUs Clusters ComputeResource Grids
MPPs
DatabasesFederatedDatabases
Sensor Sensor Nets
DataResource Grids
Information Grid
Enterprise Grid
Compute Grid
Campus Grid
R2R1
Teacher
Students
Dynamic light-weight Peer-to-peerCollaborative Training Grid
Composing anOverlay Grid
Education Grid
Inservice TeachersPreservice TeachersSchool of Education Teacher Educator
Grids
Informal Education(Museum)
Grid
Student/Parent …Community Grid
Science GridsCI Grids
Typical Science GridService such as ResearchDatabase or simulation
Transformed by Grid Filterto form suitable for education
Learning Managementor LMS Grid
Publisher Grid
Campus orEnterprise
AdministrativeGrid
Education as a Grid of Grids
DigitalLibrary
Grid
Solid Earth Research Virtual Solid Earth Research Virtual Observatory (SERVO)Observatory (SERVO)
NASA funded: JPL, UC Davis, UC Irvine, USC, Brown, IndianaWeb/Grid-services and portlet based Problem Solving Environment (PSE)Couples data with simulation, pattern recognition software, and visualizationEnable investigators to seamlessly merge multiple data sets and models, and
create new queries.
Data• Spaced-based observational data• Ground-based sensor data (GPS, seismicity)• Simulation data• Published/historical fault measurements
Analysis Software• Earthquake fault• Lithospheric modeling• Pattern recognition software
International Version iSERVO• Australia China and Japan as an APEC (Asia-Pacific Economic
Cooperation) Initiative – next meeting Beijing July 2004
Database Database
ResearchSimulations
Analysis and VisualizationPortal
RepositoriesFederated Databases
Data Filter
Services
Crisis Data
Streaming Data
Sensors
?DiscoveryServices
SERVOGridResearch Crisis
CustomizationServices
From Research
to CI/Crisis
Cell PhoneMobileNetwork
Geoscience Research andCrisis Grids
OtherInterdependentCI and CrisisGrids
GIS
Database Database
Analysis and VisualizationPortal
RepositoriesFederated Databases
Data Filter
Services
Field Trip DataStreaming Data
Sensors
?DiscoveryServices
SERVOGrid
ResearchSimulations
Research Education
CustomizationServices
From Research
to Education
EducationGrid ComputerFarm
Geoscience Research andEducation Grids
GIS
HPCSimulation
DataFilter
Data FilterD
ata
Filt
er
Data
Filter
Data
Filter
Distributed Filters massage dataFor simulation
Other
Grid
and W
eb
Servi
ces
AnalysisControl
Visualize
SERVOGrid (Complexity) Computing Model
Grid
OGSA-DAIGrid Services
This Type of Gridintegrates with
Parallel computingMultiple HPC
facilities but only use one at a time
Many simultaneous data sources and
sinks
Grid Data Assimilation
Crisis and CI Grids• We first form Critical Infrastructure CI Grids from the
“atomic” functional and resource Grids and Services– This gives water (flood), gas, postal, agricultural etc. Grids
• Then we customize, compose and overlay CI with other Grids (such as weather, census data) for – Public Health
– Emergency Response (Command and Control) or Crisis Grids
– Infrastructure Planning (e.g. transportation planning)
– Education (schools) and Training (of managers and first responders)
– Interdependency analyses between the different infrastructures
• Note need to link to “Cell Phone Grid” for first responders with services that reduce image sizes etc.
What are we trying to do• Center for Disaster and Critical Infrastructure Informatics
– IUPUI, Indiana, Purdue, openGIS Consortium project to build a generic Crisis/CI Grid infrastructure with a FloodGrid as initial prototype
• SERVOGrid (Solid Earth Research Virtual Observatory) building Grid to support Earthquake Forecasting
• GlobalMMCS is an open-source Audio-Video conferencing and collaboration system built totally on Web Services and spanning Access Grid, H323, SIP and WebCAM’s.
• NaradaBrokering is a Grid message system supporting reliable message delivery, software multicast, firewall friendliness, security. (Grid as managed web)
• Hope to partner with NISAC (National Infrastructure Simulation and Analysis Center) on core CIGrids
Minicomputer
Firewall
ComputerServer
PDA
Modem
Laptop computerWorkstationPeers
Peers
Audio/VideoConferencing Client
Audio/VideoConferencing Client
NaradaBrokering BrokerNetwork
NaradaBrokering
Queues
Web Service A
Web Service B
Stream
Reliable Routedsoftware multi-castSecure filtered Message delivery
CI/Crisis Grid of Grids Services in a CI or Crisis Grid fall into three classes 1) Those that special to Critical Infrastructure or
Crises such as biological sensor nets or evacuation planning service
2) Those that are important but can be taken from other Grids such as GIS or collaboration and security
3) Those that come from other Grids and are refactored for crises• The simulation is reduced in size• The public health database interface is simplified
GridResource
Filter CI/Crisis Grid View
of generic Grid Resource
CI/Crisis Grid
Gas Servicesand Filters
Security Workflow MessagingNotification
Metadata Data Access/StorageRegistry
Physical Network
Compute
Sensors Visualization
Collaboration
openGISServices
Flood Servicesand Filters
Flood CIGrid Gas CIGrid… Electricity CIGrid …
Critical Infrastructure CIGrids
openGIS Grid Semantics• Note GIS (Geographical Information System) Grid at heart of all
these Grids• Geography Markup Language (GML) is an XML encoding for the
specification of the geometry and properties of geographic features. GML utilizes the OpenGIS Abstract Specification geometry model which has been harmonized with the ISO geospatial geometry model. – We are building CI specific ontologies in terms of GML to define
faults, satellites etc.– http://ripvanwinkle.ucs.indiana.edu:4780/examples/download/schema/
• Styled Layer Descriptor (SLD) specifies the format of a map-styling language for portraying the output of Web Map Servers, Web Feature Servers and Web Coverage Servers etc. SLD will enable different communities in the Emergency Response area to develop a set of customized portrayal rules that best fit their mission requirements.– This becomes the specification of portals to different composite Grids
• Sensor Markup Language (SensorML) defines the information model for discovering, querying and controlling Web-resident sensors.
• Observations & Measurements (O&M) defines the information model for observations that are returned from the CrisisGrid sensors.
GIS Grid Services I• Web Feature Service (WFS) supports the query and discovery of
geographic features delivering GML representations of simple geospatial features in response to queries from HTTP clients. WFS can access geographic features including critical infrastructure features, incident locations, and flood-related geographic features including inundation areas, watershed boundaries, and demographic feature.
• Web Coverage Service (WCS) supports the query and discovery of digital geospatial information such as digital elevation models, imagery, orthophotography, weather coverages (such as predicted rainfall, air pressure, wind speed and direction), and any other space-varying flood-related phenomena.
• Web Map Service (WMS) uses a SLD portrayal to generate "pictures" of georeferenced feature or coverage data. WMS will provide a means to portray geographic information independent of the underlying data model (WFS or WCS).
• Coverage Portrayal Service (CPS) defines a standard interface for producing visual pictures from coverage data typically accessed via WCS with a SLD portrayal.
GIS Grid Services II• Web Terrain Service (WTS) augments WMS with advanced
visualization including 3D terrains. • Catalog Service - Web Profile (CS-W) is a catalog service
that will be built on a general Grid metadata service• Sensor Collection Service (SCS) fetches observations from
a sensor or group of sensors and will be integrated with research on Grid sensor services
• Sensor Planning Service (SPS) assists in 'collection feasibility plans' and to process collection requests for a sensor or group of sensors.
• Web Notification Service (WNS) will be replaced by standard Grid notification service
Grid Portals Portals need to provide services for security,
customization, layout, rendering Jetspeed and GridSphere are two well known portals
• Commercial systems like IBM WebSphere similar These Portals are very important as they encourage
“component” model for user interfaces and so this fits service model so every service can be packaged with its (document fragment) user interface
Portlets are good as they support they mixing and matching of services for a particular composed/overlay Grid to allow this to be easily reflected in user interface• New JSR168 portlet Java Standard
NSF Middleware Initiative NMI funding collection of open source Grid portlets
OGCEOGCEConsortium
The OGCE Computing Grid Portal
• Provides Portlets for– Management of user proxy
certificates– Remote file Management via
Grid FTP– News/Message systems
• for collaborations– Grid Event/Logging service– Access to OGSA services – Access to directory services– Specialized Application Factory
access• Distributed Workflow
– Access to Metadata Index tools• User searchable index
– Real Time Collaboration• Audio/Video Conferencing
Download from open source http://www.ogce.org
OGCEOGCEConsortium
Example Capability: File Management
• Grid FTP portlet– Allow User to manage remote file spaces– Uses stored proxy for
authentication– Upload and download files– Third party file transfer
• Request that GridFTP server A send a file to GridFTP server B
• Does not involve traffic through portal server
Portal Server
User
GridFTPServer A GridFTP
Server B
GridFTPService
1 of many Portlets
Jetspeed
Watershed Entry Page: Select a County
FloodGridPortal
Watershed Boundaries in Black
Open source GIS MapServer presented as a simple portlet
Data Download Portal
This is where you can download ArcView compatible data files for your watershed. Some of this gets hooked to the runoff model in our connected crisis grid model.
Flood Grid
DataArchives
DataArchives
RunoffModel
RunoffModel
FlowModel
FlowModel
FlowModel
GIS Grid Services Link Distributed
Data and Applications
SOAP MessagesAnd Events
DataArchives
DataArchives
RunoffModel
RunoffModel
FlowModel
FlowModel
FlowModel
GIS Grid Services Link Distributed
Data and Applications
SOAP MessagesAnd Events
Parallel Simulations
Sensors
GIS Data Archives provide terrain, elevation, and land use data
Grid WorkflowLinks Data, Sensors, Model
SERVOGrid HPC ApplicationsSERVOGrid HPC Applications Codes range from simple “rough estimate” codes to parallel, Codes range from simple “rough estimate” codes to parallel,
high performance applications.high performance applications.• DislocDisloc: handles multiple arbitrarily dipping dislocations (faults) in an : handles multiple arbitrarily dipping dislocations (faults) in an
elastic half-space.elastic half-space.• SimplexSimplex: inverts surface geodetic displacements for fault : inverts surface geodetic displacements for fault
parameters using simulated annealing downhill residual parameters using simulated annealing downhill residual minimization. minimization.
• GeoFESTGeoFEST: Three-dimensional viscoelastic finite element model for : Three-dimensional viscoelastic finite element model for calculating nodal displacements and tractions. Allows for realistic calculating nodal displacements and tractions. Allows for realistic fault geometry and characteristics, material properties, and body fault geometry and characteristics, material properties, and body forces. forces.
• VirtualVirtual CaliforniaCalifornia: Program to simulate interactions between vertical : Program to simulate interactions between vertical strike-slip faults using an elastic layer over a viscoelastic half-space strike-slip faults using an elastic layer over a viscoelastic half-space
• RDAHMMRDAHMM: Time series analysis program based on Hidden Markov : Time series analysis program based on Hidden Markov Modeling. Produces feature vectors and probabilities for Modeling. Produces feature vectors and probabilities for transitioning from one class to another. transitioning from one class to another.
• PARKPARK: Boundary element program to calculate fault slip velocity : Boundary element program to calculate fault slip velocity history based on fault frictional properties; a model for unstable slip history based on fault frictional properties; a model for unstable slip on a single earthquake fault.on a single earthquake fault.
• PDPCPDPC: Phase Dynamics Probability Change: Phase Dynamics Probability Change Preprocessors, mesh generatorsPreprocessors, mesh generators Visualization tools: Visualization tools: RIVARIVA, , GMTGMT
QuakeSim Portal ShotsQuakeSim Portal ShotsSERVOGrid linking HPC, Data and Visualization
SAR Data
Run Finley – Simulation from Australia
Global-MMCS 2.0 XGSP MCU We are building an open source protocol independent Web
Service “MCU” which will scale to an arbitrary number of users and provide integrated thousands of simultaneous users collaboration services.
We will deploy it globally with first release end of May 2004. The function of A/V media server will be distributed using
NaradaBrokering architecture.• Media Servers mix and convert A/V streams
Open XGSP MCU based on the following open source projects• openh323 is basis of H323 Gateway
• NIST SIP stack is basis of SIP Gateway
• NaradaBrokering is open source messaging from Indiana
• Java Media Framework basis of Media Servers
XGSP Web Service MCU Architecture
SIP H323 Access Grid Native XGSPAdmire
Gateways convert to uniform XGSP Messaging
High Performance (RTP)and XML/SOAP and ..
Media ServersFilters
Session ServerXGSP-based Control
NaradaBrokeringAll Messaging
Use Multiple Media servers to scale to many codecs and manyversions of audio/video mixing
NB Scales asdistributed
WebServices
NaradaBrokering
A/V Collaboration Systems GlobalMMCS Grid federates existing A/V Protocols H323
H.323 is defined as an umbrella standard specifying the components to be used within an H.323-based environment.
SIPThe Session Initiation Protocol (SIP) defines how to establish, maintain and terminate Internet sessions including multimedia conferences
Access Grid enhanced Mbone A/V tools ( VIC, RAT ) Internet 2 network ( Multicast support )
Grid Services in GlobalMMCS I Audio Mixing
The audio mixer creates a mixed audio stream from all the audio streams in the session
Video Mixing Video mixing makes the unicast users watch the pictures of
multiple participants in a meeting through one video stream Video Thumbnail visualize the VS set in the session, embedded into the control panel
of each endpoint, which Image grabbers capture video streams and save them as static JPEG
files. All the media processing components can be distributed among
the pool of the media servers connected to NaradaBrokering infrastructures.
This generalizes to a HPC farm of “stream servers” doing image processing etc.• Perhaps need 100 node cluster to support 10,000 simultaneous A/V
streams
H.323, SIP Gateway Servers, A/V Session Server H.323 and SIP gateway transform their protocol
specific messages into XGSP signaling messages so that H.323 and SIP A/V endpoints could communicate with the XGSP A/V session server
The session server implements session management logics • creating/destroying A/V sessions
• allowing endpoints to join/leave session
• Allowing users to make audio/video selection, managing A/V application components
0
10
20
30
40
50
60
0 200 400 600 800 1000 1200 1400 1600 1800 2000
De
lay
(Mill
ise
con
ds)
Packet Number
Average delays per packet for 50 video-clients NaradaBrokering Avg=2.23 ms, JMF Avg=3.08 ms
NaradaBrokering-RTP JMF-RTP
Polycom, Access Grid and RealVideo views of multiple streams using
GlobalMMCSA/V Web Service
Unicast AG Portlet
WSDisplay
WSViewer
WS Display
WS Viewer
Master
WSDisplay
WS Viewer
NaradaBrokeringMessage Bus
Collaboration as a WSSet up Session with GlobalMMCS
defining participants
Application orContent source
WSDL
Web Service
F
I
U
O
F
I
R
O
Shared Output Port Collaboration
OtherParticipants
Text ChatWhiteboardGIS Services etc. areAUTOMATICALLYCollaborative ifBuilt as a Grid Services
Integration of PDA, Cell phone and Desktop Grid Access