GISolve GISolve A GridA Grid--based problem solving environment for based problem solving environment for computationally intensive geographic information computationally intensive geographic information analysisanalysis

Shaowen WangShaowen Wang

Department of Geography & Department of Geography & Grid Research & educatiOn group @ IoWa (GROW)Grid Research & educatiOn group @ IoWa (GROW)

ITS Academic Technologies ITS Academic Technologies –– Research Services Research Services The University of IowaThe University of Iowa

Iowa City, IA 52242, USAIowa City, IA 52242, USA

July 24, 2005July 24, 2005

22

CollaboratorsCollaborators

Marc P. Armstrong, Professor, Department of Marc P. Armstrong, Professor, Department of Geography and Program in Applied Geography and Program in Applied Mathematical and Computational SciencesMathematical and Computational SciencesJun Ni, Ph.D., Associate Research Scientist, Jun Ni, Ph.D., Associate Research Scientist, ITS Academic Technologies ITS Academic Technologies –– Research Research ServicesServicesYan Liu, Graduate Research Assistant, ITS Yan Liu, Graduate Research Assistant, ITS Academic Technologies Academic Technologies –– Research Services Research Services

33

PurposePurpose

To address why the Grid is important to To address why the Grid is important to geographic information analysis geographic information analysis researchresearchTo demonstrate the To demonstrate the designdesign and and implementationimplementation of GISolve of GISolve –– a Grida Grid--based problem solving environment for based problem solving environment for computationally intensive geographic computationally intensive geographic information analysisinformation analysis

Purpose

Background

Design

Case Study

Implementation

Evaluation

Conclusions

44

BackgroundBackground

The amount of geographic informationThe amount of geographic information–– Ever increasingEver increasing

Application drivenApplication driven–– GPS, LBS, RSGPS, LBS, RS

Computationally intensive geographic analysisComputationally intensive geographic analysis–– Heuristic searchHeuristic search–– SimulationSimulation–– OptimizationOptimization–– Statistical methods Statistical methods

Grid computingGrid computing–– CyberinfrastructureCyberinfrastructure–– Open Grid Service Architecture (OGSA)Open Grid Service Architecture (OGSA)

Web servicesWeb servicesProblem solving environmentsProblem solving environments–– Grid portalsGrid portals

Purpose

Background

Design

Case Study

Implementation

Evaluation

Conclusions

55

GIScience Grid PortalGIScience Grid Portal

Purpose

Background

Design

Implementation

Case Study

Evaluation

Conclusions

66

Grid Resources Grid Resources –– A User ViewA User View

R

GIPPL

MA

PeerDB

R: Resource (Running Sensor), e.g., GangliaGIP: Grid Information Provider, e.g., MDSDB: Database (MYSQL in this case)MA: Monitoring AgentPL: Peer Location ComponentVO: Virtual Organization

PeerDB

Peer

Peer

DB

DB

PeerDB

VO1

VO2

User Agent

Peer communicationServing User Requests

Fully ConnectedVO Networks

View 1View 2

User Agent

View 1

Purpose

Background

Design

Implementation

Case Study

Evaluation

Conclusions

77

Grid ComplexityGrid Complexity

Grid middleware limitationGrid middleware limitation–– Provides a selected set of protocols and servicesProvides a selected set of protocols and services

Not directly userNot directly user-- or applicationor application-- orientedoriented

HeterogeneousHeterogeneousDynamicDynamicAdministrated in different security domainsAdministrated in different security domainsFaults Faults –– Usually unpredictableUsually unpredictable

Purpose

Background

Design

Case Study

Implementation

Evaluation

Conclusions

88

GISolve Architecture

Computational domain modeling

Domain decomposition

Task scheduling

Information broker and resource

discoveryData access module

Problem solving environments implemented using Grid portal technologies

Grid information services

Protocols and services for data access on the Grid,

such as the Globus GridFTP

Grid Middleware such as Globus, Legion, and Condor

Resource allocation

99

GISolve ServicesGISolve Services

SecurityDecomposition and task schedulingGeographic data accessGeographic data accessResource information brokering Resource information brokering

Purpose

Background

Design

Case Study

Implementation

Evaluation

Conclusions

1010

Classification of GISolve Classification of GISolve ServicesServices

Purpose

Background

Design

Case Study

Implementation

Evaluation

Conclusions

1111

GISolve WorkflowGISolve Workflow

Purpose

Background

Design

Implementation

Case Study

Evaluation

Conclusions

1212

A ThreeA Three--Layer Model of Layer Model of Grid Portal Technologies Grid Portal Technologies

Purpose

Background

Design

Case Study

Implementation

Evaluation

Conclusions

1313

Technology SpecificationTechnology Specification

JetspeedJetspeed portal serverportal server–– JetspeedJetspeed: portal engine : portal engine –– Turbine: MVC (ModelTurbine: MVC (Model--ViewView--Control) frameworkControl) framework–– Velocity: dynamic html generationVelocity: dynamic html generation–– Tomcat: portal containerTomcat: portal container

MultiMulti--user supportuser support–– User registration and managementUser registration and management–– UserUser--based state managementbased state management

Configurable portal interfaceConfigurable portal interface–– Customized layoutCustomized layout–– Display control of each Display control of each portletportlet interfaceinterface

Purpose

Background

Design

Case Study

Implementation

Evaluation

Conclusions

1414

APIs ImplementedAPIs Implemented

APIs developedAPIs developed–– [[org.gisolve.globusorg.gisolve.globus]: Globus Toolkit enhancement]: Globus Toolkit enhancement–– [[edu.uiowa.gisolve.ogsaedu.uiowa.gisolve.ogsa]: Grid service (interface ]: Grid service (interface

definition, userdefinition, user--level Grid service implementation)level Grid service implementation)–– [[org.apache.jetspeed.modules.actions.gisolveportlorg.apache.jetspeed.modules.actions.gisolveportl

etsets]: ]: JetspeedJetspeed portletportlet action implementationaction implementation–– [[gisolveportletsgisolveportlets]: portal support for Grid service ]: portal support for Grid service

APIs (state management, Grid service client APIs)APIs (state management, Grid service client APIs)APIs integratedAPIs integrated–– Java COG (GT2), GT3/GT4 Core, OGSA (WS Java COG (GT2), GT3/GT4 Core, OGSA (WS

GRAM, RFT), GRAM, RFT), MyProxyMyProxy, LDAP, JNI, JPL, LDAP, JNI, JPL

Purpose

Background

Design

Case Study

Implementation

Evaluation

Conclusions

1515

Defining GISolve ServicesDefining GISolve Services

••<definitions name="<definitions name="TSServiceTSService targetNamespacetargetNamespace==http://uiowa.edu/grow/namespaces/2004/04/gisolve/TShttp://uiowa.edu/grow/namespaces/2004/04/gisolve/TS …… />/>••<!<!---- import external types import external types ---->>••<import location="../<import location="../ogsi/ogsi.gwsdlogsi/ogsi.gwsdl““ ……/>/>••<!<!---- selfself--defined types defined types ---->>••<import location="<import location="gidTypes.xsdgidTypes.xsd““ namespace="http://uiowa.edu/grow/namespaces/2004/04/gisolve/gidnamespace="http://uiowa.edu/grow/namespaces/2004/04/gisolve/gidtypes"/>types"/>••<!<!---- local types schema local types schema ---->>••<types><<types><xsd:schemaxsd:schema ……/></types>/></types>••<!<!---- message definition message definition ---->>••<message name="<message name="SchedInputMessageSchedInputMessage">">……</message></message>••<message name="<message name="SchedOutputMessageSchedOutputMessage">">……</message></message>••<!<!---- service operation definition service operation definition ---->>••<<gwsdl:portTypegwsdl:portType name="name="TSPortTypeTSPortType" extends="" extends="ogsi:GridServiceogsi:GridService">">

••<operation name="schedule"><operation name="schedule">••<input message="<input message="tns:SchedInputMessagetns:SchedInputMessage"/>"/>••<output message="<output message="tns:SchedOutputMessagetns:SchedOutputMessage"/>"/>••<fault name="Fault" message="<fault name="Fault" message="ogsi:FaultMessageogsi:FaultMessage"/>"/>

••</operation> </operation> ••<<sd:serviceDatasd:serviceData name="name="TSStateTSState““ …… />/>

••</</gwsdl:portTypegwsdl:portType>>

A task scheduling service A task scheduling service gWSDLgWSDL::Purpose

Background

Design

Case Study

Implementation

Evaluation

Conclusions

1616

OGSI to WSRFOGSI to WSRFWSDL

<?xml version="1.0" encoding="UTF-8"?><definitions name="DCService" ... ...

xmlns:wsrp="http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-ResourceProperties-1.2-draft-01.xsd"xmlns:wsrpw="http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-ResourceProperties-1.2-draft-01.wsdl"xmlns:wsdlpp="http://www.globus.org/namespaces/2004/10/WSDLPreprocessor" ><wsdl:import namespace="http://docs.oasis-open.org/wsrf/2004/06/wsrf-WS-ResourceProperties....wsdl"

location="../wsrf/properties/WS-ResourceProperties.wsdl"/><types>... ...<xsd:element name="DCResourceProperties">... ...</xsd:element></types><!-- WSRF Grid service PortType definition --><portType name="DCPortType" wsdlpp:extends="wsrpw:GetResourceProperty"

wsrp:ResourceProperties="tns:DCResourceProperties"><operation name="decompose">... ...</operation>

</portType>

gWSDL<?xml version="1.0" encoding="UTF-8"?><definitions name="DCService“ … …

xmlns:ogsi="http://www.gridforum.org/namespaces/2003/03/OGSI"xmlns:gwsdl="http://www.gridforum.org/namespaces/2003/03/gridWSDLExtensions"xmlns:sd="http://www.gridforum.org/namespaces/2003/03/serviceData"><!-- import ogsi Grid Service gwsdl. Location is under gt3/schema/ogsi/ --><import location="../ogsi/ogsi.gwsdl" namespace="http://www.gridforum.org/namespaces/2003/03/OGSI"/>... ...<!-- Grid service interface definition --><gwsdl:portType name="DCPortType" extends="ogsi:GridService">

<operation name="decompose">… …</operation> <sd:serviceData name="DCState" type="xsd:int" … … />

</gwsdl:portType>

1717

GISolve Services in ActionGISolve Services in Action

Deployed GISolve services:Deployed GISolve services:Purpose

Background

[globus4@rsgridportal globus4]$ globus-start-containerStarting SOAP server at: https://128.255.162.167:8443/wsrf/services/With the following services:... ...[12]: https://128.255.162.167:8443/wsrf/services/gisolve/TSService... ...[47]: https://128.255.162.167:8443/wsrf/services/gisolve/DCService

Design

Implementation

Case Study

Evaluation

Conclusions

1818

Case Study Case Study –– GGii*(d) *(d) StatisticStatistic

Purpose

)(* dGi = 2/12**1

*

)}1/(]){[(

)(

−−

−∑nWnSs

xWxdw

ii

j ijijBackground

Design

Implementation

Case Study ReferencesReferences–– GetisGetis, A., and , A., and OrdOrd, J.K., 1992, The analysis of spatial , J.K., 1992, The analysis of spatial

association by use of distance statistics. association by use of distance statistics. Geographical Geographical AnalysisAnalysis, , 2424(3): 189(3): 189--206.206.

–– OrdOrd, J. K., and , J. K., and GetisGetis, A., 1995, Local spatial autocorrelation , A., 1995, Local spatial autocorrelation statistics: distributional issues and an application. statistics: distributional issues and an application. Geographical AnalysisGeographical Analysis, , 2727(4): 286(4): 286--306.306.

Evaluation

Conclusions

1919

Synthetic DatasetsSynthetic Datasets

Purpose

Background

Design

Implementation

Case Study

Evaluation

Conclusions

20000-point : two clusters, each of which has a normal distribution with a standard deviation of 2

20000-point a uniform random distribution

2020

User InterfaceUser Interface

Purpose

Background

Design

Implementation

Case Study

Evaluation

Conclusions

2121

An Example of Spatial An Example of Spatial Computational Domain Computational Domain DecompositionDecomposition

PurposeY

Background

0, 0 1, 1 2, 4

4, 2 5, 3

11, 13

3, 5

6, 6 7, 7

8, 8 9, 9 10, 12

12, 10 13, 11 14, 14 15, 15Design

Implementation

Case Study

Evaluation

Conclusions

X

2222

A Task Scheduling ServiceA Task Scheduling Service

Decoupled from domain decomposition Decoupled from domain decomposition servicesservicesNPNP--complete problemcomplete problem–– MaxMax--min algorithm implementedmin algorithm implemented

Using the theoretical estimate to Using the theoretical estimate to compute the Expected Time to compute the Expected Time to Compute (ETC) matrixCompute (ETC) matrix

Purpose

Background

Design

Case Study

Implementation

Evaluation

Conclusions

2323

A Diagrammatic Example A Diagrammatic Example of Static Task Schedulingof Static Task Scheduling

1, 10, 0 2, 4

4, 2 5, 3

11, 13

14, 14 15, 15

3, 5

6, 6 7, 7

8, 8 9, 9 10, 12

12, 10 13, 11

1, 10, 0 2, 4

4, 2 5, 3

11, 13

14, 14 15, 15

3, 5

6, 6 7, 7

8, 8 9, 9 10, 12

12, 10 13, 11

1, 10, 0 2, 4

4, 2 5, 3

11, 13

14, 14 15, 15

3, 5

6, 6 7, 7

8, 8 9, 9 10, 12

12, 10 13, 11

1, 10, 0 2, 4

4, 2 5, 3

11, 13

14, 14 15, 15

3, 5

6, 6 7, 7

8, 8 9, 9 10, 12

12, 10 13, 11

Resource1 Resource2 Resource3

Purpose

Background

Design

Implementation

Case Study

Evaluation

Conclusions

2424

Grid Testbed Grid Testbed Implementation Implementation -- HawkGridHawkGrid

rtgrid1.its.uiowa.edu

The Open Science Grid

High energy physics VO

beowulf.stat.uiowa.edu rsgrid3.its.uiowa.edu

A server desktop at

ITS –rsgrid1

8.5Mbits/s

62.4Mbits/s

60.5Mbits/s

27.3Mbits/s 331Kbits/s 306Kbits/s

28.3Mbits/s

48.2Mbits/s73.7Mbits/s

ITS-Stat-Geo VO

Purpose

Background

Design

Implementation

Case Study

Evaluation

Conclusions

2525

Purpose

Background

DEMODesign

Implementation

Case Study

Evaluation

Conclusions

2626

Performance Performance

The problem cannot be solved by any The problem cannot be solved by any singlesingle computer in our Grid testbedcomputer in our Grid testbed–– 3.2G memory3.2G memory

Solved within 10 minutes through Solved within 10 minutes through GISolveGISolve

Purpose

Background

Design

Case Study

Implementation

Evaluation

Conclusions

2727

ConclusionsConclusions

GISolve demonstrates how the Grid can GISolve demonstrates how the Grid can benefit research on computationally benefit research on computationally intensive geographic information intensive geographic information analysesanalysesGISolve integrates OGSAGISolve integrates OGSA--based Web based Web services to support the computational services to support the computational aspects of GIServicesaspects of GIServices

Purpose

Background

Design

Case Study

Implementation

Evaluation

Conclusions

2828

Ongoing Research Ongoing Research

Interoperability of GISolve servicesInteroperability of GISolve servicesAdaptive domain decomposition Adaptive domain decomposition servicesservicesEvaluation of GISolve performanceEvaluation of GISolve performanceExtension of the types of geographic Extension of the types of geographic information analyses GISolve supportsinformation analyses GISolve supports

2929

AcknowledgementAcknowledgement

This research is partially supported by This research is partially supported by the HawkGrid project funded by the the HawkGrid project funded by the Office of Vice President for Research at Office of Vice President for Research at The University of Iowa The University of Iowa Computational resources used for Computational resources used for experiments include a cluster supported experiments include a cluster supported by the U.S. NSF iVDGL (international by the U.S. NSF iVDGL (international Virtual Data Grid Laboratory) project Virtual Data Grid Laboratory) project