grass cloud
DESCRIPTION
The presentation of g.cloud module for GRASS GIS at FOSS4G 2011TRANSCRIPT
![Page 1: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/1.jpg)
IntroductionIdea
ImplementationConclusions
GRASS GIS in the Cloud
Luca Delucchi, Markus Neteler
Fondazione Edmund Mach – GIS & Remote Sensing Platform
http://gis.cri.fmach.it
FOSS4G 201114th September 2011, Denver (USA)
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 2: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/2.jpg)
IntroductionIdea
ImplementationConclusions
GRASS GISOur cluster
GRASS GIS
The first Open Source GIS
More than 400modules for everykind of analysis
Interoperability withother software
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 3: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/3.jpg)
IntroductionIdea
ImplementationConclusions
GRASS GISOur cluster
GRASS GIS
The first Open Source GIS
More than 400modules for everykind of analysis
Interoperability withother software
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 4: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/4.jpg)
IntroductionIdea
ImplementationConclusions
GRASS GISOur cluster
GRASS GIS
The first Open Source GIS
More than 400modules for everykind of analysis
Interoperability withother software
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 5: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/5.jpg)
IntroductionIdea
ImplementationConclusions
GRASS GISOur cluster
Our cluster: infrastructure
FEM GIS cluster consists of
300 nodes
610 GB RAM
10 TB disk storage + 15 TB tapebackup
10 Gb/s internal bus
Scientific Linux 5.6
Grid Engine
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 6: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/6.jpg)
IntroductionIdea
ImplementationConclusions
GRASS GISOur cluster
Our cluster: massive spatial data analysis
MODIS Land Surface Temperature(LST) map gap-filling (13,000 maps)
MODIS data analysis (LST, NDVI,Snow products)
LiDAR data analysis
Solar energy calculations
Neteler et al., 2011: Tiger mosquito, IJHG, doi:10.1186/1476-072X-10-49
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 7: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/7.jpg)
IntroductionIdea
ImplementationConclusions
Idea to facilitate GRASS GIS usage on a cluster
Simplify the work
Offer the possibility to use Grid Engine by all our colleagues
Launch jobs remotely without connecting every time to thecluster
Offer the possibility to external people to use the clusterwithout learning Grid Engine
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 8: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/8.jpg)
IntroductionIdea
ImplementationConclusions
Idea to facilitate GRASS GIS usage on a cluster
Simplify the work
Offer the possibility to use Grid Engine by all our colleagues
Launch jobs remotely without connecting every time to thecluster
Offer the possibility to external people to use the clusterwithout learning Grid Engine
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 9: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/9.jpg)
IntroductionIdea
ImplementationConclusions
Idea to facilitate GRASS GIS usage on a cluster
Simplify the work
Offer the possibility to use Grid Engine by all our colleagues
Launch jobs remotely without connecting every time to thecluster
Offer the possibility to external people to use the clusterwithout learning Grid Engine
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 10: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/10.jpg)
IntroductionIdea
ImplementationConclusions
Idea to facilitate GRASS GIS usage on a cluster
Simplify the work
Offer the possibility to use Grid Engine by all our colleagues
Launch jobs remotely without connecting every time to thecluster
Offer the possibility to external people to use the clusterwithout learning Grid Engine
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 11: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/11.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: requirements
GRASS GIS 7 (current development version)
Grid Engine
Python > 2.4
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 12: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/12.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: requirements
GRASS GIS 7 (current development version)
Grid Engine
Python > 2.4
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 13: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/13.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: requirements
GRASS GIS 7 (current development version)
Grid Engine
Python > 2.4
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 14: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/14.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: how does it work?
Required parameters
conf = file with username and password or stdin to connectto the cluster
server = hostname of the cluster
qsub script = file containing qsub script (template)
grass script = own GRASS GIS script
g.cloud conf=/tmp/passwd server=giscluster
qsub script=test launch SGE grassjob.sh grass script=test novariables.sh
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 15: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/15.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: how does it work?
Required parameters
conf = file with username and password or stdin to connectto the cluster
server = hostname of the cluster
qsub script = file containing qsub script (template)
grass script = own GRASS GIS script
g.cloud conf=/tmp/passwd server=giscluster
qsub script=test launch SGE grassjob.sh grass script=test novariables.sh
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 16: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/16.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: how does it work?
Required parameters
conf = file with username and password or stdin to connectto the cluster
server = hostname of the cluster
qsub script = file containing qsub script (template)
grass script = own GRASS GIS script
g.cloud conf=/tmp/passwd server=giscluster
qsub script=test launch SGE grassjob.sh grass script=test novariables.sh
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 17: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/17.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: how does it work?
Required parameters
conf = file with username and password or stdin to connectto the cluster
server = hostname of the cluster
qsub script = file containing qsub script (template)
grass script = own GRASS GIS script
g.cloud conf=/tmp/passwd server=giscluster
qsub script=test launch SGE grassjob.sh grass script=test novariables.sh
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 18: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/18.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: how does it work?
Required parameters
conf = file with username and password or stdin to connectto the cluster
server = hostname of the cluster
qsub script = file containing qsub script (template)
grass script = own GRASS GIS script
g.cloud conf=/tmp/passwd server=giscluster
qsub script=test launch SGE grassjob.sh grass script=test novariables.sh
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 19: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/19.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: how does it work?
Optional parameters
raster = name of raster(s)
vector = name of vector(s)
variables = string with name and values of variables statedas Python dictionary “’key’:[’value0’,’value1’]”
email = user’s email address
path = path to the folder which must be accessible fromcluster frontend and blades
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 20: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/20.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: how does it work?
Optional parameters
raster = name of raster(s)
vector = name of vector(s)
variables = string with name and values of variables statedas Python dictionary “’key’:[’value0’,’value1’]”
email = user’s email address
path = path to the folder which must be accessible fromcluster frontend and blades
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 21: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/21.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: how does it work?
Optional parameters
raster = name of raster(s)
vector = name of vector(s)
variables = string with name and values of variables statedas Python dictionary “’key’:[’value0’,’value1’]”
email = user’s email address
path = path to the folder which must be accessible fromcluster frontend and blades
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 22: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/22.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: how does it work?
Optional parameters
raster = name of raster(s)
vector = name of vector(s)
variables = string with name and values of variables statedas Python dictionary “’key’:[’value0’,’value1’]”
email = user’s email address
path = path to the folder which must be accessible fromcluster frontend and blades
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 23: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/23.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: how does it work?
Optional parameters
raster = name of raster(s)
vector = name of vector(s)
variables = string with name and values of variables statedas Python dictionary “’key’:[’value0’,’value1’]”
email = user’s email address
path = path to the folder which must be accessible fromcluster frontend and blades
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 24: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/24.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: more examples
North Carolina sample data
r.texture exampleg.cloud conf=/tmp/passwd server=gisclusterqsub script=test launch SGE grassjob.shgrass script=test onevariables raster.sh variables=“’TEXT’ :[’asm’,’contrast’,’corr,var’,’idm’,’sa’,’se’,’sv’,’entr’,’dv’,’de’,’moc1’,’moc2’]”raster=lsat7 2002 40
r.sun exampleg.cloud conf=/tmp/passwd server=giscluster
qsub script=test launch SGE grassjob.sh grass script=test onevariables sun.sh
variables=“’DOY’ : [1,2,3,4,5,6,7,8,9,10]” raster=elevation
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 25: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/25.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: more examples
North Carolina sample data
r.texture exampleg.cloud conf=/tmp/passwd server=gisclusterqsub script=test launch SGE grassjob.shgrass script=test onevariables raster.sh variables=“’TEXT’ :[’asm’,’contrast’,’corr,var’,’idm’,’sa’,’se’,’sv’,’entr’,’dv’,’de’,’moc1’,’moc2’]”raster=lsat7 2002 40
r.sun exampleg.cloud conf=/tmp/passwd server=giscluster
qsub script=test launch SGE grassjob.sh grass script=test onevariables sun.sh
variables=“’DOY’ : [1,2,3,4,5,6,7,8,9,10]” raster=elevation
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 26: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/26.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: future
Support for more clustering system (for example Eucalyptus)
Auto installation of GRASS and its dependencies on thecluster if not present (GRASS 7 release as stable version)
Show the status of jobs
Return outputs to the client
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 27: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/27.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: future
Support for more clustering system (for example Eucalyptus)
Auto installation of GRASS and its dependencies on thecluster if not present (GRASS 7 release as stable version)
Show the status of jobs
Return outputs to the client
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 28: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/28.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: future
Support for more clustering system (for example Eucalyptus)
Auto installation of GRASS and its dependencies on thecluster if not present (GRASS 7 release as stable version)
Show the status of jobs
Return outputs to the client
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 29: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/29.jpg)
IntroductionIdea
ImplementationConclusions
RequirementsHow does it work?Future
Implementation: future
Support for more clustering system (for example Eucalyptus)
Auto installation of GRASS and its dependencies on thecluster if not present (GRASS 7 release as stable version)
Show the status of jobs
Return outputs to the client
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 30: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/30.jpg)
IntroductionIdea
ImplementationConclusions
Conclusions
First implementation of a module to run GRASS on a cluster
Experimental version
Please test it and report bugs or improvements
Contact us for more [email protected] - [email protected]
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 31: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/31.jpg)
IntroductionIdea
ImplementationConclusions
Conclusions
First implementation of a module to run GRASS on a cluster
Experimental version
Please test it and report bugs or improvements
Contact us for more [email protected] - [email protected]
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 32: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/32.jpg)
IntroductionIdea
ImplementationConclusions
Conclusions
First implementation of a module to run GRASS on a cluster
Experimental version
Please test it and report bugs or improvements
Contact us for more [email protected] - [email protected]
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 33: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/33.jpg)
IntroductionIdea
ImplementationConclusions
Conclusions
First implementation of a module to run GRASS on a cluster
Experimental version
Please test it and report bugs or improvements
Contact us for more [email protected] - [email protected]
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver
![Page 34: Grass cloud](https://reader037.vdocuments.site/reader037/viewer/2022100304/555078f0b4c90524138b5160/html5/thumbnails/34.jpg)
IntroductionIdea
ImplementationConclusions
Conclusions
Thanks for your attention
gis.cri.fmach.it
Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver