Cluster Computing for SWAP Crop Model Parameter Identification using RS
HONDA KiyoshiMd. Shamim Akhter
Yann CheminPutchong Uthayopas
Importance of Crop model parameters identification
Agriculture Activity Monitoring Sowing date, Cropping intensity, Growth, Water stress, Yield,
etc. Production for Food Security Water Management in Irrigation Activity Remote Sensing is quite useful data, however modeling has
not been done. – empirical model Crop growth model ( SWAP )
Continuous monitoring in various aspects Prediction Spatial Parameter estimation & Calibration ->RS
Data Assimilation Technique To estimate parameters which cannot be observed by RS High-Resolution Remote Sensing High-Resolution Remote Sensing Temporal Info -> Low Resolution Remote Sensing; Mixed PixelTemporal Info -> Low Resolution Remote Sensing; Mixed Pixel
SWAP Model Diagram
Adopted from Van Dam et al. (1997)Drawn by Teerayut Horanont (AIT)
SWAP Model Parameter identificationSWAP Model Parameter identification - Data Assimilation using RS and GA -- Data Assimilation using RS and GA -
0.00
1.00
2.00
3.00
4.00
0 45 90 135 180 225 270 315 360
Day Of Year
Eva
po
tra
nsp
iratio
n L
AI
RS ObservationRS Observation
SWAP Crop Growth ModelSWAP Crop Growth Model
SWAP Input ParametersSWAP Input Parameters
sowing date, soil property, sowing date, soil property, Water management, and etc.Water management, and etc.
LAI, LAI, EvapotranspirationEvapotranspiration
0.00
1.00
2.00
3.00
4.00
0 45 90 135 180 225 270 315 360
Day Of Year
E
avpo
tran
spira
tion
LA
I
FittingFitting
LAI, LAI, EvapotranspirationEvapotranspiration
Assimilation by Assimilation by finding Optimized finding Optimized
parametersparameters
By GABy GA
RSRS ModelModel
Field photos
Longitude: 100008133. Latitude: 14388195.
Comparison of Satellite LAI and Simulated LAI
0
1
2
3
4
5
0 30 60 90 120 150 180 210 240 270 300 330 360
DOY
LA
I
LAI_sat
LAI_sim
Estimated parameters DOYCrop1 = 19 DOYCrop2 = 188 Crop.Int.Crop2 = 0.32 Fitness = 4.537 Generation found = 31 (popsize=5) Calculation time approximate 15 minutes
A Practical Problem Arises
A practical issue arises with the overall calculations time load for assimilating data with remote sensing data.
The calculation time for identify SWAP parameters only for 1 pixel (1 sq.km) takes several minutes to 30 minutes.
Thus, a RS image of 1000 x 1000 sq.km of 1000x1000 pixels will take more than 50 years (30min x 1000 x 1000) is not acceptable.
Solutions LUT Method Segmentation Parallel Computing
Objectives
Implement SWAP-GA model in cluster computers
Propose and evaluate job distribution models Population by Population Pixel by Pixel Hybrid
Develop a model to estimate calculation time
3 Clusters (AIT and Kasetsart universities Clusters) Optima 9 nodes: Front-end:
Athlon XP 1800+ 512 MB RAM 80 GB IDE Disk Compute Node:
Athlon Xp 1800+ 512 MB RAM 40 GB IDE Disk Disk Interconnection: Fast Ethernet http://optima.ait.ac.th/scmsweb/scms_home.html Magi 4 nodes:
Athlon XP 2500+ 512 MB RAM 80 GB Hard Disk Disk Interconnection: Gigabit Ethernet Interface http://magi.cpe.ku.ac.th/scmsweb/scms_home.html
Clusters Technical Specifications
Maeka 32 nodes: Front-end: Dual AMD Opteron 1400 MHz
RAM PC2100 : 6GBs Disk SCSI Ultra 320: 72GBs
15 machines Dual AMD Opteron 1400 MHz RAM PC2100 : 3GBs Hard Disk SCSI Ultra 320: 72GBs 16 machines Dual AMD Opteron 1800 MHz RAM PC2700 : 3GBs Hard Disk SCSI Ultra 320: 72GBs Gigabit Ethernet Interface2 http://maeka.ku.ac.th/spec.html
Technical Spec Continue..
Implementation Schemes
3 Implementation Schemes are proposed Population by population (Distributing population) Pixel by pixel Hybrid (combination of Population by population
and pixel by pixel)
Distributing Population by Population
Each Generation have n populations to be evaluated 1 population ( 1 set of parameters ) will be distributed
to 1 slave. If n is more than slave, more than 1 population will be distributed to 1 slave.
Slaves do the evaluation, generate fitness and send back the population (with fitness) to Master.
SLAVE n n=popsize
Population n./SWAP
MASTER
SWAP-GA 1PIXEL
SLAVE 1
Population 1./SWAP
. . . . . . .
Master
Slave
Distribute Population Without Fitness
Read Input FilesRun SWAPAnalyze ResultPopulation With Fitness
Next Generation
All pixels will be distributed among the available slaves. Each slave will evaluate total serial SWAP-GA procedure
inside itself for one pixel at a time and produce a total assimilation result for that pixel in a file in their local Hard Disk
Distributing Pixel by Pixel
SLAVE n
n=101 PIXEL
SWAP-GA
MASTER
SLAVE 1
1 PIXELSWAP-GA
Nodes
Master
Slave
Distribute Pixel
All Populations Without FitnessRead Input FilesRun SWAPAnalyze ResultAll Populations With Fitness
Assimilation result
Next Generation
Slave 11 PIXELSWAP-
GA
Slave n1 PIXELSWAP-
GA
. . .
Slave n n=popsize
Population n./SWAP
Local Master1 PIXEL
SWAP-GA
Slave 1 n=popsize
Population n./SWAP
. . .
31
0
2
75
4
6
21
0
54
3
6 Available Nodes
8 Available Nodes
Hybrid (Population by Population and Pixel by Pixel Together)
Here Node 0,3 are Master
Node 1,2,4,5 are Slaves
Here Node 0,4 are Master
Node 1,2,3,5,6,7 are Slaves
Pixel1 Pixel2
pop1 pop2 pop1 pop2
Master
Hardware Architecture Influence on Speed up :
( Pop & Pop: 1 SWAP on 1 Slave )
Distribution Population Method Copy all necessary (inputted) files (CF)=336KB, Read files (RF)=.0067KB,
Run SWAP executable file (RS)=32KB , Write files (WF)=1.46KB The parameters inside parallel SWAP-GA are fixed in all Clusters (No of
population=2, No of Generation=1, PXOVER=0.8, PMUTATION=0.25)
The Bandwidth Times for Copying Files and Executing SWAP
According to above Table it is clear that Hardware Architecture Influence on Speed Up.
Copy File Run 1 SWAP
Software Architecture Influence on Speed up:
( Total Throughput / n-SWAP : sec/SWAP)
Average SWAP Execution Time (in second) on Different Clusters
Here, a comparison between the running time (in second) of a single SWAP execution (Total Model running time / Total No. of SWAP execution) on these Clusters (by guessing that the I/O factors and GA running factors are taking constant time in all cases).
Parameters in Optima, Maeka and Magi:-No of pixel (9), No of Population (10), No of generation (20).
-9 processors are used in Optima and Maeka but due to reason that Magi is a 4 nodded Cluster so, 4 processors are used in Magi.
Hybrid model is given best performance (time domain) among the three predefined models
Compare Serial SWAP_GA with Clustering SWAP-GA
0
200
400
600
800
1000
2 3 4 5 6 7 8No of Population
Tim
e
25 Generation_Serial
25 Generation_Cluster
0
100
200
300
400
500
600
1 5 10 15 20 25No of Generation
Tim
e
Serial Timming
1 pixel
3 pixel
5 pixel
8 pixel
Figure: Timing Diagram Serial SWAP-GA and Parallel SWAP-GA by Distributed Pixel in Optima
Figure: Timing Diagram Serial SWAP-GA and Parallel SWAP-GA by Distributing Population in Optima
Pop by Pop modeln Slaves = n Pop.
Serial
Cluster
Pix by Pix modeln pixel = n slave
Scalability for Pop by Pop model ( Optima: n Slave = n Pop, 1 pix )
The symbols, Pi is the population number, T is the time in population piand generation gj. Agj and Cgj are find out with respect to the value of gj and used as constants to find out the value of T.
For i belongs to population, j belongs to generation
T =Agj Pi + Cgj (sec)
(Serial SWAP-GA)
Agj= 11.16 gj – 6.94 Cgj =-1.385.gj + 3.433
(Parallel SWAP-GA)
Agj= 0.865gj + 1.22 Cgj =5.57 gj – 1.93
However, within this experiment highest 25 generations and 8 populations are used but all the above equations have given a generic model. For any value of generation and population number the running time in both parallel SWAP-GA and serial SWAP-GA can be measured by using these equations. The above equations can solve the complication of running parallel SWAP-GA within Clusters those hold limited number of slaves (PC).
gj Agj Cgj Pi T(sec)Serial 5 48.86 -3.49 8 387
Parallel 5 5.55 25.92 8 70
Conclusion & Future Work
3 Implementation Models were proposed and evaluated on 3 clusters Hybrid model always gives the fastest result A Model to estimate calculation time on Optima was established. Models in different model will be developed.
Improve Time MeasurementImprove Time Measurement Implementation for bigger clusters ( new cluster in Kasetsrat, Titech )Implementation for bigger clusters ( new cluster in Kasetsrat, Titech ) A monitoring System ( CPU load )A monitoring System ( CPU load ) Finding idling CPU to send jobs : schedulerFinding idling CPU to send jobs : scheduler Implementation to Thai GRID.Implementation to Thai GRID. A fully SWAP-GA software moduleA fully SWAP-GA software module Package, documentation, scripting.
We believe that this research will include one new part not only in the Grid-Cluster computing arena but also in the Remote Sensing image analyzing field.