land cover game lps16.ppt
TRANSCRIPT
A Crowdsourcing-based game for land cover validation
Brovelli, Maria Antonia(1); Celino, Irene (2); Molinari, Monia (1); Venkatachalam, Vijay Charan (1)
(1) Politecnico di Milano, (2) CEFRIEL
bit.ly/foss4game
Problem Setting
● When different products are compared there are lots of disagreement● Which product is correct ?● Disagreement overall
Target
● To evaluate the classification quality of GlobeLand30 on the Italian area this dataset has been compared with more accurate Italian land cover maps.
● To develop an interactive WebGIS with gaming environment for the crowdsourcing-based validation of the non-coherent pixels.
DataSet
DUSAF(Destinazione d’Uso dei Suoli Agricoli e Forestali )● Scale 1:10'000 ● Updated through aerial photo interpretation.● Currently available in five releaseshttp://www.ersaf.lombardia.it/servizi/Menu/dinamica.aspx?idArea=16914&idCat=17254&ID=17254
Global Land30● 30 meters resolution ● Landsat (TM and ETM+) and HJ-1 satellites images .● pixel-object-knowledge-based (POK-based)
http://globallandcover.com/GLC30Download/index.aspx
Citizen Science● Public involvement
▪ To produce scientific data▪ To conduct experiments
Human Computation
● Human Computation Systems aim to solve the problems that are trivial for humans, but very challenging for computer programs.
● There are different types of Human computation systems▪ Incentive by money - Amazon Mechanical Turk ▪ Incentive by fun - Games with a Purpose▪ You have no other choices - CAPTCHA/reCAPTCHA
Land Cover Validation Game - Architecture
Land Cover Validation Game
Client ( JavaScript- Angular-Leaflet directives)
Web Portal
(PHP, Apache webserver with Linux)
Authentication
MySQL DatabaseQGIS
DUSAF
GL30
Disagreement
map
Geoserver
(CGR orthophotos)
Accuracy Assessment - Milano
Coherent
pixels
Non-coherent
pixels
Overall Accuracy
[%]
Disagreement
[%]
2000 173219 28758 85.76 14.24
2010 174604 27240 86.50 13.50
Ground Truth DUSAF 1.1 1 2 3 4 5
GL30 (2000)
1 134025 4020 989 26 267
2 11950 38139 4026 31 601
3 4677 2076 456 0 49
4 0 0 0 0 0
5 30 3 13 0 599
Ground Truth DUSAF 4.1 1 2 3 4 5
GL30 (2010)
1 138952 3199 557 0 441
2 12597 34425 2508 0 340
3 6799 635 371 0 37
4 0 0 0 0 0
5 89 26 12 0 856
CONTINGENCY TABLE
Coherent
pixels
Non-coherent
pixels
Overall Accuracy
[%]
Disagreement
[%]
Sample
Size
2000 33416 8022 80.60 19.40 1340
2010 33318 8120 80.50 19.60 1208
CONTINGENCY TABLE Accuracy Assessment - Como
Ground Truth DUSAF 4.1
1 2 3 4 5
GL30 (2010)
1 16629 1340 2626 0 1312 917 1644 872 29 93 1177 835 12824 79 464 0 0 0 0 05 57 0 2 0 2221
LAND COVER VALIDATION GAME
Evaluation of Game
(Game ability to engage Players)
(Game ability to find an agreement on pixels’ classification)
User Analysis
● Reputation of the players
Pixel Analysis
Complicated Pixels
• The land cover validation game is a novel interactive system that allows citizen
scientists to validate land cover maps while having fun.
• The Confidence level of the GlobeLand30 and DUSAF dataset is obtained.
• The More the player , the higher the number of validated pixels.
Conclusion
