validation of remote sensing classifications: a case of balans classification markus törmä

VALIDATION OF REMOTE SENSING CLASSIFICATIONS:

a case of Balans classification

Markus Törmä

STRUCTURE OF THESIS

1. Introduction

2. Methods for validation

3. Case: Validation of Balans classification

4. Success of validation

5. Conclusions

BALANS CLASSIFICATION

• The goal of the project was to develop a streamlined production line for creation of land cover information using medium resolution satellite data

• The production of land cover classification covering Baltic Sea drainage basin was tested


Organisations:

• Metria Miljöanalys

• Finnish Environment Institute

• Novosat Oy

• GRID-Arendal

• GRID-Warsaw

• Swedish Meteorological and Hydrological Institute.


Classes:• Artificial surfaces• Agricultural areas• Coniferous forest• Deciduous forest• Bare rock• Glaciers and perpetual snow• Other seminatural areas• Wetlands• Water bodies


IRS WiFS:

• Channels: RED and NIR

• Spatial resolution: 188m


Reference datasets:• Finnish National Land Use and Forest Classification

• Swedish Terrain Type Classification

• CORINE Land Cover classification

• FIRS (Forest Information from Remote Sensing project) regions and stratas

• Agricultural areas from Baltic Sea Region GIS, Maps and Statistical Database

• Baltic Sea Drainage Basin Watershed Areas from Baltic Sea Region GIS, Maps and Statistical Database

• World Atlas of Agriculture

• Digital Elevation Model (DEM), slope and wetness index


Classification:• Unsupervised classification of individual WiFS-

images• Interpretation of clusters using reference data• Second unsupervised classification to unclassified

areas• Some classes like artificial surfaces and

agricultural areas are taken directly from reference data if possible

VALIDATION

Geometric validation:

• The accuracy of a geometrically rectified image

Thematic validation:

• The correspondence between the class label assigned to a image pixel and its true class on the field

GEOMETRIC VALIDATION

• Validation points from 12 test areas

• Residual = reference crd - classification crd

• Water areas as reference data:

• Finland: Digital map 1:250 000, 4 areas

• Poland: Topographic map 1:100 000, 6 areas

• Sweden: Topographic map 1:50 000 and 1:100 000, 2 areas

GEOMETRIC VALIDATION• Water areas from classification result and reference data

were compared using INREC-software (VTT)

GEOMETRIC VALIDATION• Areal distribution of geometric control points


• Residual plots of Fuinnish, Polish and Swedish test areas


• Geometrically Balans classification was successful

Finland Poland Sweden ALL

Emean 55.3 53.3 63.4 56.2

Nmean -2.0 6.6 -22.0 -2.9

ERMSE 79.2 116.9 98.5 95.5

NRMSE 51.3 79.2 44.4 59.3

PlanimetricRMSE

94.4 141.2 108.0 112.6

THEMATIC VALIDATION

• In order to get some idea about the mixing between classes and accuracy in the different parts of database

• Check points using reference material • Classes of check points in reference material and

classification result were compared• Number of check points per class was indended to

be about 100 per country, but was as low as 20 in some case

THEMATIC VALIDATION

Reference material• Finland

Digital map 1:250000

Base map 1:20000

3 Landsat-7 ETM-images

• PolandCorine Land Cover classification

Topographic map 1:100 000

• SwedenTopographic map 1:50 000

Topographic map 1:100 000

2 Landsat-5 TM-images

THEMATIC VALIDATIONSampling• Aim: 100 or more reference points per class per

country

• Finland and Sweden:

-Visual interpretation of Landsat image or map

-It was tried to place reference point to an area with size more than five hectares in order to eliminate the possibity of border and mixed pixels and minimize the effect of geometric errors

THEMATIC VALIDATIONSampling• Poland:

-GRID Warsaw

-Center points of areas larger than 100 hectares from Corine landcover

-Random sampling for classes artificial surfaces, agricultural areas, deciduous and coniferous forests

-Classes water and wetlands were augmented using points taken from topographic map

-No reference points for class seminatural areas

THEMATIC VALIDATION

Different class-combinations were studied:

• A: Original classes

• B: Coniferous and deciduous forest classes are combined to one forest class

• C: Agricultural and seminatural areas are combined to one class

• D: One forest class and agricultural and seminatural areas are combined

THEMATIC VALIDATION

• Following information was computed from sampling points:

• Statistics of reference and classified data

• Error matrix

• Overall accuracy

• Interpretation accuracies of classes and their mean value

• Target accuracies of classes and their mean value

• Average accuracies of classes and their mean value

• Mapping accuracy of classes and their mean value

• Kappa coefficient of classification

• Tau coefficient of classification using equal a’priori probabilities for classes

• 95% confidence intervals for overall, interpretation, target and average accuracies

THEMATIC VALIDATION• Thematic accuracy of Balans classification is not that

good• Overall accuracy of original classes is 69.7% when

wetlands are not included and 65.4% with wetlands• Water is classified very well • Artificial surfaces reasonably well• Otherwise from average (agricultural land) to very poor

(seminatural areas)• The most problematic classes coniferous forest and

seminatural areas

SUCCESS OF VALIDATION• Geometric validation was successful and provides

unbiased view to geometric errors of Balans classification

• Validation was based on check points measured from independent material (digital maps and Landsat image)

• Number of check points was rather large, 294• There were 12 test areas in Finland, Poland and Sweden

so check points cover Balans classification area quite well

• The effect of transformations between different coordinate systems was not evaluated

SUCCESS OF VALIDATION• Thematic validation was unsuccessful• Sampling design was driven by the availability of

free reference data and lack of time• Effects of mixed pixels and geometric

transformations were tried to minimize

-visual interpretation from middle of homogeneous areas

• Due to the sampling process the performed sampling is not probability sampling, more like convenience sampling.

SUCCESS OF VALIDATION

• Finlandpoints for deciduous forest and wetlands unreliable due to reference data

• Polandpoints were based on CORINE which was used in classification

points for seminatural areas were missing

• Swedentwo small test areas

SUCCESS OF VALIDATION

Lessons:

• Aim of thematic validation should be stated clearly

• Sampling should be based on probability sampling

• Large pixel, there is need for the validation methods for soft classifications

VALIDATION OF OTHER CLASSIFICATIONS

• Should classification and thematic validation use hard or soft methods?

• Simple, economic, unbiased and spatially comprehensive sampling method


Spatial resolution vs. size of land cover polygons• Determine optimal spatial resolution (SR) for scene

If SR of instrument smaller than optimum SR, use hard classification and validation

If SR of instrument larger than optimum SR, use fuzzy classification and validation

• Decide acceptable proportion of mixed pixels, i.e. pixels containing more than one land cover type

If the average size of land cover polygons is known then the required spatial resolution of remote sensing instrument can be estimated.


• Simulated effect of the size of land cover polygon to the proportion of mixed pixels

• Solid line: small change causes pixel to be labeled as mixed

• Dashed line: some amount of change (10%) in pixel values is tolerated


• Sampling should be random, cover studied area and all classes

• Stratified random sampling

• If not cost-efficient, include clustering

• Sampling points from ground visits, aerial photographs, satellite images or existing maps

• More than one preson should interprete the sampling points

CONCLUSIONS• Geometric accuracy of Balans classification

was good

• Geometric validation method OK

• Thematic accuracy of Balans classification poor

• Due to sampling, it is dangerous to generalize thematic validation outside validation points

• Thematic accuracy most likely optimistically biased

validation of remote sensing classifications: a case of balans classification markus törmä

Documents