quality analysis in land cover change studies

®

QUAlity aware VIsualisation for the Global Earth QUAlity aware VIsualisation for the Global Earth Observation system of systemsObservation system of systems

Kick off meeting. February 17th, 2011

Quality analysis in Quality analysis in land cover change land cover change

studiesstudies

February, the 17th, 2011 Barcelona

Joan PinoCREAF


2

5 0 25 km

N

19562000

Aigües continentals

Basses urbanes

Boscos clars (no de ribera)

Boscos de ribera

Boscos densos (no de ribera)

Canals i basses i agrícoles

Conreus

Glaceres i congestes

Matollars

Plantacions de plàtans

Plantacions de pollancres

Platges

Prats i herbassars

Reforestacions recents

Roquissars

Sòls nus forestals

Sòls nus urbans

Tarteres

Vegetació d'aiguamolls

Vies de comunicació

Zones d'extracció minera

Zones esportives i lúdiques

Zones recent cremades

Zones urbanitzades

A case study: the Barcelona RegionA case study: the Barcelona Region


3

113253 ha(35%)

152815 ha(47.2%)

12068 ha(3.7%)

127072 ha(39.3%)

67018 ha(20.7%)

56797 ha(17.6%)

35814 ha(11.1%)

62203 ha(19.2%)

112%

174%

44%

471%

Increase 2000/1956

Changes in four land cover categoriesChanges in four land cover categories

1956 2000

Forest

Cropland

Urban

Scrubland


4

78.5 %

1.3 %

15.6 %

4.6 %

Change in forestsChange in forests

1956 2000

Forest

Cropland

Urban

Scrubland


5

36.4 %

3.1 %

51.6 %

8.8 %

Change in scrublandChange in scrubland

1956 2000

Forest

Cropland

Urban

Scrubland


6

17.5 %

43.8 %

17.4 %

21.4 %

1956 2000

Forest

Cropland

Urban

Scrubland

Change in croplandChange in cropland


7

2.5 %

2.3 %

3.5%

91.7 %

1956 2000

Forest

Cropland

Urban

Scrubland

Change in urbanChange in urban


8

But….how (un)certain are these changes?

A first approach to the problem in classified satellite images


9

Key points by Serra et al (2003)

• A significant proportion of boundary errors are expected when change detection from remote sensing data is often done by simple overlay of classified maps.

• A specific post-classification is proposed that considers the overall accuracy of the overlay (as the product of the acuracies of the overlayed classifications)

• A method is proposed to increase accuracy, by Eroding the boundaries of the polygons to avoid comparing areas with locational inaccuracy

Resampling the two layers accounting for the different pixel size and grid origin


10

What about photo-interpreted maps?


11

Some definitions

• Accuracy: the deviation around the true population mean. The standard deviation can be taken for accuracy estimation under the assumption of infinite population.

• Bias: The difference between the estimated mean and true mean of the population.

• Precision: depends on the deviation and the number of samples, obtained by a repeated sampling procedure. The standard error, sampling error or the confidence interval can be taken to quantify the precision.


12

Estimating spatial accuracy

Validation of polygon borderlines:

•A distance threshold error has to be defined first.

•One interpretation with borderlines has to be selected as accurate and all other interpretations are compared with this reference data.

•The accurate and validation lines are buffered with the acceptable distance error.

•With overlay a 2x2 cross table can be produced containing the area proportion or number of pixel for the individual combination.

Ground truth (reality)

Borderline present Borderline not present

Data to be validatedBorderline present P11 p12

Borderline not present p21 p22

pppp

pppp

22211211

12212211

Accuracy (pearson correlation)


13

Estimating thematic accuracy

A data source has to be defined as reference data (ground truth)

Changes between the reference data and the validation data are summarised in a confusion matrix, from which overall, omission and commission errors can be estimated.

Reference data

Validation data 112 121 124 211 243 312 313 Total validation

112 10 1 3 14

121 1 1

124 3 1 4

211 2 1 1 4

243 1 1

312 2 1 1 13 1 18

313 1 1 2

Total reference 14 1 4 4 1 18 2 Total samples:44


14

A first experience (2003-2006)


15

Aim: Detecting changes in Natura 2000 areas (1950’s- 2000’s)

75 Windows: 30 x 30 km (black)

59 Transects: 2 x 15 km (red)

Focussing on 4 Annex-I habitats which are found in main bio-geographical regions:

(i) Freshwater habitats,

(ii) Natural and semi-natural grassland formations,

(iii) Raised bogs and mires and fens and (iv) Forests.

Stratification:Biogeographical Regions Map of Europe (BRME)


16

Transects

CORINE land cover legend

1950 1990


17

Quality assessment of photo-interpretation: Transect re-interpretation

Points to be re-interpreted by a set of teams

Selected across a 500-m grid

Interpretation of:

•Land cover category (common manual)•Distance to the category border

Compared with reference data (local photo-interpreter)


18

Thematic accuracy

Differences between the local interpretation and the validation

Validation Data (CLC-Classes)

112 121 124 211 243 312 313 Total Truth Consistency N Consistency %

112 10 1 3 14 10 71%

121 1 1 1 100%

124 3 1 4 3 75%

211 2 1 1 4 1 25%

243 1 1 1 100%

312 2 1 1 13 1 18 13 72%

313 1 1 2 1 50%

Total Validatio

n14 1 4 4 1 18 2 44 30 68%

Ref

eren

ce D

ata

(C

LC

-Cla

sse

s)

tcp̂ = 100**

1

Lx

x

vj

L

iji

Total class consistency in percent for all interpreters, with:

L = number of validationsXij=consistent observation between observersXvj=observations of local interpreter in class j


19

Total achieved accuracy for each class

The percentage values are referring to the mean number of observations by the local interpreters.

112 121 124 211 243 312 313

112 67% 0% 1% 4% 0% 26% 2%

121 0% 100% 0% 0% 0% 0% 0%

124 4% 0% 75% 4% 0% 8% 8%

211 38% 0% 8% 29% 0% 25% 0%

243 0% 0% 0% 0% 100% 0% 0%

312 15% 0% 3% 11% 0% 69% 2%

313 17% 0% 0% 8% 0% 25% 50%


20

Spatial accuracy

Distance between the validation point and the next borderline to another class, compared with that of the reference


21

Example of distance measurements for 7 interpreters (A to G)

ID A B C D E F G Mean Total SDV Error

DE137_1 0 -4 -3 4 42 -4 -3 5 18,22 7,44

DE137_2 0 -2 6 1 1 16 1 4 6,49 2,65

DE137_3 0 87 -6 -2 83 85 3 42 47,57 19,42

DE137_4 0 4 9 5 4 3 4 5 2,14 0,87

DE137_5 0 -5 0 -1 0 -1 0 -1 1,94 0,79

DE137_6 0 424 -1 -2 425 426 4 213 232,61 94,96

DE137_7 0 5 10 21 6 5 3 8 6,62 2,70

Relative distances with their mean, standard deviation and error

Reference interpreter Quality of the individual interpreters can be measured from mean/median values with SD


22

Quality of the individual interpreter

ID A B C D E F G

DE137_1 0 -4 -3 4 42 -4 -3

DE137_2 0 -2 6 1 1 16 1

DE137_3 0 87 -6 -2 83 85 3

DE137_4 0 4 9 5 4 3 4

DE137_5 0 -5 0 -1 0 -1 0

DE137_6 0 424 -1 -2 425 426 4

DE137_7 0 5 10 21 6 5 3

interpreter mean 73 2 4 80 76 2

interpreter median 4 0 1 6 5 3

interpreter error 60 2 3 59 60 1

(figures in m)


23

Geometric accuracy for one transect

Mean Total 39

Stadv Total 111

Number 42

Error 17,07

RMSE 41

Per LC class

Overall

•Id LCCD B C D E F G Mean Stadv RMSE

DE137_4 112 4 9 5 4 3 4 2 4 3

DE137_5 112 -5 0 -1 0 -1 0

DE137_7 121 5 10 21 6 5 3 25 37 26

DE137_3 121 87 -6 -2 83 85 3

DE137_6 311 424 -1 -2 425 426 4 213 213 213

DE137_1 313 -4 -3 4 42 -4 -3 5 13 6

DE137_2 313 -2 6 1 1 16 1


24

A first step

®

QUAlity aware VIsualisation for the Global Earth QUAlity aware VIsualisation for the Global Earth Observation system of systemsObservation system of systems


Thanks !

quality analysis in land cover change studies

Documents