How to describe AccuracyAnd why does it matter

Jon Proctor, PhotoTopoGIS In The Rockies: October 10, 2013

introduction• Accuracy.• We think we understand it• But, there are more details than you think… • This discussion will help you understand what is

needed to describe accuracy• And Why it matters…

Horizontal Accuracy

• Horizontal accuracy shall be tested by comparing the planimetric coordinates of well-defined points in the dataset with coordinates of the same points from an independent source of higher accuracy. (NSSDA)

• http://www.fgdc.gov/standards/projects/FGDC-standards-projects/accuracy/part3/chapter3

What to measure• Measure and record the x and y

coordinate of the feature in the product.• Ensure that the coordinates are in the

same projection as the control

Product

Control Chip

GCP (control) coordinateObserved location

Accuracy Spreadsheet• For each point

• record the control ID, the observed (product) x and y coordinate• List the control x and y coordinate• Calculate the difference (observed – control) for x and y• Plot errors• Calculate radius

Control ID x y x y x y rPIN001 403646.84 3726715.94 403645.75 3726711.63 1.09 4.31 4.45PIN002 399591.66 3708968.52 399589.84 3708971.60 1.82 -3.08 3.57PIN003 416623.59 3702996.24 416619.04 3703001.39 4.55 -5.15 6.87PIN004 418475.52 3732674.29 418481.07 3732668.95 -5.55 5.34 7.70PIN005 426526.34 3708595.74 426523.21 3708596.75 3.13 -1.01 3.29PIN006 418639.30 3700463.95 418641.36 3700465.86 -2.06 -1.91 2.81PIN007 401694.89 3706717.06 401692.26 3706713.86 2.63 3.20 4.15PIN008 404411.45 3708702.01 404411.52 3708706.95 -0.07 -4.94 4.94PIN009 401772.10 3698027.51 401772.45 3698029.60 -0.35 -2.09 2.12PIN010 411800.33 3714986.37 411799.25 3714985.78 1.08 0.59 1.23PIN011 409516.42 3687990.49 409515.25 3687985.46 1.17 5.03 5.16PIN012 410440.31 3683672.29 410447.94 3683666.38 -7.63 5.91 9.65PIN013 404769.14 3699308.82 404773.45 3699306.92 -4.31 1.90 4.71PIN014 403481.50 3715698.82 403476.33 3715696.69 5.17 2.13 5.59PIN015 403883.45 3706603.99 403882.24 3706607.80 1.21 -3.81 4.00PIN016 420934.28 3722887.13 420931.48 3722883.95 2.80 3.18 4.23PIN017 415720.34 3688609.70 415720.73 3688612.26 -0.39 -2.56 2.59PIN018 429095.38 3718136.62 429101.03 3718139.58 -5.65 -2.96 6.38PIN019 415910.20 3705249.24 415910.33 3705249.24 -0.13 0.00 0.13PIN020 424121.51 3723034.15 424123.85 3723035.09 -2.34 -0.94 2.52

ControlObserved Difference

-10.00

-8.00

-6.00

-4.00

-2.00

0.00

2.00

4.00

6.00

8.00

10.00

-10.00 -8.00 -6.00 -4.00 -2.00 0.00 2.00 4.00 6.00 8.00 10.00

Scatter Plot of product error

Is this the accuracy?• So far, we only now the errors of the points that were

measured• We don’t have the accuracy for the product• How do we describe the accuracy?

Control ID x y x y x y rPIN001 403646.84 3726715.94 403645.75 3726711.63 1.09 4.31 4.45PIN002 399591.66 3708968.52 399589.84 3708971.60 1.82 -3.08 3.57PIN003 416623.59 3702996.24 416619.04 3703001.39 4.55 -5.15 6.87PIN004 418475.52 3732674.29 418481.07 3732668.95 -5.55 5.34 7.70PIN005 426526.34 3708595.74 426523.21 3708596.75 3.13 -1.01 3.29PIN006 418639.30 3700463.95 418641.36 3700465.86 -2.06 -1.91 2.81PIN007 401694.89 3706717.06 401692.26 3706713.86 2.63 3.20 4.15PIN008 404411.45 3708702.01 404411.52 3708706.95 -0.07 -4.94 4.94PIN009 401772.10 3698027.51 401772.45 3698029.60 -0.35 -2.09 2.12PIN010 411800.33 3714986.37 411799.25 3714985.78 1.08 0.59 1.23PIN011 409516.42 3687990.49 409515.25 3687985.46 1.17 5.03 5.16PIN012 410440.31 3683672.29 410447.94 3683666.38 -7.63 5.91 9.65PIN013 404769.14 3699308.82 404773.45 3699306.92 -4.31 1.90 4.71PIN014 403481.50 3715698.82 403476.33 3715696.69 5.17 2.13 5.59PIN015 403883.45 3706603.99 403882.24 3706607.80 1.21 -3.81 4.00PIN016 420934.28 3722887.13 420931.48 3722883.95 2.80 3.18 4.23PIN017 415720.34 3688609.70 415720.73 3688612.26 -0.39 -2.56 2.59PIN018 429095.38 3718136.62 429101.03 3718139.58 -5.65 -2.96 6.38PIN019 415910.20 3705249.24 415910.33 3705249.24 -0.13 0.00 0.13PIN020 424121.51 3723034.15 424123.85 3723035.09 -2.34 -0.94 2.52

ControlObserved Difference

-10.00

-8.00

-6.00

-4.00

-2.00

0.00

2.00

4.00

6.00

8.00

10.00

-10.00 -8.00 -6.00 -4.00 -2.00 0.00 2.00 4.00 6.00 8.00 10.00

Scatter Plot of product error

What is Positional Accuracy?• Positional accuracy is a statistical measure of a features location.

• We are not saying that the product is off by 4 meters to the East.• Rather we have 4 meters of uncertainty.• And we need to describe that uncertainty.

• It represents the probability that a feature is within a given distance of its true location. • Probability and distance• Not a specific direction, not a specific distance

• This probability, statistical description can be represented in numerous ways such as: • CE90, CE95, RMSE, StDev, and many more…

What is needed to Describe accuracy?

• We need 3 descriptors:• Measure: 4, 8.5, 10• Unit: meter, feet• Statistical description: RMSEr, StDevr, CE90

Definitions• RMSE: Root Mean Square error

• RMS of the absolute error

• StDev: Standard Deviation

• Deviation from the average

• For non-biased datasets (where the average is 0), StDev and RMSE will be the same

RMSE vs StDev• RMSEr

• Measure radial distance from control (0,0) to data point

• StDevr• Measure radial value from

cluster center to data point

RMSE vs StDev• For an Un-biased dataset, where the Center of scatter plot is near 0,0

• Measures are almost exactly the same• RMSE = StDev

(more) Definitions• CE90

• A CE90 of 10.0 meters is an accuracy in which 90% of the well defined, measured image points are statistically expected to be within 10.0 meters from their surveyed locations.

• assumes that the survey or truth point is (much) more accurate than the dataset being sampled.

• CE95• The distance for a 95% probability

• CEP• Circular Error Probable. • Circle of Equal Probability.• The distance for a 50% probability. • Or CE50

Example of 10m CE90

• But what if we forget the statistical descriptor of CE90?

• If the accuracy was only described as 10m, what would the scatter look like?

Why it matters• These plots are all at 10m• The scatter size changes based

on the statistical descriptor• CE99.99• CE99• CE95 • CE90• CE50• RMSEr• RMSExy

• Without the descriptor, we would not know the tolerance for error on the project

Why it matters (2)

• Another way to show why the statistical description matters is shown here.• These are equivalent

values for this error scatter plot

• 20.00 m CE99.99

• 14.14 m CE99

• 12.91 m CE95

• 10.00 m CE90

• 8.27 m CE50

• 6.59 m RMSer

• 4.66 m RMSexy

xy compared to r• When calculating RMSE or StDev, you can measure the

x, y offset or the radial offset, and use these offsets in the equations• RMSEr equals the horizontal radial RMSE, • Use notation that clarifies which value is being

measured, such as:• RMSE(xy), StDev(xy)• RMSE(r), StDev(r)

Or

• RMSExy, StDevxy • RMSEr, StDevr

xy Compared to r offset with biased dataset• Measure xy or r?• Note: radial values are always positive

• can’t have a negative radius

• RMSEr =

• Note: histogram for RMSE is only for the x measures

xy Compared to r offset with unbiased dataset• With an unbiased dataset, the x values are centered on 0• Radial values are all positive

Types of Error• Blunder: A gross error. A careless mistake. An obvious error. Blunders are individual errors that effect each measurement

differently. Blunders that can be identified should be dis-regarded, or removed from the report or solution.• Marking a control point at the wrong corner for an intersection.• Bad image correlation

• Systematic: An error that tends to shift all measurements in a systematic way. If the systematic error is identified, it can be removed from the results, or the project can be re-processed with the corrected information.

• A bold of shadowed line used to depict a feature• A burr at the end of a measuring stick• Sensor calibration error• Incorrect interior orientation• Incorrect re-projection parameters

• Random: An error that shifts measurements in a haphazard on inconsistent manner.• Rounding of significant figures• Coarse DEM postings, not capturing terrain change• Noise in signal processing

To goal is to identify and remove the blunders, and remove systematic errors. As a result, only Random errors are left in the project.

• If we could say the positional accuracy was 4 meter to the east, then we could just shift the data set to the east, and improve the accuracy.

• But in a good project, you will remove the blunders, and bias.• All that is left is random

error

Random error at 10m CE90• Example of Random

error• Observations = • 10• 100• 1,000• 10,000

• Distribution is circular and non-biased

Conclusions• Specify a measure: 4, 8.5, 10• Specify Unit: meter, feet• Use a statistical description

• RMSE• StDev• CE50• CE90• Etc…

• And, be precise• RMSExy vs RMSEr• StDevxy vs StDevr

• Important to know because the conversion factor from Xr to CE90 is different than the conversion factor of Xxy to CE90

Conversion factors• Since these descriptors are

all describing 2-dimensional error offsets, we can convert between the different values.

* CE99.99 CE99.9 CE99 CE95 CE90 CE50 RMSEr RMSExy

CE99.99 1 0.8660 0.7071 0.5703 0.5000 0.2743 0.3295 0.2330ce99.9 1.1547 1 0.8165 0.6585 0.5774 0.3168 0.3805 0.2690ce99 1.4142 1.2247 1 0.8065 0.7071 0.3880 0.4660 0.3295ce95 1.7534 1.5185 1.2399 1 0.8767 0.4810 0.5778 0.4085ce90 2.0000 1.7320 1.4142 1.1406 1 0.5486 0.6590 0.4660ce50 3.6452 3.1569 2.5776 2.0789 1.8227 1 1.2011 0.8493RMSEr 3.0348 2.6282 2.1460 1.7308 1.5175 0.8325 1 0.7071RMSExy 4.2919 3.7169 3.0349 2.4477 2.1460 1.1774 1.4142 1

To

From

Backup Slides

• 3 methods for measuring CE90• Direct rank• XY• Radius

• Show the difference with 5 check points, 10, 20

• Really, we should say “I am 73% confident that 90% of the well defined points are within 10 meters of their true location”• Where my confidence level is based on the sample size, and

variance measured from the population

• DRAFT ASPRS accuracy standards• Tested __ (meters, feet) Non-vegetated Vertical Accuracy

(NVA) at 95 percent confidence level in all open and non-vegetated land cover categories combined using RMSEz x 1.96,” and• “Tested __ (meters, feet) Vegetated Vertical Accuracy

(VVA) at the 95th percentile in all vegetated land cover categories combined using the absolute value 95th percentile error.”

Variation in estimated CE90

• These plots show variation in the estimated CE90 based on 3 different methods and a sample size of 20

• Estimated CE90 ranges from 6.5 and 12 m CE90 in these 5 samples