Classification Accuracy Assessment
Dr. Sao Hone Pha
Director and Head
Remote Sensing and GIS Research Center
Yangon Technological University
Date: 15th, October 2018
Certificate Training Course on Basic Concept of Remote Sensing & GIS (1.11.2018) to (30.11.2018)
2
Accuracy Assessment
Accuracy
The degree (often expressed as a percentage) of correspondence between observation and reality.
We usually judge accuracy against existing maps, large scale aerial photos, or field checks
(information used as reference, considered “truth”).
Comparison to two sources of information
Remote Sensing derived classification map
Reference Test information
The relationship between the two sets of information's is expressed as a matrix known as
ERROR MATRIX / CONFUSION MATRIX / CONTINGENCY TABLE
3
Classification Error Matrix
Error matrices compare, on a category-by-category basis, the relationship between known
reference data (ground truth) and the corresponding results of an automated classification.
Square matrix with the number of rows and columns equal to the number of categories
whose classification accuracy is being assessed.
Error matrix stems from classifying the sampled training set pixels and listing the known cover
types for training (columns) versus the pixel actually classified into each land cover category
by the classifier (rows).
4
Several Characteristics about Classification Performance
Major diagonal of the matrix The training set pixels that are classified into the proper
and cover categories
All nondiagonal elements Errors of Omission of the matrix (or)Errors of Commission
5
Errors
Omission Error (related to “producer accuracy”)
Error of exclusion
Pixels are not assigned to its appropriate class, or
Pixels are omitted from the actual class they belong
Correspond to non diagonal column element
Commission Error (related to “user accuracy”)
error of inclusion
pixel is assigned to a class to which it does not belong,
(or) pixels were improperly included in that category
represented by non diagonal row element
6
ERROR of Omission Vs. Commission
(From a Class1 Perspective)
Commission error: pixels in Class2 Omission error: pixels in Class1
erroneously assigned to Class 1 erroneously assigned to Class 2
Class 1 Class 2
Number of
pixels
0 Digital Number 255
7
ERROR of Omission Vs. Commission
(From a Class2 Perspective)
Omission error: pixels in Commission error: pixels in
Class2 erroneously assigned to Class1 erroneously assigned to
Class 1 Class 2
Class 1 Class 2
Number of
pixels
0 Digital Number 255
8
Overall Accuracy =
The total number of correctly classified pixels
The total number of reference pixels
Accuracy of individual category =
The number of correctly classified pixel in each category
The total number of pixels in the corresponding row or column
9
Producer’s accuracy = (1-Omission error)
Omission error of a class = 100 – (% Accuracy of a class)
Producer Accuracy indicates how well training set pixels of the given cover types are classified
The number of correctly classified pixels in each category (major diagonal)
The number of training set pixels used for that category (column total)
User Accuracy indicates the probability that a pixel classified into a given category actually represented
that category on the ground
The number of correctly classified pixels in each category (major diagonal)
The total number of pixels that were classified in that category (row total)
User’s Accuracy = (1-Commission Error)
Commission error of a class = 100 – (% Accuracy of a class)
10
So, from this assessment we have three measures of accuracy which address subtly different
issues:
Overall accuracy :
takes no account of source of error (errors of omission or commission)
User accuracy :
measures the proportion of each TM class which is correct.
Producer accuracy
measures the proportion of the land base which is correctly classified.
11
∧
Kappa Coefficient ( 𝒌𝒌 or “KHAT”)
K statistics is a measure of the difference between the actual agreement between the
reference data and an automated classifier and the chance agreement between the reference
data and a random classifier.
∧
Conceptually 𝑘𝑘 can be defined as
Observed accuracy - chance agreement
k =
ˆ
1 - chance agreement
12
∧
Kappa Coefficient ( 𝒌𝒌 or “KHAT”)
Observed accuracy - chance agreement
k =
ˆ
1 - chance agreement
The statistic serves as an indicator of the extent to which the percentage correct values of an
error matrix are due to the “true” agreement versus “chance” agreement
A true agreement (observed accuracy) approaches 1 and chance agreement approaches 0
then Kappa approaches 1
A kappa of 0 suggest that the classification is very poor and the given classifier is no better
than a random assignment of pixels
13
∑x
i =1
ii
Observed Accuracy =
N
r
1 ∑ (x i+ ⋅ x+ i )
Change Agreement = i =1
N N
Where
r = number of rows in the error matrix
xii = number of observations in row i and column i (on the major diagonal)
xi+ = total of observations in row i (shown as marginal total to right of the matrix)
x+i = total of observations in column i (shown as marginal total at bottom of the matrix)
N = total number of observations included in matrix
14
The KHAT statistic is computed as
Observed accuracy - chance agreement
k =
ˆ
1 - chance agreement
r r
∑x ii
1 ∑ (x i+ ⋅ x+ i )
i =1
- i =1
N N N
k̂ = r
1 ∑ (x i+ ⋅ x+ i )
1 - i =1
N N
r r
N ∑ xii − ∑ ( xi + ⋅ x+ i )
kˆ = i =1
r
i =1
N − ∑ ( xi + ⋅ x+ i )
2
i =1
15
• Note : Overall Accuracy is biased on the correctly classified pixels (the major
diagonal) and excludes the omission and commission errors
• Kappa in addition to the correctly classified pixels, includes the off diagonal elements
as a product of Row total and column total
16
CLASSIFIED IMAGE REFERENCE DATA
From airphotos, field data, etc.
1 Generate a random sample of points
2 For each point, compare the map class with the “true” class
3 Construct and analyze an error matrix...
17
Error Matrix Resulting from Classifying Random ly Sam pled Test
Reference Data
Classification Data Water Sand Forest Urban Corn Hay Row total
Water 226 0 0 12 0 1 239
Sand 0 216 0 92 1 0 309
Forest 3 0 360 228 3 5 599
Urban 2 108 2 397 8 4 521
Corn 1 4 48 132 190 78 453
Hay 1 0 19 84 36 219 359
Column total 233 328 429 945 238 307 2480
Producer’s accuracy User’s accuracy
W = 226/233 = 97% U = 397/945 = 42% W =226/239= 94% U = 397/521 = 76%
S = 216/328 = 66% C= 190/238= 80% S = 216/309 = 70% C = 190/453 = 42%
F = 360/429 = 84% H = 219/307 =71% F = 360/599 =60% H = 219/359 = 61%
Overall accuracy = (226+ 216+ 360 + 397+ 190+219)/2480= 65%
18
Error Matrix Resulting from Classifying Random ly Sam pled Test
r r
N ∑ xii − ∑ ( xi + ⋅ x+ i )
kˆ = i =1
r
i =1
N 2 − ∑ ( xi + ⋅ x+ i )
i =1
∑x
i =1
ii = 226 + 216 + 360 + 397 + 190 + 219 = 1608
∑ (x
i =1
i+ ⋅ x+ i ) = (239 × 233) + (309 × 328) + (599 × 429)
+ (521× 945) + (453 × 238) + (359 × 307) =
1,124, 382
(2480 × 1608) − 1,124, 382
kˆ = 0.57
(2480) 2 − 1,124, 382
19
How accurate should the classified map be?
General rule of thumb is 85% accuracy
What’s a good Kappa?
General range
K < 0.4: poor
0.4 < K < 0.75: good
K > 0.75: excellent
