Learning Objectives
• Continuing discussion on KNN
• Performance metrics for Classification
• Significance of different metrics
1
kNN: Classification
Effect of Outliers:
● Consider k=1.
● Sensitive to outliers: Decision boundary
changes drastically with outliers.
● Solution?
○ Increase k
3
kNN: Classification
Effect of k:
● Low k: overfitting, highly
unstable decision boundary
● Good k: Smooth boundary, no
overfitting/underfitting
● Higher k: Everything classified
as most probable class
● How to find a good k?
k=1 k=15
4
kNN: Classification
Effect of k:
● Low k: overfitting, highly
unstable decision boundary
● Good k: Smooth boundary, no
overfitting/underfitting
● Higher k: Everything classified
as most probable class
● How to find a good k?
k=1 k=15
Cross validation is our friend!
5
kNN: Classification
What if we have same votes from
both classes?
Potential solutions for tie-
breaking:
● Take k odd
● Randomly select
● Use the class with larger prior k=1 k=15
6
kNN: Classification
A probabilistic variant: Probabilistic kNN
E.g. k=4, c=3
P=[3/4, 0, 1/4]
y=1 y=2 y=3
8
kNN: Classification
A probabilistic variant: Probabilistic kNN
E.g. k=4, c=3 Variant with pseudo counts:
P=[(3+1)/(4+3), (0+1)/(4+3),(1+1)/(4+3)]
=[4/7, 1/7, 2/7]
y=1 y=2 y=3
9
kNN: Regression
A simple regression algorithm:
● Train examples: where is a continuous real valued target
● Given test input
● Find distances to the n training examples using a distance metric
● Select k closest training examples and their target values
● The output is the mean of the target values of the k neighbours
Can be used for interpolation.
12
kNN: Challenges
Computationally expensive:
● Need to store all training example
● Required to compute distances to all training examples:
There are ways to optimize kNN computation:
● Reduce dimensionality using dimensionality reduction techniques
● Reduce number of comparisons:
○ kD tree implementation
○ Locality sensitive hashing
14
kNN: Computational Complexity
Brute force method
● Training time complexity: O(1)
● Training space complexity: O(1)
● Prediction time complexity: O(k * n * d)
● Prediction space complexity: O(1)
15
kNN: Computational Complexity
k-d tree method
● Training time complexity: O(d * n * log(n))
● Training space complexity: O(d * n)
● Prediction time complexity: O(k * log(n))
● Prediction space complexity: O(1)
16
kNN: k-d Tree
● K Dimensional tree (or k-d tree) is a tree data structure that is used to represent
points in a k-dimensional space.
● Used for various applications like nearest point (in k-dimensional space), efficient
storage of spatial data, range search etc.
18
kNN: k-d Tree
Example:
19
kNN: k-d Tree
Example:
20
kNN: Computational Complexity
The more “traditional” application of the kNN is the classification of data. It
often has quite a lot of points, e. g. MNIST has 60k training images and 10k test
images. Classification is done offline, which means we first do the training
phase, then just use the results during prediction. Therefore, if we want to
construct the data structure, we only need to do so once. For 10k test images,
let’s compare the brute force (which calculates all distances every time) and
k-d tree for 3 neighbors.
21
kNN: Computational Complexity
● Brute force (O(k * n)): 3 * 10,000 = 30,000
● k-d tree (O(k * log(n))): 3 * log(10,000) ~ 3 * 13 = 39
22
Classification Metrics
How to measure the performance of a classification model?
24
Classification Metrics
Most widely used metrics and tools to access classification models:
● Confusion matrix
● Accuracy
● Precision/Recall/F1-score
● Area under the ROC curve
25
Classification Metrics
Confusion Matrix
A table to summarize how successful the classification model is at
predicting examples belonging to various classes.
26
Classification Metrics
Confusion Matrix
E.g., For binary classification, a model predicts two classes: “spam” and
“not_spam” from a given email.
prediction
spam not_spam
spam True Positive False Negative
actual
(TP) (FN)
not_spam False Positive True Negative
(FP) (TN)
27
Classification Metrics
Confusion Matrix
Exercise 1: Consider a Cricket Tournament. Find
the mapping.
True Positive False Negative
(TP) (FN)
1. You had predicted that India would win and it won.
2. You had predicted that England would not win and it
False Positive True Negative
lost. (FP) (TN)
3. You had predicted that England would win, but it lost.
4. You had predicted that India would not win, but it won.
28
Classification Metrics
Confusion Matrix
Exercise 2:
prediction
1 0
1 TP=? FN=?
actual
0 FP=? TN=?
29
Classification Metrics
Confusion Matrix
Exercise 2:
prediction
1 0
1 TP=6 FN=2
actual
0 FP=1 TN=3
30
Classification Metrics
Confusion Matrix
Multiclass Classification: E.g. emotion classification
prediction
Happy Sad Angry Surprise Disgust Neutral
Happy
Sad
actual
=? Angry
Surprise
Disgust
Neutral
31
Classification Metrics
Accuracy
Accuracy is given by the number of correctly classified examples divided by the
total number of classified examples.
prediction
spam not_spam
TP+TN
Acc = spam True Positive False Negative
TP + TN + FP + FN
actual
(TP) (FN)
not_spam False Positive True Negative
(FP) (TN)
33
Classification Metrics
Accuracy
Accuracy is given by the number of correctly classified examples divided by the
total number of classified examples.
prediction
1 0
actual
1 TP=6 FN=2
0 FP=1 TN=3
Accuracy = ?
34
Classification Metrics
Precision
Precision is the ratio of correct positive predictions to the overall number of
positive predictions
prediction
spam not_spam
TP
Precisions = spam True Positive False Negative
TP + FP
actual
(TP) (FN)
FP is costly! not_spam False Positive True Negative
(FP) (TN)
35
Classification Metrics
Recall
Recall is the ratio of correct positive predictions to the overall number of positive
examples.
prediction
spam not_spam
TP spam True Positive False Negative
Recall =
actual
(TP) (FN)
TP + FN
not_spam False Positive True Negative
(FP) (TN)
FN is costly!
36
Classification Metrics
F1-Score
● The formula for the standard F1-score
is the harmonic mean of the precision
and recall.
● Best of both worlds
● A perfect model has an F-score of 1.
● FP & FN both are costly!
37
Classification Metrics
Visualizing Precision/Recall
© wikipedia
38
Classification Metrics
Precision/Recall/F1-score
prediction
1 0
1 P=6 FN=2
Precision = ?
actual
0 FP=1 TN=3
Recall=?
F1-score=?
39
Classification Metrics
Examples: It all depends on the problem!
Diagnosis of cancer.
prediction
It is important in medical cases
cancer no_cancer where it doesn’t matter
whether we raise a false alarm
cancer Perfect X
actual
but the actual positive cases
should not go undetected!
no _cancer OK Perfect
What metric would you pick?
41
Classification Metrics
Examples: It all depends on the problem!
Diagnosis of cancer.
prediction
It is important in medical cases
cancer no_cancer where it doesn’t matter
whether we raise a false alarm
cancer Perfect X
actual
but the actual positive cases
should not go undetected!
no _cancer OK Perfect
TP
Recall =
TP + FN
42
Classification Metrics
Examples: It all depends on the problem!
Detecting if an email spam or no spam.
prediction
It is important in emails where
spam no_spam it is more important that we
don’t miss any important email
spam Perfect OK
actual
as spam than receiving an
occasional spam as no spam.
no_spam X Perfect
What metric would you pick?
43
Classification Metrics
Examples: It all depends on the problem!
Detecting if an email spam or no spam.
prediction
It is important in emails where
spam no_spam it is more important that we
don’t miss any important email
spam Perfect OK
actual
as spam than receiving an
occasional spam as no spam.
no_spam X Perfect
TP
Precision =
TP + FP
44
Classification Metrics
Multiclass Classification
prediction
Happy Sad Angry Surprise Disgust Neutral
Happy
● Can you define recall (Happy)?
Sad
● Can you define precision (Happy)?
actual
Angry
Surprise
Disgust
Neutral
45
Classification Metrics
Multiclass Classification
prediction
Happy Sad Angry Surprise Disgust Neutral
recall(Happy)
Happy
Sad
actual
Angry
Surprise
Disgust
Neutral
46
Classification Metrics
Multiclass Classification
prediction
Happy Sad Angry Surprise Disgust Neutral
precision(Happy)
Happy
Sad
actual
Angry
Surprise
Disgust
Neutral
47
Classification Metrics
Multiclass Classification
prediction
Happy Sad Angry Surprise Disgust Neutral
Can you define accuracy?
Happy
Sad
actual
Angry
Surprise
Disgust
Neutral
48
Classification Metrics
Multiclass Classification
prediction
Happy Sad Angry Surprise Disgust Neutral
Happy
Sad
actual
Angry
Surprise
Disgust
Neutral
49
Classification Metrics
Area under the ROC Curve (AUC)
● The ROC curve (ROC stands for “receiver operating characteristic,” the term
comes from radar engineering. The method was originally developed for
operators of military radar receivers starting in 1941, which led to its name.) is
a commonly used method to assess the performance of binary classification
models.
● ROC curves use a combination of:
(1) true positive rate (the proportion of positive examples predicted correctly,
defined exactly as recall) and
(2) false positive rate (the proportion of negative examples predicted
incorrectly)
to build up a summary picture of the classification performance.
51
Classification Metrics
Area under the ROC Curve (AUC)
● ROC curves use a combination of:
(1) true positive rate (the proportion of positive examples predicted correctly,
defined exactly as recall) and
(2) false positive rate (the proportion of negative examples predicted
incorrectly)
to build up a summary picture of the classification performance.
TP
FP
TPR =
FPR =
TP + FN
FP + TN
52
Classification Metrics
Area under the ROC Curve (AUC)
prediction
TP spam not_spam
TPR =
TP + FN spam True Positive False Negative
actual
(TP) (FN)
FP not_spam False Positive True Negative
FPR = (FP) (TN)
FP + TN
Specificity = 1 - FPR = TN / (TN + FP)
Sensitivity/Recall = TPR
53
Classification Metrics
Area under the ROC Curve (AUC)
TP
TPR =
TP + FN
FP
FPR =
FP + TN
54
Classification Metrics
Area under the ROC Curve (AUC)
● We used a threshold for
classification in many classification
models
● Typically for models that give
probabilistic output score.
55
Classification Metrics
Area under the ROC Curve (AUC)
● To compare different classifiers, it can
be useful to summarize the
performance of each classifier into a
single measure.
● One common approach is to
calculate the area under the ROC
curve, which is abbreviated to AUC.
56
Classification Metrics
Area under the ROC Curve (AUC)
● AUC ranges in value from 0 to 1
● A model whose predictions are 100%
wrong has an AUC of 0.0
● One whose predictions are 100%
correct has an AUC of 1.0
● AUC is classification-threshold-
invariant and suitable for comparison
58
Classification Metrics
Area under the ROC Curve (AUC)
prediction
spam not_spam
actual
spam 10 0
not_spam 10 0
All predictions say “spam”.
(1) TPR=?
(2) FPR=?
(3) Where is the point in ROC curve? 59
Classification Metrics
Area under the ROC Curve (AUC)
prediction
spam not_spam
actual
spam 0 10
not_spam 0 10
All predictions say “not_spam”.
(1) TPR=?
(2) FPR=?
(3) Where is the point in ROC curve? 60
Classification Metrics
Area under the ROC Curve (AUC)
prediction
spam not_spam
actual
spam 10 0
not_spam 0 10
All predictions are perfect.
(1) TPR=?
(2) FPR=?
(3) Where is the point in ROC curve? 61
Classification Metrics
Area under the ROC Curve (AUC)
prediction
spam not_spam
actual
spam 5 5
not_spam 5 5
Some random predictions.
(1) TPR=?
(2) FPR=?
(3) Where is the point in ROC curve? 62
