Evaluation chapter (4 marks)
1. A social media company has developed an AI model to predict which users are likely to churn (cancel
their account). During testing, the AI model came up with the following predictions.
CONFUSION MATRIX REALITY
YES NO
PREDICTED YES 60 25
NO 5 10
(i) Calculate precision, recall and F1 Score for churn prediction
(ii) How many total tests were performed in the above scenario?
Ans:
Precision = True Positives / (True Positives + False Positives) = 60 / (60 + 5) = 0.923
Recall = True Positives / (True Positives + False Negatives) = 60 / (60 + 25) = 0.706
F1 Score = 2 * (Precision * Recall) / (Precision + Recall) = 2 * (0.923 * 0.706) / (0.923 + 0.706) ≈ 0.801
Note: No need to multiply by 100 since percentage is not asked
(ii) Total tests performed = Sum of all entries in the confusion matrix = 60 + 25 + 5 + 10 = 100
2. Automated trade industry has developed an AI model which predicts the selling and purchasing of
automobiles. During testing, the AI model came up with the following predictions.
i) How many total tests have been performed in the above scenario?
ii) Calculate accuracy, precision, recall and F1 Score.
CONFUSION MATRIX REALITY
YES NO
PREDICTED YES 45 (tp) 15 (fp)
NO 20 (fn) 420 (tn)
(i) TP=45, TN=420, FP=15, FN=20
45+420+15+20 = 500 total cases have been performed.
Note: For calculating accuracy, Precision, Recall and F1 score, we need not multiply the formula by 100 as
all these parameters need to range between 0 to
ii) Accuracy=(TP+TN)/TOTAL NO OF CASES : 465/500 = 0.93
Precision =TP/(TP+FP) = 45/(45+15) = 45/60 = 0.75
Recall=TP/(TP+FN) = 45/(45+420) = 45/465 = 0.09
F1 Score=2*(Precision*Recall)/ (Precision+Recall) = 2*(0.75*0.09/(0.75+0.09) = 2*(0.06/0.84.)=2*0.07=0.14
