Class X- Artificial Intelligence

Part B: Unit-3 Evaluating Models

Homework Questions:
1. Suppose you are developing an AI model to detect fraudulent
transaction risk. Describe True Positive, False Positive, False Negative
and True Negative in this context. (CBSE 2025)
2. Why is it not recommended to use the training dataset as the testing
dataset? What is this scenario called?
3. Are there any ethical concerns we need to keep in mind when
performing model evaluation?
4. What is classification accuracy? Can it be used all times for evaluating
AI models?
5. “Understanding both error and accuracy is crucial for effectively
evaluating and improving AI models.” Justify this statement.
6. A machine learning model is used to classify emails as ‘Spam’ or ‘Not
Spam’. The model’s classification results on the testing set are as
follows:
• True Positives (TP): 150
• True Negatives (TN): 200
• False Positives (FP): 50
• False Negatives (FN): 30
(a) Calculate the accuracy and error rate of the model.
(b) Explain the significance of True Positives and False Positives in this
scenario.
(c) Based on the results, what recommendations can you provide to
improve the model’s performance?
Answer the following (02 marks):
1. Which two parameters are considered for Evaluation of a model?
Ans: Prediction and Reality are the two parameters considered for
Evaluation of a model. The “Prediction” is the output which is given by
the machine and the “Reality” is the real scenario, when the prediction
has been made?
 Class X- Artificial Intelligence
Part B: Unit-3 Evaluating Models

2. People of a village are totally dependent on the farmers for their daily
food items. Farmers grow new seeds by checking the weather
conditions every year. An AI model is being deployed in the village
which predicts the chances of heavy rain to alert farmers which helps
them in doing the farming at the right time. Which evaluation
parameter out of precision, recall and F1 Score is best to evaluate the
performance of this AI model? Explain.
Ans: Let us take each of the factor into consideration at once,
• If precision is considered, FN cases will not be taken into account,
so it will be of great loss as if the machine will predict there will be
no heavy rain, but if the rain occurred, it will be a big monetary loss
due to damage to crops.
• If only recall is considered, then FP cases will not be taken into
account. This situation will also cause a big amount of loss, as all
people of the village are dependent on farmers for food, and if the
model predicts there will be heavy rain and the farmers may not
grow crops, it will affect the basic needs of the people.
Hence F1 Score is the best suited parameter to test this AI model, which
is the balance between Precision and Recall.

3. Give an example of a situation wherein false positive would have a high

cost associated with it.
Ans: Let us consider a model that predicts that a mail is spam or not. If
the model always predicts that the mail is spam, people would not look
at it and eventually might lose important information. Here False Positive
condition (Predicting the mail as spam while the mail is not spam) would
have a high cost.

4. What is Evaluation of an AI model? Why is evaluation important?

Ans: Model evaluation is the process of using different evaluation metrics
to understand a machine learning model’s performance. Its purpose is to
make judgments about a model, to improve its effectiveness, and/or to
inform programming decisions.
 Class X- Artificial Intelligence
Part B: Unit-3 Evaluating Models

• Evaluation is important to ensure that the model is operating

correctly and optimally.
• Evaluation is an initiative to understand how well it achieves its
goals.
• Evaluations help to determine what works well and what could be
improved in a program

5. Which evaluation metric would be crucial in the following cases? Justify

your answer.
a. Mail Spamming
b. Gold Mining
c. Viral Outbreak
Ans:
a. If the model always predicts that the mail is spam, people would not
look at it and eventually might lose important information. False
Positive condition would have a high cost. (predicting the mail as
spam while the mail is not spam). So, Precision would be a crucial
evaluation metric in this case as formula for Precision is:
Precision = True Positive/(True Positive + False Positive)
b. A model saying that there exists treasure at a point and you keep on
digging there but it turns out that it is a false alarm. False Positive case
is very costly. (predicting there is a treasure but there is no treasure).
So, Precision would be a crucial evaluation metric in this case also.
c. A deadly virus has started spreading and the model which is supposed
to predict a viral outbreak does not detect it. The virus might spread
widely and infect a lot of people. Hence, False Negative can be
dangerous. Recall will be considered a crucial metrics in this case as
formula for Recall is;
Recall = True Positive/(True Positive + False Positive)
6. Which evaluation metric is more important for any model?
Or, Why is F1 score an important evaluation metric?
Ans: F1 Evaluation metric is more important for any AI Model. F1 score
maintains a balance between the precision and recall for the classifier.
 Class X- Artificial Intelligence
Part B: Unit-3 Evaluating Models

If the precision is low, the F1 is low and if the recall is low again F1 score
is low.
The F1 score is a number between 0 and 1 and is the harmonic mean of
precision and recall.

When we have a value of 1 (that is 100%) for both Precision and Recall.
The F1 score would also be an ideal 1 (100%). It is known as the perfect
value for F1 Score.

Answer the following (04 marks):

7. Explain Train-Test Split and its importance.
The Train-Test Split is a fundamental technique used to evaluate the
performance of a supervised machine learning model. It ensures that the
model can generalize well to unseen data by dividing the original dataset
into two distinct subsets:
• Training Set: Used to train the machine learning model.
• Testing Set: Used to evaluate the trained model’s performance.
This method is applicable across all supervised learning tasks, including
both classification and regression problems.
• The training dataset allows the model to learn patterns and
relationships from the input features and their corresponding output
labels.
• Once trained, the testing dataset (which the model hasn’t seen before)
is used to assess how well the model performs on new, unseen data.
This evaluation process involves:
a) Feeding input features from the test set into the trained model.
b) Predicting outputs based on those inputs.
c) Comparing predicted outputs to the actual values to calculate
metrics such as accuracy, precision, recall, or mean squared error.
 Class X- Artificial Intelligence
Part B: Unit-3 Evaluating Models

8. Imagine that you have come up with an AI based prediction model

which has been deployed on the roads to check traffic jams. Now, the
objective of the model is to predict whether there will be a traffic jam
or not. Now, to understand the efficiency of this model, we need to
check if the predictions which it makes are correct or not. Thus, there
exist two conditions which we need to ponder upon: Prediction and
Reality.
Traffic Jams have become a common part of our lives nowadays. Living
in an urban area means you have to face traffic each and every time you
get out on the road. Mostly, school students opt for buses to go to
school. Many times, the bus gets late due to such jams and the students
are not able to reach their school on time.
Considering all the possible situations make a Confusion Matrix for the
above situation.
Ans:

The Confusion Matrix Prediction

Yes No
Reality Yes TP FN
No FP TN

Case 1: Is there a traffic Jam? Case 3: Is there a traffic Jam?

Prediction: Yes Reality: Yes Prediction: Yes Reality: No
True Positive False Positive
Case 2: Is there a traffic Jam? Case 4: Is there a traffic Jam?
Prediction: No Reality: No Prediction: No Reality: Yes
True Negative False Negative

9. What are the possible reasons for an AI model not being efficient? Explain.
Reasons of an AI model not being efficient:
a. Lack of Training Data: If the data is not sufficient for developing an AI
Model, or if the data is missed while training the model, it will not be
efficient.
 Class X- Artificial Intelligence
Part B: Unit-3 Evaluating Models
b. Unauthenticated Data / Wrong Data: If the data is not authenticated and
correct, then the model will not give good results.
c. Inefficient coding / Wrong Algorithms: If the written algorithms are not
correct and relevant, Model will not give desired output. Not Tested: If the
model is not tested properly, then it will not be efficient.
d. Not Easy: If it is not easy to be implemented in production or scalable.
e. Less Accuracy: A model is not efficient if it gives less accuracy scores in
production or test data or if it is not able to generalize well on unseen data.

10. What is a confusion matrix? What are the four key components of a
Confusion matrix?
Ans: A confusion matrix is a table used to evaluate the performance of a
classification model. It compares the actual target values with those
predicted by the model. It represents the actual values on the y-axis and
predicted values on the x-axis.

• The numbers in each cell represents the

number of predictions made by a machine learning algorithm that falls into
that particular category.
There are four key categories of Confusion Matrix are:
(i) True Positive (TP): True Positive (TP) represents the number of
instances where the AI model correctly predicts the positive class. It
indicates that the model’s prediction aligns with reality and it
accurately identifies positive cases.
(ii) True Negative (TN): True Negative (TN) is the number of instances
where the AI model correctly predicts the negative class. It occurs
when the model makes a negative prediction and, in reality, the
outcome is also negative.
(iii) False Positive (FP): False Positive (FP) is the number of instances where
the AI model incorrectly predicts the positive class. It occurs when the
 Class X- Artificial Intelligence
Part B: Unit-3 Evaluating Models
model makes a positive prediction but, in reality, the outcome is
negative.
(iv) False Negative (FN): False Negative (FN) is the number of instances
where the AI model incorrectly predicts the negative class. It occurs
when the model makes a negative prediction but, in reality, the
outcome is positive.

11. What is the difference between Precision and Recall?

Precision Recall
Precision is a metric that Recall measures the ability of a
measures the accuracy of model to find all the positive
positive predictions made by a instances in a dataset.
model.
To minimize False Positives.
To minimize False Negatives.
Ex: YouTube recommendation Ex: Covid-19 detection.
system.

12. In cases with higher False Negative predictions, which metrics would
be suitable out of Precision and Recall? Why?
Ans. In cases with higher False Negative predictions, recall metrics is most
suitable because it considers all true positives as well as all false negatives. So,
for situations where human lives or disaster is involved, it is better to consider
recall metrics in such case.
13. In cases with higher False Positive predictions, which metrics would
be suitable out of Precision and Recall? Why?
Ans. In cases with higher False positive predictions, precision metrics is most
suitable because it considers all true and false positives. So, for situations
where danger to human lives or disaster is predicted and does not occur, it will
be a big comfort and reassurance.
14. Calculate Precision, Recall and F1 Score for the following Confusion
Matrix on Heart Attack Risk. Also suggest which metric would not be a
good evaluation parameter here and why?
 Class X- Artificial Intelligence
Part B: Unit-3 Evaluating Models

The Confusion Prediction: 1 Prediction: 0

Matrix
Reality: 1 50 60
Reality: 0 20 20

Ans:
Precision:
Precision is defined as the percentage of true positive cases versus all the
cases where the prediction is true.

= (50 / (50 + 20))

= (50/70)
= 0.714
Recall: It is defined as the fraction of positive cases that are correctly
identified.

= 50 / (50 + 60)
= 50 / 110
= 0.45
F1 Score:
F1 score is defined as the measure of balance between precision and
recall.
 Class X- Artificial Intelligence
Part B: Unit-3 Evaluating Models

= 2 * (0.714 *0.45) / (0.714 + 0.45)

= 2 * (0.321 / 1.164)
= 2* (0.27577)
= 0.55
Therefore,
Precision=0.714 Recall=0.5
F1 Score=0.55
Here within the test, there is a tradeoff. But Recall is not a good Evaluation
metric in this case. Recall metric needs to improve more. Because,
False Positive (impacts Precision): A person is predicted as high risk but does
not have heart attack.
False Negative (impacts Recall): A person is predicted as low risk but has
heart attack.
Therefore, False Negatives miss actual heart patients which is more
dangerous, hence recall metric need more improvement.

****************************