CS 188: Artificial Intelligence

Naïve Bayes

Spring 2023
[These slides were created by Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley. All CS188 materials are available at http://ai.berkeley.edu.]
 Machine Learning

 Up until now: how use a model to make optimal decisions

 Machine learning: how to acquire a model from data / experience

 Learning parameters (e.g. probabilities)
 Learning structure (e.g. BN graphs)
 Learning hidden concepts (e.g. clustering, neural nets)

 Today: model-based classification with Naive Bayes

Classification
 Classification and Machine Learning
 Dataset: each data point, x, is associated with some label (aka class), y
 Goal of classification: given inputs x, write an algorithm to predict labels y
 Workflow of classification process:
 Input is provided to you
 Extract features from the input: attributes of the input that characterize each x and hopefully
help with classification
 Run some machine learning algorithm on the features: today, Naïve Bayes
 Output a predicted label y

Feature Machine
extraction Features learning y
x (input)
(attributes of x) (predicted output)
 Training and Machine Learning
 Big idea: ML algorithms learn patterns between features and labels from data
 You don’t have to reason about the data yourself
 You’re given training data: lots of example datapoints and their actual labels

Training: Learn patterns from labeled data, and Eventually, use your algorithm to
periodically test how well you’re doing predict labels for unlabeled data
 Example: Spam Filter
 Input: an email Dear Sir.
 Output: spam/ham
First, I must solicit your confidence in
this transaction, this is by virture of its
 Setup: nature as being utterly confidencial and
 Get a large collection of example emails, each labeled top secret. …
“spam” or “ham” TO BE REMOVED FROM FUTURE
 Note: someone has to hand label all this data! MAILINGS, SIMPLY REPLY TO THIS
 Want to learn to predict labels of new, future emails MESSAGE AND PUT "REMOVE" IN THE
SUBJECT.

 Features: The attributes used to make the ham / 99 MILLION EMAIL ADDRESSES
spam decision FOR ONLY $99
 Words: FREE! Ok, Iknow this is blatantly OT but I'm
 Text Patterns: $dd, CAPS beginning to go insane. Had an old Dell
 Non-text: SenderInContacts, WidelyBroadcast Dimension XPS sitting in the corner and
 decided to put it to use, I know it was
… working pre being stuck in the corner,
but when I plugged it in, hit the power
nothing happened.
 Example: Digit Recognition
 Input: images / pixel grids
0
 Output: a digit 0-9
1
 Setup:
 Get a large collection of example images, each labeled with a digit
 Note: someone has to hand label all this data! 2
 Want to learn to predict labels of new, future digit images

1
 Features: The attributes used to make the digit decision
 Pixels: (6,8)=ON
 Shape Patterns: NumComponents, AspectRatio, NumLoops
 … ??
 Features are increasingly induced rather than crafted
 Other Classification Tasks
 Classification: given inputs x, predict labels (classes) y

 Examples:
 Medical diagnosis (input: symptoms,
classes: diseases)
 Fraud detection (input: account activity,
classes: fraud / no fraud)
 Automatic essay grading (input: document,
classes: grades)
 Customer service email routing
 Review sentiment
 Language ID
 … many more

 Classification is an important commercial technology!

Model-Based Classification
 Model-Based Classification
 Model-based approach
 Build a model (e.g. Bayes’ net) where
both the output label and input
features are random variables
 Instantiate any observed features
 Query for the distribution of the label
conditioned on the features

 Challenges
 What structure should the BN have?
 How should we learn its parameters?
 Naïve Bayes Model
 Random variables in this Bayes’ net:
 Y = The label Y
 F1, F2, …, Fn = The n features
 Probability tables in this Bayes’ net:
 P(Y) = Probability of each label occurring, given no information about
the features. Sometimes called the prior. F1 F2 Fn
 P(Fi|Y) = One table per feature. Probability distribution over a feature,
given the label.
 Naïve Bayes Model
 To perform training:
 Use the training dataset to estimate the probability tables. Y
 Estimate P(Y) = how often does each label occur?
 Estimate P(Fi|Y) = how does the label affect the feature?
 To perform classification:
 Instantiate all features. You know the input features, so they’re your F1 F2 Fn
evidence.
 Query for P(Y|f1, f2, …, fn). Probability of label, given all the input features.
Use an inference algorithm (e.g. variable elimination) to compute this.
 Example: Naïve Bayes for Spam Filter
 Step 1: Select a ML algorithm. We choose to model the problem with Naïve Bayes.
 Step 2: Choose features to use. Y: The label (spam or ham)
Y P(Y)
Y ham ?
spam ?

F1: A feature F2: Another feature

(do I know the sender?) (# of occurrences of FREE)
F1 F2 F1 Y P(F1|Y) F2 Y P(F2|Y)
yes ham ? 0 ham ?
no ham ? 1 ham ?
yes spam ? 2 ham ?
no spam ? 0 spam ?
1 spam ?
2 spam ?
 Example: Naïve Bayes for Spam Filter
 Step 3: Training: Use training data to fill in the probability tables.

F2: # of occurrences of FREE Training Data

F2 Y P(F2|Y) # Email Text Label
0 ham 0.5 1 Attached is my portfolio. ham
1 ham 0.5 2 Are you free for a meeting tomorrow? ham
2 ham 0.0 3 Free unlimited credit cards!!!! spam
0 spam 0.25 4 Mail $10,000 check to this address spam
1 spam 0.50 5 Sign up now for 1 free Bitcoin spam
2 spam 0.25 6 Free money free money spam

Row 4: P(F2=0 | Y=spam) = 0.25 because 1 out of 4 spam emails contains “free” 0 times.
Row 5: P(F2=1 | Y=spam) = 0.50 because 2 out of 4 spam emails contains “free” 1 time.
Row 6: P(F2=2 | Y=spam) = 0.25 because 1 out of 4 spam emails contains “free” 2 times.
 Example: Naïve Bayes for Spam Filter
 Model trained on a larger dataset:
Y: The label (spam or ham)
Y P(Y)
Y ham 0.6
spam 0.4

F1: A feature F2: Another feature

(do I know the sender?) (# of occurrences of FREE)
F1 F2 F1 Y P(F1|Y) F2 Y P(F2|Y)
yes ham 0.7 0 ham 0.85
no ham 0.3 1 ham 0.07
yes spam 0.1 2 ham 0.08
no spam 0.9 0 spam 0.75
1 spam 0.12
2 spam 0.13
 Example: Naïve Bayes for Spam Filter
 Step 4: Classification
 Suppose you want to label this email from a known sender: Y

“Free food in Soda 430 today”

 Step 4.1: Feature extraction:
 F1 = yes, known sender
F1 F2
 F2 = 1 occurrence of “free”
 Example: Naïve Bayes for Spam Filter
Y: The label (spam or ham)
 Step 4.2: Inference Y P(Y)
ham 0.6
 Instantiate features (evidence): spam 0.4
 F1 = yes
F1: do I know the sender?
 F2 = 1
F1 Y P(F1|Y)
 Compute joint probabilities: yes ham 0.7
no ham 0.3
 P(Y = spam, F1 = yes, F2 = 1) = P(Y = spam) P(F1 = yes | spam) P(F2 = 1 | spam)
yes spam 0.1
= 0.4 * 0.1 * 0.12 = 0.0048 no spam 0.9
 P(Y = ham, F1 = yes, F2 = 1) = P(Y = ham) P(F1 = yes | ham) P(F2 = 1 | ham)
= 0.6 * 0.7 * 0.07 = 0.0294 F2: # of occurrences of FREE
 Normalize: F2 Y P(F2|Y)
0 ham 0.85
 P(Y = spam | F1 = yes, F2 = 1) = 0.0048 / (0.0048+0.0294) = 0.14
1 ham 0.07
 P(Y = ham | F1 = yes, F2 = 1) = 0.0294 / (0.0048+0.0294) = 0.86 2 ham 0.08
0 spam 0.75
 Classification result: 1 spam 0.12
 14% chance the email is spam. 86% chance it’s ham. 2 spam 0.13

 Or, if you don’t need probabilities, note that 0.0294 > 0.0048 and guess ham.
 Naïve Bayes for Digits
 Simple digit recognition version:
 One feature (variable) Fij for each grid position <i,j>
 Feature values are on / off, based on whether intensity Y

is more or less than 0.5 in underlying image

 Each input maps to a feature vector, e.g.

F1 F2 Fn
 Here: lots of features, each is binary valued

 Naïve Bayes model:

 What do we need to learn?

 General Naïve Bayes
 Naïve Bayes assumes that all features are independent effects of the label
 A general Naive Bayes model:
Y

|Y| parameters

F1 F2 Fn

|Y| x |F|n values n x |F| x |Y|

parameters

 We only have to specify how each feature depends on the class

 Total number of parameters is linear in n
 Model is very simplistic, but often works anyway
 Inference for Naïve Bayes
 Goal: compute posterior distribution over label variable Y
 Step 1: get joint probability of label and evidence for each label

+
 Step 2: sum to get probability of evidence

 Step 3: normalize by dividing Step 1 by Step 2

Example: Conditional Probabilities

1 0.1 1 0.01 1 0.05

2 0.1 2 0.05 2 0.01
3 0.1 3 0.05 3 0.90
4 0.1 4 0.30 4 0.80
5 0.1 5 0.80 5 0.90
6 0.1 6 0.90 6 0.90
7 0.1 7 0.05 7 0.25
8 0.1 8 0.60 8 0.85
9 0.1 9 0.50 9 0.60
0 0.1 0 0.80 0 0.80
 A Spam Filter
Dear Sir.
 Naïve Bayes spam filter
First, I must solicit your confidence in this
transaction, this is by virture of its nature
 Data: as being utterly confidencial and top
 Collection of emails, labeled secret. …

spam or ham
 Note: someone has to hand TO BE REMOVED FROM FUTURE
MAILINGS, SIMPLY REPLY TO THIS
label all this data! MESSAGE AND PUT "REMOVE" IN THE
 Split into training, held-out, SUBJECT.
test sets
99 MILLION EMAIL ADDRESSES
FOR ONLY $99
 Classifiers
 Learn on the training set Ok, Iknow this is blatantly OT but I'm
 (Tune it on a held-out set) beginning to go insane. Had an old Dell
Dimension XPS sitting in the corner and
 Test it on new emails decided to put it to use, I know it was
working pre being stuck in the corner, but
when I plugged it in, hit the power nothing
happened.
 Naïve Bayes for Text
 Bag-of-words Naïve Bayes:
 Features: Wi is the word at position i
 As before: predict label conditioned on feature variables (spam vs. ham)
 As before: assume features are conditionally independent given label
 New: each Wi is identically distributed Word at position
i, not ith word in
the dictionary!
 Generative model:

 “Tied” distributions and bag-of-words

 Usually, each variable gets its own conditional probability distribution P(F|Y)
 In a bag-of-words model
 Each position is identically distributed
 All positions share the same conditional probs P(W|Y)
 Why make this assumption?
 Called “bag-of-words” because model is insensitive to word order or reordering
 Example: Spam Filtering
 Model:
 What are the parameters?

ham : 0.66 the : 0.0156 the : 0.0210

spam: 0.33 to : 0.0153 to : 0.0133
and : 0.0115 of : 0.0119
of : 0.0095 2002: 0.0110
you : 0.0093 with: 0.0108
a : 0.0086 from: 0.0107
with: 0.0080 and : 0.0105
from: 0.0075 a : 0.0100
... ...

 Where do these tables come from?

 Spam Example
Word P(w|spam) P(w|ham) Tot Spam Tot Ham
(prior) 0.33333 0.66666 -1.1 -0.4
Gary 0.00002 0.00021 -11.8 -8.9
would 0.00069 0.00084 -19.1 -16.0
you 0.00881 0.00304 -23.8 -21.8
like 0.00086 0.00083 -30.9 -28.9
to 0.01517 0.01339 -35.1 -33.2
lose 0.00008 0.00002 -44.5 -44.0
weight 0.00016 0.00002 -53.3 -55.0
while 0.00027 0.00027 -61.5 -63.2
you 0.00881 0.00304 -66.2 -69.0
sleep 0.00006 0.00001 -76.0 -80.5

P(spam | w) = 98.9
 General Naïve Bayes
 What do we need in order to use Naïve Bayes?
 Inference method
 Start with a bunch of probabilities: P(Y) and the P(Fi|Y) tables
 Use standard inference to compute P(Y|F1…Fn)
 Nothing new here

 Estimates of local conditional probability tables

 P(Y), the prior over labels
 P(Fi|Y) for each feature (evidence variable)
 These probabilities are collectively called the parameters of the model
and denoted by 
 Up until now, we assumed these appeared by magic, but they typically
come from training data counts
Training and Testing
 Empirical Risk Minimization
 Empirical risk minimization
 Basic principle of machine learning
 We want the model (classifier, etc) that does best on the true test distribution
 Don’t know the true distribution so pick the best model on our actual training set
 Finding “the best” model on the training set is phrased as an optimization problem

 Main worry: overfitting to the training set

 Better with more training data (less sampling variance, training more like test)
 Better if we limit the complexity of our hypotheses (regularization and/or small
hypothesis spaces)
 Important Concepts
 How do we check that we’re not overfitting during training?
 Split training data into 3 different sets:
 Training set
 Held out set (more on this later)
 Test set
 Experimentation cycle Training
 Learn parameters (e.g. model probabilities) on training set Data
 Compute accuracy of test set
 Very important: never “peek” at the test set!
 Evaluation (many metrics possible, e.g. accuracy)
 Accuracy: fraction of instances predicted correctly
 Overfitting and generalization
 Want a classifier which does well on test data Held-Out
 Overfitting: fitting the training data very closely, but not Data
generalizing well
 We’ll investigate overfitting and generalization formally in a few
lectures Test
Data
Generalization and Overfitting
 Overfitting
30

25

20
Degree 15 polynomial
15

10

-5

-10

-15
0 2 4 6 8 10 12 14 16 18 20
Example: Overfitting

2 wins!!
 Example: Overfitting
 Posteriors determined by relative probabilities (odds ratios):

south-west : inf screens : inf

nation : inf minute : inf
morally : inf guaranteed : inf
nicely : inf $205.00 : inf
extent : inf delivery : inf
seriously : inf signature : inf
... ...

What went wrong here?

 Generalization and Overfitting
 Relative frequency parameters will overfit the training data!
 Just because we never saw a 3 with pixel (15,15) on during training doesn’t mean we won’t see it at test time
 Unlikely that every occurrence of “minute” is 100% spam
 Unlikely that every occurrence of “seriously” is 100% ham
 What about all the words that don’t occur in the training set at all?
 In general, we can’t go around giving unseen events zero probability

 As an extreme case, imagine using the entire email as the only feature (e.g. document ID)
 Would get the training data perfect (if deterministic labeling)
 Wouldn’t generalize at all
 Just making the bag-of-words assumption gives us some generalization, but isn’t enough

 To generalize better: we need to smooth or regularize the estimates

Parameter Estimation
 Parameter Estimation
 Estimating the distribution of a random variable
 Elicitation: ask a human (why is this hard?) b
r b
br
b
br
r b b
r
b

b b

 Empirically: use training data (learning!)

 E.g.: for each outcome x, look at the empirical rate of that value:

r r b

 This is the estimate that maximizes the likelihood of the data

Smoothing
 Maximum Likelihood?
 Relative frequencies are the maximum likelihood estimates

 Another option is to consider the most likely parameter value given the data

????
Unseen Events
 Laplace Smoothing

 Laplace’s estimate:
 Pretend you saw every outcome
r r b
once more than you actually did

 Can derive this estimate with

Dirichlet priors (see cs281a)
 Laplace Smoothing
 Laplace’s estimate (extended)– “Add-k
smoothing”:
 Pretend you saw every outcome k extra times r r b

 What’s Laplace with k = 0?

 k is the strength of the prior

 Laplace for conditionals:

 Smooth each condition independently:
 Real NB: Smoothing
 For real classification problems, smoothing is critical
 New odds ratios:

helvetica : 11.4 verdana : 28.8

seems : 10.8 Credit : 28.4
group : 10.2 ORDER : 27.2
ago : 8.4 <FONT> : 26.9
areas : 8.3 money : 26.5
... ...

Do these make more sense?

Tuning
 Tuning on Held-Out Data

 Now we’ve got two kinds of unknowns

 Parameters: the probabilities P(X|Y), P(Y)
 Hyperparameters: e.g. the amount / type of
smoothing to do, k, 

 What should we learn where?

 Learn parameters from training data
 Tune hyperparameters on different data
 Why?
 For each value of the hyperparameters, train
and test on the held-out data
 Choose the best value and do a final test on
the test data
Features
 Errors, and What to Do
 Examples of errors
Dear GlobalSCAPE Customer,
GlobalSCAPE has partnered with ScanSoft to offer you the
latest version of OmniPage Pro, for just $99.99* - the
regular list price is $499! The most common question we've
received about this offer is - Is this genuine? We would like
to assure you that this offer is authorized by ScanSoft, is
genuine and valid. You can get the . . .

. . . To receive your $30 Amazon.com promotional certificate,

click through to
http://www.amazon.com/apparel
and see the prominent link for the $30 offer. All details are
there. We hope you enjoyed receiving this message. However,
if you'd rather not receive future e-mails announcing new
store launches, please click . . .
 What to Do About Errors?
 Need more features– words aren’t enough!
 Have you emailed the sender before?
 Have 1K other people just gotten the same email?
 Is the sending information consistent?
 Is the email in ALL CAPS?
 Do inline URLs point where they say they point?
 Does the email address you by (your) name?

 Can add these information sources as new

variables in the NB model

 Later this week we’ll talk about classifiers which

let you easily add arbitrary features more easily,
and, later, how to induce new features
 Baselines
 First step: get a baseline
 Baselines are very simple “straw man” procedures
 Help determine how hard the task is
 Help know what a “good” accuracy is

 Weak baseline: most frequent label classifier

 Gives all test instances whatever label was most common in the training set
 E.g. for spam filtering, might label everything as ham
 Accuracy might be very high if the problem is skewed
 E.g. calling everything “ham” gets 66%, so a classifier that gets 70% isn’t very good…

 For real research, usually use previous work as a (strong) baseline

 Confidences from a Classifier
 The confidence of a probabilistic classifier:
 Posterior probability of the top label

 Represents how sure the classifier is of the classification

 Any probabilistic model will have confidences
 No guarantee confidence is correct

 Calibration
 Weak calibration: higher confidences mean higher accuracy
 Strong calibration: confidence predicts accuracy rate
 What’s the value of calibration?
 Summary
 Bayes rule lets us do diagnostic queries with causal probabilities

 The naïve Bayes assumption takes all features to be independent given the class label

 We can build classifiers out of a naïve Bayes model using training data

 Smoothing estimates is important in real systems

 Classifier confidences are useful, when you can get them