CLASSIFICATION: Bayesian Classifiers
Uses Bayes (Thomas Bayes, 1701-1781) Theorem to
build probabilistic models of relationships between
attributes and classes
Statistical principle for combining prior class
knowledge with new evidence from data
Multiple implementations
Nave Bayes
Bayesian networks
CLASSIFICATION: Bayesian Classifiers
Requires concept of conditional probability
Measures the probability of an event given that (by
evidence or information) another event has
occurred
Notation: P(A|B) = Probability of A given that
knowledge of B occurred
P(A|B) = P(AB)/P(B)
Equivalently if P(B) 0, = P(AB) = P(A|B)P(B)
BAYESIAN CLASSIFIERS: Conditional
Probability
Example:
Suppose 1% of a specific population has a form of cancer
A new diagnostic test
produces correct positive results for those with the
cancer of 99% of the time
produces correct negative results for those without the
cancer of 98% of the time
P(cancer) = 0.01
P(cancer | positive test) = 0.99
P(no cancer | negative test) = 0.98
BAYESIAN CLASSIFIERS: Conditional
Probability
Example:
But what if you tested positive? What is the
probability that you actually have cancer?
Bayes Theorem reverses the process to provide
us with an answer.
BAYESIAN CLASSIFIERS: Bayes Theorem
P(B|A) = P(BA)/P(A), if P(A)0
= P(AB)/P(A)
= P(A|B)P(B)/P(A)
Application to our example
P(cancer | test positive) =
P(test positive | cancer)*P(cancer)/P(test positive) =
0.01*0.99/(0.01*0.99+0.99*0.98) = 0.01
BAYESIAN CLASSIFIERS: Bayes Theorem
0.99
cancer
0.01
0.99
0.01
No cancer
Test positive
Test negative
0.98
Test positive
0.02
Test negative
BAYESIAN CLASSIFIERS: Nave Bayes
Bayes Theorem Interpretation
P(class C| F1, F2, , Fn) =
P(class C) P(F1, F2, , Fn| C)/P(F1, F2, , Fn)
posterior = prior likelihood/evidence
BAYESIAN CLASSIFIERS: Nave Bayes
Key concepts
Denominator independent of class C
Denominator effectively constant
Numerator equivalent to joint probability model
P(C, F1, F2, , Fn)
Nave conditional independence assumptions
P(C|F1, F2, , Fn) P(C)P(F1|C) P(F
n
2|C) P(Fn|C)
P C P Fi | C
i1
BAYESIAN CLASSIFIERS: Nave Bayes
Multiple distributional assumptions possible
Gaussian
Multinomial
Bernoulli
BAYESIAN CLASSIFIERS: Nave Bayes
Example
Training set (example from Wikipedia)
Sex
Height(feet)
Weight(pounds)
Footsize(inches)
male
male
male
male
female
female
female
female
6
5.92 (5'11")
5.58 (5'7")
5.92 (5'11")
5
5.5(5'6")
5.42 (5'5")
5.75 (5'9")
180
190
170
165
100
150
130
150
12
11
12
10
6
8
7
9
BAYESIAN CLASSIFIERS: Nave Bayes
Example
Assumptions
Continuous data
Gaussian (Normal) distribution
2
x
1
p
exp
2
2
2
P(male) = P(female) = 0.5
BAYESIAN CLASSIFIERS: Nave Bayes
Example
Classifier generated from training set
Sex
Heightmean
Heightvariance
Weightmean
Weightvariance
Footsizemean
Footsizevariance
male
5.855
0.035033
176.25
122.92
11.25
0.91667
female
5.4175
0.097225
132.5
558.33
7.5
1.6667
BAYESIAN CLASSIFIERS: Nave Bayes
Example
Test sample
Sex
Height
Weight
Footsize
sample
6
130
8
BAYESIAN CLASSIFIERS: Nave Bayes
Example
Calculate posterior probabilities for both
genders
Posterior(male) = P(male)P(height|
male)P(weight|male)P(foot size|
male)/evidence
Posterior(female) = P(female)P(height|
female)P(weight|female)P(foot size|
female)/evidence
Evidence is constant and same so we ignore
denominators
BAYESIAN CLASSIFIERS: Nave Bayes
Example
Calculations for male
P(male) = 0.5 (assumed)
1
P(height|male) =
2 0.035033
P(weight|male) =
6 5.855 2
1.5789
exp
2 0.035033
130 176.25 2
5.988110 6
exp
2 122.92
2 122.92
8 11.25 2
1.3112 10 3
exp
2 0.91667
2 0.91667
1
P(foot size|male) =
6.1984 10 9
Posterior numerator (male)
BAYESIAN CLASSIFIERS: Nave Bayes
Example
Calculations for female
P(male) = 0.5 (assumed)
6 5.4175
1
P(height|fwmale) = 2 0.097225 exp 2 0.097225
P(weight|female) =
P(foot size|female) =
0.22346
130 132.5 2
0.016789
exp
2 558.33
2 558.33
8 7.5 2
1.3112 10 3
exp
2 1.6667
2 1.6667
1
5.3778104
Posterior numerator (female)
BAYESIAN CLASSIFIERS: Nave Bayes
Example
Conclusion
Posterior numerator (significantly) greater for
female classification than for male, so classify
sample as female
BAYESIAN CLASSIFIERS: Nave Bayes
Example
Note
We did not calculate P(evidence) [normalizing
constant] since not needed, but could
P(evidence) = P(male)P(height|
male)P(weight|male)P(foot size|male) +
P(female)P(height|female)P(weight|
female)P(foot size|female)
BAYESIAN CLASSIFIERS: Bayesian Networks
Judea Pearl (UCLA Computer Science, Cognitive
Systems Lab): one of the pioneers of Bayesian
Networks
Author: Probabilistic Reasoning in Intelligent
Systems,1988
Father of journalist Daniel Pearl
Kidnapped and murdered in Pakistan in 2002
by Al-Queda
BAYESIAN CLASSIFIERS: Bayesian Networks
Probabilistic graphical model
Represents random variables and conditional
dependencies using a directed acyclic graph
(DAG)
Nodes of graph represent random variables
BAYESIAN CLASSIFIERS: Bayesian Networks
Edges of graph represent conditional
dependencies
Unconnected nodes conditionally independent
of each other
Does not require all attributes to be
conditionally independent
BAYESIAN CLASSIFIERS: Bayesian Networks
Probability table associating each node to its
immediate parent nodes
If node X has no immediate parents, table
contains only prior probability P(X)
If one parent Y, table contains P(X|Y)
If multiple parents {Y1, Y2, , Yn}, table
contains P(X|Y1, Y2, , Yn)
BAYESIAN CLASSIFIERS: Bayesian Networks
BAYESIAN CLASSIFIERS: Bayesian Networks
Model encodes relevant probabilities from
which probabilistic inferences can then be
calculated
Joint probability: P(G, S, R) = P(R)P(S|R)*P(G|S, R)
G = Grass wet
S = Sprinkler on
R = Raining
BAYESIAN CLASSIFIERS: Bayesian Networks
We can then calculate, for example:
P it is raining AND grass is wet
P it is raining | grass is wet
P grass is wet
sprinkler T,F
P grass is wet = T AND sprinkler AND raining=T
sprinkler T,F , raining T,F
P grass is wet = T AND sprinkler AND raining
BAYESIAN CLASSIFIERS: Bayesian Networks
That is
P it is raining | grass is wet
P TTT P TFT
P TTT P TTF P TFT P TFF
0.198 0.154
0.3577
0.198 0.288 0.1584 0.0
BAYESIAN CLASSIFIERS: Bayesian Networks
Building the model
Create network structure (graph)
Determine probability values of tables
Simplest case
Network defined by user
Most real-world cases
Defining network too com[plex
Use machine learning: many algorithms
BAYESIAN CLASSIFIERS: Bayesian Networks
Algorithms built into Weka
User defined network
Conditional independence tests
Genetic search
Hill climber
K2
Simulated annealing
Maximum weight spanning tree
Tabu search
BAYESIAN CLASSIFIERS: Bayesian Networks
Many other versions online
BNT (Bayes Net Tree) Matlab toolbox
Kevin Murphy, University of British Columbia
http://www.cs.ubc.ca/~murphyk/Software/
