1
Machine Learning
Chapter 1: Introduction
孫民
清華大學
Credit : 林嘉文 (Chia-Wen Lin)
2/22/25
3 Polynomial Curve Fitting
Data Set Size:
𝑁 = 10
2/22/25
4 Sum-of-Squares Error Function
2/22/25
5 0th Order Polynomial
2/22/25
6 1st Order Polynomial
2/22/25
7 3rd Order Polynomial
2/22/25
8 9th Order Polynomial
2/22/25
9 Over-fitting
Root-Mean-Square (RMS) Error:
2/22/25
10 Polynomial Coefficients
2/22/25
11 Data Set Size: 𝑁 = 15
9th Order Polynomial
2/22/25
12 Data Set Size: 𝑁 = 100
9th Order Polynomial
2/22/25
13 Regularization
´ Penalize large coefficient values
2/22/25
14 Regularization: ln 𝜆 = −18
2/22/25
15 Regularization: ln 𝜆 = 0
2/22/25
16 Regularization: 𝐸!"# vs. ln 𝜆
2/22/25
17 Polynomial Coefficients
2/22/25
18 Probability Theory
Apples and Oranges
𝐵 𝑜𝑥 𝑖𝑠 𝑏 𝑙𝑢𝑒 𝑜𝑟 𝑟 𝑒𝑑
(F)Ruit is (a)pple or (o)range
2/22/25
19 Probability Theory – two random variables
´Marginal Probability
´Conditional Probability
Joint Probability
2/22/25
20 Probability Theory
´Sum Rule
Product Rule
2/22/25
21 The Rules of Probability
´ Sum Rule
´ Product Rule
2/22/25
22 Bayes’ Theorem
𝑃 𝑋, 𝑌 = 𝑃 𝑌 𝑋 𝑃 𝑋 Product Rule
Since P(X,Y) = P(Y,X), and
𝑃 𝑌, 𝑋 = 𝑃 𝑋 𝑌 𝑃 𝑌
Hence, 𝑃 𝑌 𝑋 𝑃 𝑋 = 𝑃 𝑋 𝑌 𝑃 𝑌
Posterior Likelihood Prior
Evidence
Posterior µ Likelihood × Prior 2/22/25
23 Probability Theory
Apples and Oranges
4
𝑝 𝐵=𝑟 = = 2/5 𝑂𝑣𝑒𝑟𝑎𝑙𝑙 𝑝𝑟𝑜𝑏 𝑜𝑓 𝑝𝑖𝑐𝑘𝑖𝑛𝑔 𝑎𝑛 𝑎𝑝𝑝𝑙𝑒?
10
p(B = b) = 6/10=3/5 p(F=a)?
p(F=a|B=r) = 2/8 = ¼ Use Sum Rule:
P(F=a|B=b) = 3/4 p(F=a) = p(F=a,B=r)+p(F=a,B=b)
Use Product Rule:
p(F=a,B=r) = p(F=a|B=r)p(B=r)
p(F=a,B=b) = p(F=a|B=b)p(B=b)
Hence,
p(F=a) = p(F=a|B=r)p(B=r)+ p(F=a|B=b)p(B=b)
= ¼*2/5+3/4*3/5 = 2/20+9/20=11/20
p(F=o) = 1- p(F=a) = 9/20 2/22/25
24 Probability Theory
Apples and Oranges
4
𝑝 𝐵=𝑟 = = 2/5 𝐼𝑓 𝑓𝑟𝑢𝑖𝑡 𝑖𝑠 𝑜𝑟𝑎𝑛𝑔𝑒, 𝑤ℎ𝑖𝑐ℎ 𝑏𝑜𝑥?
10
p(B = b) = 6/10=3/5 p(B|F=o)?
p(F=a|B=r) = 2/8 = ¼ Use Bayes’ Rule:
P(F=a|B=b) = 3/4 < 𝐹 = 𝑜 𝐵 <(>)
p(B|F=o) =
<(@AB)
!
∗D/F
p(B=r|F=o) = "
G/DH
= 6/9=2/3
p(B=b|F=o) = 1 - p(B=r|F=o)= 1/3
2/22/25
25 Probability Densities (continuous variable)
Probability density Cumulative distribution function
2/22/25
26 Transformed Densities
x = g(y)
dx/dy = d g(y)/ dy = g’(y)
2/22/25
27 Expectations
Approximate Expectation
(discrete and continuous)
Conditional Expectation
(discrete)
2/22/25
28 Variances and Covariances
𝑊ℎ𝑒𝑛 𝑥, 𝑦 𝑎𝑟𝑒 𝑖𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑡
E(xy) = E(x)E(y). Hence,
Cov[x,y] = 0
2/22/25
29 The Gaussian Distribution
2/22/25
30 Gaussian Mean and Variance
2/22/25
31 The Multivariate Gaussian
Σ 𝑖𝑠 𝑐𝑜𝑣𝑎𝑟𝑖𝑎𝑛𝑐𝑒 𝑚𝑎𝑡𝑟𝑖𝑥. 𝐷𝑖𝑎𝑔𝑛𝑎𝑙 𝑣𝑎𝑙𝑢𝑒𝑠 𝑎𝑟𝑒 𝑣𝑎𝑟𝑖𝑎𝑛𝑐𝑒𝑠 𝜎
2/22/25
32 Gaussian Parameter Estimation
Likelihood function
𝐱 = 𝑥! , 𝑥" , ⋯ , 𝑥# If x is i.i.d.
2/22/25
33 Two Principles for Estimating Parameters
´Maximum likelihood estimation (ML)
Choose 𝛉 that maximizes the probability (likelihood)
of observed data
.!" = argmax 𝑃(𝐷|𝛉)
𝛉
𝛉
´Maximum a posteriori estimation (MAP)
Choose 𝛉 that is most probable given prior
probability and data
𝑃 𝐷𝛉 𝑃 𝛉
.!$%
𝛉 = argmax 𝑃 𝛉 𝐷 = argmax
𝛉 𝛉 𝑃(𝐷)
2/22/25
34 Maximum (Log) Likelihood
𝐱 = 𝑥! , 𝑥" , ⋯ , 𝑥# , 𝐱 is i.i.d.
𝛉$% = argmax 𝑝 𝐱 𝛉 ?
𝛉
(log-likelihood)
𝛉$% = argmax ln 𝑝 𝐱 𝛉
𝛉
(sample mean) (sample variance)
2/22/25
%
35 Properties of 𝜇"$ and 𝜎"$
(unbiased)
(biased)
2/22/25
(unbiased)
36 Curve Fitting Re-visited
𝛽: inverse variance (precision)
2/22/25
37 Maximum Likelihood
Determine 𝐰$% by minimizing sum-of-squares error, 𝐸 𝐰 .
1 #
"
𝐰$% = arg min 5 𝑦 𝑥( , 𝐰 − 𝑡(
𝐰 2 ()!
2/22/25
ML Curve Fitting
38
Green: Actual model 2/22/25
Red: Predicted model
39 MAP: A Step towards Bayes
Posterior Likelihood Prior
Determine 𝐰$*+ by minimizing regularized sum-of-squares error, 𝐸9 𝐰 .
Eq. (1.4) 2/22/25
40 Bayesian Curve Fitting
Predictive
Distribution
W for both mu
And beta. 𝑝 𝑡 𝑥, 𝐰, 𝛽 = 𝒩 𝑡 𝑦 𝑥, 𝐰 , 𝛽 ,!
(Refer to Sec. 3.3 for detailed derivation)
2/22/25
ML Curve Fitting:
41 Bayesian Predictive Distribution
2/22/25
ML Curve Fitting Bayesian Curve Fitting
42 Cross Validation for Model Selection
´5-fold cross-validation -> split the training
data into 5 equal folds
´4 of them for training and 1 for validation
2/22/25
43 Cross Validation
2/22/25
44 Curse of Dimensionality
2/22/25
Curse of Dimensionality
45
Polynomial curve fitting, M = 3
𝐷R # 𝑐𝑜𝑒𝑓𝑓𝑖𝑐𝑖𝑒𝑛𝑡𝑠
𝑉𝑜𝑙𝑢𝑚𝑛 𝑜𝑓 𝑆𝑝ℎ𝑒𝑟𝑒 𝑤𝑖𝑡ℎ 𝑟𝑎𝑑𝑖𝑢𝑠 𝑟
VD 𝑟 = 𝐾𝐷rD
[VD(1)-VD(1-∈)]/VD(1) = 1-(1- ∈)D
2/22/25
46 Decision Theory
Given (x,t), predict t give new x.
´ Inference step
´ Determine either or .
´ Decision step
´ For given x, determine optimal t or decision/action based
on t.
2/22/25
47 Minimum Misclassification Rate
Assuming t as C1 or C2 class
Change x_hat to
X0, blue/green fixed,
but red reduced.
2/22/25
Red/Green Blue
48 Minimum Expected Loss
´ Example: classify medical images as ‘cancer’
or ‘normal’
Decision
Truth
When a cancer patient is classified as normal -> 1000 loss
2/22/25
49 Minimum Expected Loss
True class k, but
Assign class j
Regions are chosen to minimize
Eliminate common factor p(x)
2/22/25
50 Reject Option – avoid making decision
2/22/25
51 Generative vs Discriminative
´ Generative approach:
Model
Use Bayes’ theorem
´ Discriminative approach:
Model directly
2/22/25
52 Why Separate Inference and Decision?
• Minimizing risk (loss matrix may change over time)
• Reject option
• Unbalanced class priors
• Combining models
2/22/25
53 Decision Theory for Regression
´ Inference step
´ Determine 𝑝 𝐱, 𝑡 .
´ Decision step
´ For given x, make optimal prediction, y(x), for t.
´ Loss function:
2/22/25
54 The Squared Loss Function
𝔼 𝐿 is minimized when
2/22/25
55
Information Theory
2/22/25
56 Entropy
h(x) is a monotonic function of p(x),
and expresses the information content (>=0).
ℎ 𝑥 = −𝑙𝑜𝑔2 𝑝(𝑥)
If x,y independent, p(x,y) = p(x) p(y),
h(x,y) = -log2p(x) -log2p(y) = h(x)+h(y)
H(x) is the expectation of h(x)
2/22/25
57 Entropy
Important quantity in
• coding theory
• statistical physics
• machine learning
2/22/25
58 Entropy
2/22/25
59 Entropy - coding theory
´ Coding theory: x discrete with 8 possible
states; how many bits to transmit the
state of x?
´ All states equally likely
Code: 000, 001, 010, 011, 100, 101, 110, 111 2/22/25
60 Entropy
2/22/25
61 Entropy - statistical physics
In how many ways can N identical
objects be allocated M bins?
Note that ni balls in ith bin.
# ways to allocate (multiplicity)
pi is the prob that ball assigned to ith bin.
Entropy maximized when
2/22/25
64 Differential Entropy – continuous x
Put bins of width ¢ along the real line
Differential entropy maximized (for fixed
& 𝜇) when
in which case (only related to 𝜎)
2/22/25
65 Conditional Entropy
ℎ 𝑦|𝑥 = −𝑙𝑜𝑔2 𝑝(𝑦|𝑥)
H[x]
2/22/25
66 The Kullback-Leibler Divergence (Relative Entropy)
Unknow p(x) modeled by q(x). Additional info required
2/22/25
67 Mutual Information
ℎ 𝑥 = −𝑙𝑜𝑔2 𝑝(𝑥)
If x,y independent, p(x,y) = p(x) p(y),
h(x,y) = -log2p(x) -log2p(y) = h(x)+h(y)
If x, y not independent
2/22/25
