Description
LDA and QDA are inconsistent in how they estimate the class wise covariance matrix.
For QDA the relevant line of code is:
|
S2 = (S ** 2) / (len(Xg) - 1) |
i.e. the normalization 1/(N-1) is used, whereas LDA calls
|
def empirical_covariance(X, assume_centered=False): |
which by default uses np.cov with bias=True, which is equivalent to ddof=0, corresponding to the normalization 1/N
Steps/Code to Reproduce
import numpy as np
np.set_printoptions(4)
data = np.array([
[ 1, 1, "A"],
[ 2 , 1 , "A"],
[ 2 , 2 , "A"],
[ 1, -1, "B"],
[-1, -1, "B"],
[-2, -2, "B"]])
X = data[:, :-1].astype(float)
Y = data[:, -1]
classes = np.unique(Y)
pi_A = len(X[Y=="A"])/len(X)
pi_B = len(X[Y=="B"])/len(X)
cov_A = np.cov(X[Y=="A"], rowvar=False, ddof=0)
cov_B = np.cov(X[Y=="B"], rowvar=False, ddof=0)
cov_A2 = np.cov(X[Y=="A"], rowvar=False, ddof=1)
cov_B2 = np.cov(X[Y=="B"], rowvar=False, ddof=1)
sigmahat = pi_A*cov_A + pi_B*cov_B
sigmahat2 = pi_A*cov_A2 + pi_B*cov_B2
print("**ddof=0**", "cov_A:", cov_A, "cov_B:", cov_B, "sigmahat:", sigmahat,"\n", sep="\n")
print("**ddof=1**", "cov_A:", cov_A2, "cov_B:", cov_B2, "sigmahat:", sigmahat2,"\n", sep="\n")
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
model = LDA(store_covariance=True)
model.fit(X, Y)
print("**LDA**","sigmahat:", model.covariance_,"\n", sep="\n")
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis as QDA
model = QDA(store_covariance=True)
model.fit(X, Y)
covs=model.covariance_
print("**QDA**", "cov_A:", covs[0], "cov_B:", covs[1],"\n", sep="\n")Expected Results
There are two common ways to estimate the joint covariance matrix in LDA:
- estimate all class covariance individually and aggregate them as
sigmahat = sum prior(class)*covariance(class)
- estimate the common covariance "jointly" (https://en.wikipedia.org/wiki/Pooled_variance)
sigmahat = 1/(N-num_classes) sum_{c in classes} sum_{x : y=c} (x_i - mu_c)*(x_i -mu_c).T
The sklearn LDA model seems to do the former by calling
|
def _class_cov(X, y, priors, shrinkage=None): |
In this case, both LDA and QDA should estimate the class covariance with he same estimator. ddof=0 is appropriate, because the model presupposed that the classes are normally distributed and ddof=0 is the maximum likelihood estimator in this case.
Actual Results
LDA uses ddof=0 and QDA uses ddof=1
Versions
System:
python: 3.7.5 (default, Oct 25 2019, 15:51:11) [GCC 7.3.0]
executable: /home/rscholz/miniconda3/bin/python
machine: Linux-5.3.0-23-generic-x86_64-with-debian-buster-sid
Python deps:
pip: 19.3.1
setuptools: 41.6.0.post20191030
sklearn: 0.21.3
numpy: 1.17.3
scipy: 1.3.1
Cython: None
pandas: 0.25.2
Description
LDA and QDA are inconsistent in how they estimate the class wise covariance matrix.
For QDA the relevant line of code is:
scikit-learn/sklearn/discriminant_analysis.py
Line 694 in 1495f69
i.e. the normalization
1/(N-1)is used, whereas LDA callsscikit-learn/sklearn/covariance/empirical_covariance_.py
Line 49 in 1495f69
which by default uses
np.covwithbias=True, which is equivalent toddof=0, corresponding to the normalization1/NSteps/Code to Reproduce
Expected Results
There are two common ways to estimate the joint covariance matrix in LDA:
sigmahat = sum prior(class)*covariance(class)sigmahat = 1/(N-num_classes) sum_{c in classes} sum_{x : y=c} (x_i - mu_c)*(x_i -mu_c).TThe sklearn LDA model seems to do the former by calling
scikit-learn/sklearn/discriminant_analysis.py
Line 96 in 1495f69
In this case, both LDA and QDA should estimate the class covariance with he same estimator.
ddof=0is appropriate, because the model presupposed that the classes are normally distributed andddof=0is the maximum likelihood estimator in this case.Actual Results
LDA uses
ddof=0and QDA usesddof=1Versions