Normalization in SVD solver for LinearDiscriminantAnalysis can produce large scaling coefficients #6725

Description

The SVD solver contains the following code

std = Xc.std(axis=0)
# avoid division by zero in normalization
std[std == 0] = 1.
fac = 1. / (n_samples - n_classes)
#2) Within variance scaling
X = np.sqrt(fac) * (Xc / std)

This can result in very large scaling coefficients if std contains small values not exactly zero.

Steps/Code to Reproduce

Example:

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

lda = LinearDiscriminantAnalysis()
data = [[ 1, 2, 0, 4],
             [5, 6, 1e-7, 8],
             [9, 10, 0, 12]]
labels = [0,0,1]

lda.fit(data, labels)
print lda.scalings_

array([[  8.83883476e-02],
       [  8.83883476e-02],
       [  3.53553391e+06],
       [  8.83883476e-02]])

The scaling value resulting from the normalization could ( and does ) result in poor classification performance. I haven't as yet checked this with other implementations to compare the results.

Versions

Windows-7-6.1.7601-SP1
('Python', '2.7.11 |Anaconda 2.5.0 (32-bit)| (default, Mar 4 2016, 15:18:41) [MSC v.1500 32 bit (Intel)]')
('NumPy', '1.10.4')
('SciPy', '0.17.0')
('Scikit-Learn', '0.17')