FIX Inconsistencies between solvers in LinearDiscriminantAnalysis and QuadraticDiscriminantAnalysis - #33626

Thanks for this PR and the detailed description. Let's see if we can find someone who can help review this.

The code changes are simple and make sense. So the reviewer should focus on the methodological aspects.

@ogrisel do you have time and enough knowledge about solvers to evaluate this? Who else could we ask to evaluate this?

@agramfort ;)

Regarding the unsolved issue mentioned at the end, I think that it should be possible to make _solve_svd() be equivalent to _solve_eigen() with the prior-weighted covariance. This should be achievable by some kind of weighting of the data samples by their class prior.

This change would:

1. Make the two solvers consistent across more scenarios
2. Prevent possibly undesired behaviors. E.g. we recorded many more samples of class 1 than of class 2, but set the prior to [0.5, 0.5]. The LDA within-class covariance will still be approximately that of class 1 in the current scenario when using svd, while we might want it to be an average of the two classes.

This could be added in the present PR, since it fits within "fixing solver inconsistencies". If there is interest it exploring this further change, I can study whether such solution exists.

PD. I wonder whether such sample-weighting approach could also be used for the eigen solver in order to fix #12662.

The following code might help review this contribution. The final all_close() comparing SVD and Eigen solvers returns False in main and True in this branch. If the priors are modified, then the comparison fails in the new branch too, because of the unresolved issue described in the PR.

import numpy as np
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

rng = np.random.default_rng(0)
n_class0 = 20
n_class1 = 20
n_features = 5
priors = np.array([0.5, 0.5])
#priors = None

X0 = rng.normal(loc=-1.0, scale=1.0, size=(n_class0, n_features))
X1 = rng.normal(loc=1.0, scale=1.2, size=(n_class1, n_features))
X = np.vstack([X0, X1])
y = np.array([0] * n_class0 + [1] * n_class1)

lda_svd = LinearDiscriminantAnalysis(solver="svd", store_covariance=True, priors=priors)
lda_eigen = LinearDiscriminantAnalysis(solver="eigen", priors=priors)

lda_svd.fit(X, y)
lda_eigen.fit(X, y)

decision_svd = lda_svd.decision_function(X)
decision_eigen = lda_eigen.decision_function(X)

np.allclose(decision_svd, decision_eigen)

I realized that the same disagreement existed for the QuadraticDiscriminantAnalysis() solvers. I also edited the svd solver for QDA to use the ML covariance.

I think that this should also fix #15640 then.

@ogrisel @betatim do you think it's possible to try to start moving this and #33738 forward? I also have the intention to contribute adding array-api compatibility for the full discriminant analysis module, but I'd like for this review to be advanced before starting that.

@ogrisel @betatim do you think it's possible to try to start moving this and #33738 forward? I also have the intention to contribute adding array-api compatibility for the full discriminant analysis module, but I'd like for this review to be advanced before starting that.

The thing we need is people who can review the changes on a methodological level (is that a word?). Questions like "is there a reason for things being different in main?" and "what consequences for users will this have?" or "do we need to let users know that their results are changing?". For someone like me who is keen to have your contributions to array API'ifying this but has only surface level knowledge of LDA and QDA it would take a lot of time to read up and figure out what is what. Which makes it hard to even answer the simple question of "is investing a few hours in this PR worth it or is this a nice to have tweak that basically effects no one?" (as in, should I invest time here or somewhere else).Far from an ideal situation :(

Thanks @betatim. I understand the situation, there's a lot more work to be done than maintainers available time.

Re "is investing a few hours in this PR worth it or is this a nice to have tweak that basically effects no one?", this PR's fix is needed to add something @ogrisel had requested, so I hope it is worth someone's time eventually, although I don't know the balance for you specifically.

I just wanted to bump the PR to keep it from oblivion, if there's some potential reviewers you can think wouldn't mind being tagged.

Also, always feel free to tag me to review PR's if you think I could be helpful. I've had merged PR's on the discriminant analysis and covariance modules before.

I had a little bit of time and interest so I asked Opus (a LLM) to do some research on this topic and write me some material about:

• the general topic,
• research what the "standard references" for this topic implement, and
• find cases where this change makes visible change to the results a user gets.

This was quite an interesting experience. But clearly I am still just a beginner in this topic (LDA and QDA solvers).

My take away from reading what the LLM produced is that the formulation in main is closer to what textbooks like Elements of Statistical Learning use. The reason why they teach it is maybe that they are more in the business of hypothesis testing/classical statistics. It also seems to be the way things were formulated when the tools were first "discovered"/derived.

The scikit-learn library is about classification, so using Maximum Likelihood (ML) makes sense. The LLM also points out that other covariance related tools in scikit-learn already use ML, so making the switch here would increase consistency across the library.

For LDA it suggests that users will basically not notice this change. However for QDA with imbalanced classes you might see a change (I asked for a minimal reproducer for this, which is at the end of this post). It seems like a percent level change?

I think I am 👍 this change. The open questions for me are:

1. is there a meaningful change in the QDA results?
2. if yes, how do we package this? Is this a bug fix (the bug being that we overlooked this difference for a few years) or is it an improvement?

The reason I am not sure on the bug fix vs improvement is that it seems like both approaches (main and this PR) are mathematically correct. Which you prefer depends more on your personal priors/biases. If we class this as a bug fix, then changing behaviour is not a problem. If we class this as an improvement then I am not sure how we should handle this (is a change log note enough?). What do others think?

The coding agent also noticed that doc/modules/lda_qda.rst (lines 234–237) needs updating.

import numpy as np
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis

rng = np.random.default_rng(42)
n0, n1 = 4, 60
X0 = rng.multivariate_normal([0.0, 0.0], np.eye(2),       size=n0)
X1 = rng.multivariate_normal([2.0, 0.0], 0.5 * np.eye(2), size=n1)
X  = np.vstack([X0, X1])
y  = np.array([0] * n0 + [1] * n1)


def qda_predict(X_test, X_train, y_train, ddof):
    """Plain-numpy QDA. ddof=1 -> unbiased (current main). ddof=0 -> ML (the PR)."""
    classes = np.unique(y_train)
    means, inv_covs, log_dets, log_priors = [], [], [], []
    for k in classes:
        Xk = X_train[y_train == k]
        mu = Xk.mean(axis=0)
        Xc = Xk - mu
        Sigma = (Xc.T @ Xc) / (len(Xk) - ddof)
        means.append(mu)
        inv_covs.append(np.linalg.inv(Sigma))
        _, logdet = np.linalg.slogdet(Sigma)
        log_dets.append(logdet)
        log_priors.append(np.log(len(Xk) / len(y_train)))
    scores = []
    for k in range(len(classes)):
        m = X_test - means[k]
        mahal = np.einsum("ij,jk,ik->i", m, inv_covs[k], m)
        scores.append(-0.5 * log_dets[k] - 0.5 * mahal + log_priors[k])
    return np.argmax(np.column_stack(scores), axis=1)


unbiased = qda_predict(X, X, y, ddof=1)   # what main does
ml       = qda_predict(X, X, y, ddof=0)   # what the PR does

sklearn_pred = QuadraticDiscriminantAnalysis().fit(X, y).predict(X)
print("sklearn matches the unbiased convention:",
      np.array_equal(sklearn_pred, unbiased))   # True on main, False on PR branch

flip_idx = np.where(unbiased != ml)[0]
print(f"On training set: predictions flip on {len(flip_idx)} / {len(y)} points")
print("Flipping rows (X, y_true, main, PR):")
for i in flip_idx:
    print(f"  X={X[i].round(3)}  y={y[i]}  main={unbiased[i]}  PR={ml[i]}")

xx, yy = np.meshgrid(np.linspace(-3, 5, 200), np.linspace(-3, 4, 200))
grid = np.column_stack([xx.ravel(), yy.ravel()])
boundary_diff = (qda_predict(grid, X, y, 1) != qda_predict(grid, X, y, 0)).sum()
print(f"Decision regions disagree on {boundary_diff} / {len(grid)} grid points "
      f"({100 * boundary_diff / len(grid):.1f}% of the input space)")

I feel like this was a somewhat useful exercise, but maybe the above is all totally obvious and uninteresting for someone who is a bit more involved in the business of these solvers and estimators?

Thanks for the comment @betatim.

Your understanding of the issue is pretty spot on. I expect the users to not notice, except in some edge cases with very few samples, since the difference between 1/n and 1/(n-1) will be largest when n is small.

What I would highlight, is that the the current default using the unbiased covariance estimate is not only inconsistent with scikit-learn in general: it is specifically inconsistent with the same LDA and QDA methods when using other solvers within scikit-learn.

To illustrate this, the code below uses the same example you shared, but to compare QuadraticDiscriminantAnalysis(solver="svd") to QuadraticDiscriminantAnalysis(solver="eigen") from the current main branch. The two solvers show the same disagreement as the two methods in your example. In practice, users encountering these edge cases might be confused that they get different results with different solvers. With this PR the solvers give the exact same solution.

The example above makes me think of this as a bug fix: a user would report a bug if they observed different QDA or LDA outcomes with the different solvers in an edge case.

Regarding the change in QDA/LDA behavior, I wouldn't worry too much about it because: 1) there doesn't seem to be a "mathematically correct" choice, 2) it should mostly affect edge cases, 3) that behavior already exists in scikit-learn with the other solvers.

I'll add that, while I think that consistency is good to have for its own sake, my main interest in this change is to enable testing that compares the different solvers, to validate shrinkage with SVD in #33738, which should have a more practical impact.

import numpy as np
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis

rng = np.random.default_rng(42)
n0, n1 = 4, 60
X0 = rng.multivariate_normal([0.0, 0.0], np.eye(2), size=n0)
X1 = rng.multivariate_normal([2.0, 0.0], 0.5 * np.eye(2), size=n1)
X = np.vstack([X0, X1])
y = np.array([0] * n0 + [1] * n1)

svd_clf = QuadraticDiscriminantAnalysis(solver="svd").fit(X, y)
eigen_clf = QuadraticDiscriminantAnalysis(solver="eigen").fit(X, y)

svd_pred = svd_clf.predict(X)
eigen_pred = eigen_clf.predict(X)

print(
    'training predictions agree between solver="svd" and solver="eigen":',
    np.array_equal(svd_pred, eigen_pred),
)
flip_idx = np.where(svd_pred != eigen_pred)[0]
print(f"On training set: predictions flip on {len(flip_idx)} / {len(y)} points")
print('Flipping rows (X, y_true, solver="svd", solver="eigen"):')
for i in flip_idx:
    print(f'  X={X[i].round(3)}  y={y[i]}  svd={svd_pred[i]}  eigen={eigen_pred[i]}')

xx, yy = np.meshgrid(np.linspace(-3, 5, 200), np.linspace(-3, 4, 200))
grid = np.column_stack([xx.ravel(), yy.ravel()])
boundary_diff = (svd_clf.predict(grid) != eigen_clf.predict(grid)).sum()
print(
    f"Decision regions disagree on {boundary_diff} / {len(grid)} grid points "
    f"({100 * boundary_diff / len(grid):.1f}% of the input space)"
)

The coding agent also noticed that doc/modules/lda_qda.rst (lines 234–237) needs updating.

Do you agree that this needs updating? If yes, will you do it?

I think I am 👍 making this change and calling it a bug fix/a changelog entry is enough to inform people about this change.

I think I've convinced myself that very few people will notice this. So even though I don't like it when code from one version to another changes behaviour (Imagine you are running a system with several dependencies, if each makes some minute changes you end up chasing after weird behaviour a lot), I think we should make this change.

More philosophically, while it might be surprising/annoying that different solvers find different solutions I am not so worried about that. I guess my bias is that "different solvers exist because they are good at different things, so they will give different answers sometimes". So my worry is more about changing results for someone who didn't change their code, but probably not an issue here.

Thanks @betatim. I agree that those docs required editing, I pushed the update.

And thanks for sharing the perspective on the tradeoffs, this is a useful learning experience.

Hi @betatim, just checking if you're still interested in continuing with this review, after I incorporated the docs changes.

@ogrisel @betatim, I am wondering if I should close this PR, and the associated PR #33738, together with Issue #32590 that requests this feature. I'm not sure these contributions will ever get feedback, and they prevent me from trying to submit other contributions. Any thoughts?

I've added you to the whitelist so that you can open more than one PR at a time. This fixes the immediate problem, because I think even though this PR has been hanging for a long time, closing it seems like a waste (or might trigger someone else to create a PR implementing the same). So I prefer to keep it open.

Thanks @virchan! I forgot to give access to edit the branch, done now. I resolved the conflicts, it was just using the recently renamed array_device() instead of device(). So, ready to merge from my end.