FIX: Reduce bias of covariance.MinCovDet with consistency correction
#32117
+58
−18
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MinCovDetMinCovDet, to reduce bias
MinCovDet, to reduce biascovariance.MinCovDet, to reduce bias
covariance.MinCovDet, to reduce biascovariance.MinCovDet with consistency correction
covariance.MinCovDet with consistency correctioncovariance.MinCovDet with consistency correction
|
In some cases, this test fails under the new implementation: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/covariance/tests/test_robust_covariance.py#L35. The specific test that fails has a much lower tolerance threshold than the other tests of this function, so I suggest increasing the threshold. In the example code of the Issue #23162 it is shown that the new implementation is much less biased than the original implementation. The test failure should not be blamed on the new implementation, but rather on the variability of the results + low threshold for that specific test. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference Issues/PRs
Partially fixes Issue #23162
What does this implement/fix? Explain your changes.
Background:
The output of the
covariance.MinCovDetestimator is strongly biased (see Issue #23162), because it is lacking a consistency correction. This PR adds the missing correction, reducing the bias. In my comment to the Issue, I explain the problem and show that this PR generates less biased output.Changes:
I added the function
_consistency_correctionto compute the multiplicative consistency factor, and use it to correct the robust covariance estimate here.Also, the correction is also needed in another place of the code. The original implementation used an adhoc correction from the original paper, which I substituted for the correction factor obtained with
_consistency_correctionhere. This change increases code consistency, and the new correction is more theoretically grounded.
Any other comments?
The estimate is still slightly biased because it lacks a finite sample correction. This should be added in the future, for which the MinCovDet implementation in R can be used as a template https://rdrr.io/cran/robustbase/src/R/covMcd.R.