permutation_test_score documentation: pvalue when scoring is a loss function #8564
Comments
|
I think that note applies to a long-since deprecated API. Since scoring has
been available, the convention is always "great is better" in which case
you should please submit a PR to remove that note. Thanks!
…On 10 Mar 2017 3:36 am, "leereeves" ***@***.***> wrote:
While reviewing issue #8379
<#8379> I noticed this
statement in the documentation for permutation_test_score:
"If scoring is rather a loss function (i.e. when lower is better such as
with mean_squared_error) then this is actually the complement of the
p-value: 1 - p-value."
If I understand correctly that's not quite true. permutation_test_score
returns:
pvalue = (np.sum(permutation_scores >= score) + 1.0) / (n_permutations + 1)
But one minus this value is not the same as the value obtained with the
appropriate comparison for a scoring function where lower is better:
pvalue = (np.sum(permutation_scores <= score) + 1.0) / (n_permutations + 1)
In particular, one minus this value can be 0.0, while the minimum p-value
is 1 / ( n_permutations + 1 ) as was discussed in issue #8379
<#8379>.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#8564>, or mute the
thread
<https://github.com/notifications/unsubscribe-auth/AAEz69hUfQ94WhKUPWilyLKk-vaDG1Bmks5rkCqkgaJpZM4MYUzS>
.
|
|
Done. I also verified that scoring with |
jnothman
pushed a commit
that referenced
this issue
Mar 12, 2017
Closed
herilalaina
pushed a commit
to herilalaina/scikit-learn
that referenced
this issue
Mar 26, 2017
massich
pushed a commit
to massich/scikit-learn
that referenced
this issue
Apr 26, 2017
Sundrique
pushed a commit
to Sundrique/scikit-learn
that referenced
this issue
Jun 14, 2017
NelleV
pushed a commit
to NelleV/scikit-learn
that referenced
this issue
Aug 11, 2017
paulha
pushed a commit
to paulha/scikit-learn
that referenced
this issue
Aug 19, 2017
maskani-moh
pushed a commit
to maskani-moh/scikit-learn
that referenced
this issue
Nov 15, 2017
jwjohnson314
pushed a commit
to jwjohnson314/scikit-learn
that referenced
this issue
Dec 18, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
While reviewing issue #8379 I noticed this statement in the docstring explaining the return value from permutation_test_score:
If I understand correctly that's not quite true. permutation_test_score returns:
pvalue = (np.sum(permutation_scores >= score) + 1.0) / (n_permutations + 1)But one minus this value is:
pvalue = np.sum(permutation_scores < score) / (n_permutations + 1)which is not the same as the value obtained with the appropriate comparison for a scoring function where lower is better:
pvalue = (np.sum(permutation_scores <= score) + 1.0) / (n_permutations + 1)In particular, one minus this value can be 0.0, while the minimum p-value is
1 / ( n_permutations + 1 )as was discussed in issue #8379.The text was updated successfully, but these errors were encountered: