[WIP] : Added assert_consistent_docs() and related tests #10323

amanp10 · 2017-12-14T18:28:14Z

Added a function to check for consistency between docstring of objects.

In this approach there is a python dictionary for each of Parameters, Attributes and Returns. Indexed by the name of the parameter/attribute/return with the value being a Python dictionary containing its type definition and description. The function checks for each object if the docstring is identical (excluding whitespaces) for a given parameter/attribute/return in the main dictionary. If a new parameter/attribute/return are found they are added in the main dictionary.

amueller · 2017-12-14T20:23:47Z

Do we expect this to be reasonable for everything, or only for parts of the library? something like random_state might or might not be good to document consistently - saying what the randomness is might be good. Also, we have parameters like alpha that can mean basically anything

amueller · 2017-12-14T20:24:25Z

Or is the idea to call this with very small families, like "only linear models" or something like that?

amueller · 2017-12-14T20:25:10Z

numpydoc is not found? (are we not using numpydoc on master... I lost track of that)

jnothman · 2017-12-15T02:08:59Z

can you choose a few related objects from one module and add tests for their parameters as an example?

jnothman · 2017-12-15T03:23:22Z

In sklearn/tests/test_docstring_parameters.py we skip the test if numpydoc is not found. We should do so here also. Only one of our Travis runs installs numpydoc for testing.

jnothman · 2017-12-15T03:24:08Z

Yes, Andy, the intention is to use this for families of objects, e.g. precision_score, recall_score, f1_score, etc..

amanp10 · 2017-12-15T07:40:13Z

The doctest for the example is failing as I have used try and except for import numpydoc in the unit test and not in the function itself. What should I do?

jnothman · 2017-12-16T21:07:29Z

do the import locally in the function is one easy solution

amanp10 · 2017-12-17T05:11:02Z

The doctest is still failing with
UNEXPECTED EXCEPTION: SkipTest('numpydoc is required to test the docstrings, as well as python version >= 3.5',)
Should we skip the doctest?

jnothman · 2017-12-17T10:33:05Z

I think just skip, or remove, the doctest.

…

On 17 December 2017 at 16:11, Aman Pratik ***@***.***> wrote: The doctest is still failing with UNEXPECTED EXCEPTION: SkipTest('numpydoc is required to test the docstrings, as well as python version >= 3.5',) Should we skip the doctest? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10323 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz61TOYUDWAaKm6cvUFC7AjYZ3ggi2ks5tBKJngaJpZM4RCetf> .

amanp10 · 2017-12-18T07:12:54Z

I will add more tests to improve the coverage shortly.

amanp10 · 2017-12-18T08:32:13Z

After seeing the coverage report I feel because of numpydoc not being available many tests are not running, hence the bad coverage.
What do you say @jnothman @amueller ?

jnothman

Nice work so far, thanks!

jnothman · 2017-12-20T08:34:23Z

sklearn/utils/testing.py

+    include_returns : list, '*' or None (default)
+        List of Returns to be included. '*' for including all returns.
+
+    exclude_params : list, '*' or None (default)


Please use the same order as in the function signature

'*' is meaningless for exclusion, isn't it?

What if we have to ignore all the, lets say, attributes. exclude_attribs='*' would be a nice way since we have to set either include or exclude.

jnothman · 2017-12-20T08:36:03Z

sklearn/utils/testing.py

+        objects (classes, functions, descriptors) with docstrings that can be
+        parsed as numpydoc.
+
+    include_params : list, '*' or None (default)


It's tempting to make include_params='*' the default.

I thought so too.

jnothman · 2017-12-20T08:40:28Z

sklearn/utils/testing.py

+    """
+    Checks consistency between the docstring of ``objects``.
+
+    Checks if types and descriptions of Parameters/Attributes/Returns are


We need to clarify behaviour when one of the params/attribs/returns is present in one and not another. Do we just ignore it and only compare for all pairs where they are common? I think so, but this should be documented.

Yes. We compare only those with the same name, else do nothing. I will document it.

jnothman · 2017-12-20T08:44:04Z

sklearn/utils/testing.py

@@ -882,3 +882,128 @@ def check_docstring_parameters(func, doc=None, ignore=None, class_name=None):
            if n1 != n2:
                incorrect += [func_name + ' ' + n1 + ' != ' + n2]
    return incorrect
+
+
+def check_data(doc_list, type_dict, type_name, object_name, include, exclude):


I think this should be _check_matching_docstrings or something. Definitely start with a _

I also think this deserves a succinct docstring

jnothman · 2017-12-20T08:45:55Z

sklearn/utils/testing.py

+def check_data(doc_list, type_dict, type_name, object_name, include, exclude):
+    for name, type_definition, description in doc_list:
+        # remove all whitespaces
+        type_definition = type_definition.replace(' ', '')


White space is significant. How about using ' '.join(s.split()) to normalise whitespace?

jnothman · 2017-12-20T08:59:05Z

sklearn/utils/testing.py

+
+def check_data(doc_list, type_dict, type_name, object_name, include, exclude):
+    for name, type_definition, description in doc_list:
+        # remove all whitespaces


Do the include/exclude logic before this

In excluded cases just continue. Otherwise branch depending on whether it's been seen in a previous object.

jnothman · 2017-12-20T09:04:40Z

sklearn/utils/testing.py

+
+        if name in type_dict:
+            u_dict = type_dict[name]
+            if (u_dict['type_definition'] != type_definition or


Use plain old

assert actual == expected, msg

This way pytest can help with more or less verbosity. Msg might be "type for parameter random_state in SVC differs from in SVR".

jnothman · 2017-12-20T09:06:06Z

sklearn/utils/testing.py

+                                             object_name+" has inconsistency.")
+        else:
+            if include is None:
+                if name not in exclude:


What if exclude is None?

Currently we are assuming that either include or exclude is set(not None) so if include is None then exclude is not. We might change this if we change the default for include.

jnothman · 2017-12-20T09:09:59Z

sklearn/utils/testing.py

+                    add_dict = {}
+                    add_dict['type_definition'] = type_definition
+                    add_dict['description'] = description
+                    type_dict[name] = add_dict


This would be more readable if you just defined add_dict here rather than a series of insertions

jnothman · 2017-12-20T09:12:23Z

sklearn/utils/tests/test_testing.py

@@ -491,3 +496,49 @@ def test_check_docstring_parameters():
         'type definition for param: "c " (type definition was "")',
         'sklearn.utils.tests.test_testing.f_check_param_definition There was '
         'no space between the param name and colon ("d:int")'])
+
+
+def test_assert_consistent_docs():


Here we should be testing that the assert function works properly, i.e. by inventing or copying docstrings to test ordinary and tricky cases.

Tests for metric docstrings belong with metric tests, I think

This might take some time. I will start working on it with all the other changes.

amanp10 · 2017-12-20T15:00:32Z

@jnothman What are we supposed to do finally about the include and exclude concept?
As I believe,

default for include can be set to '*'.
We can keep it mandatory to set either include or exclude only.

jnothman · 2017-12-20T20:33:56Z

Let's get rid of '*' and replace it with True:

Include=True by default
Include=False disables that section
Include=collection tests only named elements
Exclude=None (equivalent to ()) by default
Exclude otherwise can only be set if include is True

Yes, make a helper to only include a test when docstring testing is enabled

amanp10 · 2018-01-02T14:34:21Z

@jnothman I have made the changes and added tests. Need your opinion on the tests.

jnothman · 2018-01-09T07:16:49Z

sklearn/utils/testing.py

+
+    """
+    from numpydoc import docscrape
+


Should validate include and exclude make sense

It would be helpful and do no harm, I think we should add it.

jnothman · 2018-01-09T07:17:41Z

sklearn/utils/testing.py

+    for name, type_definition, description in doc_list:
+        if exclude is not None and name in exclude:
+            pass
+        elif include is not True and name not in include:


This will raise TypeError if include=False

I think we are going with include and exclude validation in the very beginning, so it wont be necessary here.

jnothman · 2018-01-09T07:20:30Z

sklearn/utils/testing.py

+            type_definition = " ".join(type_definition.split())
+            description = [" ".join(s.split()) for s in description]
+            try:
+                description.remove('')


This will only remove the first. You could use list(filter(None, description))

jnothman · 2018-01-09T07:21:03Z

sklearn/utils/testing.py

+            if name in type_dict:
+                u_dict = type_dict[name]
+                msg1 = (type_name + " " + name + " of " + object_name +
+                        " has inconsistent type definition.")


Inconsistent with what?

jnothman · 2018-01-09T07:24:55Z

sklearn/utils/tests/test_testing.py

+@if_numpydoc
+def test_assert_consistent_docs():
+    # Test for consistent parameters
+    assert_consistent_docs([func_doc1, func_doc2, func_doc3],


I think you can just test once or twice with actual dummy functions, and then just hack the data in NumpyDocString instances to test intricacies of the implementation

I will work on it.

amanp10 · 2018-01-14T10:40:59Z

I have added validation test for include_ and exclude_ , but I need your opinion on it. Also, I am not sure on how to test the corner cases. Few examples might help.

jnothman

Hmm. Your tests aren't being run in CI currently. Hopefully #10473 will change that.

It's currently a bit hard to follow what your tests are trying to check. Testing the error message will improve this. But make sure they are systematic and commented such that each change of parameter value (e.g. include_returns=True) is clear to the reader.

How you can structure the tests: test all meaningful valid and invalid settings of {include_params, exclude_params}, then using a NumpyDocString object, set doc['Returns'] = doc['Parameters'] and doc['Parameters'] = [], and run the same tests with {include_returns, exclude_returns}, to make sure that behaviour there is identical. Same with {include_attribs, exclude_attribs}. Do so with loops or pytest.mark.parametrize to avoid repeating yourself. Then in a separate test function, assert things about precedence: make sure that assertions about parameters happen first, then those about attribs, then those about returns.

jnothman · 2018-01-14T23:21:52Z

sklearn/utils/testing.py

+        objects (classes, functions, descriptors) with docstrings that can be
+        parsed as numpydoc.
+
+    include_params : list, False or True (default)


could change list -> collection. All we care is that in works (or in some other implementation, iteration may be used; but still, a collection is sufficient).

So I should just allow collections to be passed as arguments. Wont it break somewhere? It might need some testing as well. What do you say?

jnothman · 2018-01-14T23:21:55Z

sklearn/utils/testing.py

+    AssertionError: Parameter y_true of mean_squared_error has inconsistency.
+
+    """
+    if ((isinstance(exclude_params, list) and include_params is not True) or


I think we should allow exclude_* to be a set too. And generally, isinstance should be avoided in preference for duck typing. I think in this case (exclude_params and include_params is not True) is sufficient.

I thought this was little messy.
About exclude_, I would prefer the current scheme since it is simpler and fulfills our purpose, I will document it better if needed. Why do we need exclude_ ?

I'm not sure what you mean. Actually I think you've misunderstood that by exclude_* I just mean exclude_params etc. All I mean is that we should not strictly be checking for a list; a set would also be an appropriate collection

Sorry, I get it now. As per your previous comment we are not just allowing lists but collections and sets as well. I will make the changes.

jnothman · 2018-01-14T23:23:42Z

sklearn/utils/tests/test_testing.py

+    doc1 = docscrape.NumpyDocString(inspect.getdoc(func_doc1))
+    doc2 = docscrape.NumpyDocString(inspect.getdoc(func_doc2))
+
+    assert_raises(AssertionError, assert_consistent_docs, [doc1, doc2],


we should test the message too. Please use assert_raises_regex (or with pytest.raises(AssertionError, match=regex))

I will do it.

jnothman · 2018-01-14T23:24:22Z

sklearn/utils/tests/test_testing.py

+                  include_attribs=True)
+
+    # Test with actual classification metrics
+    assert_consistent_docs([precision_recall_fscore_support, precision_score,


Can you please put this in a function test_docstrings() in sklearn/metrics/tests/test_classification.py? I think that's where we want it.

Should I add one for regression metrics as well?

No hurry to test all metrics, it's just an example

jnothman

Hmm. Your tests aren't being run in CI currently. Hopefully #10473 will change that.

It's currently a bit hard to follow what your tests are trying to check. Testing the error message will improve this. But make sure they are systematic and commented such that each change of parameter value (e.g. include_returns=True) is clear to the reader.

How you can structure the tests: test all meaningful valid and invalid settings of {include_params, exclude_params}, then using a NumpyDocString object, set doc['Returns'] = doc['Parameters'] and doc['Parameters'] = [], and run the same tests with {include_returns, exclude_returns}, to make sure that behaviour there is identical. Same with {include_attribs, exclude_attribs}. Do so with loops or pytest.mark.parametrize to avoid repeating yourself. Then in a separate test function, assert things about precedence: make sure that assertions about parameters happen first, then those about attribs, then those about returns.

amanp10 · 2018-01-19T20:26:30Z

I have tried to change the tests accordingly. I have used assert_raise_message instead of assert_raises_regex since I was having difficulty with regular expressions.

jnothman

This is looking pretty good. Just have a think about edge cases that are not currently tested, and which might fail if the code we're written differently. One case I can imagine is having three docstrings, where some param is shared by two but not three of them

jnothman · 2018-01-22T21:20:32Z

sklearn/metrics/tests/test_classification.py

+                           include_returns=False,
+                           exclude_params=['labels', 'average', 'beta'])
+
+    error_str = ("Parameter 'labels' of 'precision_score' has inconsistent "


I assume we don't want this inconsistency to exist? The docs should be fixed then.

In precision_score description there seems to be an addition,
.. versionchanged:: 0.17 parameter *labels* improved for multiclass problem.
Should this be added in precision_recall_fscore_support? If yes then would this PR be appropriate?

I suppose so.

amanp10 · 2018-01-23T08:22:54Z

I will have to work on the edge cases. I get confused sometimes on which tests would be reasonable and which wont.

jnothman · 2018-01-23T09:29:08Z

it's a hard balance to get right, and you get better with practice. writing tests before implementation, and extending them during implementation helps think about how to assert desired functionality. Good tests should IMO look a bit like a proof by induction. You test the base case, and then assume everything like what's been tested so far works, upon which you extend with variants (different parameters etc)

jnothman · 2018-02-13T22:13:05Z

sklearn/metrics/tests/test_classification.py

+                           include_returns=False,
+                           exclude_params=['labels', 'average', 'beta'])
+
+    error_str = ("Parameter 'labels' of 'precision_score' has inconsistent "


I suppose so.

jnothman · 2018-02-13T22:13:51Z

sklearn/metrics/tests/test_classification.py

+    assert_consistent_docs([precision_recall_fscore_support, precision_score,
+                            recall_score, f1_score, fbeta_score],
+                           include_returns=False,
+                           exclude_params=['labels', 'average', 'beta'])


why not beta?

we want to test average for precision_score, recall_score, f1_score, fbeta_score. Can use a separate assertion, I suppose.

We have inconsistency in parameter beta.
For precision_recall_fscore_support,
beta : float, 1.0 by default
The strength of recall versus precision in the F-score.

For fbeta_score,
beta : float
Weight of precision in harmonic mean.

jnothman

Have you checked that if there are multiple differences in a section, the one reported is deterministic?

Otherwise this is looking good

jnothman · 2018-02-13T22:14:16Z

sklearn/metrics/tests/test_classification.py

+    assert_consistent_docs([precision_recall_fscore_support, precision_score,
+                            recall_score, f1_score, fbeta_score],
+                           include_returns=False,
+                           exclude_params=['labels', 'average', 'beta'])


we want to test average for precision_score, recall_score, f1_score, fbeta_score. Can use a separate assertion, I suppose.

jnothman · 2018-02-13T22:14:26Z

sklearn/utils/testing.py

+def if_numpydoc(func):
+    """
+    Decorator to check if numpydoc is available and python version is
+    atleast 3.5.


jnothman · 2018-02-13T22:16:33Z

sklearn/utils/testing.py

+
+            if name in type_dict:
+                u_dict = type_dict[name]
+                msg1 = (type_name + " '" + name + "' of '" + object_name +


using one (.format) or another (%) kind of formatting string would be clearer

jnothman · 2018-02-13T22:18:20Z

sklearn/utils/testing.py

+    for u in objects:
+        if isinstance(u, docscrape.NumpyDocString):
+            doc = u
+            name = 'Object '+str(i)


space around +. I think we should allow the user to pass in names somehow...

Perhaps objects can be (name, numpydocstring) pairs

That would be appropriate for numpydocstring objects.
So, now objects can be a callable (function,class etc.) or tuple of type (string, NumpyDocString). Am I right?

jnothman · 2018-02-13T22:19:41Z

sklearn/utils/testing.py

+    attrib_dict = {}
+    return_dict = {}
+
+    i = 1  # sequence of object in the collection


use enumerate(objects, 1) instead

jnothman · 2018-02-13T22:22:10Z

sklearn/utils/tests/test_testing.py

+                         [doc1, doc2], include_returns=['precision'],
+                         include_params=False)  # type definition mismatch
+
+    doc3 = doc1  # both doc1 and doc3 return 'recall' whereas doc2 does not


a thought inspired by this: I wonder if we should raise an error/warning if an explicitly included name is only in one of the input docstrings...

I am not sure it would be very necessary. Also, if a name is present only in a few of the objects, maybe there should be a warning for that as well.

jnothman

Do you test that the error is deterministic if there are multiple inconsistencies in one section?

Apart from these, this is looking good

amanp10 · 2018-02-27T16:06:21Z

Well, the code would show an error on the very first inconsistency it finds. The error message would be enough to locate the exact place of inconsistency i.e. the error message shows the Parameter name, name of the concerned objects and error type(type definition or description).

adrinjalali · 2024-03-06T12:05:03Z

@glemaitre @scikit-learn/documentation-team what do we think of this now? There are a few usecases, worth continuing this work?

ArturoAmorQ · 2024-03-06T13:02:32Z

I guess this PR was already looking in good shape, so the extra step may be worth it.

lucyleeow · 2024-03-07T00:26:37Z

+1, I think this is useful

adrinjalali · 2024-03-07T06:24:47Z

@lucyleeow @ArturoAmorQ would you have bandwidth to push this forward? pretty please? 😁

lucyleeow · 2024-03-07T06:45:17Z

I'm happy to work on this 😄

Charlie-XIAO · 2024-03-07T07:39:22Z

Please feel free to ping me as well if you need reviews :)

glemaitre · 2024-03-07T11:53:22Z

Yep this looks like a step ahead for consistency and having the right docstring.

Added assert_consistent_docs and related tests

b0474f0

amanp10 changed the title ~~[WIP] : Added assert_consistent_docs and related tests~~ [WIP] : Added assert_consistent_docs() and related tests Dec 14, 2017

amanp10 added 2 commits December 15, 2017 10:13

Corrected numpydoc import error

96e2822

Added examples

5f7c14a

Corrected doctest error

d503f3c

SKIP doctests

0633b66

Added tests

aed37aa

jnothman reviewed Dec 20, 2017

View reviewed changes

amanp10 added 2 commits December 22, 2017 23:13

Review Changes

aafb172

Added tests.

ea58e83

jnothman reviewed Jan 9, 2018

View reviewed changes

amanp10 added 3 commits January 9, 2018 14:41

Partial Review changes

1661dd2

Partial Review changes

149ae20

Review changes and tests

2844572

jnothman reviewed Jan 14, 2018

View reviewed changes

jnothman requested changes Jan 14, 2018

View reviewed changes

Review changes and added Tests

0d00f51

amanp10 added 2 commits January 20, 2018 20:13

Build error correction

97210fa

Build error correction

a416dc0

jnothman reviewed Jan 22, 2018

View reviewed changes

Added Test

7b53b63

jnothman reviewed Feb 13, 2018

View reviewed changes

amueller added the Waiting for Reviewer label Aug 5, 2019

github-actions bot added module:metrics module:utils labels Mar 2, 2020

cmarmo added help wanted Stalled and removed Waiting for Reviewer labels Aug 24, 2020

Base automatically changed from master to main January 22, 2021 10:49

lucyleeow mentioned this pull request Mar 22, 2024

TST check if docstring items are equal between objects (functions, classes, etc.) #28678

Merged

glemaitre closed this in #28678 Sep 5, 2024

Uh oh!

[WIP] : Added assert_consistent_docs() and related tests #10323

[WIP] : Added assert_consistent_docs() and related tests #10323

Uh oh!

Conversation

amanp10 commented Dec 14, 2017

Uh oh!

amueller commented Dec 14, 2017

Uh oh!

amueller commented Dec 14, 2017

Uh oh!

amueller commented Dec 14, 2017

Uh oh!

jnothman commented Dec 15, 2017 via email

Uh oh!

jnothman commented Dec 15, 2017

Uh oh!

jnothman commented Dec 15, 2017

Uh oh!

amanp10 commented Dec 15, 2017

Uh oh!

jnothman commented Dec 16, 2017 via email

Uh oh!

amanp10 commented Dec 17, 2017

Uh oh!

jnothman commented Dec 17, 2017 via email

Uh oh!

amanp10 commented Dec 18, 2017

Uh oh!

amanp10 commented Dec 18, 2017

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amanp10 commented Dec 20, 2017

Uh oh!

jnothman commented Dec 20, 2017

Uh oh!

amanp10 commented Jan 2, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amanp10 commented Jan 14, 2018 •

edited

Loading