✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 15145b8. Link to the linter CI: here}

The incl/excl params are a bit confusing so I made this table, which hopefully helps.

It might be nice to have a diff output, especially for long docstrings. Maybe using difflib.Differ and its compare method?

Good idea, I didn't want to implement something complex myself. I will have a play with that. I think I would want to print only the line that is different (it would be a very long output for e.g., the average param), but as I join every line of the description together (to normalise for line breaks), I need to think if there is a way to do this.

I also need to deal with the situation when there are >2 different strings. I could just use the first one as reference and compare each of the others to the first. As long as I am able to only print the line/section that is different, this should be okay.

For instance, the average parameter of the metrics are almost the same but differs only a bit, and we would have to exclude it from the comparison.

I had the same thought. average mostly differs because for recall_score we add an extra line: "Weighted recall is equal to accuracy" (the other differences can be resolved). This particular case could be something we could account for. If difflib gives the indices that are different, we could look at them and match with an input. The the user would need to specify that for obj x, allow added line of "xxx" ?

There are other scenarios that would not be so easy to deal with, e.g., using a different word in the description ('metric' instead of 'score').

I've had a go at print diffs. I've used difflib.context_diff and compared between words (not characters). Here is an example output (note I've grouped words with "+" at the start into one line to shorten the message):

E               AssertionError: The description of Parameter 'average' is inconsistent between ['precision_recall_fscore_support'] and ['f1_score', 'fbeta_score', 'precision_score'] and ['recall_score']:
E               
E               
E               
E               *** ['precision_recall_fscore_support']
E               --- ['f1_score', 'fbeta_score', 'precision_score']
E               ***************
E               
E               *** 10,12 ****
E               
E                 the
E               ! metrics
E                 for
E               --- 10,12 ----
E               
E                 the
E               ! scores
E                 for
E               
E               *** ['precision_recall_fscore_support']
E               --- ['recall_score']
E               ***************
E               
E               *** 10,12 ****
E               
E                 the
E               ! metrics
E                 for
E               --- 10,12 ----
E               
E                 the
E               ! scores
E                 for
E               ***************
E               
E               *** 122,123 ****
E               
E               --- 122,129 ----
E               
E                 recall.
E               + Weighted recall is equal to accuracy.
E                 ``'samples'``:

This is getting a bit old but @adrinjalali do you think we're still interested in having this test?

From the tool that has been written here, I'm thinking if we could reuse some part of the machinery to check that the types of a parameter is consistent with the _parameter_constraints.

Apart of making sure that we have consistent information, we could then safely have something like https://github.com/scientific-python/docstub to use the type from the documentation to generate automatically the stubs (that up to now, we don't maintain) useful for IDEs.

From the tool that has been written here, I'm thinking if we could reuse some part of the machinery to check that the types of a parameter is consistent with the _parameter_constraints.

Good idea, I think I could definitely make that work. We may need to get consensus on terms used (e.g., 'estimator object' vs 'estimator instance' or 'array-like' vs 'ndarray', whether to include 'or None'), but this is probably a good thing.

Looks good. Thanks @lucyleeow

Starting to use this function in #29831, I came across two parameters, where it would be nice to compare only part of the description, e.g.,:

and

Maybe a nice addition is to be able to specify which part of the description to compare? This may be a better than setting number of words that can be different, as you won't be able to know which words end up being different.

Note when we use it we'd probably run this function on one specific parameter, and set the 'description_subset' for it specifically.

WDYT @glemaitre @adrinjalali

It seems like a nice variation would be to check for a specific text be present in the docstring. It would cover this case as well, WDYT?

Interesting, hadn't thought of that! Let me summarise potential solutions:

• Check for specific text to be present in description
  • con: adds an extra place to amend when you change a docstring
• Check only subset of the description (e.g., let the user pass a list index or list of indicies)
  • con: can be brittle (?), e.g., if passing a complex list of indicies ([0:10, 14:20])
• Check type only and not description (suggested: TST Add test to check docstring consistency of stacking estimators #29831 (comment))
  • con: description not checked...

Another idea: we could automatically ignore a difference that is a switch between any of the words "estimator", "regressor" and "classifier".

Another idea: we could automatically ignore a difference that is a switch between any of the words "estimator", "regressor" and "classifier".

We could check against a regex, that would support a subset, and variations in a word. Do you think that'd work?

Love it, I think it would cover all use cases. I'll open a PR and see what people think?

Following on from #30854, where we discovered we need to re-work the assert function before we can use it more widely.

With regards to matching descriptions where the whole description should not match, we can instead have a regex to capture group (e.g., first 2 sentences, excluding the 4th word etc), that will be matched between params.

• this is better than the current descr_regex_pattern as you will not need to write out the whole description sentence(s) to match
• the regex can be a little more complicated, as we are dealing with groups

This parameter could also take a dict, allowing us to also specify which param/attr/return this regex matches, e.g.,:

# match only first 2 sentences
descr_regex={'parameter: test': r'^(.*?[.])\s+(.*?[.])'}

(not sure if we have to specify whether the item is a param/attr/return, as it is possible that e.g., a parameter and a return both have the same name)

This means that we don't need to use a 2nd assert, when one of the items don't need to be matched completely.

With regards to matching types that differ slightly, I see 2 cases:

• default value is different, type otherwise consistent
• one of the objects has an additional type value (I think this is uncommon, found for BaggingRegressor and IsolationForest max_samples)

Possible solutions:

• add option to ignore type (for a specific param/attr/return)
  • simple but no checking
• add ignore_default option
  • would not work for the rare cases where type is also not the same
  • we may want to specify the param/attr/return for this to apply to
• add flexible_default option, where it passes if default is different, and/or some min number of types match (not sure on this one)
  • most complicated but most flexible
  • we would want to specify the param/attr/return for this to apply to

ping @glemaitre @StefanieSenger (and maybe @adrinjalali ) WDYT?

I like the sound of these options, but I'd need to see a PR to have a better idea of what the implications are. And a side note, as for different types, we might sometimes actually make them more consistent if we see those differences.

I have given this some thought and here are my 2 cents.

Unfortunately I don't have a solution (except to not use regex at all), but I have a very big concern about maintainability: There will be many good reasons for differing param and attribute descriptions and thus after we have added all these new tests, they will have a lot of regex expressions, including very condensed/flexible ones. And having a lot of regex patterns means that adding documentation comes with needing to adapt the regex expressions of this test (not only if the PR author forgot to add the change to related classes, but possibly also if the addition is totally valid, because we defined the regex too narrowly). This puts a very high barrier to any change in the docstrings and I am not sure if we can afford that. Also, doc PRs are often done by new contributors and we make it very hard for new people to add something to the docs, even possibly a typo correction.

In general, I have the impression that in scikit-learn we are building a castle, which is safer but comes with the cost of inflexibility and - at some point - stagnation.

From the more technical standpoint (and totally disregarding what I wrote before):

1. I think that passing several regex expressions into the same test case would be very handy and will allow us to keep an overview of what had already been included in the test better than if we can only test for one regex per test and as a result have the tests for the same base class sprinkled over several tests. This could be done by passing a dict into descr_regex_pattern, but also by passing a regex directly into include_params, include_attrs and include_returns, which would be a bit easier to read.

2. If I understand correctly, then passing a very condensed and at the same time flexible regex pattern (that would allow variations on a word) is currently possible already? I didn't understand your comment on not needing the 2nd assert, @lucyleeow.

3. I think the type checking could be part of the description check?

About making this new contributor issue: I am not sure if with the regex it should be a good first issue. I feel with the need to make a judgement on which params/attributes/returns to include and the decision on how flexible the regex should be it is not an issue for people who don't know the project well and it might require us to define what exactly people should include and what we can add afterwards. What about putting the lables "moderate" and "meta-issue" instead of "good first issue"? Unprecedented idea: Maybe people can pair with maintainers who then also push to their branches to finish the regex. We could offer this to people who have proven they know how to handle the git workflow and other skills in other good first issues first, so to say as a follow up.

I know that a lot of thought and effort went into this a long time before I even became aware of it; so I feel quite guilty for coming in and voicing concerns so late in the process. Please use my comments the way that suits you best, @lucyleeow, I don't want to constrain you in any way.

There will be many good reasons for differing param and attribute descriptions and thus after we have added all these new tests, they will have a lot of regex expressions,

I may have been looking at different objects than you, but from my experience, for objects we want to test, parameters are mostly the same?

But I agree that I can see this getting out of hand. @glemaitre did mention making a "wanted list" of objects for which we want to add this test for, so maybe we could only work on those to start with? We don't want/need to add tests for everything, as I said in the issue, the list is just a starting point and some (many?) items do not warrant a test to be added.

I think:

• if the descriptions differ by an additional sentence, or different word, we should add a test if we feel it is worth it for these objects
• if the regex is more complicated than above, we probably shouldn't add a test for that param/object

I didn't envisage a lot of regex use, but we can see after working on the "wanted list".

I think the type checking could be part of the description check?

Sorry I didn't make this clear (it's only obvious if you look at the code) but the description text and the type are dealt with separately in the code, because numpydoc splits these. The regex (descr_regex_pattern) only applies to the description and NOT the type. This is why I suggested a separate parameter to allow types to be more flexible (second part of #28678 (comment))

I didn't understand your comment on not needing the 2nd assert, @lucyleeow.

It's just want you described in item 1, if we allow >1 regex per 'test' we can have less 'tests'. I just used the term assert because we can run assert_docstring_consistency several times within one test.

About making this new contributor issue:

Yes it probably is a bit more complicated than our usual good first issues, even without the regex part, they have to know how to use pytest etc. Not sure if we should have this has a general open issue or use for sprint etc.

Let me make some draft PRs, and we can re-assess from there?

Thanks for the explanations, @lucyleeow.

I had imagined a very broad use of these tests and in Bagging* I have encountered three params (only checked params, not attributes and returns) that needed a regex. I see you are thinking of a more selected use of the test, and I agree that (given a beginner friendly error message) the test can be enriching. In any case, no need to directly adjust your approach to my opinions/concerns. I am not a maintainer and I just wanted to give some feedback from when I had tested this issue.

Your insights are just as useful and valid as a maintainers, and I'm happy to take them on board.

I had a look at the Bagging estimators and note:

• BaggingClassifier and BaggingRegressor are pretty similar, whereas IsolationForest differs a bit, which makes sense. Potentially for this item, we would only want to test BaggingClassifier and BaggingRegressor. Or at least test more params for these two, and test fewer params for all 3 objects.
• warm_start - differs due to different versionadded. I wonder if we should add an option to ignore these. For some objects, the version added is the same, and this check actually helped me add one of these tags for one of the objects.
• bootstrap - wording differs but AFAICT the meaning is the same, so we should re-word such that they are consistent. The default value differs but this would not be a regex issue (as I explain above the description text and types are handled differently), I'd add an option to make type checking more flexible.

As for versionadded and other directives, I think we can ignore them all.

As for versionadded and other directives, I think we can ignore them all.

I would tend to agree. I conservatively kept it because once it helped me pick up that we missed a versionadded for an object, but overall it may be easier to just always ignore.

You can take a look at draft PR #30926, the regex option there would allow us to ignore a 'version...' if desired, but it is extra regex work so it may be worth it to ignore by default there too.