We would need a test to check that check_estimator passes on a minimal estimator class that does not inherit from the scikit-learn base classes (at least for mode="compatible" in #18582).

I remember we removed _safe_tags some time ago and I was really sure we had a test for an estimator not inheriting from BaseEstimator in sklearn/utils/tests/test_estimator_checks.py. Maybe we "fixed" the test at the same time, not sure...

#16950 but it looks like there was no test indeed.

The goal would be to honor our hold duck typing contract which I find fundamental to the spirit of the scikit-learn API design:

https://scikit-learn.org/dev/developers/develop.html#rolling-your-own-estimator

BaseEstimator and mixins:

We tend to use “duck typing”, so building an estimator which follows the API suffices for compatibility, without needing to inherit from or even import any scikit-learn classes.

However, if a dependency on scikit-learn is acceptable in your code, you can prevent a lot of boilerplate code by deriving a class from BaseEstimator and optionally the mixin classes in sklearn.base. For example, below is a custom classifier, with more examples included in the scikit-learn-contrib project template.

The goal would be to honor our hold duck typing contract which I find fundamental to the spirit of the scikit-learn API design

IMHO, having merged #16950 does not violate our duck typing contract: estimators implementing fit, predict etc will work perfectly with the internal tools like cross_validate, etc.

What #16950 does is that it forces estimators to inherit from BaseEstimator in order to run check_estimator. Which is quite different, and I would say it's a reasonable constraint.

We don't force a dependency on sklearn for third-party libraries, we just force a dependency on sklearn for their developers.

So I'd be +0 to bring this back. If we do, let's please explicitly write a comment indicating that this should not be removed, possibly with a link here

is that it forces estimators to inherit from BaseEstimator in order to run check_estimator.

A lot of projects wouldn't want to add scikit-learn as a dependency (e.g. lightfgbm, scorch) which in turns means they have no way to way to check that their API is compliant.

+1 to put it back, especially that's it's a minor change on our side.

@NicolasHug here (https://scikit-learn.org/stable/developers/develop.html#rolling-your-own-estimator) it says that in order to be a scikit-learn compatible estimator, an estimator must pass check_estimator

Though yes, they could actually define a _get_tags manually. If they don't set tags and don't inherit from sklearn, the determination of regressor vs classifier for tests won't work anyway, would it?

Edit: indeed, is_classifier or is_regressor helpers won't produce anything meanigful with arbitrary third party python classes at present. And then the risk is that check_estimator might pass, but not actually run any of the relevant checks which is another reason I think check_estimator should be avoided in favor of parametrize_with_checks #18750 (comment)

Edit: indeed, is_classifier or is_regressor helpers won't produce anything meanigful with arbitrary third party python classes at present. And then the risk is that check_estimator might pass, but not actually run any of the relevant checks..

We could add a check that fails if the estimator has a predict method but both is_classifier and is_regressor return False, WDYT?

@NicolasHug here (scikit-learn.org/stable/developers/develop.html#rolling-your-own-estimator) it says that in order to be a scikit-learn compatible estimator, an estimator must pass check_estimator

I understand that but I'm not sure what your point is @jeremiedbb ?

Mine is that scikit-learn is not a dependency for users of third-party libraries. It's only a dependency for their developers. And as the comments above suggest, I doubt that one can get anything meaningful out of check_estimator without inheriting from at least some of our Mixins. (It's the same for parametrize_with_checks BTW)

But I'm not opposing to adding _safe_tags back anyway.

which is another reason I think check_estimator should be avoided in favor of parametrize_with_checks

+1 for pushing more 3rd party libraries to use parametrize_with_checks if they use pytest but check_estimator is still useful for interactive checks / demos in ipython sessions or jupyter notebooks.

I understand that but I'm not sure what your point is @jeremiedbb ?

@NicolasHug I'm feeling that neither us and our doc is sure about what it requires for a third party estimator to be scikit-learn compat. In the doc it says it needs to pass check_estimator. But in order check_estimator kind of require a dependency on sklearn which we don't want to enforce. My comment was just that I'm a bit confused here :)

I'm -0 on this one. I see the tags as a part of our API, and estimators should implement them. I'd say if users want to pass check_estimator, they should also inherit from the right classes. They can choose not to pass these tests and their estimators may work w/o an issue in most usecases, but they are not completely scikit-learn compatible.

This PR adds IMO unnecessary complexity to our codebase. This means we should always use _safe_tags instead of getting the tags, everywhere in the codebase, and that I don't see why that's necessary where we can avoid if if we require users to inherit from the right class or implement everything they should, themselves.

This means we should always use _safe_tags instead of getting the tags, everywhere in the codebase

This a very good point. If we were limiting the use of tags in the common test, we would not have this issue. However, we started to use the tag elsewhere in the estimators themselves.

As I mentioned there, #18798 (comment), if we want solely to make tags part of our API, we might want to isolate this functionality in a Mixin. I am not sure this a good solution though; it will force to already make multiple inheritances in the common case (i.e. BaseEstimator, TagsMixin)

Do you have a case where the user would want to inherit TagsMixin but not BaseEstimator?

I'd say if users want to pass check_estimator, they should also inherit from the right classes
[..] or implement everything they should, themselves.

I don't think we should suggest that they inherit from scikit-learn. Being compatible with scikit-learn API is and should be unrelated to depending on scikit-learn.

However yes, _safe_tags also has its issues. We can provide default tags, but will they be appropriate for the estimator in question? Possibly but not sure until someone actually looks into tags, so they might as well re-implement them. However then for contrib projects the issue is that they would need to also implement tag inheritance via _get_tags, if they have many estimators to avoid having repeated N tags in each estimator.

So in that sense maybe _safe_tags is still useful as it would minimize the amount of work a contrib project would need to do (only tags that are different from defaults).

This means we should always use _safe_tags instead of getting the tags, everywhere in the codebase,

Not everywhere, just in common tests and meta-estimators. We did it before. It's certainly not ideal, but also not such a big deal maybe? Though the issue there is that is can start to be used in contrib projects as well, since they would find it as the way of getting tags in our code base :/

It would be someone that needs to redefine set/get/params/states and does not want to check_X_y within _validate_data. Of course, they could inherit their base class from it and overwrite these methods but the easiest way would be to implement your base from scratch for your use case. This looks like the use-case of cuml but I agree that it is not the most common and expected use case.

Even if we were to accept a third-party estimator as scikit-learn compatible w/o them inheriting from the right classes, I don't see why we'd need to do extra work for them to be compatible while they don't implement the API we require. This puts severe burden on us moving forward while designing our API.

I don't understand why we introduce this PR: the goal seems to be that we don't want to force third-party libraries to depend on scikit-learn to pass the check suite.

But to pass check_estimator, one needs to call check_estimator, so they need to have scikit-learn, right? Now if their estimator MyEst doesn't inherit from BaseEstimator, all they need to do is

class MyWrappedEst(BaseEstimator, MyEst):
     pass

and call check_estimator on MyWrappedEst instead of MyEst

BTW, I agree with most of what @adrinjalali said but I don't think this is true:

This means we should always use _safe_tags instead of getting the tags, everywhere in the codebase

This is only related to the check suite, not the rest of the code-base

But to pass check_estimator, one needs to call check_estimator, so they need to have scikit-learn, right?

test dependencies are not necessarily runtime dependencies.

test dependencies are not necessarily runtime dependencies.

Yes this is what I'm trying to say since the beginning: currently in master, we don't force third party libraries to have scikit-learn as a runtime dependency.

This PR doesn't change anything w.r.t. dependency. All it does is removing the need for estimators to inherit from BaseEstimator in order to pass check_estimator.

My understanding is that check_estimator is a tool that can be used by third party estimators to check that such an estimator follows the scikit-learn api and can be used in scikit-learn model evaluation and model selection tools. So we don't want to force an estimator to inherit from BaseEstimator to pass check_estimator.

On the other hand, if a third party estimator doesn't want to inherit from BaseEstimator, it needs to implement all important parts of the scikit-learn api. tags is a part of that (as listed here https://scikit-learn.org/stable/developers/develop.html#rolling-your-own-estimator).

So I join Nicolas and Adrin and I don't think this PR is necessary since it's clear in the docs that implementing tags is mandatory to be a compatible estimator.

this goes against our docs:
> Note however that all tags must be present in the dict

I think it's coherent with the proposed changes to the doc:

 you will need to implement a `_get_tags()` method which returns a dict that
  should contains all the necessary tags for that estimator, including the
  default tags typically defined in :class:`~sklearn.base.BaseEstimator` and
  other scikit-learn mixin classes. Note however that **all tags must be
  present in the dict**. If any of the keys documented above is not present in
  the output of `_get_tags()`, an error might occur.

I don't think I agree: when doing estimator._get_tags().get(key, _DEFAULT_TAGS[key]), the following isn't true anymore:

Note however that all tags must be
present in the dict. If any of the keys documented above is not present in
the output of _get_tags(), an error might occur

No error will ever be raised if a tag isn't returned by _get_tags(), and so we don't require all keys to exist. In effect, implementing _get_tags() is exactly the same as implementing _more_tags() for 3rd parties

Strictly following the docs would mean doing estimator._get_tags()[key] (which is what I would prefer).

Right I did not understand what you were reffering to in the first place. I agree that the code does not reflect the doc currently. Guillaume is working on it :)

In other words, do we even want to document "you can also override _get_tags()"? We could just say "you need to define _more_tags() if you want to override the defaults, and if you want to access tags values that you don't override (i.e. that are not in your own-defined _more_tags()), you'll need to inherit from BaseEstimator."

How do you deal with CuML case: inheriting is not an option. If they want to use tags (for new checks for instance) internally, we are forcing them to call _more_tags instead of their own implementation of _get_tags.

IMO, it is not a burden to mention that if you want to access your tags by implementing _get_tags you need to have all scikit-learn defaults because we are going to raise error otherwise.

@NicolasHug would it be fine with you if we merge this PR as you are fine with the code. This would allow us to branch 0.24.X and start the release PR for 0.24.0rc1.

We can always fine tune the doc before 0.24.0 final if needed.

Merged thanks all!