DOC improve metadata routing example #27357

StefanieSenger · 2023-09-13T12:30:43Z

What does this implement/fix? Explain your changes.

This PR aims to improve the Metadata Routing example for clarity and readability. I've also enhanced two error messages in sklearn/utils/_metadata_requests.py to make them better understandable for inexperienced users.

I feel that this example is already pretty comprehensible and contains all the information needed (though it's a slow read). I've mainly worked on wording and variable naming.

I'm looking forward to your reviews, @adrinjalali and @glemaitre. :)

Any other comments?

Edit: It was a draft, but now it's ready to be a real PR.

github-actions · 2023-09-13T12:32:24Z

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 7e53771. Link to the linter CI: here}

sklearn/utils/_metadata_requests.py

examples/miscellaneous/plot_metadata_routing.py

StefanieSenger · 2023-09-13T22:32:41Z

examples/miscellaneous/plot_metadata_routing.py

-#     meta_est.fit(X, y, aliased_sample_weight=my_weights):
-#         ...  # this estimator (est), expects aliased_sample_weight as seen above
+#     meta_meta_est.fit(X, y, aliased_sample_weight=my_weights):
+#         ...  # this estimator (meta_est), expects aliased_sample_weight as seen above
 #         self.estimator_.fit(X, y, aliased_sample_weight=aliased_sample_weight):
-#             ...  # now est passes aliased_sample_weight's value as sample_weight,
+#             ...  # now meta_est passes aliased_sample_weight's value as sample_weight,
 #                  # which is expected by the sub-estimator
 #             self.estimator_.fit(X, y, sample_weight=aliased_sample_weight)
 #    ...


I'm very confused about this part. Do the comments belong to the code lines below or above them? Had est and meta_est been used correctly? What are the ... supposed to mean?

In the next push, I'll suggest to change that part the way I would think it was meant. Please, let me know if this makes sense.

StefanieSenger · 2023-09-13T23:30:05Z

examples/miscellaneous/plot_metadata_routing.py

-est.fit(X, y, sample_weight=my_weights, clf_sample_weight=my_other_weights)
+meta_est.fit(
+    X, y, sample_weight=my_weights, clf_sample_weight=my_other_weights
+)  # <-- `meta_clf_sample_weight=my_weights`?


I find it very confusing why this would throw an error:

meta_est.fit(X, y, meta_clf_sample_weight=my_weights, clf_sample_weight=my_other_weights)

From the phrase above, I don't understand the reasoning and also its wording ("only needs") it doesn't seem so strict that it would not be possible to to otherwise.

I find it very confusing why this would throw an error:

Yes, that is confusing, we agree. But consumers do no validation and routing, hence they need non-aliased arguments. The routing information is only for the router that they'll be a part of, not for themselves.

The "only needs" refers to aliased metadata instead of non-aliased ones. As in, the aliases are relevant only for sub-estimators of the meta-estimator.

Okay, understood. Though I'm afraid this will be a point where every user stumbles.

StefanieSenger · 2023-09-13T23:37:44Z

examples/miscellaneous/plot_metadata_routing.py

-# - Alias only on the sub-estimator. This is useful if we don't want the
-#   meta-estimator to use the metadata, and we only want the metadata to be used
-#   by the sub-estimator.
-est = RouterConsumerClassifier(
+# - Alias only on the sub-estimator:
+#
+# This is useful when we don't want the meta-estimator to use the metadata, but
+# the sub-estimator should.
+meta_est = RouterConsumerClassifier(
    estimator=ExampleClassifier().set_fit_request(sample_weight="aliased_sample_weight")
-).set_fit_request(sample_weight=True)
-print_routing(est)
+).set_fit_request(
+    sample_weight=True
+)  # <-- why is `sample_weight=True` if `meta_est` shouldn't consume it?
+print_routing(meta_est)


I don't understand the intend here.

Why not simply set the sub-estimators set_fit_requests to True and the meta-estimator's to False? Why do we need an alias here at all? Why is sample_weight=True if the meta-estimator shouldn't consume it?

If we do True and False here, then in effect you're saying sample_weight should be passed to the meta-estimator, and also shouldn't. Note that the consumer doesn't do any validation / routing, so if it receives sample_weight, it would use it. If we want to pass something which only goes to the sub-estimator, then it needs to have another name. You could also set sample_weight=False for the meta-estimator, that's irrelevant here.

Then let's have it like this:

meta_est = RouterConsumerClassifier( estimator=ExampleClassifier().set_fit_request(sample_weight="aliased_sample_weight")

This way, it's less confusing on the first look and I'll add a sentence explaining the special case.

examples/miscellaneous/plot_metadata_routing.py

adrinjalali

Thanks @StefanieSenger

examples/miscellaneous/plot_metadata_routing.py

adrinjalali · 2023-09-14T11:43:47Z

examples/miscellaneous/plot_metadata_routing.py

-        # meta-estimators are responsible for validating the given metadata.
-        # `get_routing_for_object` is a safe way to construct a
-        # `MetadataRouter` or a `MetadataRequest` from the given object.
+        # `get_routing_for_object` returns a copy of the MetadataRouter


Suggested change

# `get_routing_for_object` returns a copy of the MetadataRouter

# `get_routing_for_object` returns a copy of the `MetadataRouter`

also, I think it's important to mention that the function can return a MetadataRequest if the object is a consumer.

Yes, this comes a little below:

# First, the :meth:`~utils.metadata_routing.get_routing_for_object` takes a # meta-estimator (``self``) and returns a # :class:`~utils.metadata_routing.MetadataRouter` or, a # :class:`~utils.metadata_routing.MetadataRequest` if the object is a consumer, # based on the output of the estimator's ``get_metadata_routing`` method.

Here in this case, it's a MetadataRouter.

examples/miscellaneous/plot_metadata_routing.py

sklearn/utils/_metadata_requests.py

StefanieSenger

Thank you for reviewing, @adrinjalali. Now, I've gone through the whole document again and resolved everything. There are no open questions from my side.
I'd be happy if you could take another look. :)

StefanieSenger · 2024-01-04T10:10:22Z

examples/miscellaneous/plot_metadata_routing.py

-est.fit(X, y, sample_weight=my_weights, clf_sample_weight=my_other_weights)
+meta_est.fit(
+    X, y, sample_weight=my_weights, clf_sample_weight=my_other_weights
+)  # <-- `meta_clf_sample_weight=my_weights`?


Okay, understood. Though I'm afraid this will be a point where every user stumbles.

StefanieSenger · 2024-01-04T10:24:00Z

examples/miscellaneous/plot_metadata_routing.py

-# - Alias only on the sub-estimator. This is useful if we don't want the
-#   meta-estimator to use the metadata, and we only want the metadata to be used
-#   by the sub-estimator.
-est = RouterConsumerClassifier(
+# - Alias only on the sub-estimator:
+#
+# This is useful when we don't want the meta-estimator to use the metadata, but
+# the sub-estimator should.
+meta_est = RouterConsumerClassifier(
    estimator=ExampleClassifier().set_fit_request(sample_weight="aliased_sample_weight")
-).set_fit_request(sample_weight=True)
-print_routing(est)
+).set_fit_request(
+    sample_weight=True
+)  # <-- why is `sample_weight=True` if `meta_est` shouldn't consume it?
+print_routing(meta_est)


Then let's have it like this:

meta_est = RouterConsumerClassifier( estimator=ExampleClassifier().set_fit_request(sample_weight="aliased_sample_weight")

This way, it's less confusing on the first look and I'll add a sentence explaining the special case.

adrinjalali · 2024-01-22T13:22:27Z

CI Failing.

…Senger/scikit-learn into doc_developing_metadata_routing

StefanieSenger · 2024-01-26T09:53:23Z

This should fix the CI failure.
@adrinjalali, please have a look into this PR.

adrinjalali

Pff, long one, nice. A few points, but mostly looks okay.

adrinjalali · 2024-02-06T11:43:48Z

examples/miscellaneous/plot_metadata_routing.py

+# metadata routing as a consumer and how a meta-estimator can be upgraded to be
+# a router. Imagine a simple classifier accepting ``sample_weight`` as a
+# metadata on its ``fit`` and ``groups`` in its ``predict`` method:


I'm not sure if "upgrade" is the right notion here

here we're only talking about a consumer, I don't understand why you've added the router part in the description here O_o

Agreed, I will move this sentence down.

examples/miscellaneous/plot_metadata_routing.py

adrinjalali · 2024-02-06T11:54:06Z

examples/miscellaneous/plot_metadata_routing.py

+        # Meta-estimators are responsible for validating the given metadata.
+        # `MetadataRouter.validate_metadata` maps the given metadata to the
+        # metadata required by the underlying estimator. `method` refers to the


the mapping is done by route_params, not validate_metadata.

Yes, that makes sense.

adrinjalali · 2024-02-06T11:56:10Z

examples/miscellaneous/plot_metadata_routing.py

+        # For demonstration purpose, only one sub-estimator is fitted and its
+        # classes are attributed to the meta-estimator.


not really, only metaestimators like ColumnTransformer and Pipeline have more than one subestimator. Most other ones take only one subestimator.

StefanieSenger

@adrinjalali
I've modified like suggested.
Fingers crossed for the CI failure not appearing again.

Would you also like to have a look, @glemaitre?

StefanieSenger · 2024-02-06T12:27:54Z

examples/miscellaneous/plot_metadata_routing.py

+# metadata routing as a consumer and how a meta-estimator can be upgraded to be
+# a router. Imagine a simple classifier accepting ``sample_weight`` as a
+# metadata on its ``fit`` and ``groups`` in its ``predict`` method:


Agreed, I will move this sentence down.

StefanieSenger · 2024-02-06T13:02:24Z

examples/miscellaneous/plot_metadata_routing.py

+        # Meta-estimators are responsible for validating the given metadata.
+        # `MetadataRouter.validate_metadata` maps the given metadata to the
+        # metadata required by the underlying estimator. `method` refers to the


Yes, that makes sense.

StefanieSenger · 2024-02-06T13:06:03Z

examples/miscellaneous/plot_metadata_routing.py

+        # For demonstration purpose, only one sub-estimator is fitted and its
+        # classes are attributed to the meta-estimator.


Oh yes, now I can see that, too.

glemaitre

A couple of typo but it looks good. We will need #28371 to remove the current failure.

examples/miscellaneous/plot_metadata_routing.py

glemaitre · 2024-02-10T10:50:25Z

examples/miscellaneous/plot_metadata_routing.py

+parameter and forwards data and metadata, is also a router. A consumer, on the
+other hand, is an object which accepts and uses a certain given metadata. For
+instance, an estimator taking into account ``sample_weight`` in its :term:`fit`
+method is a consumer of ``sample_weight``. It is possible for an object to be


I would also break the paragraph when explaining that an estimator can consume and route data.

examples/miscellaneous/plot_metadata_routing.py

Co-authored-by: Guillaume Lemaitre <[email protected]>

glemaitre · 2024-02-19T10:04:10Z

So now, it is green. @adrinjalali you want to have a last pass on it.

glemaitre · 2024-03-05T18:53:38Z

So we need to fix something before to merge in main.

StefanieSenger · 2024-03-07T12:05:08Z

This PR is essentially waiting for #28517 to be merged. Then afterwards I will come back here. This is because I had originally tried to modify the UnsetMetadataPassedError here already, then did something better on the other PR. So, the _metadata_requests.py file and some of the test files will change, after merging the other PR.

The changes in metadata_routing.rst can still be reviewed, @ArturoAmorQ (I have seen that you've self requested a review today).

ArturoAmorQ · 2024-03-07T14:27:32Z

Ok, then I'll wait for that to happen. Thanks for letting me know, @StefanieSenger :)

StefanieSenger · 2024-04-09T12:29:29Z

@ArturoAmorQ: The other PR is now merged and this one here is cleaned up and up to date. You can now review this PR if you wish.

improve developing metadata routing example

1ebba28

github-actions bot added module:utils Documentation labels Sep 13, 2023

StefanieSenger commented Sep 13, 2023

View reviewed changes

sklearn/utils/_metadata_requests.py Show resolved Hide resolved

StefanieSenger commented Sep 13, 2023

View reviewed changes

sklearn/utils/_metadata_requests.py Outdated Show resolved Hide resolved

Merge branch 'main' into doc_developing_metadata_routing

7149120

StefanieSenger commented Sep 13, 2023

View reviewed changes

examples/miscellaneous/plot_metadata_routing.py Outdated Show resolved Hide resolved

StefanieSenger commented Sep 13, 2023

View reviewed changes

examples/miscellaneous/plot_metadata_routing.py Show resolved Hide resolved

StefanieSenger commented Sep 13, 2023

View reviewed changes

examples/miscellaneous/plot_metadata_routing.py Show resolved Hide resolved

StefanieSenger and others added 2 commits September 14, 2023 02:18

little changes after going through again

2782bcf

Merge branch 'main' into doc_developing_metadata_routing

d7752db

adrinjalali reviewed Sep 14, 2023

View reviewed changes

Merge branch 'main' into doc_developing_metadata_routing

00eaabe

StefanieSenger mentioned this pull request Jan 3, 2024

DOC Improve User Guide for metadata routing #27282

Merged

StefanieSenger and others added 2 commits January 3, 2024 15:27

Merge branch 'main' into doc_developing_metadata_routing

0ac12a0

changes after review

0fb417c

StefanieSenger commented Jan 4, 2024

View reviewed changes

StefanieSenger marked this pull request as ready for review January 4, 2024 12:09

StefanieSenger and others added 2 commits January 4, 2024 14:46

adjusted error message in tests

fb3f45c

Merge branch 'main' into doc_developing_metadata_routing

c9d6f72

StefanieSenger added 2 commits January 26, 2024 10:50

fix failing regex match test

85da957

Merge branch 'doc_developing_metadata_routing' of github.com:Stefanie…

de91b58

…Senger/scikit-learn into doc_developing_metadata_routing

adrinjalali reviewed Feb 6, 2024

View reviewed changes

Merge branch 'main' into doc_developing_metadata_routing

906937f

improvements after review

e22bf6d

StefanieSenger commented Feb 6, 2024

View reviewed changes

glemaitre added the No Changelog Needed label Feb 10, 2024

glemaitre self-requested a review February 10, 2024 10:33

Merge branch 'main' into doc_developing_metadata_routing

e543310

glemaitre approved these changes Feb 10, 2024

View reviewed changes

StefanieSenger and others added 6 commits February 12, 2024 11:47

Apply suggestions from code review

22a5599

Co-authored-by: Guillaume Lemaitre <[email protected]>

line breaks intro paragraph

d49ce16

solve merge conflict

8b34af0

Merge branch 'main' into doc_developing_metadata_routing

2cadd8b

correct output in rst file

f465864

Merge branch 'main' into doc_developing_metadata_routing

7282848

StefanieSenger added a commit to StefanieSenger/scikit-learn that referenced this pull request Feb 23, 2024

disable routing like in scikit-learn#27357

497ec93

StefanieSenger and others added 3 commits February 23, 2024 14:18

Merge branch 'main' into doc_developing_metadata_routing

e49b7d5

handle test failure

655c938

Merge branch 'main' into doc_developing_metadata_routing

81b9bdb

ArturoAmorQ self-requested a review March 7, 2024 09:37

StefanieSenger added 2 commits April 8, 2024 13:01

Merge branch 'main' into doc_developing_metadata_routing

ab226a3

Merge branch 'main' into doc_developing_metadata_routing

7e53771

StefanieSenger added a commit to StefanieSenger/scikit-learn that referenced this pull request Apr 12, 2024

disable routing like in scikit-learn#27357

d228816

adrinjalali approved these changes Apr 23, 2024

View reviewed changes

adrinjalali merged commit 9985c0b into scikit-learn:main Apr 23, 2024

	# `get_routing_for_object` returns a copy of the MetadataRouter
	# `get_routing_for_object` returns a copy of the `MetadataRouter`

		# For demonstration purpose, only one sub-estimator is fitted and its
		# classes are attributed to the meta-estimator.

Uh oh!

DOC improve metadata routing example #27357

DOC improve metadata routing example #27357

Uh oh!

Conversation

StefanieSenger commented Sep 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

github-actions bot commented Sep 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✔️ Linting Passed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

StefanieSenger Sep 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

StefanieSenger Sep 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adrinjalali left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

StefanieSenger left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adrinjalali commented Jan 22, 2024

Uh oh!

StefanieSenger commented Jan 26, 2024

Uh oh!

adrinjalali left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

StefanieSenger left a comment

Choose a reason for hiding this comment

StefanieSenger commented Sep 13, 2023 •

edited

Loading

github-actions bot commented Sep 13, 2023 •

edited

Loading

StefanieSenger Sep 13, 2023 •

edited

Loading

StefanieSenger Sep 13, 2023 •

edited

Loading