MNT more informative error message for `UnsetMetadataPassedError` #28517

StefanieSenger · 2024-02-23T10:36:33Z

What does this implement/fix? Explain your changes.

This PR aims to improve the information provided when UnsetMetadataPassedError is raised.

The error message will now explicitly state, that the method they are asked to set a metadata request is called from which method in the meta-estimator object.

More detailed explanation

This is especially important when a method in the router is routing to several methods in a consumer. See #28261 for RANSACRegressor as an example, where RANSACRegressor.fit() routes to both estimator.fit() and estimator.score(). The user will receive the following error:

sklearn.exceptions.UnsetMetadataPassedError: [sample_weight] are passed but are not explicitly set as requested or not requested for LinearRegression.score, which is used within RANSACRegressor.fit. Call LinearRegression.set_score_request({metadata}=True) for each metadata.

when they have done LinearRegression.set_fit_request(sample_weight=True), but where not aware that they need to do set_score_request(sample_weight=True) as well. At the moment, the error message they would receive in such a case would confuse many people.

Additionally, in case of composed methods like fit_transform and fit_predict the message is now improved to suggest to use set_fit_request, instead of setting the request for the composed method (which would result in an error).

It also works over several layers.

@adrinjalali :)

github-actions · 2024-02-23T10:37:47Z

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 96e42a0. Link to the linter CI: here}

adrinjalali

And I would like to see a test where we test basically this case:

est = MetaEstimator1(
    MetaEstimator2(
        Consumer()
    )
).fit(X, y, metadata="blah")

to see what the error message looks like.

doc/metadata_routing.rst

Co-authored-by: Adrin Jalali <[email protected]>

StefanieSenger · 2024-02-23T16:11:21Z

Thank you @adrinjalali.
I've added a test, please have a look.
metadata, sample_weight and so on are not explicitly translated in the error message. They come as a list and can be one or many.

adrinjalali

I think it's okay that the message only includes the immediate parent and not the whole tree, i.e. in the test, the message doesn't mention that the meta estimator was inside a pipeline.

adrinjalali · 2024-02-27T11:10:14Z

sklearn/utils/_metadata_requests.py

@@ -403,7 +403,7 @@ def _check_warnings(self, *, params):
                "warning, or to True to consume and use the metadata."
            )

-    def _route_params(self, params):
+    def _route_params(self, params, parent, caller_method):


the parent, caller order is not consistent here. I think parent, caller makes more sense than caller, parent. Also, caller would be enought, doesn't need to be caller_method.

True, I fixed that an now it's parent, caller everywhere.

adrinjalali · 2024-02-27T11:12:10Z

sklearn/utils/_metadata_requests.py

+            if self.method in COMPOSITE_METHODS:
+                callee_method = COMPOSITE_METHODS[self.method][0]


This does'nt seem right. The issuue might be in transform of a fit_transform, it's not always fit.

Also, the order of method in COMPOSITE_METHODS[self.method] shouldn't matter.

This was to prevent a message like that one:

sklearn.exceptions.UnsetMetadataPassedError: [metadata] are passed but are not explicitly set as requested or not requested for ConsumingTransformer.fit_transform, which is used within Pipeline.fit_transform. Call ConsumingTransformer.set_fit_transform_request({metadata}=True/False) for each metadata you want to request/ignore.

Where set_fit_transform_request() is wrongly suggested as existing.

If either set_fit_request or set_transform_request is set, the user would get this error message:

raise ValueError( f"Conflicting metadata requests for {', '.join(conflicts)} while" f" composing the requests for {name}. Metadata with the same name" f" for methods {', '.join(COMPOSITE_METHODS[name])} should have the" " same request value."

So here, we'd have to deal with the case that both composite methods don't have a metadata request set. I've modified the message that so both method requests are now displayed in it and I've also added a test for that.

sklearn/utils/_metadata_requests.py

adrinjalali · 2024-02-27T11:22:00Z

sklearn/utils/_metadata_requests.py

@@ -987,7 +1022,7 @@ def _route_params(self, *, params, method):
        res.update(child_params)
        return res

-    def route_params(self, *, caller, params):
+    def route_params(self, *, caller, params, parent=None):


I don't think we should have this default then, the user should always pass. If it's none, we'll fail.

Alright, and I have adjusted the places where it was used in the code, especially where the scorer functions are tested.

sklearn/tests/test_metadata_routing.py

StefanieSenger

@adrinjalali Thanks for your review.
I've addressed all your suggestions, please have another look.

sklearn/utils/_metadata_requests.py

StefanieSenger · 2024-03-01T12:21:14Z

sklearn/utils/_metadata_requests.py

@@ -403,7 +403,7 @@ def _check_warnings(self, *, params):
                "warning, or to True to consume and use the metadata."
            )

-    def _route_params(self, params):
+    def _route_params(self, params, parent, caller_method):


True, I fixed that an now it's parent, caller everywhere.

sklearn/tests/test_metadata_routing.py

StefanieSenger · 2024-03-01T14:53:53Z

sklearn/utils/_metadata_requests.py

+            if self.method in COMPOSITE_METHODS:
+                callee_method = COMPOSITE_METHODS[self.method][0]


This was to prevent a message like that one:

sklearn.exceptions.UnsetMetadataPassedError: [metadata] are passed but are not explicitly set as requested or not requested for ConsumingTransformer.fit_transform, which is used within Pipeline.fit_transform. Call ConsumingTransformer.set_fit_transform_request({metadata}=True/False) for each metadata you want to request/ignore.

Where set_fit_transform_request() is wrongly suggested as existing.

If either set_fit_request or set_transform_request is set, the user would get this error message:

raise ValueError( f"Conflicting metadata requests for {', '.join(conflicts)} while" f" composing the requests for {name}. Metadata with the same name" f" for methods {', '.join(COMPOSITE_METHODS[name])} should have the" " same request value."

So here, we'd have to deal with the case that both composite methods don't have a metadata request set. I've modified the message that so both method requests are now displayed in it and I've also added a test for that.

StefanieSenger · 2024-03-05T12:08:53Z

I have also slightly changed the error message raised in MetadataRouter.validate_metadata (which has caused me a lot of confusion today in another PR). The reason this error raises here, is that no child method expects those metadata, not that the user forgot the set_method_request.

CodeCov is failing, but I suspect that we can ignore that?

glemaitre · 2024-03-05T17:46:19Z

Yep codecov seems to be a false positive.

glemaitre

I'm happy with the improved error message.

However, I would like to make sure that the API is something that we want. I'm under the impression that parent is usuallay representing self and is actually the request.owner. But I might have overlooked something.

glemaitre · 2024-03-05T18:00:01Z

sklearn/utils/_metadata_requests.py

@@ -434,12 +440,28 @@ def _route_params(self, params):
            elif alias in args:
                res[prop] = args[alias]
        if unrequested:
+            if parent.__class__.__name__ != "str":


This should be documented in the docstring. Do you have a concrete example where parent is a string. Also what is the problem in this case?

I know this part is awkward. It's there to make sure we only look into the parent's class name if we use an actual scikit-learn-made parent class.
I had encountered problems with the scoring functions when used for routing metadata. In sklearn/metrics/tests/test_score_objects.py the name of the parent class would be str (not a string), if I remember correctly. I don't manage to recreate that anymore though, it had popped up when running tests, and I cannot figure out which one and how.

The change I made was this one.

At the moment, I don't know what to do about it.

Now this distinction is not necessary anymore, since all MetadataRouter.owner is always a string.

glemaitre · 2024-03-05T18:01:47Z

sklearn/utils/_metadata_requests.py

+            if parent.__class__.__name__ != "str":
+                parent = parent.__class__.__name__
+            if self.method in COMPOSITE_METHODS:
+                callee_methods = list(COMPOSITE_METHODS[self.method])


Do we need really need to call the list constructor?

You're right, it doesn't need to be wrapped into a list. I'm super surprised though, that both, list(COMPOSITE_METHODS[self.method]) and COMPOSITE_METHODS[self.method], behave in the same way.
They're both ['fit', 'transform'], and not [['fit', 'transform']] and ['fit', 'transform'] respectively ...

I've changed that in the code.

sklearn/utils/_metadata_requests.py

glemaitre · 2024-03-05T18:10:13Z

examples/miscellaneous/plot_metadata_routing.py

@@ -169,7 +169,9 @@ def fit(self, X, y, **fit_params):
        # we can use provided utility methods to map the given metadata to what
        # is required by the underlying estimator. Here `method` refers to the
        # parent's method, i.e. `fit` in this example.
-        routed_params = request_router.route_params(params=fit_params, caller="fit")
+        routed_params = request_router.route_params(
+            params=fit_params, caller="fit", parent=self.__class__


I'm confused here. I thought that parent should be self directly?

Another thing that I'm realising now: parent different from router.owner? I'm under the impression that this the is same information and thus we would not need to have parent.

I'm not sure about this. I will have to come back to this next week.

StefanieSenger

However, I would like to make sure that the API is something that we want. I'm under the impression that parent is usuallay representing self and is actually the request.owner. But I might have overlooked something.

I am not sure. Right now, I'm passing parent through the callstack within _metadata_request.py via:

li. 1609 process_routing()
li.1075 MetadataRouter.route_params()
li. 644 MetadataRequest._route_params() and
li. 464 MethodMetadataRequest._route_params()

I have to check if I can skip the first step and pick the information from MetadataRouter.self.owner. That might be well possible. However, there will be another problem for routers that are functions then.

StefanieSenger · 2024-03-07T12:45:04Z

sklearn/utils/_metadata_requests.py

@@ -434,12 +440,28 @@ def _route_params(self, params):
            elif alias in args:
                res[prop] = args[alias]
        if unrequested:
+            if parent.__class__.__name__ != "str":


I know this part is awkward. It's there to make sure we only look into the parent's class name if we use an actual scikit-learn-made parent class.
I had encountered problems with the scoring functions when used for routing metadata. In sklearn/metrics/tests/test_score_objects.py the name of the parent class would be str (not a string), if I remember correctly. I don't manage to recreate that anymore though, it had popped up when running tests, and I cannot figure out which one and how.

The change I made was this one.

At the moment, I don't know what to do about it.

StefanieSenger · 2024-03-07T13:46:19Z

sklearn/utils/_metadata_requests.py

+            if parent.__class__.__name__ != "str":
+                parent = parent.__class__.__name__
+            if self.method in COMPOSITE_METHODS:
+                callee_methods = list(COMPOSITE_METHODS[self.method])


You're right, it doesn't need to be wrapped into a list. I'm super surprised though, that both, list(COMPOSITE_METHODS[self.method]) and COMPOSITE_METHODS[self.method], behave in the same way.
They're both ['fit', 'transform'], and not [['fit', 'transform']] and ['fit', 'transform'] respectively ...

I've changed that in the code.

sklearn/utils/_metadata_requests.py

StefanieSenger · 2024-03-07T14:29:51Z

examples/miscellaneous/plot_metadata_routing.py

@@ -169,7 +169,9 @@ def fit(self, X, y, **fit_params):
        # we can use provided utility methods to map the given metadata to what
        # is required by the underlying estimator. Here `method` refers to the
        # parent's method, i.e. `fit` in this example.
-        routed_params = request_router.route_params(params=fit_params, caller="fit")
+        routed_params = request_router.route_params(
+            params=fit_params, caller="fit", parent=self.__class__


I'm not sure about this. I will have to come back to this next week.

…outer.owner

StefanieSenger

I went through it with fresh eyes and: yes! The parent param is the same as MetadataRouter.owner. We can take the information from there (which facilitates our test and example files) and as its a string, we don't even need the distinction before raising the UnsetMetadataPassedError. This solves several problems at once. Good that you hinted that, @glemaitre. Maybe you want to have another look now?

glemaitre · 2024-03-15T16:04:33Z

I'll have another look now.

glemaitre

LGTM. I would have maybe think of calling parent with another name but it seems that through the documentation this is the way that we chose so let's keep it.

adrinjalali

I'm still somewhat on the edge about this. But if y'all think it helps users, then okay.

Note that this only provides the parent info, and in nested cases that itself might not be enough. Example:

est = BaggingClassifier(Pipeline([("inner", LogisticRegression())]))
est.fit([[1], [1]], [1,1], sample_weight=[1])

...
UnsetMetadataPassedError: [sample_weight] are passed but are not explicitly set as requested or not requested for LogisticRegression.fit, which is used within Pipeline.fit. Call `LogisticRegression.set_fit_request({metadata}=True/False)` for each metadata you want to request/ignore.

The information about BaggingClassifier is not there. Happy to merge this, but I think we could do better.

more informative error message for UnsetMetadataPassedError

53255f1

github-actions bot added the module:utils label Feb 23, 2024

handle test errors

6934579

StefanieSenger changed the title ~~ENH more informative error message for UnsetMetadataPassedError~~ MNT more informative error message for UnsetMetadataPassedError Feb 23, 2024

StefanieSenger added 4 commits February 23, 2024 12:44

satisfied test

bef6606

handleing test errors

7c567f7

disable routing like in scikit-learn#27357

497ec93

fix doctest failure

fcb4325

adrinjalali added the No Changelog Needed label Feb 23, 2024

adrinjalali reviewed Feb 23, 2024

View reviewed changes

doc/metadata_routing.rst Outdated Show resolved Hide resolved

StefanieSenger and others added 2 commits February 23, 2024 16:36

Update doc/metadata_routing.rst

ea44b7e

Co-authored-by: Adrin Jalali <[email protected]>

added test

2c1e8d1

adrinjalali self-requested a review February 27, 2024 11:07

adrinjalali reviewed Feb 27, 2024

View reviewed changes

StefanieSenger added 2 commits March 1, 2024 14:11

changes after review

8ecb239

correct error message for composite methods

2073a27

StefanieSenger commented Mar 1, 2024

View reviewed changes

StefanieSenger and others added 4 commits March 1, 2024 17:17

adjust for parent param without default value

0ea9e4d

Merge branch 'main' into informative_set_request_error

e98be66

fix expected error message

5f7d734

clearer error message for TypeError in validate_metadata

f7bff85

glemaitre self-requested a review March 5, 2024 17:46

glemaitre reviewed Mar 5, 2024

View reviewed changes

StefanieSenger mentioned this pull request Mar 7, 2024

DOC improve metadata routing example #27357

Merged

StefanieSenger commented Mar 7, 2024

View reviewed changes

removed wrapping in list

24483d2

StefanieSenger added 2 commits March 12, 2024 16:26

move parent param one function down, get string directly by MetadataR…

cfec100

…outer.owner

same signature for methods with same name

e0eef01

StefanieSenger commented Mar 12, 2024

View reviewed changes

StefanieSenger and others added 2 commits March 15, 2024 15:14

Merge branch 'main' into informative_set_request_error

85bd091

fix test

f7614e2

glemaitre self-requested a review March 15, 2024 16:04

glemaitre approved these changes Mar 15, 2024

View reviewed changes

StefanieSenger and others added 2 commits April 8, 2024 12:58

Merge branch 'main' into informative_set_request_error

9c11d3e

satisfy linting

96e42a0

adrinjalali approved these changes Apr 8, 2024

View reviewed changes

adrinjalali merged commit 2f5361a into scikit-learn:main Apr 8, 2024

StefanieSenger deleted the informative_set_request_error branch April 9, 2024 11:30

antoinecollas mentioned this pull request May 24, 2024

Scikit-learn 1.5 breaks skada scikit-adaptation/skada#207

Closed

StefanieSenger mentioned this pull request Mar 18, 2025

MNT fix error message for UnsetMetadataPassedError in validation #31014

Merged

		if self.method in COMPOSITE_METHODS:
		callee_method = COMPOSITE_METHODS[self.method][0]

Uh oh!

MNT more informative error message for UnsetMetadataPassedError #28517

MNT more informative error message for UnsetMetadataPassedError #28517

Uh oh!

Conversation

StefanieSenger commented Feb 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this implement/fix? Explain your changes.

More detailed explanation

Uh oh!

github-actions bot commented Feb 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✔️ Linting Passed

Uh oh!

adrinjalali left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

StefanieSenger commented Feb 23, 2024

Uh oh!

adrinjalali left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

StefanieSenger left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

StefanieSenger commented Mar 5, 2024

Uh oh!

glemaitre commented Mar 5, 2024

Uh oh!

glemaitre left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

StefanieSenger left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MNT more informative error message for `UnsetMetadataPassedError` #28517

MNT more informative error message for `UnsetMetadataPassedError` #28517

StefanieSenger commented Feb 23, 2024 •

edited

Loading

github-actions bot commented Feb 23, 2024 •

edited

Loading