RFC Defining the auto-request strategy for scikit-learn estimators / sample_weight handling #34901

ACA Where do we want to use sample_weight by default?

With #31413 soon to be merged, we have defined the mechanism for the auto-routing strategy in scikit-learn (and for use in third party libraries). It will be easily possible to define per estimator which methods of the estimator will request which metadata if users set set_config(metadata_request_policy="auto"). (Context: At one point in the future this setting will become the default, but for now it is opt-in.)

The way to do this is to call add_auto_request inside an estimator's get_metadata_routing method:

class XYZEstimator(BaseEstimator):

    def get_metadata_routing(self):
        ...
        requests.fit.add_auto_request("sample_weight") # <--- one-liner to add auto-requests
        ...

The effect of adding an auto-request is that then XYZEstimator.fit() automatically requests and uses sample_weight if the user has passed it into the top-level meta-estimator or cross validation.

This issue is to define the scope of the auto-requests.

We have discussed this in our bi-weekly meeting on pushing the subject of auto-requests and here we want to give everybody who is interested the opportunity to share their ideas on the following questions:

• Which metadata should get auto-requested ("sample_weigh", "X_val", "sample_weight_val", custom metadata, other?)
• Should we add auto-requests to fit and score everywhere or are there estimators where that doesn't make any sense?
• Which type of tests do we need?

My personal take would be to work these questions out in separate POC PRs.

CC @scikit-learn/core-devs for gathering opinions. I'm will especially be interested in the opinions of the statistically more inclined core devs (since we can figure out the engineering part pretty easily). Happy to discuss this in calls also.