FEAT allow RFE(CV) be used with pemutation_importance
#32251
Conversation
pemutation_importance
There was a problem hiding this comment.
Thanks for the PR. This is very useful. Please indeed update the example and add a changelog entry.
EDIT: here are the instructions for the changelog entry: https://github.com/scikit-learn/scikit-learn/blob/main/doc/whats_new/upcoming_changes/README.md
| This allows methods like :func:`permutation_importance` to extract the relevant features | ||
| from its test set. |
There was a problem hiding this comment.
| This allows methods like :func:`permutation_importance` to extract the relevant features | |
| from its test set. | |
| This allows methods like :func:`permutation_importance` and similar tools to | |
| iteratively extract the previously selected features from a test set. |
There was a problem hiding this comment.
yeah I realized it was not very clear so I planned to change it to:
" This allows methods that need a test set, like :func:permutation_importance, to know which
features of to use in their predictions."
There was a problem hiding this comment.
I found a small problem in the example + small suggestions for further improvement in the docstrings.
Also @ArturoAmorQ might be interested in reviewing this PR (the updated example in particular).
| If `callable`, overrides the default feature importance getter. | ||
| The callable is passed with the fitted estimator and it should | ||
| return importance for each feature. | ||
| return importance for each feature. When it accepts it, the callable is passed |
There was a problem hiding this comment.
| return importance for each feature. When it accepts it, the callable is passed | |
| return importance for each feature. When it accepts it, the callable is passed |
| return importance for each feature. | ||
| return importance for each feature. When it accepts it, the callable is passed | ||
| `feature_indices` which stores the index of the features in the full dataset | ||
| that have not been eliminated yet. |
There was a problem hiding this comment.
| that have not been eliminated yet. | |
| that have not yet been eliminated in previous iterations. |
| return importance for each feature. | ||
| return importance for each feature. When it accepts it, the callable is passed | ||
| `feature_indices` which stores the index of the features in the full dataset | ||
| that have not been eliminated yet. |
There was a problem hiding this comment.
| that have not been eliminated yet. | |
| that have not yet been eliminated in previous iterations. |
| shown at the end of | ||
| :ref:`sphx_glr_auto_examples_feature_selection_plot_rfe_with_cross_validation.py`. | ||
| .. versionadded:: 0.24 |
There was a problem hiding this comment.
I think we should add a .. versionchanged:: 1.8 and mention that support for passing feature_indices to the callable when part of its signature.
There was a problem hiding this comment.
And similarly for the docstring of the other class.
| n_classes=8, | ||
| n_clusters_per_class=1, | ||
| class_sep=0.8, | ||
| random_state=0, |
There was a problem hiding this comment.
We need to make sure that we do not sample from the training set:
| random_state=0, | |
| random_state=1, # Use a different seed to sample different points. |
Reference Issues/PRs
Fixes #32201
What does this implement/fix? Explain your changes.
To be used in RFE and RFECV,
permutation_importanceneeds to be aware of which features were already eliminated by the procedure to reduce its test dataset.This PR adds a
feature_indicesparameter tosklearn.feature_selection._base._get_feature_importancesthat is given to theimportance_getterso that it is aware of which features to compute the importance of.Any other comments?
The new feature is added to the test suite and illustrated in the
RFECVdoc example.@glemaitre @ogrisel