DOC HalvingGridSearchCV/HalvingRandomSearchCV: clarify best_estimator_ reflects last halving iteration - #33723
Conversation
…_ reflects last halving iteration best_estimator_, best_score_, and best_params_ select the winner from the final halving iteration only (via the _select_best_index callable), not the globally highest-ranked entry in cv_results_. The global rank_test_score can show earlier-iteration candidates ranked scikit-learn#1 because ranking spans all iterations, but those scores come from fewer resources and are less reliable. Added a .. note:: block to best_estimator_ explaining the distinction and showing how to filter cv_results_ by the final iteration. Also clarified best_score_ and best_params_ docstrings to make this explicit. Closes scikit-learn#24901 Built by Rudrendu Paul, developed with Claude Code
There was a problem hiding this comment.
Thanks for this documentation enhancement, @RudrenduPaul.
best_estimator_ etc. in Halving*Search are indeed only chosen from within the last halving iteration and it is helpful to be clear about it in the docs. I think these changes are clear and understandable.
betatim
left a comment
There was a problem hiding this comment.
Some small clean up comment and one about shortening the docstring a bit if possible.
Otherwise I like it
| on the left out data. Not available if ``refit=False``. | ||
| Estimator that was chosen by the search, i.e. estimator which gave | ||
| the highest score (or smallest loss if specified) on the left out data | ||
| **in the last halving iteration**. Not available if ``refit=False``. |
There was a problem hiding this comment.
| **in the last halving iteration**. Not available if ``refit=False``. | |
| in the last halving iteration. Not available if ``refit=False``. |
Not sure we need this
| Estimator that was chosen by the search, i.e. estimator | ||
| which gave highest score (or smallest loss if specified) | ||
| on the left out data. Not available if ``refit=False``. | ||
| Estimator that was chosen by the search, i.e. estimator which gave |
There was a problem hiding this comment.
Would it be possible to make the edit so that the unchanged lines are not changed? It helps keep the amount of churn in the git blame history low.
Same comment for the other instances where this happened.
Per @betatim review: restore original line wrapping to minimize git blame churn, remove bold from 'in the last halving iteration', and shorten the note by removing parenthetical about global ranking. Built by Rudrendu Paul, developed with Claude Code
|
Hi @RudrenduPaul, can you give us a summary feedback please? Did you integrate everything @betatim had suggested to you? |
Combine the two-line ``Not available if refit=False`` sentence back onto one line in both HalvingGridSearchCV and HalvingRandomSearchCV, exactly as betatim's inline suggestion shows. Built by Rudrendu Paul, developed with Claude Code
|
Hi @StefanieSenger and @betatim — apologies for the delayed follow-up. Summary of what I've integrated:
All three suggestions are now integrated. Happy to make any further adjustments. |
Remove "Not available if ``refit=False``." from best_estimator_ in both HalvingGridSearchCV and HalvingRandomSearchCV. This was flagged by betatim as "not sure we need this" and caused E501 lint violations (94 > 88 chars) introduced by the previous commit. Built by Rudrendu Paul, developed with Claude Code
|
Pushed a follow-up commit to fix an E501 linting error introduced in the previous commit. The merged line |
Co-authored-by: Tim Head <[email protected]>
|
I've added the last lacking line. I think this PR is now ready to merge, @betatim. Thanks a lot for your work, @RudrenduPaul! |
betatim
left a comment
There was a problem hiding this comment.
Thanks for improving the documentation and the last little fix
…_ reflects last halving iteration (scikit-learn#33723) Co-authored-by: Stefanie Senger <[email protected]> Co-authored-by: Tim Head <[email protected]> Co-authored-by: Stefanie Senger <[email protected]>
…_ reflects last halving iteration (#33723) Co-authored-by: Stefanie Senger <[email protected]> Co-authored-by: Tim Head <[email protected]> Co-authored-by: Stefanie Senger <[email protected]>
Description
Closes #24901.
HalvingGridSearchCVandHalvingRandomSearchCVuse a custom_select_best_indexcallable (line 194–214 of_search_successive_halving.py) that picks the best candidate from the final halving iteration only — not the globally best-ranked row incv_results_. However, the existing docstrings forbest_estimator_,best_score_, andbest_params_did not mention this, leading users to expect thatbest_params_matches therank_test_score == 1row incv_results_(which may come from an earlier iteration with fewer resources).Changes
best_estimator_: Added "in the last halving iteration" to the description + a.. note::block explaining why earlier-iteration candidates can outrank the final winner incv_results_, and showing how to filtercv_results_['iter'] == n_iterations_ - 1to see only final-iteration results.best_score_: Clarified it is "computed over the candidates from the last halving iteration only."best_params_: Added "in the last halving iteration" to the description.Both
HalvingGridSearchCVandHalvingRandomSearchCVshare identical docstrings for these attributes, so both were updated.Motivation
Discussed in #24901: multiple users were confused because
cv_results_['rank_test_score'] == 1pointed to an early-iteration candidate (with a high but unreliable score from a small sample), whilebest_params_pointed to a different set of parameters. The root cause is documented in the code but not in the public API documentation.