Thanks to visit codestin.com
Credit goes to github.com

Skip to content

chore(skore): Align default metrics with is_computable and discouraged - #3195

Draft
glemaitre wants to merge 5 commits into
probabl-ai:mainfrom
glemaitre:cursor/metrics-factory-1866
Draft

chore(skore): Align default metrics with is_computable and discouraged#3195
glemaitre wants to merge 5 commits into
probabl-ai:mainfrom
glemaitre:cursor/metrics-factory-1866

Conversation

@glemaitre

@glemaitre glemaitre commented Aug 4, 2026

Copy link
Copy Markdown
Member

Summary

  • Replace the task-branching default_metrics factory with a flat DEFAULT_METRICS catalog.
  • Rename Metric.availableMetric.is_computable (technical: can we compute it?).
  • Add Metric.discouraged(report) -> str | None (methodological: should we register it?). Defaults skip discouraged metrics silently; metrics.add(..., force=False) raises with the reason unless force=True.
  • Move the has_default_score display filter onto Score.discouraged, so sklearn's default score is no longer registered for LogisticRegression / RandomForest / etc.
  • Allow metrics.add(Score(), force=True) to put a discouraged score back into summarize / get; metrics.score() remains the dedicated escape hatch.

Behavior change

Defaults only register metrics that are both computable and not discouraged. User .add never checks is_computable; only discouraged can block it (unless force=True).

For estimators that use scikit-learn's default ClassifierMixin.score / RegressorMixin.score, "score" is skipped from the registry for the same reason (use report.metrics.score(), or add(Score(), force=True) to put it back).

Example

The point of discouraged is to refuse registering a metric that can be computed but should not be reported for the problem at hand. Override it on your metric class; .add raises with that reason unless the caller passes force=True:

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from skore import evaluate
from skore._sklearn.metrics import FunctionKind, Metric

class AccuracyOnImbalanced(Metric):
    """Accuracy that refuses to be registered on binary classification reports."""

    name = "accuracy_on_imbalanced"
    verbose_name = "Accuracy (imbalanced)"
    function = staticmethod(accuracy_score)
    response_method = "predict"
    greater_is_better = True
    function_kind = FunctionKind.METRIC

    @staticmethod
    def discouraged(report):
        if report._ml_task == "binary-classification":
            return (
                "accuracy is often misleading for binary classification; "
                "prefer precision, recall, or ROC AUC"
            )
        return None

X, y = load_breast_cancer(return_X_y=True)
report = evaluate(LogisticRegression(max_iter=10_000), X, y, splitter=0.2)

# Refused with the reason from discouraged().
report.metrics.add(AccuracyOnImbalanced())
# ValueError: Cannot add 'accuracy_on_imbalanced': accuracy is often misleading
# for binary classification; prefer precision, recall, or ROC AUC. Pass
# force=True to register it anyway.

# Opt in explicitly when you still want it.
report.metrics.add(AccuracyOnImbalanced(), force=True)
report.metrics.accuracy_on_imbalanced()
# 0.94...

Custom metrics without a discouraged override keep today's behavior: .add registers them unconditionally (still not filtered by is_computable).

Out of scope (flagged, not fixed here)

  • The neg_ alias claim in summarize / get docstrings that lookup does not implement
  • Metric.new silently dropping some arguments
  • MetricRow / MetricsSummaryRow duplication
  • Stale 'score' listing in accepted design record docs/design/0007-show-neg-when-used.md (left as history)

Test plan

  • Registry / discouraged / force-add unit tests
  • CV and comparison metrics.score() still work when score is not registered
  • Doctests on metrics accessors
  • No circular registry ↔ report reference (state size test)
  • Merged with main (#3196 registry-key row names kept via resolved.name)

@glemaitre
glemaitre marked this pull request as ready for review August 4, 2026 19:26
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Documentation preview @ e9c5b77

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report for skore/
FileStmtsMissBranchBrPartCoverMissing
skore/src/skore
   __init__.py4122095%118–119
   _config.py58312194%70, 117–118
   exceptions.py44000%4, 15, 19, 23
skore/src/skore/_plugins
   __init__.py12000100% 
   serde.py32080100% 
skore/src/skore/_plugins/hub
   __init__.py920077%15, 20
   exception.py2000100% 
   json.py1012190%16
   metric.py3426294%74, 82
skore/src/skore/_plugins/hub/artifact
   __init__.py0000100% 
   artifact.py23040100% 
   serializer.py27020100% 
   upload.py26040100% 
skore/src/skore/_plugins/hub/artifact/media
   __init__.py5000100% 
   data.py23000100% 
   inspection.py580100100% 
   media.py12000100% 
   model.py10000100% 
   performance.py57000100% 
skore/src/skore/_plugins/hub/artifact/pickle
   __init__.py2000100% 
   pickle.py24020100% 
skore/src/skore/_plugins/hub/authentication
   __init__.py0000100% 
   apikey.py7000100% 
   login.py2844285%37, 42–43, 52
   token.py80080100% 
   uri.py6000100% 
skore/src/skore/_plugins/hub/client
   __init__.py0000100% 
   client.py881018388%140, 187–189, 191–192, 194, 196, 198, 230
skore/src/skore/_plugins/hub/project
   __init__.py0000100% 
   project.py143628595%88, 113, 128, 327, 425, 455
skore/src/skore/_plugins/hub/report
   __init__.py3000100% 
   cross_validation_report.py185638396%185–189, 204
   estimator_report.py17000100% 
   report.py56040100% 
skore/src/skore/_plugins/local
   __init__.py2000100% 
   project.py250358298%455, 497, 518
skore/src/skore/_plugins/mlflow
   __init__.py5000100% 
   project.py28123721091%47, 49–51, 129, 236, 254, 272–273, 362, 413, 521, 523, 539, 541–546, 548–550
   reports.py160636596%129, 176, 214–215, 277, 288
skore/src/skore/_project
   __init__.py0000100% 
   _summary.py118148199%108
   dependencies.py19060100% 
   git.py30040100% 
   login.py1736282%57, 66–67
   plugin.py9000100% 
   project.py65118198%151
   protocol.py55000%3, 5, 7, 13, 16
   types.py4000100% 
skore/src/skore/_sklearn
   __init__.py8000100% 
   _base.py158642696%71, 222, 299, 308, 314, 349
   compare.py5000100% 
   evaluate.py440220100% 
   feature_names.py280120100% 
   find_ml_task.py610461100% 
   metrics.py4041104199%484
   train_test_split.py17000100% 
   types.py2010095%31
skore/src/skore/_sklearn/_checks
   __init__.py3000100% 
   _utils.py1260541100% 
   accessor.py350140100% 
   base.py1040282100% 
   model_checks.py4681152299%896
   tunable_hyperparameters.py5000100% 
skore/src/skore/_sklearn/_comparison
   __init__.py9000100% 
   inspection_accessor.py25020100% 
   metrics_accessor.py116226298%433, 688
   report.py191790696%276–277, 481, 600, 682–684
skore/src/skore/_sklearn/_cross_validation
   __init__.py11000100% 
   data_accessor.py40214295%49, 75
   inspection_accessor.py25020100% 
   metrics_accessor.py101518295%107–108, 110, 113, 637
   report.py205946595%230, 235, 246, 251, 353, 595, 736–738
skore/src/skore/_sklearn/_estimator
   __init__.py11000100% 
   data_accessor.py61128198%65
   inspection_accessor.py4218297%298
   metrics_accessor.py1160220100% 
   report.py3241088796%64, 80, 299, 381, 435, 685, 770, 865–867
skore/src/skore/_sklearn/_plot
   __init__.py3000100% 
   base.py75214197%72–73
   utils.py160376498%254–255, 454
skore/src/skore/_sklearn/_plot/data
   __init__.py2000100% 
   table_report.py189160299%727
skore/src/skore/_sklearn/_plot/inspection
   __init__.py0000100% 
   calibration_curve.py821114886%120–123, 125, 130–131, 234, 237–238, 268
   coefficients.py2020981100% 
   impurity_decrease.py104234398%444, 488
   permutation_importance.py201190199%659
   utils.py320100100% 
skore/src/skore/_sklearn/_plot/metrics
   __init__.py6000100% 
   confusion_matrix.py2000662100% 
   metrics_summary_display.py1760621100% 
   precision_recall_curve.py1180321100% 
   prediction_error.py1780582100% 
   roc_curve.py1220342100% 
skore/src/skore/_utils
   __init__.py620066%8, 13
   _accessor.py10620301181%13, 36, 61–65, 68, 70–71, 76, 81, 83, 85, 92–94, 164, 216, 236
   _cache.py37021100% 
   _cache_key.py38624684%22, 24, 26, 53, 61, 70
   _callable.py9040100% 
   _dataframe.py550220100% 
   _environment.py33110296%49
   _fixes.py8020100% 
   _index.py15080100% 
   _measure_time.py10000100% 
   _parallel.py17000100% 
   _patch.py21128842%30, 35–39, 42–43, 46–47, 58, 60
   _progress_bar.py4244090%53–54, 64–65
   _show_versions.py660220100% 
   _skrub.py4618197%37
   _testing.py1281412289%24, 33, 71–72, 94, 193, 202, 213–218, 220
   _uuid.py18410477%7, 17, 23, 26
   docscrape.py44212195%32–33
skore/src/skore/_utils/repr
   __init__.py2000100% 
   base.py54040100% 
   data.py1500361100% 
   html_repr.py40000100% 
   markdown.py580220100% 
   rich_repr.py940343100% 
   utils.py20020100% 
TOTAL7777214214214697% 

Tests Skipped Failures Errors Time
2911 3 💤 0 ❌ 0 🔥 4m 18s ⏱️

thomass-dev
thomass-dev previously approved these changes Aug 5, 2026

@thomass-dev thomass-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly i have no strong opinion on that.

Making the availability tests in a factory is the same as delegating to each metric.available.
The only benefit is one compute in the factory vs re-doing many computes in each metric.available.

LGTM, wait for the @auguste-probabl review.

@auguste-probabl auguste-probabl left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I understand it, the point of default_metrics is to allow more flexibility in defining which metrics should be shown: available tells you if the metric is technically computable, whereas default_metrics would tell you what we maintainers recommend showing. The thing is:

  • I can't think of a motivating example for this change. What would you like to do with this that you can't do with available?
  • If we accept that we need this distinction, I don't see why that information couldn't be defined within each metric class, e.g. with a new recommended method.

Comment thread skore/src/skore/_sklearn/metrics.py Outdated
list of :class:`Metric`
The metric instances to register, in default display order.
"""
ml_task = report._ml_task

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inline

@glemaitre

Copy link
Copy Markdown
Member Author

available for me is related to what we had with the available_if in scikit-learn. It answers to the question "is this metric can be computed on the given predictions".

The factory with the default choices is different semantic to me. It answers to the following question "From the available metric for the problem, is this particular metric advisable to be used given the problem, data, and model at hand".

So in short, available is for a technical/engineering purpose while the factory is for a methodological purpose.

@glemaitre

Copy link
Copy Markdown
Member Author

I need to think a bit more for this problem. Basically, what I don't like here is more the language actually: built-in and available are the two terms that I'm concerned with.

However, the fact that the metric itself, given the report, can tell you what to do (that could be extended for which reason), is actually nice.

I'll think about more about the metric API to see if we can make this distinction between the available and for instance something like is_recommended. Registering a metric will be a combination of the two and is_recommended could return a sort of reason why it is not recommended.

@thomass-dev

Copy link
Copy Markdown
Collaborator

@glemaitre if a user adds a metric, do you want available to filter it out or you want it in the summary whether it's computable or not?

@glemaitre
glemaitre marked this pull request as draft August 5, 2026 13:00
@glemaitre

Copy link
Copy Markdown
Member Author

So I'm putting in draft mode because it seems that it is not as urgent as what I thought.

@glemaitre

Copy link
Copy Markdown
Member Author

@glemaitre if a user adds a metric, do you want available to filter it out or you want it in the summary whether it's computable or not?

My last thought where that the .add could have a parameter that would bypass or not a discouraged function that could return the reason for which a metric should not be added.

If the user is bypassing this, the metric is added in the registry and will be computed. If not bypassing the check, then we raise an error with the reason. In the factory, we could add any available method and make sure it does not raise any error then.

@glemaitre glemaitre changed the title chore(skore): Replace BUILTIN_METRICS with a task-driven metrics factory chore(skore): Align default metrics with is_computable and discouraged Aug 5, 2026
@glemaitre

Copy link
Copy Markdown
Member Author

I open #3202 to isolate the part that should not require discussion for the moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants