HistGradientBoosting* and DecisionTree* (native categorical support) both encode a categorical column such that a category never seen during fit is mapped to the exact same internal value as an actual missing value, inheriting whatever meaning missing values happened to have during training. Let's call this policy A.
This is fine when missingness carries no signal. When it does (a common real-world pattern), it's actively harmful: a brand new, unrelated category at prediction time gets confidently assigned the missing-value prediction instead of an uncertain/neutral one.
I experimented with an alternative policy (B): consider missing values as a category of its own (so just a normal category) and encode it as an integer, and encode unknown values as NaNs.
I could not find a real case example where B underperforms A, but I did find an example where B outperforms A
(KDD cup 2009 dataset, churn target, select one single column with informative missing values as X)
I propose to adopt policy B. It's a small behavior change but I don't think it can break or hurt any existing user code.
Note that we made a bigger and ill-documented one on HGB native categorical support and nobody complained... (and this one could have terribly hurt some ML pipelines).
I think the current impact is low because the likelihood of having unknown categories is low for categorical features of at most 255 distinct categories (which is the current limitation for native categorical support). But with #33972, this 255 limitation is removed for extra-trees (which is great) and the impact of policy A vs B becomes more important.
Note: implementing policy B cleanly with an OrdinalEncoder is currently not possible, which is one of the thing that makes me think that we should move forwards on those two issues:
HistGradientBoosting*andDecisionTree*(native categorical support) both encode a categorical column such that a category never seen duringfitis mapped to the exact same internal value as an actual missing value, inheriting whatever meaning missing values happened to have during training. Let's call this policy A.This is fine when missingness carries no signal. When it does (a common real-world pattern), it's actively harmful: a brand new, unrelated category at prediction time gets confidently assigned the missing-value prediction instead of an uncertain/neutral one.
I experimented with an alternative policy (B): consider missing values as a category of its own (so just a normal category) and encode it as an integer, and encode unknown values as NaNs.
I could not find a real case example where B underperforms A, but I did find an example where B outperforms A
(KDD cup 2009 dataset, churn target, select one single column with informative missing values as X)
I propose to adopt policy B. It's a small behavior change but I don't think it can break or hurt any existing user code.
Note that we made a bigger and ill-documented one on HGB native categorical support and nobody complained... (and this one could have terribly hurt some ML pipelines).
I think the current impact is low because the likelihood of having unknown categories is low for categorical features of at most 255 distinct categories (which is the current limitation for native categorical support). But with #33972, this 255 limitation is removed for extra-trees (which is great) and the impact of policy A vs B becomes more important.
Note: implementing policy B cleanly with an
OrdinalEncoderis currently not possible, which is one of the thing that makes me think that we should move forwards on those two issues:OrdinalEncodersurprising behavior: missing values are treated as unknown categories first #34387