Thanks to visit codestin.com
Credit goes to github.com

Skip to content

META OHE / OrdinalEncoder: NaN support, unfrequent cat. and pd.Categorical #15796

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
rth opened this issue Dec 4, 2019 · 4 comments
Open

META OHE / OrdinalEncoder: NaN support, unfrequent cat. and pd.Categorical #15796

rth opened this issue Dec 4, 2019 · 4 comments
Labels
module:preprocessing

Comments

@rth
Copy link
Member

rth commented Dec 4, 2019

There have been 3 significant improvements proposed for OneHotEncoder (and to a lesser extent OrdinalEncoder), often with an associated PR,

the goal of this issue to have a high level agreement on the desired solution, that it is consistent/compatible for different available encoders, or encoders that we may want to add in the near future (e.g. target encoder, #5853). Some of the possible solutions for the above 3 features are mutually exclusive. Also putting aside aside backward compatibility constraints for a start, what default options we would want ideally.

I have not followed in detail all past discussions about encoders (in particular about ordering concerns #15050). Following are some of the observations / open questions I have, please add more if I missed something/link with existing comments.

NaN support

  • Mainly if we want to implement support directly or suggest to use an imputer for the pre-processing step (as one can do now).

pd.Categorical support

  • Do we say that categories='dtype' would make OHE categories match categories in the dtype
    ? Including the ordering? But then, this means only at fit since, for transform, the test set could have unknown categories.
  • Actually, if one does a train test split, some categories from the dtype can be missing from the train set as well. Do we then create a column with 0s in fit_transform, or disregard this column breaking the assumption of conforming to dtype categories? Solution proposed by @thomasjpfan in [MRG] ENH Adds categories='dtypes' option to OrdinalEncoder and OneHotEncoder #15396 (comment)
  • Finally, if we do not conform to the dtype categories, and only use pd.Categorical for computational efficiency internally, what is the point of defining a categories='dtype' in the first place (or warning that categories order doesn't match the order of categories in dtype [MRG] ENH Adds warning with pandas category does not match lexicon ordering #15050).

Infrequent categories

Overall the plan seems fairly clear in #12153 (comment)

NaN & pd.Categorical

  • Handing NaN as a separate category, means we are no longer using purely the categories from the dtype (even with dtype='category') which can be fine as long as we agree on it.
  • Another possibility is to implement a say CategoricalPreprocessor to add NaN as a new dtype category in a separate preprocessing step, 
    df[column].cat.add_categories("NaN", inplace=True)
df[column].fillna("NaN", inplace=True)
    which can make interpretation simpler. Say df[column].value_counts() would then show NaN properly and one might want to do it for exploratory analysis in any case.

NaN & infrequent categories

  • Do infrequent categories rules apply to NaN or is it always a separate category (even if passes the infrequent criteria)?

pd.Categorical and infrequent categories

  • Similarly we could consider a preprocessor that would add infrequent as a category to dtype, instead of doing that internally in encoders. Say to evaluate how many infrequent elements one has, I find that doing (approximately), 
    ohe = OneHotEncoder(categories='dtype', min_frequency=5)
X = ohe.fit_transform(df['col'])
unfrequent_idx = ohe.get_feature_names().tolist().find("infrequent")
print(X.sum(axis=0).A1[unfrequent_idx])
    very awkward as opposed to, 
    df['col'].value_counts()["infrequent"]
    where it was properly added to df['col'].cat.categories previously and we are using all the nice features of pd.Categorical.

So there is some tension here between adding these features to scikit-learn and keeping exploratory analysis with pandas user friendly (and not asking users to implement the same thing twice).

Given the complexity of this interaction, maybe separating "Imputer + Unfrequent categories conversion with pd.Categorical support" and "OneHot, Ordinal, Target etc encoder" into 2 or 3 estimators might be easier to understand? Not sure about usability though. The alternative that would mean we also plan to add these features (and enforce consistency) for any future encoder.

cc @thomasjpfan @NicolasHug @glemaitre @jorisvandenbossche @amueller @jnothman @ogrisel
@rth rth mentioned this issue Dec 4, 2019
Closed
@jnothman
Copy link
Member

jnothman commented Dec 5, 2019

handling of infrequent categories (issue #12153, PR #13833)

I note that OrdinalEncoder does not currently have any handling of categories not seen at fit.

Mainly if we want to implement support directly or suggest to use an imputer for the pre-processing step (as one can do now).

If we want to support estimators, here and elsewhere, that learn directly from data with missing values, we're best propagating the missingness rather than imputing.

Do we say that categories='dtype' would make OHE categories match categories in the dtype
? Including the ordering?

I think so. I think ordering should be respected even when ordered=False, since ordered=False is the default, and since it would be easier to use s.codes than to remap them. But I could be persuaded to respect the order only when ordered=True.

But then, this means only at fit since, for transform, the test set could have unknown categories.

I think if there is a categorical dtype at fit time, then unknown categories appearing at transform time should raise an error. This should be included in _validate_data across the board.

some categories from the dtype can be missing from the train set as well. Do we then create a column with 0s in fit_transform, or disregard this column breaking the assumption of conforming to dtype categories?

I'm not certain, but maybe just make the column with 0s, for correspondence with when one explicitly sets the categories.

Infrequent categories: Overall the plan seems fairly clear in #12153 (comment)

This does not pertain to OrdinalEncoder, where we don't even know how to handle unknowns. But infrequent category handling is more straightforward than unknowns when ordinal encoding, because at least in the case of infrequents, we "know" the order where the value should appear, and either conflate with the greater or lesser category.

Handling NaN

I think in OrdinalEncoder, the only encoding of NaN should be as NaN.

I don't think that handling NaN as a separate category is a problem in OHE (and I don't see why we need to modify the dtype). When absent from training, I would just output a row of zeros.

Do infrequent categories rules apply to NaN or is it always a separate category (even if passes the infrequent criteria)?

My instinct is to treat it as special, not an infrequent cat, but I am not wedded to this position.

Similarly we could consider a preprocessor that would add infrequent as a category to dtype, instead of doing that internally in encoders.

I don't get why we're modifying the dtype, except for efficiency. And if for efficiency, then we do whatever is pragmatic, as long as it's hidden from the user, i.e. creates no API stability issues.

ohe.get_feature_names().tolist()

The convention is to return a list from get_feature_names()

as opposed to, df['col'].value_counts()["infrequent"] where it was properly added to df['col'].cat.categories previously and we are using all the nice features of pd.Categorical.

It's fine to ask the user to handle infrequents themselves (with the caveat that they'll likely mess up the train-test split)... but if we want to handle it in our encoders, then I don't think we can do that by modifying the input... so I don't understand this suggestion, until I read the next paragraph where I see you are considering handling infrequent categories in a different transformer.

If we can design an InfrequentCategoryTransformer, I'm okay with that... I'm not yet convinced that it will work without duplicating a lot of the stuff (either the implementation, or the need to specify parameters redundantly) from OHE or OrdinalEncoder, nor am I convinced that we can decouple the handling of pd.Categorical from encoders entirely.

I'd be keen to see this framed around some test cases as @thomasjpfan has done for one small piece of the pie.

@thomasjpfan thomasjpfan mentioned this issue May 23, 2020
Merged
@mfeurer mfeurer mentioned this issue Jul 22, 2020
Merged
@mtorabirad
Copy link

Are the above-proposed improvements now implemented in OneHotEncoder ?

@glemaitre
Copy link
Member

Still in progress

@mfeurer mfeurer mentioned this issue Nov 16, 2020
Closed
@kbattocchi kbattocchi mentioned this issue Mar 16, 2023
Open
@ogrisel
Copy link
Member

ogrisel commented Dec 1, 2023

There is also another feature request for a small but usability improvement to make it possible to use a shared encoding value for unknowns and infrequent features (as done in OneHotEncoder):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
module:preprocessing
Projects
Missing value and nan support
Status: No status
Support for categorical variable
Status: Discussion
Milestone
No milestone
Development

No branches or pull requests
6 participants
@jnothman @ogrisel @rth @cmarmo @glemaitre @mtorabirad