Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Include drop='last' to OneHotEncoder #23436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
WittmannF opened this issue May 20, 2022 · 5 comments
Closed

Include drop='last' to OneHotEncoder #23436

WittmannF opened this issue May 20, 2022 · 5 comments
Labels

Comments

@WittmannF
Copy link

WittmannF commented May 20, 2022

Describe the workflow you want to enable

When using SimpleImputer + OneHotEncoder, I am able to add a new constant category for NaN values like the example below:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
import numpy as np

categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='constant')),
    ('encoder', OneHotEncoder(handle_unknown='ignore'))])

preprocessor = ColumnTransformer(
    transformers=[
        ('cat', categorical_transformer, [0])
    ])

df = pd.DataFrame(['Male', 'Female', np.nan])
preprocessor.fit_transform(df)

# array([[0., 1., 0.],
#       [1., 0., 0.],
#       [0., 0., 1.]])

However, I wanted to have an argument like OneHotEncoder(drop='last') in order to have an output like:

array([[0., 1.],
       [1., 0.],
       [0., 0.]])

This would allow all NaNs to be filled with zeros.

Describe your proposed solution

categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='constant')),
    ('encoder', OneHotEncoder(drop='last'))])

Describe alternatives you've considered, if relevant

There's no good alternative for compatibility with sklearn's pipelines. I was following the issue #11996 of adding a handle_missing to OneHotEncoder but it has been ignored in favor of using a "constant" strategy on the categorical columns. But the constant strategy will add an unnecessary new column that could be dropped in this scenario.

Additional context

No response

@WittmannF WittmannF added Needs Triage Issue requires triage New Feature labels May 20, 2022
@lesteve
Copy link
Member

lesteve commented May 25, 2022

There is a drop argument in OneHotEncoder which you can pass a array to (one category to drop for each feature), can you use this for you use case? Adapting your snippet, something like this:

import numpy as np
import pandas as pd
from sklearn.preprocessing import OneHotEncoder

df = pd.DataFrame(['Male', 'Female', np.nan])
ohe = OneHotEncoder(drop=[np.nan])
ohe.fit_transform(df).toarray()

Output:

array([[0., 1.],
       [1., 0.],
       [0., 0.]])

@ogrisel
Copy link
Member

ogrisel commented Jun 2, 2022

I think the original request could be a convenience in some cases but it's a bit weird to have a drop strategy that depends on the lexicographical ordering of the categories.

For this particular case, I think @lesteve's solution is the right approach.

We could also offer to drop the "most_frequent" (and maybe also "least_frequent" although the latter is probably very unstable under CV/resampling) to drop a category based on its frequency in the training set.

Since the choice of the category to drop has an impact on the effect of regularization in a downstream linear model, dropping based on the frequency on the training set might be a good strategy to reduce collinearity for non-binary categorical variables in general.

Maybe @lorentzenchr has an opinion on this.

@thomasjpfan thomasjpfan added module:preprocessing Needs Decision - Include Feature Requires decision regarding including feature and removed Needs Triage Issue requires triage labels Jun 2, 2022
@lorentzenchr
Copy link
Member

lorentzenchr commented Jun 5, 2022

IIRC, R‘s formula drops the first level by default. That corresponds to our option "first". Having first, why not have an option "last". Edit: maybe not worth it.

More striking is the argument to have "most_frequent". This is a strategy that I‘ve seen in other GLM software. Model coefficients are then the effect relative to this most frequent level which seems the most natural choice (but is irrelevant for most other things).

We could also consider to support sample weights with "most_frequent", i.e. choose the level with highest sum of sample weights.

To the best of my knowledge, for linear models with penalties, one should never drop a level. For unpenalized linear models, the converse is true: always drop one level (otherwise one has perfect collinearity and solvers have a much harder job finding the unique minimum norm solution).

@ogrisel
Copy link
Member

ogrisel commented Sep 9, 2022

There is a (stalled) open PR for drop="most_frequent" here: #18679.

@thomasjpfan
Copy link
Member

I agree it is not worth it to have a drop="last". The drop="most_frequent" feature is being tracked in #18553.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants