API Forbid pd.NA in ColumnTransformer output unless transform output is configured as "pandas" #27734
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ogrisel
reviewed
Dec 1, 2023
Member
ogrisel
left a comment
There was a problem hiding this comment.
This looks good to me, but I do not understand why the following line is not covered by the CI. Could you please investigate what's wrong?
Contributor
Author
|
TODO:
|
This reverts commit 1313a0c.
ogrisel
approved these changes
Jan 10, 2024
Member
ogrisel
left a comment
There was a problem hiding this comment.
LGTM. If merged on time this should be backported to 1.4.0 or 1.4.1.
Member
|
I enable the auto-merge. |
jeremiedbb
approved these changes
Jan 16, 2024
jeremiedbb
pushed a commit
to jeremiedbb/scikit-learn
that referenced
this pull request
Jan 17, 2024
…is configured as "pandas" (scikit-learn#27734) Co-authored-by: Guillaume Lemaitre <[email protected]>
jeremiedbb
pushed a commit
that referenced
this pull request
Jan 17, 2024
…is configured as "pandas" (#27734) Co-authored-by: Guillaume Lemaitre <[email protected]>
glemaitre
added a commit
to glemaitre/scikit-learn
that referenced
this pull request
Feb 10, 2024
…is configured as "pandas" (scikit-learn#27734) Co-authored-by: Guillaume Lemaitre <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference Issues/PRs
Closes #27482
What does this implement/fix? Explain your changes.
The
ColumnTransformer(withoutset_output(transform="pandas")), when it transforms pandas columns that contain pd.NA, outputs arrays of dtype object that contain pd.NA, which can cause subsequent estimators in a pipeline to fail becausecheck_array(dtype="numeric")raises a TypeError on such arrays.#27671 looked into fixing the problem by converting
pd.NAtonp.nanby applyingcheck_arrayon transformer outputs, but this causes other issues when the outputs have mixed dtypes so it was decided during the scikit-learn developers meeting that instead we should raise an error when the output of some transformer containspd.NAand the output is not set to "pandas". Therefore this PR supersedes #27671Here we first check the dtypes of the output to only search for
pd.NAin columns that have a chance of containing any.ATM this PR only searches for
pd.NAin dataframes; I'm not sure if we should check numpy arrays (of dtype object) and sparse arrays and matrices as well