Thanks to visit codestin.com
Credit goes to github.com

Skip to content

FIX Fix OpenML dataset parsing with pandas>=3.1.0.dev0 - #34902

Merged
jeremiedbb merged 9 commits into
scikit-learn:mainfrom
DeaMariaLeon:ci-pandas
Sep 10, 2026
Merged

FIX Fix OpenML dataset parsing with pandas>=3.1.0.dev0#34902
jeremiedbb merged 9 commits into
scikit-learn:mainfrom
DeaMariaLeon:ci-pandas

Conversation

@DeaMariaLeon

@DeaMariaLeon DeaMariaLeon commented Sep 7, 2026 •
edited by jeremiedbb
Loading

Copy link
Copy Markdown
Member

Reference Issues/PRs

Closes #34894
Fixes #34892

What does this implement/fix? Explain your changes.

The bug that was found by the CI failure is in sklearn/datasets/_arff_parser.py line 435-447:

_pandas_arff_parser strips single quotes on categories as a post-processing steps if needed.
This part of the code (on the main branch) assumes that every categorical value is a str.

But pd.read_csv(..., dtype="category") on unquoted numeric values yields object with pandas 2.3.3.
And it yields int64 with pandas 3.1.0.dev0.

The pandas' PR that seems to have changed the behaviour: pandas-dev/pandas#64659

One can verify that with:

for env in ~/mambaforge/envs/sklearn-dev ~/venvs/sklearn-scipy-dev; do
  "$env/bin/python" -c "
import io, pandas as pd
col = pd.read_csv(io.StringIO('1\n2\n3\n'), header=None, dtype={0: 'category'})[0]
print(f'pandas {pd.__version__}: {col.cat.categories.dtype}')"
done

Where sklearn-scipy-dev was my local environment with the nightly builds.
The output of the block above is:

pandas 2.3.3: object
pandas 3.1.0.dev0+1803.gc26625d6e7: int64

The test that I added fails without this check on _pandas_arff_parser: if not isinstance(input_string, str)

For this PR I run this command and committed the generated files:

python build_tools/update_environments_and_lock_files.py --select-tag scipy-dev

AI usage disclosure

I used AI assistance for:

  • Code generation (e.g., when writing an implementation or fixing a bug)
  • Test generation
  • Research and understanding

Any other comments?

Not ready to review - I want to see if there are other failures

DeaMariaLeon
DeaMariaLeon commented Sep 7, 2026
View reviewed changes
Comment thread sklearn/datasets/_arff_parser.py
@@ -432,6 +432,9 @@ def _pandas_arff_parser(
single_quote_pattern = re.compile(r"^'(?P<contents>.*)'$")

def strip_single_quotes(input_string):

@DeaMariaLeon DeaMariaLeon Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On line 438 we try to do a regex to an int (with the nightly build version of pandas).

With the environment I created locally using the nightly build, one can see the following at this point in the code when running the new test on test_arff_parser.py:test_pandas_arff_parser_numeric_nominal_categories:

(Pdb) input_string
1
(Pdb) match = re.search(single_quote_pattern, input_string)
*** TypeError: expected string or bytes-like object, got 'int'

Note: this error happens when we comment out lines 435 and 436 and run the test.

@DeaMariaLeon
DeaMariaLeon marked this pull request as ready for review September 7, 2026 15:52
@DeaMariaLeon DeaMariaLeon changed the title Fix: scipy-dev CI build CI: fix scipy-dev build due to the change in pandas' nightly version Sep 7, 2026
@DeaMariaLeon DeaMariaLeon changed the title CI: fix scipy-dev build due to the change in pandas' nightly version CI: fix scipy-dev build due to the change in pandas' Sep 7, 2026
@DeaMariaLeon DeaMariaLeon changed the title CI: fix scipy-dev build due to the change in pandas' CI: fix scipy-dev build due to the change in pandas Sep 7, 2026
@DeaMariaLeon
DeaMariaLeon marked this pull request as draft September 8, 2026 06:11
@DeaMariaLeon
DeaMariaLeon marked this pull request as ready for review September 8, 2026 07:03
@lesteve

lesteve commented Sep 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Thanks for the PR!

I think adding a changelog makes sense.

Also if this is not too much work it would be nice to find the pandas PR that changed the behaviour, in principle this is a recent PR, since it started failing a few days ago in #34892 and pandas development wheels are updated often (daily by a quick look at https://anaconda.org/channels/scientific-python-nightly-wheels/packages/pandas/files?sort=-updated_at)

@lesteve lesteve changed the title CI: fix scipy-dev build due to the change in pandas CI Fix OpenML parsing with pandas>=3.1.0.dev0 Sep 8, 2026
@lesteve lesteve changed the title CI Fix OpenML parsing with pandas>=3.1.0.dev0 FIX Fix OpenML dataset parsing with pandas>=3.1.0.dev0 Sep 8, 2026
@DeaMariaLeon

DeaMariaLeon commented Sep 8, 2026

Copy link
Copy Markdown
Member Author

I think that the pandas PR that changed the behaviour is this one: pandas-dev/pandas#64659

@DeaMariaLeon

DeaMariaLeon commented Sep 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

With the pandas' PR I realised that leading zeros are gone.

I need to investigate more but it seems that if a category that OpenML calls "01", it will become 1:

>>> pd.read_csv(io.StringIO("01\n02\n03\n"), header=None, dtype="category")
   0
0  1
1  2
2  3

Edit: I'm searching if this change affects other places in scikit-learn

cakedev0
cakedev0 approved these changes Sep 9, 2026
View reviewed changes

@cakedev0 cakedev0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, simple fix, good test 👍

@DeaMariaLeon

DeaMariaLeon commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

Thanks!

@DeaMariaLeon DeaMariaLeon added the Waiting for Second Reviewer First reviewer is done, need a second one! label Sep 9, 2026
@DeaMariaLeon DeaMariaLeon moved this to In progress in Labs Sep 9, 2026
@DeaMariaLeon DeaMariaLeon added this to Labs Sep 9, 2026
@DeaMariaLeon DeaMariaLeon moved this from In progress to PR waiting for reviews in Labs Sep 9, 2026
jeremiedbb
jeremiedbb approved these changes Sep 10, 2026
View reviewed changes

@jeremiedbb jeremiedbb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix

@jeremiedbb
jeremiedbb merged commit f01834a into scikit-learn:main Sep 10, 2026
44 checks passed
@github-project-automation github-project-automation Bot moved this from PR waiting for reviews to Done in Labs Sep 10, 2026
@DeaMariaLeon
DeaMariaLeon deleted the ci-pandas branch September 10, 2026 10:05
@DeaMariaLeon DeaMariaLeon mentioned this pull request Sep 11, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jeremiedbb jeremiedbb jeremiedbb approved these changes
+1 more reviewer
@cakedev0 cakedev0 cakedev0 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Build / CI module:datasets Waiting for Second Reviewer First reviewer is done, need a second one!

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

⚠️ CI failed on Unit tests Linux x86-64 pylatest_pip_scipy_dev (last failure: Sep 10, 2026) ⚠️

4 participants

@DeaMariaLeon @lesteve @cakedev0 @jeremiedbb