BUG: read_csv dtype="category" now infers numeric categories - #7

Use explicit dtype mapping instead of accessing .numpy_dtype
on a union type that the type checkers can't narrow.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…type_backend is defined

The block referenced `dtype_backend` inside `_normalize_timezone_dtypes`,
which only takes `df`. Every pyarrow-backed IO path hit a NameError at
runtime, breaking Type Checking, Doctests, Doc Build, and pyarrow IO
tests. Moved the block into `_post_convert_dtypes` (after the astype),
where `dtype_backend` is already a parameter.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

…read_csv dtype="category"

- Merge categories when distinct strings convert to the same number
  (e.g. "1" and "1.0") instead of raising on duplicate categories
- Skip numeric inference when thousands/decimal are set, since
  to_numeric is unaware of those options
- Only downcast IntegerDtype categories that the pyarrow workaround
  introduced, not categories the user explicitly requested; restrict
  the downcast to read_csv (dtype is not None) and make it safe for
  duplicate column labels
- Add tests for the above plus large-integer behavior
- Move the whatsnew entry to the notable bug fixes section

Co-Authored-By: Claude Fable 5 <[email protected]>

The c and python engines produced string categories for boolean columns
under dtype="category" while the pyarrow engine inferred bool categories.
All engines now infer boolean categories, mirroring each engine's
non-categorical inference (numeric first, then boolean). GH#56044

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

…ory"

- Defer category-dtype inference in low_memory mode until chunks are
  concatenated; per-chunk inference could produce differing category
  dtypes, breaking union_categoricals
- Keep string categories when to_numeric turns "" into NaN
  (na_filter=False / keep_default_na=False)
- Keep object dtype for empty inferred categories (all-NA columns)

Co-Authored-By: Claude Fable 5 <[email protected]>

… docs

- factor _from_converted_categories out of _from_inferred_categories so
  the low_memory deferred path runs to_numeric once on the unioned
  categories
- update stale io.rst note obsoleted by GH#56044 inference
- pin dtype_backend and QUOTE_NONNUMERIC category behavior in tests;
  scope the whatsnew claim to the default dtype_backend

Co-Authored-By: Claude Fable 5 <[email protected]>

Co-Authored-By: Claude Opus 4.8 <[email protected]>

…tegory"

With dtype_backend="pyarrow" the python engine's values arrive as
ArrowDtype strings, whose kind is "U" rather than "O", so the guard in
_maybe_convert_categories bailed and no inference was applied at all.
Categories now infer to int64[pyarrow], matching a non-categorical read.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

Co-Authored-By: Claude Opus 4.8 <[email protected]>

# Conflicts:
#	doc/source/whatsnew/v3.1.0.rst
#	pandas/_libs/parsers.pyx

…tegories unsorted

With the c and python engines, a CategoricalDtype specifying ordered=True but
not categories silently produced an unordered result, while the pyarrow engine
honored it. Separately, low_memory=True (the c engine default) left string
categories in the order the file's buffer chunks happened to produce them,
rather than sorted as every other path gives.

Also restore the pre-GH#56044 defaults of Categorical._from_inferred_categories
so callers other than the parsers are unaffected, and collapse
_from_converted_categories to a single recode pass.

Co-Authored-By: Claude Opus 5 <[email protected]>

Co-Authored-By: Claude Opus 5 <[email protected]>

Co-Authored-By: Claude Opus 5 <[email protected]>

The pyarrow engine parsed a categorical with no explicit categories as
large_string, so the categories were the raw text from the file instead of
the inferred values the other engines now give (GH#56044).

This reverses one branch of GH-62242, which aligned the pyarrow engine with
the c and python engines as they behaved before this PR changed them. Its
dtype=str and dtype=object handling, and every issue it cites, are
unaffected.

Co-Authored-By: Claude Opus 5 <[email protected]>

# Conflicts:
#	doc/source/whatsnew/v3.1.0.rst
#	pandas/io/parsers/c_parser_wrapper.py
#	pandas/tests/io/excel/test_readers.py
#	pandas/tests/io/parser/dtypes/test_categorical.py
#	pandas/tests/io/parser/test_read_fwf.py