BUG: read_csv dtype="category" now infers numeric and boolean categories - #64659

• closes API: categories.dtype with pd.read_csv(..., dtype='category') #56044

Summary

read_csv(..., dtype="category") with the c and python engines produced string categories for numeric and boolean columns (e.g. categories=['1', '2'] with object dtype), while the pyarrow engine inferred them. Now all three engines infer numeric and boolean categories, matching the types those columns get without dtype="category".

import pandas as pd
from io import StringIO

data = "a,b,c\n1,a,3.4\n1,a,3.4\n2,b,4.5"

res = pd.read_csv(StringIO(data), dtype="category")

res["a"].cat.categories
# Before (c/python): Index(['1', '2'], dtype='object')
# After (all engines): Index([1, 2], dtype='int64')

Changes

• Categorical._maybe_convert_categories / _from_converted_categories: new helpers that infer numeric and boolean categories when dtype does not provide them. Distinct strings that convert to the same value (e.g. "1" and "1.0", or "True" and "TRUE") are merged and recoded, matching the pyarrow engine. The c engine's tokenizer matches the default True/False spellings case-insensitively, so its category inference does too -- after consulting true_values/false_values, which the tokenizer checks first. The parsers call them; _from_inferred_categories keeps its previous signature.
• Parsers skip the numeric inference when thousands/decimal are non-default, since to_numeric is unaware of those options — such columns keep string categories rather than mis-parsing (e.g. "1.000" with thousands="." becoming 1.0). Under quoting=csv.QUOTE_NONNUMERIC the inference produces float64 instead, matching the csv module's cast-every-unquoted-field-to-float semantics and the non-categorical read.
• PythonParser._cast_types: removed the ensure_string_array hack that cast values back to strings "for consistency with the c-parser".
• _maybe_convert_string_to_object: with future.infer_string disabled, an ordered categorical column kept str categories where an unordered one got object, because CategoricalDtype.__eq__ ignores the categories' dtype when ordered and the astype no-opped. This reaches every reader that builds a frame from an Arrow table (read_parquet, read_feather, read_orc, ...), not just read_csv, so it has its own whatsnew line and a read_feather test.
• _post_convert_dtypes: convert IntegerDtype categories (from the BUG: read_csv loses precision when engine='pyarrow' and dtype Int64 #56136 lossy-float64 workaround) back to numpy, but only for read_csv with a user dtype, and never for categories the user explicitly requested.
• _pyarrow_parse_type: a CategoricalDtype with no explicit categories is left to pyarrow's type inference instead of being parsed as large_string, so the pyarrow engine infers categories like the other two. See the note below.

Three further inconsistencies in the c/python engines are fixed along the way:

• A CategoricalDtype specifying ordered=True without categories no longer discards ordered.
• low_memory=True (the c engine default) no longer leaves categories in buffer-chunk order.
• An all-NA buffer chunk no longer raises TypeError: dtype of categories must be the same when other chunks did infer categories.

Notes for reviewers

• This is a direct behavior change (notable-bug-fix whatsnew entry), not a deprecation — a warning would fire on essentially every dtype="category" read. Flagging since API: categories.dtype with pd.read_csv(..., dtype='category') #56044 has no recorded discussion.
• The inference lives in the shared text parser, so it also reaches read_table, read_fwf and read_excel, which already inferred numeric/boolean types for these columns without dtype="category". Covered by tests.
• Integers too large for int64/uint64 become object-dtype categories of Python ints on c/python (lossless); pyarrow gives lossy float64. Pinned in tests at two magnitudes, including integers beyond the float64 range — those used to trip the position-dependent inference bug addressed upstream in BUG: position-dependent integer inference in maybe_convert_objects #66519.
• Like every other dtype, the inference sees one chunk at a time under chunksize/iterator=True, so chunks of one column can now come back with different category dtypes; previously they were always strings and so always concat-able. The buffer chunking low_memory=True does inside a single read adds no such divergence -- it infers once, after those chunks are combined. Noted in the whatsnew and pinned in tests.
• Interaction with BUG: correctly honor specified dtype=str in read_csv pyarrow engine #62242, merged while this was open: it made the pyarrow engine parse dtype="category" as large_string so the categories are the raw file text, aligning pyarrow with the c/python behavior as it stood at the time. This PR changes that behavior, so one branch of _pyarrow_parse_type is reversed here and the dtype="category" clause is dropped from its whatsnew entry. Its dtype=str and dtype=object handling is untouched, and all three issues it cites (index_col in read_csv and read_table ignores dtype argument #9435, BUG: pyarrow read_csv engine stripping leading zeros with dtype=str #57666, BUG: read_csv() with engine="pyarrow" converts numeric string even when dtype=str is specified #58260) are about dtype=str/index_col, not categoricals — its own tests for them still pass. Concretely: with the raw-text rule there is no way to ask for a categorical with inferred value type at read time, while string categories stay available via dtype="str" plus .astype("category"), or an explicit CategoricalDtype. One consequence worth calling out: zero-padded identifier columns (zip codes, account numbers) now lose their leading zeros under dtype="category", exactly as they already do without it. Noted in the whatsnew with the escape hatch.
• Datetime-like strings still diverge (pyarrow infers, c/python keep string categories) — out of scope here.
• A non-default dtype_backend is only partly honored for the categories: the c engine always gives numpy-backed categories, as does the python engine with dtype_backend="numpy_nullable" (BUG: read_csv(dtype="category") ignores dtype_backend for the c and python engines #66382). The current behavior is pinned in tests.