Thanks to visit codestin.com
Credit goes to github.com

Skip to content

UInt128 handling in cudf-polars only covers one specific polars rewrite shape #24108

Description

@Matt711

polars 1.43+ rewrites concat(a, b, ...).select(len()) into col("len").cast(UInt128).sum().cast(IDX_DTYPE). libcudf has no 128-bit integer type (size_type is 32-bit, #13159), so UInt128 fails to translate. The sum never actually needs 128 bits, it's just widened so accumulation can't overflow, then narrowed back to IDX_DTYPE before materializing. No GPU-resident table holds anywhere near 2**64 rows, so cudf-polars represents it as UInt64 instead.

The fix (translate.py) only recognizes this one shape: an Agg named "sum", or a Cast whose child is Column("len"). Any other UInt128 usage still hits the generic NotImplementedError in datatype.py. If polars restructures this rewrite or introduces UInt128 elsewhere, the new shape won't match, and the query silently loses GPU acceleration instead of running there.

Also: polars' own Series.to_arrow() can't export real UInt128 data (ArrowInvalid: Invalid or unsupported format string: '_plu128'), so the dtype isn't fully Arrow-interoperable in polars itself yet either.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Needs TriageNeed team to review and classifycudf-polarsIssues specific to cudf-polars

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions