Bug report

Bug description:

When unpickling using _pickle.c or pickle.py through load/loads, an encoding can be specified using the encoding argument, with the default being ASCII. However, pickletools does not support custom encodings and instead makes assumptions about what encoding it uses, which can lead to either incorrect data being displayed or erroring/not erroring when the normal unpickling process would error.

The three opcodes that I have found this in are STRING, BINSTRING, and SHORT_BINSTRING:

• On line 359, it assumes ASCII for STRING
• On line 456, it assumes latin-1 for BINSTRING
• On line 422, it assumes latin-1 for SHORT_BINSTRING

I think the best solution would be to support encodings as an optional argument in pickletools.py, with the default being set to ASCII (since that's the default encoding for pickle.py and _pickle.c).

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Linked PRs

• gh-126997: Fix support of non-ASCII strings in pickletools #127062
• [3.13] gh-126997: Fix support of non-ASCII strings in pickletools (GH-127062) #127094
• [3.12] gh-126997: Fix support of non-ASCII strings in pickletools (GH-127062) #127095