Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-136702: Deprecate passing non-ascii *encoding* (str) to encodings.normalize_encoding #140030

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
malemburg merged 11 commits into from
Nov 9, 2025
Merged

gh-136702: Deprecate passing non-ascii *encoding* (str) to encodings.normalize_encoding #140030

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
c8fc658
deprecate non-ascii
StanFromIreland Oct 13, 2025
5b50daa
Relocate import
StanFromIreland Oct 13, 2025
95f2e65
sanitize charset names in email
StanFromIreland Oct 13, 2025
fad52cd
Use table, replace with 'ascii'
StanFromIreland Oct 16, 2025
3ac0804
Merge branch 'main' into encodings/non-ascii
StanFromIreland Oct 31, 2025
9d6f06e
Review
StanFromIreland Nov 1, 2025
16697dc
Fix second warning
StanFromIreland Nov 1, 2025
e4036f8
Convert to asserts
StanFromIreland Nov 1, 2025
b8fc5f4
Fix for platforms with ordered tests
StanFromIreland Nov 1, 2025
7592af8
!fixup
StanFromIreland Nov 1, 2025
8c59899
Fix CI on Android and iOS
StanFromIreland Nov 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
sanitize charset names in email
  • Loading branch information
@StanFromIreland
StanFromIreland committed Oct 13, 2025
commit 95f2e65dbdee909c88cd8b6276ad9c803c4115cb
1 change: 1 addition & 0 deletions Lib/email/_header_value_parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -796,6 +796,7 @@ def params(self):
value = urllib.parse.unquote(value, encoding='latin-1')
else:
try:
charset = utils._sanitize_charset_name(charset, 'us-ascii')
Comment thread
StanFromIreland marked this conversation as resolved.
Outdated
value = value.decode(charset, 'surrogateescape')
except (LookupError, UnicodeEncodeError):
# XXX: there should really be a custom defect for
Expand Down
10 changes: 10 additions & 0 deletions Lib/email/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -446,6 +446,15 @@ def decode_params(params):
new_params.append((name, '"%s"' % value))
return new_params

def _sanitize_charset_name(charset, fallback_charset):
if not charset:
return charset
sanitized = ''.join(
c for c in charset
if (ord(c) < 0xDC80 or ord(c) > 0xDCFF) and c.isascii()
)
Comment thread
StanFromIreland marked this conversation as resolved.
Outdated
return sanitized if sanitized else fallback_charset

Copy link
Copy Markdown
Member

@bitdancer bitdancer Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the trigger for this change? Do I actually have a test that uses a non-ascii charset name? If I did it should be an error case, since non-ascii is not permitted in charset names per the RFCs. I'm surprised I don't appear to be registering a defect for that, though I didn't go through the code enough to be sure I don't ;)

Regardless it isn't clear to me that 'sanitizing' is a useful operation. It isn't likely to produce a valid charset name, we should just be falling back to ascii at that point. What led you to choose this approach?

Copy link
Copy Markdown
Member Author

@StanFromIreland StanFromIreland Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is currently done by normalize_encoding.

Copy link
Copy Markdown
Member

@bitdancer bitdancer Nov 1, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. emal doesn't call lookup directly and no tests fail without the changes.

I presume you did this to preserve backward compatibility. Unless I'm missing something, I don't think we should bother to do that. Given a non-ascii charset name, there are two possible outcomes from the current code: the name after sanitizing is not a valid codec name, or it is. If it is valid after sanitizing, there are two cases: the sanitized name results in successful decoding, or it does not. It is only the first of these second two cases that would be affected by the post-deprecation change.

How often would that case occur in reality? I would guess it would be a vanishingly small number of cases, if it ever occurs at all.

I think it will be better to remove the changes to the email package from this PR. If anyone sees the deprecation warning maybe they'll open an issue, but I'm betting nobody ever sees it from the email package. The behavior after the deprecation is over is the behavior we want: if the codec name contains non-ascii it is not a valid codec name, so any non-ascii in the text being decoded using that charset name will ultimately get turned into the 'unknown character' glyph when decoded by the email package.

Copy link
Copy Markdown
Member Author

@StanFromIreland StanFromIreland Nov 1, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I presume you did this to preserve backward compatibility.

Yes, I'm no email expert and I did not dig into the specifications, so I did this to not change any behaviour. I can remove it.

Copy link
Copy Markdown
Member

@malemburg malemburg Nov 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What't the conclusion here ? I still see the email package changes in place, but they look pretty harmless to me.

def collapse_rfc2231_value(value, errors='replace',
fallback_charset='us-ascii'):
if not isinstance(value, tuple) or len(value) != 3:
Expand All @@ -458,6 +467,7 @@ def collapse_rfc2231_value(value, errors='replace',
# Issue 17369: if charset/lang is None, decode_rfc2231 couldn't parse
# the value, so use the fallback_charset.
charset = fallback_charset
charset = _sanitize_charset_name(charset, fallback_charset)
rawbytes = bytes(text, 'raw-unicode-escape')
try:
return str(rawbytes, charset, errors)
Expand Down
Loading