gh-67022: Document bytes/str inconsistency in email.header.decode_header() and suggest email.headerregistry.HeaderRegistry as a sane alternative #92900

dlenski · 2022-05-17T20:52:48Z

This function's possible return types have been surprising and error-prone
for the entirety of its Python 3.x history. It can return either:

typing.List[typing.Tuple[str, None]], of length exactly 1
or typing.List[typing.Tuple[bytes, typing.Optional[str]]]

This function can't be rewritten to be more consistent in a backwards-compatible way, because some users of this function depend on the existing return type(s).

This PR addresses the inconsistencies:

as suggested by @JelleZijlstra in The decode_header() function decodes raw part to bytes or str, depending on encoded part #67022 (comment):

we should document the surprising return type at https://docs.python.org/3.10/library/email.header.html.
by suggesting email.headerregistry.HeaderRegistry as a replacement, per gh-67022: Document bytes/str inconsistency in email.header.decode_header() and suggest email.headerregistry.HeaderRegistry as a sane alternative #92900

Example of the old/inconsistent (decode_header) vs. modern/sane approaches:

>>> from email import decode_header
>>> from email.headerregistry import HeaderRegistry
>>>
>>> # decode_header exposes differences in sub-encodings AND a str/bytes inconsistency in return type:
>>> print(decode_header('hello foo bar')
 [('hello foo bar', None)])
>>> print(decode_header('hello =?utf-8?B?ZsOzbw==?= bar'))
[(b'hello ', None), (b'f\xc3\xb3o', 'utf-8'), (b' bar', None)]
>>> print(decode_header('=?iso-8859-1?q?hello_f=F3o_bar?='))
[(b'hello f\xf3o bar', 'iso-8859-1')]
>>>
>>> # HeaderRegistry has a sane and consistent interface:
>>> decoder = HeaderRegistry()
>>> decoder('Subject', 'hello foo bar')
'hello foo bar'
>>> decoder('Subject', 'hello =?utf-8?B?ZsOzbw==?= bar')
'hello fóo bar'
>>> decoder('Subject', '=?iso-8859-1?q?hello_f=F3o_bar?=')
'hello fóo bar'

(Closes #30548 and replaces it.)

Issue: The decode_header() function decodes raw part to bytes or str, depending on encoded part #67022

ghost · 2022-05-17T20:52:50Z

All commit authors signed the Contributor License Agreement.

warsaw

In general, I think this would help users of the legacy API, although I think we should also steer people to the new API. What does @bitdancer think?

Doc/library/email.header.rst

Lib/email/header.py

Misc/NEWS.d/next/Library/2022-01-11-21-40-14.bpo-22833.WB-JWw.rst

Lib/email/header.py

bedevere-bot · 2022-07-20T19:28:15Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

And if you don't make the requested changes, you will be put in the comfy chair!

dlenski · 2022-07-20T21:24:34Z

I have made the requested changes; please review again, @warsaw.

And if you don't make the requested changes, you will be put in the comfy chair!

😂

bedevere-bot · 2022-07-20T21:24:38Z

Thanks for making the requested changes!

@warsaw: please review the changes made to this pull request.

dlenski · 2023-02-21T01:43:40Z

I have made the requested changes; please review again

bedevere-bot · 2023-02-21T01:43:42Z

Thanks for making the requested changes!

@warsaw: please review the changes made to this pull request.

…email.header.make_header' functions. Recommend use of `email.headerregistry.HeaderRegistry` instead, as suggested in python#92900 (comment)

Per python#92900 (comment), not wanted for doc-only PRs.

…de_header() This function's possible return types have been surprising and error-prone for the entirety of its Python 3.x history. It can return either: 1. `typing.List[typing.Tuple[bytes, typing.Optional[str]]]` of length >1 2. or `typing.List[typing.Tuple[str, None]]`, of length exactly 1 This means that any user of this function must be prepared to accept either `bytes` or `str` for the first member of the 2-tuples it returns, which is a very surprising behavior in Python 3.x, particularly given that the second member of the tuple is supposed to represent the charset/encoding of the first member. This patch documents the behavior of this function, and adds test cases to demonstrate it. As discussed in bpo-22833, this cannot be changed in a backwards-compatible way, and some users of this function depend precisely on the existing behavior.

…email.header.make_header' functions. Recommend use of `email.headerregistry.HeaderRegistry` instead, as suggested in python#92900 (comment)

Per python#92900 (comment), not wanted for doc-only PRs.

bitdancer

LGTM

miss-islington-app · 2025-06-15T19:30:12Z

Thanks @dlenski for the PR, and @bitdancer for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

miss-islington-app · 2025-06-15T19:30:12Z

Thanks @dlenski for the PR, and @bitdancer for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

…de_header() and suggest email.headerregistry.HeaderRegistry as a sane alternative (pythonGH-92900) * pythongh-67022: Document bytes/str inconsistency in email.header.decode_header() This function's possible return types have been surprising and error-prone for the entirety of its Python 3.x history. It can return either: 1. `typing.List[typing.Tuple[bytes, typing.Optional[str]]]` of length >1 2. or `typing.List[typing.Tuple[str, None]]`, of length exactly 1 This means that any user of this function must be prepared to accept either `bytes` or `str` for the first member of the 2-tuples it returns, which is a very surprising behavior in Python 3.x, particularly given that the second member of the tuple is supposed to represent the charset/encoding of the first member. This patch documents the behavior of this function, and adds test cases to demonstrate it. As discussed in bpo-22833, this cannot be changed in a backwards-compatible way, and some users of this function depend precisely on the existing behavior. Add warnings about obsolescence of 'email.header.decode_header' and 'email.header.make_header' functions. Recommend use of `email.headerregistry.HeaderRegistry` instead, as suggested in python#92900 (comment) (cherry picked from commit 60181f4) Co-authored-by: Dan Lenski <[email protected]>

bedevere-app · 2025-06-15T19:30:21Z

GH-135548 is a backport of this pull request to the 3.14 branch.

bedevere-app · 2025-06-15T19:30:25Z

GH-135549 is a backport of this pull request to the 3.13 branch.

…ode_header() and suggest email.headerregistry.HeaderRegistry as a sane alternative (GH-92900) (#135548) gh-67022: Document bytes/str inconsistency in email.header.decode_header() and suggest email.headerregistry.HeaderRegistry as a sane alternative (GH-92900) * gh-67022: Document bytes/str inconsistency in email.header.decode_header() This function's possible return types have been surprising and error-prone for the entirety of its Python 3.x history. It can return either: 1. `typing.List[typing.Tuple[bytes, typing.Optional[str]]]` of length >1 2. or `typing.List[typing.Tuple[str, None]]`, of length exactly 1 This means that any user of this function must be prepared to accept either `bytes` or `str` for the first member of the 2-tuples it returns, which is a very surprising behavior in Python 3.x, particularly given that the second member of the tuple is supposed to represent the charset/encoding of the first member. This patch documents the behavior of this function, and adds test cases to demonstrate it. As discussed in bpo-22833, this cannot be changed in a backwards-compatible way, and some users of this function depend precisely on the existing behavior. Add warnings about obsolescence of 'email.header.decode_header' and 'email.header.make_header' functions. Recommend use of `email.headerregistry.HeaderRegistry` instead, as suggested in #92900 (comment) (cherry picked from commit 60181f4) Co-authored-by: Dan Lenski <[email protected]>

…ode_header() and suggest email.headerregistry.HeaderRegistry as a sane alternative (GH-92900) (#135549) gh-67022: Document bytes/str inconsistency in email.header.decode_header() and suggest email.headerregistry.HeaderRegistry as a sane alternative (GH-92900) * gh-67022: Document bytes/str inconsistency in email.header.decode_header() This function's possible return types have been surprising and error-prone for the entirety of its Python 3.x history. It can return either: 1. `typing.List[typing.Tuple[bytes, typing.Optional[str]]]` of length >1 2. or `typing.List[typing.Tuple[str, None]]`, of length exactly 1 This means that any user of this function must be prepared to accept either `bytes` or `str` for the first member of the 2-tuples it returns, which is a very surprising behavior in Python 3.x, particularly given that the second member of the tuple is supposed to represent the charset/encoding of the first member. This patch documents the behavior of this function, and adds test cases to demonstrate it. As discussed in bpo-22833, this cannot be changed in a backwards-compatible way, and some users of this function depend precisely on the existing behavior. Add warnings about obsolescence of 'email.header.decode_header' and 'email.header.make_header' functions. Recommend use of `email.headerregistry.HeaderRegistry` instead, as suggested in #92900 (comment) (cherry picked from commit 60181f4) Co-authored-by: Dan Lenski <[email protected]>

…de_header() and suggest email.headerregistry.HeaderRegistry as a sane alternative (python#92900) * pythongh-67022: Document bytes/str inconsistency in email.header.decode_header() This function's possible return types have been surprising and error-prone for the entirety of its Python 3.x history. It can return either: 1. `typing.List[typing.Tuple[bytes, typing.Optional[str]]]` of length >1 2. or `typing.List[typing.Tuple[str, None]]`, of length exactly 1 This means that any user of this function must be prepared to accept either `bytes` or `str` for the first member of the 2-tuples it returns, which is a very surprising behavior in Python 3.x, particularly given that the second member of the tuple is supposed to represent the charset/encoding of the first member. This patch documents the behavior of this function, and adds test cases to demonstrate it. As discussed in bpo-22833, this cannot be changed in a backwards-compatible way, and some users of this function depend precisely on the existing behavior. Add warnings about obsolescence of 'email.header.decode_header' and 'email.header.make_header' functions. Recommend use of `email.headerregistry.HeaderRegistry` instead, as suggested in python#92900 (comment)

dlenski requested a review from a team as a code owner May 17, 2022 20:52

bedevere-bot added the awaiting review label May 17, 2022

dlenski force-pushed the gh60722 branch from 6b35dc0 to 712d83d Compare May 17, 2022 20:54

dlenski mentioned this pull request May 17, 2022

bpo-22833: Fix bytes/str inconsistency in email.header.decode_header() #30548

Closed

dlenski force-pushed the gh60722 branch from 712d83d to 9a8a34c Compare May 17, 2022 20:59

dlenski mentioned this pull request Jul 19, 2022

The decode_header() function decodes raw part to bytes or str, depending on encoded part #67022

Closed

srittau mentioned this pull request Jul 19, 2022

gh-67022: Improve email.header.decode_header() documentation #95020

Closed

warsaw requested changes Jul 20, 2022

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting review labels Jul 20, 2022

dlenski force-pushed the gh60722 branch from 9a8a34c to e760911 Compare July 20, 2022 21:11

bedevere-bot added awaiting change review and removed awaiting changes labels Jul 20, 2022

bedevere-bot requested a review from warsaw July 20, 2022 21:24

dlenski force-pushed the gh60722 branch from 0ced18b to e07b36d Compare June 11, 2025 20:26

dlenski force-pushed the gh60722 branch from e07b36d to 90db289 Compare June 11, 2025 20:27

dlenski force-pushed the gh60722 branch from 90db289 to 0eac759 Compare June 11, 2025 20:33

dlenski force-pushed the gh60722 branch from 0eac759 to 0752be8 Compare June 11, 2025 20:46

bedevere-app bot requested a review from bitdancer June 14, 2025 04:48

dlenski force-pushed the gh60722 branch from a712bc8 to 4b6143e Compare June 14, 2025 04:49

dlenski added a commit to dlenski/cpython that referenced this pull request Jun 14, 2025

Removed NEWS item

a5bc399

Per python#92900 (comment), not wanted for doc-only PRs.

dlenski force-pushed the gh60722 branch from 4b6143e to a5bc399 Compare June 14, 2025 04:51

dlenski added 3 commits June 13, 2025 22:09

Add warnings about obsolescence of 'email.header.decode_header' and '…

9c0c56f

…email.header.make_header' functions. Recommend use of `email.headerregistry.HeaderRegistry` instead, as suggested in python#92900 (comment)

Removed NEWS item

1216bde

Per python#92900 (comment), not wanted for doc-only PRs.

dlenski force-pushed the gh60722 branch from a5bc399 to 1216bde Compare June 14, 2025 05:09

Merge branch 'main' into gh60722

4c506e5

bitdancer approved these changes Jun 15, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting change review labels Jun 15, 2025

bitdancer merged commit 60181f4 into python:main Jun 15, 2025
42 of 43 checks passed

bedevere-app bot removed the awaiting merge label Jun 15, 2025

bitdancer added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Jun 15, 2025

miss-islington mentioned this pull request Jun 15, 2025

[3.14] gh-67022: Document bytes/str inconsistency in email.header.decode_header() and suggest email.headerregistry.HeaderRegistry as a sane alternative (GH-92900) #135548

Merged

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Jun 15, 2025

miss-islington mentioned this pull request Jun 15, 2025

[3.13] gh-67022: Document bytes/str inconsistency in email.header.decode_header() and suggest email.headerregistry.HeaderRegistry as a sane alternative (GH-92900) #135549

Merged

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Jun 15, 2025

Uh oh!

gh-67022: Document bytes/str inconsistency in email.header.decode_header() and suggest email.headerregistry.HeaderRegistry as a sane alternative #92900

gh-67022: Document bytes/str inconsistency in email.header.decode_header() and suggest email.headerregistry.HeaderRegistry as a sane alternative #92900

Uh oh!

Conversation

dlenski commented May 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented May 17, 2022 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

warsaw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-bot commented Jul 20, 2022

Uh oh!

dlenski commented Jul 20, 2022

Uh oh!

bedevere-bot commented Jul 20, 2022

Uh oh!

dlenski commented Feb 21, 2023

Uh oh!

bedevere-bot commented Feb 21, 2023

Uh oh!

bitdancer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app bot commented Jun 15, 2025

Uh oh!

miss-islington-app bot commented Jun 15, 2025

Uh oh!

bedevere-app bot commented Jun 15, 2025

Uh oh!

bedevere-app bot commented Jun 15, 2025

Uh oh!

Uh oh!

dlenski commented May 17, 2022 •

edited

Loading

ghost commented May 17, 2022 •

edited by ghost

Loading