Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-154719: Preserve trailing whitespace in t-string interpolation expressions - #154762

Merged
pablogsal merged 6 commits into
python:mainfrom
lpyu001:fix-tstring
Sep 13, 2026
Merged

gh-154719: Preserve trailing whitespace in t-string interpolation expressions#154762
pablogsal merged 6 commits into
python:mainfrom
lpyu001:fix-tstring

Conversation

@lpyu001

@lpyu001 lpyu001 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

1. Interpolation.expression did not match its documented contract

Doc/library/string.templatelib.rst states that, for interpolations created by t-string literals, expression is

the expression text found inside the curly brackets ({ & }), including
any whitespace
, excluding the curly brackets themselves, and ending before
the first !, :, or = if any is present.

Whitespace after { was preserved, but whitespace before }, !, : or a debug = was removed:

>>> t"{ x }".interpolations[0].expression
' x'      # expected ' x '
>>> t"{  x  }".interpolations[0].expression
'  x'     # expected '  x  '
>>> t"""{
...   x
... }""".interpolations[0].expression
'\n  x'   # expected '\n  x\n'

The asymmetry is not documented anywhere: leading whitespace was kept, trailing whitespace was dropped.

2. ast.unparse() could emit code that no longer parses

When the interpolation ended with an explicit line continuation, stripping the trailing whitespace also removed the newline that terminated it, leaving a dangling backslash in Interpolation.str:

>>> src = 't"""{x \\\n}"""'
>>> ast.parse(src).body[0].value.values[0].str
'x \\'                     # the line continuation lost its newline
>>> ast.unparse(ast.parse(src))
"t'{x \\}'"
>>> ast.parse(ast.unparse(ast.parse(src)))
SyntaxError: unexpected character after line continuation character

Root cause

_strip_interpolation_expr() in Parser/action_helpers.c was applied to every interpolation, not just debug ones. It walked backwards over the lexer metadata removing every trailing whitespace character and =, which is only correct for
the debug (=) form, where the trailing = and the text after it belong to the debug text rather than to the expression.

Solution

  • Regular interpolations now use the lexer metadata verbatim.
  • Debug interpolations keep the text that precedes the debug = and discard only the = itself plus the whitespace and explicit line continuations that follow it. Per the documentation the expression ends before the =, so t"""{value =\<newline>}""" yields 'value ', and the \<newline> stays in the debug text.
  • If the debug marker turns out to be absent, the metadata is returned unchanged instead of silently dropping source text.

Comment thread Parser/action_helpers.c Outdated
}
len--;
}
/* The debug marker may be absent from reconstructed lexer metadata. */

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure this comment is accurate: after the lexer fix, removing comments can never drop the debug = (the marker cannot be inside a comment and the metadata always extends past it), so IIUC this branch is only reachable via things like a line continuation right after the =. Can you adjust the comment or add a test that reaches this branch?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I will address it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A valid case that reached
this branch is an explicit line continuation after the debug marker:

t'''{value =\
}'''

The first patch returned "value =\\\n" as the expression. This conflicts with the [Interpolation.expression documentation(https://docs.python.org/3.16/library/string.templatelib.html#string.templatelib.Interpolation.expression),which says the expression ends before =. The expected expression is "value ".

I updated the code to strip explicit line continuations after the debug marker and added regression tests.

Comment thread Parser/lexer/string.c Outdated

// Copy escaped characters without interpreting the escaped
// character as a quote or comment marker.
if (ch == '\\') {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lexer fix also changes the debug text for f-strings (_PyLexer_set_ftstring_expr is shared when in_debug is set), e.g. a debug expression combining a comment with an escaped quote was truncated before this. Can you add a test for the f-string case in test_fstring as well? Maybe worth mentioning f-strings in the NEWS entry too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added the tests and updated the NEWS file

@JelleZijlstra
JelleZijlstra removed their request for review August 15, 2026 01:45
@pablogsal pablogsal added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes needs backport to 3.15 pre-release feature fixes, bugs and security fixes and removed needs backport to 3.13 bugs and security fixes labels Sep 13, 2026
@pablogsal
pablogsal merged commit 1a703ab into python:main Sep 13, 2026
66 checks passed
@miss-islington-app

Copy link
Copy Markdown

Thanks @lpyu001 for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14, 3.15.
🐍🍒⛏🤖

@pablogsal

Copy link
Copy Markdown
Member

Thanks a lot for the fix!

@miss-islington-app

Copy link
Copy Markdown

Sorry, @lpyu001 and @pablogsal, I could not cleanly backport this to 3.15 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 1a703ab9e6a050b40c469788ec7758713c6bf62b 3.15

@miss-islington-app

Copy link
Copy Markdown

Sorry, @lpyu001 and @pablogsal, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 1a703ab9e6a050b40c469788ec7758713c6bf62b 3.14

@pablogsal

Copy link
Copy Markdown
Member

@lpyu001 can you do the backports following @miss-islington instructions?

@lpyu001

lpyu001 commented Sep 13, 2026

Copy link
Copy Markdown
Contributor Author

@lpyu001 can you do the backports following @miss-islington instructions?

Sure, I can do that.

@bedevere-app

bedevere-app Bot commented Sep 13, 2026

Copy link
Copy Markdown

GH-157392 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Sep 13, 2026
@bedevere-app

bedevere-app Bot commented Sep 13, 2026

Copy link
Copy Markdown

GH-157392 is a backport of this pull request to the 3.14 branch.

@bedevere-app

bedevere-app Bot commented Sep 13, 2026

Copy link
Copy Markdown

GH-157394 is a backport of this pull request to the 3.15 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Sep 13, 2026
pablogsal pushed a commit that referenced this pull request Sep 13, 2026
pablogsal pushed a commit that referenced this pull request Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants