py: Add PEP 750 template strings support #17557

koxudaxi · 2025-06-24T19:32:57Z

Implements PEP 750 template strings for MicroPython.

Started in discussion #17497. Template strings return Template objects instead of strings, allowing access to literal parts and expressions separately.

Changes:

Lexer/parser support for t-string syntax
Template and Interpolation types
Public type definitions in py/objtstring.h
Minimal builtin string.templatelib module
Conditional build with MICROPY_PY_TSTRINGS

Usage:

from string.templatelib import Template

t = t"Hello {name}!"
t.strings         # ('Hello ', '!')
t.values          # ('World',)
t.interpolations[0].expression  # 'name'

t"{x!r:>10}"      # Conversions: !r, !s, !a
rt"C:\{file}"     # Raw t-strings

Implementation:

Lexer (py/lexer.c) and parser (py/parse.c) handle tokenization
Template/Interpolation types in py/modtstring.c, py/objinterpolation.c
Public header py/objtstring.h eliminates code duplication
Minimal string module in py/modstring.c (no manifest.py changes)
Dynamic array doubling (4→8→16→...→2048), GC-safe
CPython 3.14 compatible error messages

Testing:

Tested on unix (coverage, standard, minimal), windows, webassembly.

New tests in tests/basics/:

string_module_tstring.py
tstring_basic.py, tstring_parser.py, tstring_errors.py
tstring_format.py, tstring_operations.py, tstring_coverage.py

Feature detection in tests/feature_check/tstring.py - automatic skip when MICROPY_PY_TSTRINGS=0.

All existing tests pass.

Trade-offs:

Code size: ~10 KB when enabled, zero when disabled.

String module: minimal builtin with only templatelib blocks micropython-lib's full string module. Future: coordinate with micropython-lib.

Config: enabled when MICROPY_CONFIG_ROM_LEVEL >= EXTRA_FEATURES, requires MICROPY_PY_FSTRINGS=1.

To disable: make CFLAGS_EXTRA=-DMICROPY_PY_TSTRINGS=0

WebReflection · 2025-06-24T19:54:35Z

for what is worth it, we'd love to have this available at least for the PyScript WASM variant as this unlocks tons of UI related use cases we'd like to deliver to our users.

/cc @dpgeorge @ntoll

codecov · 2025-06-25T02:23:46Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.43%. Comparing base (6773051) to head (d324425).
⚠️ Report is 22 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #17557      +/-   ##
==========================================
+ Coverage   98.38%   98.43%   +0.05%     
==========================================
  Files         171      174       +3     
  Lines       22297    23031     +734     
==========================================
+ Hits        21936    22670     +734     
  Misses        361      361

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-06-25T04:56:57Z

Code size report:

Reference:  esp32/boards: Add Silicognition ManT1S board definition. [27544a2]
Comparison: tests: Add .exp outputs for template string suite. [merge of d324425]
  mpy-cross: +7128 +1.880% [incl +128(data)]
   bare-arm:   +16 +0.028% 
minimal x86:  +126 +0.067% 
   unix x64: +14656 +1.708% standard[incl +896(data)]
      stm32:   +72 +0.018% PYBV10
     mimxrt:   +64 +0.017% TEENSY40
        rp2:   +88 +0.010% RPI_PICO_W
       samd:   +72 +0.026% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:   +60 +0.013% VIRT_RV32

dpgeorge

Thanks for the contribution! This will be nice to have for the webassembly port.

I didn't do a review yet, but there will need to be tests to get full coverage of the new code.

docs/library/btree.rst

py/modbuiltins.c

koxudaxi · 2025-06-25T06:33:07Z

@dpgeorge Thank you for the feedback and for taking time to look at this! I'm glad this will be useful for the webassembly port.

I'll add more tests to ensure full coverage when I find time.

ntoll · 2025-06-25T08:59:43Z

@koxudaxi slightly off topic - but I notice you'll be at EuroPython in Prague, as will I. We should look out for each other and have a coffee or lunch together! 🇪🇺 🐍

koxudaxi · 2025-06-25T18:05:58Z

@ntoll Sounds good! See you in Prague. ☕

frozen/string/templatelib.py

koxudaxi · 2025-07-02T16:04:39Z

@dpgeorge

I didn't do a review yet, but there will need to be tests to get full coverage of the new code.

I tried really hard to make 100% coverage but some code is never called and I cannot cover it. Do you know how to fix this?

docs/library/btree.rst

koxudaxi · 2025-07-02T16:44:06Z

@dpgeorge
I'm not very familiar with the micropython codebase, so please let me know if you notice any issues with how I'm handling memory constraints. I think everything looks good, but I'd appreciate your review.

py/objtype.c

ports/unix/coverage.c

py/modtstring.c

dpgeorge · 2025-07-03T01:45:58Z

py/modtstring.c

+            return MP_OBJ_FROM_PTR(result);
+        }
+
+        case MP_BINARY_OP_REVERSE_ADD: {


You can just remove this case, it's not needed.

py/parse.c

AJMansfield · 2025-10-29T16:54:10Z

Feel free to ping me once you've finished working through this to a state that's ready to be re-reviewed.

WebReflection · 2025-10-29T20:29:28Z

@kumekay are you paying attention to hints? if you need any help at all, mind stating what is it?

I am following pushes and I feel like you are stuck in some testing loop and there's zero shame in that ... but maybe others can help with? Surely not helping your case but I see tons of pushes daily and I kinda sympathize with your possible frustration, specially when I see tests added or removed to make CI green .... maybe there's a better way, as everything else feels right in place so that asking "how do I make CI happy in there?" might be a faster way out.

kumekay · 2025-10-30T04:44:12Z

@kumekay are you paying attention to hints? if you need any help at all, mind stating what is it?

I am following pushes and I feel like you are stuck in some testing loop and there's zero shame in that ... but maybe others can help with? Surely not helping your case but I see tons of pushes daily and I kinda sympathize with your possible frustration, specially when I see tests added or removed to make CI green .... maybe there's a better way, as everything else feels right in place so that asking "how do I make CI happy in there?" might be a faster way out.

@WebReflection
You probably need wanted to mention @koxudaxi here ;)

WebReflection · 2025-10-30T08:20:12Z

@kumekay apologies and thank you for bringing my comment to @koxudaxi attention ... too many @k, my mistake 😅

koxudaxi · 2025-10-30T16:54:57Z

@WebReflection
Thank you for your kind comment.
I'm indeed struggling a bit, but I'm fine-tuning the detailed behavior to match CPython and PEP 750 as closely as possible while also aligning with MicroPython's design principles.

I think I've squashed most of the bugs, and what remains is the module path issue.

@AJMansfield @dpgeorge
I'd like to discuss the module structure.
In t-strings, we need to provide the string.templatelib namespace from which classes like Template and Interpolation are imported.
from string.templatelib import Template
In CPython, this is provided as a Python file in the stdlib, but I'm wondering what the best way to arrange this in MicroPython would be. I was thinking of placing string.templatelib as a builtin, but I realized that if someone imports string from micropython-lib, it would overwrite these attributes. Is there a good approach for this?
I also considered moving templatelib entirely to micropython-lib, but then I realized this could complicate the build configuration.

https://docs.python.org/3/library/string.templatelib.html
https://github.com/python/cpython/blob/95a3564869daa3cc083e4d4603f5f03ee5f53e31/Lib/string/templatelib.py#L5

I believe everything else should be fine, so I'd appreciate another review when you have a chance.

AJMansfield · 2025-10-30T21:22:15Z

I'd like to discuss the module structure. In t-strings, we need to provide the string.templatelib namespace from which classes like Template and Interpolation are imported. from string.templatelib import Template In CPython, this is provided as a Python file in the stdlib, but I'm wondering what the best way to arrange this in MicroPython would be. I was thinking of placing string.templatelib as a builtin, but I realized that if someone imports string from micropython-lib, it would overwrite these attributes. Is there a good approach for this? I also considered moving templatelib entirely to micropython-lib, but then I realized this could complicate the build configuration.

Normally, implementing a module as frozen mpy bytecode will actually be smaller than the equivalent implementation in C. It's also a common pattern to define an underscore-prefixed module in C to provide a minimal set of primitives to access some interpreter feature or api or perform the performance-critical hot loop of a package, and then implement the rest of that module in python. For an example of this, take a look at how the asyncio module is implemented. extmod/modasyncio.c doesn't actually define a python-api-compatible asyncio package, it just defines _asyncio to give C definitions for Task and TaskQueue, and then implements everything else in a python module --- extmod/asyncio/__init__.py and its siblings.

Instead of defining the recursive module structure manually in modstring.c and special-casing the import handling, this package structure could be implemented in python code, with a extmod/string/__init__.py, a extmod/string/templatelib.py, and then a modtstring.c that exports a bare _tstring module that exports the mp_obj_interpolation_t and mp_obj_template_t types.

Also, given the massive code size increase at stake here, it might even be worth considering if there's a way to move even more of the parse logic into something that can run from bytecode --- perhaps even, for some ports, as an externally mip-installable component rather than something to freeze in. For instance, have MicroPython's parser just minimally recognize t"(whatever)", and then invoke something along the lines of:

static mp_obj_t tstring_parse_func = NULL;
mp_obj_t tstring_parse(vstr_t *str) {
    if (tstring_parse == NULL) {
        mp_obj_t modtstring = mp_builtin___import__(1, &MP_OBJ_NEW_QSTR(MP_QSTR_mp_tstring_support));
        // NLR this to a "tstring support not installed" error? Maybe with directions to run `mip.install("mp_tstring_support")`?
        tstring_parse_func = mp_obj_dict_get(mp_obj_module_get_globals(modtstring), MP_QSTR_parse);
    }
    return mp_call_function_1(tstring_parse_func, mp_obj_new_str_from_vstr(str));
}

koxudaxi · 2025-11-02T19:54:26Z

@AJMansfield
Thank you for the detailed comment. I applied your suggestions. Could you please review it?
Regarding the parser implementation, I had a misunderstanding. I thought writing it in C was better aligned with MicroPython's approach.
However, I tried to follow the f-strings implementation. Since we have enough tests, rewriting the parser in Python as you suggested is feasible.
About the mip-installable approach, my original plan was making t-strings a build option to avoid size increase for unnecessary targets. Could you clarify your suggestion?
Should it be mip-installable instead of a build option, or in addition to it?

(If I understand correctly, no changes to micropython-lib are needed.)

AJMansfield · 2025-11-02T21:12:43Z

Thank you for the detailed comment. I applied your suggestions. Could you please review it?

Will do! Hopefully I'll be able to get to this on Monday.

I thought writing it in C was better aligned with MicroPython's approach.

That's my assumption as well --- I wouldn't put too much stock in my musings on that idea, for whether or not it might be possible to let portions of the parser operate from bytecode. If that's anything at all, it's certainly not something I'd put in the way of the current version being merged.

Signed-off-by: Koudai Aono <[email protected]>

koxudaxi · 2025-11-10T17:28:22Z

@AJMansfield
I've reorganized the commits based on your previous feedback. If the commit granularity is still too coarse or inappropriate, please let me know and I'll revise them.

AJMansfield · 2025-11-15T16:59:53Z

Apologies for the delay reviewing, finally found the time though. And that's a well-organized set of commits I'll have no trouble examining.

AJMansfield

On examination of the code and after running it on my machine, I can say that the functional code in this PR seems substantively correct, but there's a few details to still clean up especially with regards to testing.

Most of the tests here do not pass when run in comparison with a real CPython 3.14 instance (i.e. --- but with only a small number of lines differing, on error behavior and other areas where differences are commonly accepted. Usually, when testing a feature that's broadly compatible but with a minor deviation, it's a common pattern to separate out the compatible and incompatible cases between e.g. some_test.py and some_test1.py, so that you can provide a some_test1.py.exp without removing the cross-compare for all of it.

It's reasonable enough to also be using .exp files here to bridge the gap to 3.14-less testing environments, but I'd like to still see these tests split into separate fully-3.14-compatible and non-3.14-compatible test files. Note that the .exp's for the non-3.14-compatible ones should go in the same commit with the tests --- i.e. it should be a complete commit with everything needed to run and pass a comparison in an environment that does have the right CPython, with the follow-up commit containing only exactly the files generated by running e.g. python3.14 some_test.py &> some_test.py.exp for the relevant tests.

(As a related note, we've actually been needing a mechanism to address this scenario for a while --- ever since CPython changed tracing formats making MicroPython's outputs on the tests for sys.settrace no longer match on newer versions. If you'd be interested in taking on adding that --- maybe a table similar to tests_with_regex_output that gives CPython version ranges to use or disregard exp files on, that would be and then use or disregard their exp file when appropriate --- it would be quite useful indeed to have. If not though, I'll be satisfied with just having the tests split.)

AJMansfield · 2025-11-15T17:20:23Z

tests/feature_check/tstring.py.exp

This exp should be in the commit that adds the feature check; it's needed to even run the tstring tests at all.

AJMansfield · 2025-11-15T17:26:40Z

tests/micropython/tstring_validation.py

This should probably be a micropython test rather than a basics test, as __template__ is a micropython-specific implementation detail.

AJMansfield · 2025-11-15T17:41:23Z

py/modbuiltins.c

    #endif
    { MP_ROM_QSTR(MP_QSTR_ZeroDivisionError), MP_ROM_PTR(&mp_type_ZeroDivisionError) },
+    #if MICROPY_PY_TSTRINGS
+    { MP_ROM_QSTR(MP_QSTR___template__), MP_ROM_PTR(&mp_builtin___template___obj) },


What's the reasoning for putting this at the end, rather than with the other double-underscore builtins? (Note that __print_repl__ is actually also a micropython-specific builtin.)

I've moved __template__ to group it with the other double-underscore builtins.

Signed-off-by: Koudai Aono <[email protected]>

koxudaxi changed the title ~~py: Add PEP 750 template strings support.~~ py: Add PEP 750 template strings support Jun 24, 2025

dpgeorge added the py-core Relates to py/ directory in source label Jun 25, 2025

dpgeorge reviewed Jun 25, 2025

View reviewed changes

docs/library/btree.rst Show resolved Hide resolved

py/modbuiltins.c Outdated Show resolved Hide resolved

dpgeorge reviewed Jun 26, 2025

View reviewed changes

frozen/string/templatelib.py Outdated Show resolved Hide resolved

koxudaxi force-pushed the feature/pep750-template-strings branch 10 times, most recently from 7a1dc11 to 50dd4d1 Compare July 2, 2025 15:42

koxudaxi requested a review from dpgeorge July 2, 2025 15:59

koxudaxi commented Jul 2, 2025

View reviewed changes

docs/library/btree.rst Show resolved Hide resolved