py: Add PEP 750 template strings support #17557

koxudaxi · 2025-06-24T19:32:57Z

Summary

Implements PEP 750 template strings for MicroPython.

Started in discussion #17497. Template strings (t-strings) are new in Python 3.14 - they return Template objects instead of strings, so you can access the literal parts and expressions separately.

Changes:

t-string parsing in lexer/parser
Template and Interpolation objects
string.templatelib module
Conditional build with MICROPY_PY_TSTRINGS

Usage:

t = t"Hello {name}!"
str(t)  # "Hello World!"
t.args[1].expression  # "name" 
t.args[1].value  # "World"

Testing

Tested on unix port with both MICROPY_PY_TSTRINGS enabled and disabled.

Test coverage:

Basic template string functionality
Error handling and edge cases
Import system integration
Nested expression parsing

All existing tests continue to pass. New tests added in tests/basics/string_template*.py.

Ports tested: Unix (other ports need testing)

Trade-offs and Alternatives

Adds ~240 bytes when enabled on unix port. When disabled, no impact.

Worth it because:

Part of Python 3.14 standard
Optional feature (MICROPY_PY_TSTRINGS)
Useful for template processing

Config: Enabled by default when MICROPY_CONFIG_ROM_LEVEL >= EXTRA_FEATURES.
Needs MICROPY_PY_FSTRINGS=1.

To disable: make CFLAGS_EXTRA=-DMICROPY_PY_TSTRINGS=0

WebReflection · 2025-06-24T19:54:35Z

for what is worth it, we'd love to have this available at least for the PyScript WASM variant as this unlocks tons of UI related use cases we'd like to deliver to our users.

/cc @dpgeorge @ntoll

codecov · 2025-06-25T02:23:46Z

Codecov Report

Attention: Patch coverage is 98.44021% with 9 lines in your changes missing coverage. Please review.

Project coverage is 98.44%. Comparing base (df05cae) to head (198a846).
Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
py/parse.c	97.50%	4 Missing ⚠️
py/tstring_expr_parser.c	95.16%	3 Missing ⚠️
py/lexer.c	97.93%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #17557      +/-   ##
==========================================
- Coverage   98.44%   98.44%   -0.01%     
==========================================
  Files         171      174       +3     
  Lines       22192    22771     +579     
==========================================
+ Hits        21847    22416     +569     
- Misses        345      355      +10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-06-25T04:56:57Z

Code size report:

   bare-arm:   +68 +0.120% 
minimal x86:  +117 +0.062% [incl +32(data)]
   unix x64:   +56 +0.007% standard[incl +32(data)]
      stm32:   +12 +0.003% PYBV10
     mimxrt:   +24 +0.006% TEENSY40
        rp2:   +16 +0.002% RPI_PICO_W
       samd:   +48 +0.018% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:   +23 +0.005% VIRT_RV32

dpgeorge

Thanks for the contribution! This will be nice to have for the webassembly port.

I didn't do a review yet, but there will need to be tests to get full coverage of the new code.

docs/library/btree.rst

py/modbuiltins.c

koxudaxi · 2025-06-25T06:33:07Z

@dpgeorge Thank you for the feedback and for taking time to look at this! I'm glad this will be useful for the webassembly port.

I'll add more tests to ensure full coverage when I find time.

ntoll · 2025-06-25T08:59:43Z

@koxudaxi slightly off topic - but I notice you'll be at EuroPython in Prague, as will I. We should look out for each other and have a coffee or lunch together! 🇪🇺 🐍

koxudaxi · 2025-06-25T18:05:58Z

@ntoll Sounds good! See you in Prague. ☕

frozen/string/templatelib.py

koxudaxi · 2025-07-02T16:04:39Z

@dpgeorge

I didn't do a review yet, but there will need to be tests to get full coverage of the new code.

I tried really hard to make 100% coverage but some code is never called and I cannot cover it. Do you know how to fix this?

koxudaxi · 2025-07-02T16:08:53Z

docs/library/btree.rst


-   A flag for `keys()`, `values()`, `items()` methods to specify that
+   A flag for :meth:`btree.keys`, :meth:`btree.values`, :meth:`btree.items` methods to specify that
   scanning should be inclusive of the end key.

 .. data:: DESC

-   A flag for `keys()`, `values()`, `items()` methods to specify that
+   A flag for :meth:`btree.keys`, :meth:`btree.values`, :meth:`btree.items` methods to specify that


I faced an error when building with Sphinx because it conflicted with string.templatelib.Template.values. To solve this problem, I renamed the values attribute in btree to be more explicit.

It seems like a useful change, then --- it's just not a change related to PEP-750 t-string support, so it needs to be a separate commit and separate PR.
Git cherry-pick and rebase -i are your friends!

koxudaxi · 2025-07-02T16:44:06Z

@dpgeorge
I'm not very familiar with the micropython codebase, so please let me know if you notice any issues with how I'm handling memory constraints. I think everything looks good, but I'd appreciate your review.

dpgeorge · 2025-07-03T01:35:13Z

py/objtype.c

@@ -542,7 +542,18 @@ const byte mp_binary_op_method_name[MP_BINARY_OP_NUM_RUNTIME] = {
 };

 static mp_obj_t instance_binary_op(mp_binary_op_t op, mp_obj_t lhs_in, mp_obj_t rhs_in) {
+    #if MICROPY_PY_TSTRINGS
+    if (op == MP_BINARY_OP_ADD || op == MP_BINARY_OP_INPLACE_ADD) {


This whole bit can be removed, it's not needed. There is already a check in template_binary_op().

dpgeorge · 2025-07-03T01:36:42Z

ports/unix/coverage.c

@@ -822,6 +847,356 @@ static mp_obj_t extra_coverage(void) {
            MICROPY_STACK_CHECK == 0 || old_stack_limit == new_stack_limit);
    }

+
+    #if MICROPY_PY_TSTRINGS


I'm surprised all of this coverage code is needed here, in C.

You should be able to move most (if not all) of this to pure Python tests. Did you have trouble doing that?

dpgeorge · 2025-07-03T01:39:07Z

py/modtstring.c

+                    mp_print_t repr_print;
+                    vstr_init_print(&repr_vstr, 16, &repr_print);
+                    mp_obj_print_helper(&repr_print, value, PRINT_REPR);
+                    conv_value = mp_obj_new_str_from_vstr(&repr_vstr);


These 3 cases for r/s/a all look the same. Can you factor the code to reduce code duplication?

dpgeorge · 2025-07-03T01:42:58Z

py/modtstring.c

+                }
+                dest[0] = mp_obj_new_tuple(interps->len, values);
+            } else {
+                mp_obj_t *values = m_new(mp_obj_t, interps->len);


You can simplify the code here and not have any temporary memory by the following:

mp_obj_tuple_t *tuple = MP_OBJ_TO_PTR(mp_obj_new_tuple(interps->len, NULL)); for (size_t i = 0; i < interps->len; i++) { tuple->items[i] = interp->value; } dest[0] = MP_OBJ_FROM_PTR(tuple);

dpgeorge · 2025-07-03T01:45:58Z

py/modtstring.c

+            return MP_OBJ_FROM_PTR(result);
+        }
+
+        case MP_BINARY_OP_REVERSE_ADD: {


You can just remove this case, it's not needed.

dpgeorge · 2025-07-03T01:48:13Z

py/parse.c

+        est_bytes += est_seg_cnt * sizeof(mp_parse_node_t) * 2; // Assume some growth
+        est_bytes += est_interp_cnt * sizeof(mp_parse_node_t) * 2;
+
+        if (est_bytes > MICROPY_PY_TSTRING_MAX_BYTES) {


I don't think this test is needed, what's wrong with the tstring growing large?

Change t-strings from being automatically enabled at EXTRA_FEATURES level to opt-in only. This prevents token number shifts in lexer that were causing CI test failures for ports that don't need t-strings. T-strings are now explicitly enabled only for: - Unix/macOS port - Windows port - WebAssembly port This keeps t-strings available for desktop environments while avoiding memory overhead and test conflicts in embedded ports. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

Change t-strings from being automatically enabled at EXTRA_FEATURES level to opt-in only. This prevents token number shifts in lexer that were causing CI test failures for ports that don't need t-strings. T-strings are now explicitly enabled only for: - Unix/macOS port - Windows port - WebAssembly port This keeps t-strings available for desktop environments while avoiding memory overhead and test conflicts in embedded ports. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

The token number changed from 16 to 17 due to t-strings being made opt-in, which shifted the token numbering scheme. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

Change {expr=:fmt} to use conversion=None instead of 's' to match the updated PEP-750 specification. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

Add tests for t-string debug specifier (=) to verify compliance with updated PEP-750 specification: - {expr=} uses conversion='r' - {expr=:fmt} uses conversion=None - {expr=!s} preserves explicit conversion Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

- Make t-strings conditional on ROM_LEVEL for Unix port - Add skip_tstring check to run-tests.py Fixes test failures on minimal builds and embedded ports. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

Replace math.pi/e/sqrt(2) with integer values to avoid precision differences between single and double precision builds. Also show interpolation count and format specs instead of actual float values in the output. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

T-strings use recursive parsing that requires significant stack space. In constrained builds (standard/standard_v2), thread tests create threads with small 2KB stacks which overflow when t-strings are enabled. This changes MICROPY_PY_TSTRINGS to only be enabled when MICROPY_CONFIG_ROM_LEVEL_AT_LEAST_EXTRA_FEATURES. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

T-strings were failing in threads because __template__ is not in the thread's global namespace. Add special handling in mp_load_global to return the builtin __template__ directly when not found in globals, similar to how __build_class__ is handled. This fixes NameError when using t-strings in threads and also resolves the select_poll_eintr.py test failure. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

This commit implements PEP-750 compliant format specification evaluation for t-strings. Format specs containing expressions (e.g., {width}) are now evaluated at template creation time, matching Python 3.14 behavior. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

This commit adds extensive test coverage for escape sequence handling in t-strings to improve lexer.c coverage Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

When running in a thread context, __template__ was not being found if it wasn't in the thread's local globals. This adds a fallback to check the main thread's globals dictionary when __template__ is not found in the current globals, ensuring t-strings work correctly in threaded environments. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

__template__ is already registered as a builtin, so special handling in mp_load_global() is redundant. The select_poll_eintr.py test failure that this code was trying to fix is actually a pre-existing issue unrelated to t-strings. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

T-strings use recursive parsing which requires significant stack space. This causes thread initialization failures in constrained environments when threads create small stacks (e.g., 8-16KB). The failure manifests as NameError when accessing variables in thread functions. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

This commit fixes a critical GC vulnerability in modtstring.c where temporary arrays were allocated but not protected from garbage collection. The issue manifested as random thread initialization failures in CI environments. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

- Test empty and whitespace-only expressions in t-strings - Test format spec and conversion specifier ordering validation - Test integer overflow protection in template size calculation - Test Unicode/octal escape sequences and invalid escape handling - Test high byte values (0x80-0xFF) and lexer error states - Test memory allocation failures in expression parser - Test debug format with invalid conversion syntax - Add coverage for all error branches in t-string implementation Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

koxudaxi changed the title ~~py: Add PEP 750 template strings support.~~ py: Add PEP 750 template strings support Jun 24, 2025

dpgeorge added the py-core Relates to py/ directory in source label Jun 25, 2025

dpgeorge reviewed Jun 25, 2025

View reviewed changes

docs/library/btree.rst Outdated Show resolved Hide resolved

py/modbuiltins.c Outdated Show resolved Hide resolved

dpgeorge reviewed Jun 26, 2025

View reviewed changes

frozen/string/templatelib.py Outdated Show resolved Hide resolved

koxudaxi force-pushed the feature/pep750-template-strings branch 10 times, most recently from 7a1dc11 to 50dd4d1 Compare July 2, 2025 15:42

koxudaxi requested a review from dpgeorge July 2, 2025 15:59

koxudaxi commented Jul 2, 2025

View reviewed changes

dpgeorge reviewed Jul 3, 2025

View reviewed changes

koxudaxi force-pushed the feature/pep750-template-strings branch from e5f68df to 8ecf945 Compare July 12, 2025 01:26

koxudaxi added 3 commits July 12, 2025 01:38

docs: Fix method links in btree.rst.

12e6225

Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

tests/cmdline: Update cmd_parsetree.py.exp for new token numbering.

9e6fa7d

The token number changed from 16 to 17 due to t-strings being made opt-in, which shifted the token numbering scheme. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

koxudaxi force-pushed the feature/pep750-template-strings branch from 20fbafe to 9e6fa7d Compare July 12, 2025 02:29

koxudaxi added 2 commits July 12, 2025 15:06

py/parse: Update t-string debug specifier per PEP-750.

4bd4ecd

Change {expr=:fmt} to use conversion=None instead of 's' to match the updated PEP-750 specification. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

koxudaxi force-pushed the feature/pep750-template-strings branch from 3b55c40 to 966c931 Compare July 12, 2025 16:22

koxudaxi added 2 commits July 12, 2025 17:10

tests: Add t-string feature detection to fix CI failures.

c9b36f1

- Make t-strings conditional on ROM_LEVEL for Unix port - Add skip_tstring check to run-tests.py Fixes test failures on minimal builds and embedded ports. Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

koxudaxi force-pushed the feature/pep750-template-strings branch 3 times, most recently from 79f6712 to 51d8c21 Compare July 14, 2025 01:23

koxudaxi force-pushed the feature/pep750-template-strings branch from 5601a24 to 29245de Compare July 14, 2025 03:18

koxudaxi added 4 commits July 14, 2025 05:48

py: Revert MICROPY_STACK_CHECK_MARGIN changes.

8e88961

Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

tests/basics: Add comprehensive escape sequence tests for t-strings.

08697eb

This commit adds extensive test coverage for escape sequence handling in t-strings to improve lexer.c coverage Signed-off-by: Koudai Aono <koxudaxi@gmail.com>

koxudaxi force-pushed the feature/pep750-template-strings branch from adb0c30 to 08697eb Compare July 14, 2025 08:45

koxudaxi added 4 commits July 14, 2025 10:28

koxudaxi force-pushed the feature/pep750-template-strings branch 3 times, most recently from e0a7b89 to e28b3d9 Compare July 15, 2025 14:27

koxudaxi force-pushed the feature/pep750-template-strings branch from e28b3d9 to 198a846 Compare July 15, 2025 15:49

Uh oh!

py: Add PEP 750 template strings support #17557

Are you sure you want to change the base?

py: Add PEP 750 template strings support #17557

Conversation

koxudaxi commented Jun 24, 2025

Summary

Testing

Trade-offs and Alternatives

Uh oh!

WebReflection commented Jun 24, 2025

Uh oh!

codecov bot commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dpgeorge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

koxudaxi commented Jun 25, 2025

Uh oh!

ntoll commented Jun 25, 2025

Uh oh!

koxudaxi commented Jun 25, 2025

Uh oh!

Uh oh!

koxudaxi commented Jul 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

koxudaxi commented Jul 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

codecov bot commented Jun 25, 2025 •

edited

Loading

github-actions bot commented Jun 25, 2025 •

edited

Loading