gh-132661: Disallow `Template`/`str` concatenation after PEP 750 spec update #135996

davepeck · 2025-06-26T18:23:53Z

Following the steering council decision and corresponding update to PEP750, we are removing support for both implicit and explicit Template/str concatenation.

Issue: Implement PEP 750 – Template Strings #132661

sobolevn

Thank you!

Parser/action_helpers.c

Objects/unicodeobject.c

Lib/test/test_tstring.py

lysnikolaou · 2025-06-30T15:35:06Z

Happy to do this with your input, or happy to hand it off to you!

Happy to help out / review this PR if you wanna take a stab at it. Feel free to reach out if you have any specific questions.

Today was, alas, not that day.

davepeck · 2025-07-08T23:28:49Z

@lysnikolaou Think this is ready. The only question for me is whether we want to go further with _PyPegen_concatenate_strings().

sobolevn

Thank you! Several small nitpicks :)

Lib/test/test_tstring.py

Misc/NEWS.d/next/Core_and_Builtins/2025-07-08-23-22-08.gh-issue-132661.34ftJl.rst

Objects/templateobject.c

lysnikolaou

Thanks @davepeck for working on this! Looks great in general. Left a few unimportant comments and one more significant one about how to implement this in the parser. Also, we'll need to change ast_unparse.c around https://github.com/python/cpython/blob/main/Python/ast_unparse.c#L715.

Lib/test/test_tstring.py

Objects/templateobject.c

lysnikolaou · 2025-07-09T16:06:47Z

Parser/action_helpers.c

+    // Cannot mix strings/f-strings and t-strings
+    if ((unicode_string_found || f_string_found) && t_string_found) {
+        RAISE_SYNTAX_ERROR("cannot mix str and Template literals");
+        return NULL;
+    }


This is not the way to do this. Instead, we should be doing this at the parser level. Right now we're doing this:

atom[expr_ty]: | NAME | 'True' { _PyAST_Constant(Py_True, NULL, EXTRA) } | 'False' { _PyAST_Constant(Py_False, NULL, EXTRA) } | 'None' { _PyAST_Constant(Py_None, NULL, EXTRA) } | &(STRING|FSTRING_START|TSTRING_START) strings # This'll change | NUMBER | &'(' (tuple | group | genexp) | &'[' (list | listcomp) | &'{' (dict | set | dictcomp | setcomp) | '...' { _PyAST_Constant(Py_Ellipsis, NULL, EXTRA) } strings[expr_ty] (memo): a[asdl_expr_seq*]=(fstring|string|tstring)+ { _PyPegen_concatenate_strings(p, a, EXTRA) }

Instead, we should be doing something like:

atom[expr_ty]: | NAME | 'True' { _PyAST_Constant(Py_True, NULL, EXTRA) } | 'False' { _PyAST_Constant(Py_False, NULL, EXTRA) } | 'None' { _PyAST_Constant(Py_None, NULL, EXTRA) } | &(STRING|FSTRING_START) strings # We now have two alternatives | &TSTRING_START tstrings | NUMBER | &'(' (tuple | group | genexp) | &'[' (list | listcomp) | &'{' (dict | set | dictcomp | setcomp) | '...' { _PyAST_Constant(Py_Ellipsis, NULL, EXTRA) } tstrings[expr_ty] (memo): a[asdl_expr_seq*]=tstring+ { <new action here> } strings[expr_ty] (memo): a[asdl_expr_seq*]=(fstring|string)+ { _PyPegen_concatenate_strings(p, a, EXTRA) }

Notice how instead of intermixing the three rules in one sequence, we can now only mix strings and fstrings, but tstrings can only be concatenated with other tstrings. The new action should probably be _build_concatenated_template_str renamed with a _PyPegen prefix, but I'm not sure whether that'll work unchanged, it might need some editing. Also, some of the invalid_* rules might need changing.

Sure thing, will do.

I just wasn't sure we wanted the extra complexity in the grammar.

@lysnikolaou Would you be okay with something more along the lines of:

strings[expr_ty] (memo): | invalid_string_tstring_concat | a[asdl_expr_seq*]=(fstring|string)+ { _PyPegen_concatenate_strings(p, a, EXTRA) } | a[asdl_expr_seq*]=tstring+ { _PyPegen_concatenate_tstrings(p, a, EXTRA) } ... invalid_string_tstring_concat: | (fstring|string)+ a[expr_ty]=tstring { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "cannot mix t-strings with strings or f-strings") } | tstring+ a[expr_ty]=(fstring|string) { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "cannot mix t-strings with strings or f-strings") }

or do we prefer separating strings from tstrings (and I guess having two different invalid rules)

Sure, this looks even better. Can we use RAISE_SYNTAX_ERROR_KNOWN_RANGE instead of KNOWN_LOCATION so that the error is a bit clearer?

Also, is this ready for another review or should I wait for a ping from you?

Can we use RAISE_SYNTAX_ERROR_KNOWN_RANGE

Ah, yeah, that'd be nicer; will do. I guess the "ideal" range would include exactly two neighboring literals that conflict.

Also, is this ready for another review or should I wait for a ping from you?

Not yet, I'll ping you. Thanks! (Still have more to tease apart in action_helpers.c and also need to tackle ast_unparse.)

bedevere-app · 2025-07-09T16:09:22Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

Co-authored-by: sobolevn <mail@sobolevn.me>

…e-132661.34ftJl.rst Co-authored-by: sobolevn <mail@sobolevn.me>

Squashed commit of the following: commit 0941a3088578c1a8002e58ea6e11527bedf0e81e Author: Dave <davepeck@gmail.com> Date: Thu Jul 10 01:49:24 2025 +0000 Maybe cleaner? commit e5a69c851a0f43a4e84325051a37de20282dd810 Author: Dave <davepeck@gmail.com> Date: Wed Jul 9 23:17:58 2025 +0000 In progress: update grammar

pablogsal · 2025-07-10T15:31:44Z

The universe is at peace :)

…nto pep750-concat-update

davepeck · 2025-07-10T16:02:10Z

The universe is at peace :)

Getting closer:

I will not rest until it is perfectly at peace, even I if have to commit uncompilable atrocities. :-P

davepeck · 2025-07-10T17:03:19Z

@lysnikolaou

Okay, I think this is ready! Updated the grammar and actions; fixed up ast_unparse.c.

Two questions:

I suppose we could break _build_concatenated_str() in action_helpers.c into two separate routines. But given how much duplicated logic there would be around flattening/concatenating/re-folding, I decided to leave it untouched. Let me know if you'd like me to tease this apart further.
Because t-string/non-t-string concat errors are handled in the grammar, but bytes/nonbytes are handled in action_helpers.c, t"foo" b"bar" results in the syntax error "cannot mix t-strings with strings or f-strings". Are we okay with that?

(PS: dear Bedevere: I have made the requested changes; please review again)

bedevere-app · 2025-07-10T17:03:25Z

Thanks for making the requested changes!

@lysnikolaou, @JelleZijlstra: please review the changes made to this pull request.

davepeck added 2 commits June 26, 2025 17:46

Disallow explicit concat between Template and str

5d5e187

First pass at removing implicit Template/str concat

1cc778f

bedevere-app bot mentioned this pull request Jun 26, 2025

Implement PEP 750 – Template Strings #132661

Open

sobolevn reviewed Jun 26, 2025

View reviewed changes

Parser/action_helpers.c Outdated Show resolved Hide resolved

Objects/unicodeobject.c Show resolved Hide resolved

Lib/test/test_tstring.py Outdated Show resolved Hide resolved

Merge branch 'main' into pep750-concat-update

7ea498c

dylwil3 mentioned this pull request Jul 4, 2025

Add t-string fixtures for rules that do not need to be modified astral-sh/ruff#19146

Open

davepeck added 8 commits July 8, 2025 12:50

Merge branch 'main' into pep750-concat-update

d08c8b8

Remove extraneous newline

c662041

Add expected exception messages to tests

b17b7c9

Remove _ast_unparse t/f implicit concat helper code

a0c1bb6

Merge branch 'main' into pep750-concat-update

23988e8

Add a parallel comment

d7eadec

Add news blurb for PR.

cb1ad63

One day, I may develop RST muscle memory.

741eaf2

Today was, alas, not that day.

davepeck marked this pull request as ready for review July 8, 2025 23:27

davepeck requested review from lysnikolaou, Eclips4, tomasr8, isidentical, JelleZijlstra and pablogsal as code owners July 8, 2025 23:27

bedevere-app bot added the awaiting review label Jul 8, 2025

Merge branch 'main' into pep750-concat-update

8aeb512

JelleZijlstra approved these changes Jul 9, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Jul 9, 2025

sobolevn reviewed Jul 9, 2025

View reviewed changes

lysnikolaou requested changes Jul 9, 2025

View reviewed changes

bedevere-app bot added awaiting changes and removed awaiting merge labels Jul 9, 2025

davepeck and others added 10 commits July 9, 2025 12:20

Merge branch 'main' into pep750-concat-update

bf5447b

Merge branch 'main' into pep750-concat-update

7e7ea5a

Two minor tweaks

2f79337

Use subtests; test both directions.

dacafdb

Update Objects/templateobject.c

b272287

Co-authored-by: sobolevn <mail@sobolevn.me>

Update Misc/NEWS.d/next/Core_and_Builtins/2025-07-08-23-22-08.gh-issu…

ffae8e9

…e-132661.34ftJl.rst Co-authored-by: sobolevn <mail@sobolevn.me>

Fix bug in suggested change

c6ca45a

Merge branch 'main' into pep750-concat-update

091a2a9

Merge branch 'main' into pep750-concat-update

6a6f734

davepeck added 4 commits July 10, 2025 15:34

Simplify grammar

860f81d

Use KNOWN_RANGE for string/t-string mix error messages

e0fe608

Merge branch 'pep750-concat-update' of github.com:t-strings/cpython i…

c9ed06b

…nto pep750-concat-update

Merge branch 'main' into pep750-concat-update

3ebdd97

Clean up ast_unparse.c append_tstring()

ef1cd5c

bedevere-app bot added awaiting change review and removed awaiting changes labels Jul 10, 2025

bedevere-app bot requested review from JelleZijlstra and lysnikolaou July 10, 2025 17:03

Uh oh!

gh-132661: Disallow Template/str concatenation after PEP 750 spec update #135996

Are you sure you want to change the base?

gh-132661: Disallow Template/str concatenation after PEP 750 spec update #135996

Uh oh!

Conversation

davepeck commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sobolevn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lysnikolaou commented Jun 30, 2025

Uh oh!

davepeck commented Jul 8, 2025

Uh oh!

sobolevn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lysnikolaou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lysnikolaou Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

davepeck Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davepeck Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lysnikolaou Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

lysnikolaou Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

davepeck Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-app bot commented Jul 9, 2025

Uh oh!

pablogsal commented Jul 10, 2025

Uh oh!

davepeck commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davepeck commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app bot commented Jul 10, 2025

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

gh-132661: Disallow `Template`/`str` concatenation after PEP 750 spec update #135996

gh-132661: Disallow `Template`/`str` concatenation after PEP 750 spec update #135996

davepeck commented Jun 26, 2025 •

edited

Loading

davepeck Jul 9, 2025 •

edited

Loading

davepeck Jul 10, 2025 •

edited

Loading

davepeck Jul 10, 2025 •

edited

Loading

davepeck commented Jul 10, 2025 •

edited

Loading

davepeck commented Jul 10, 2025 •

edited

Loading