def test(code):
tokens = list(tokenize.tokenize(iter([code]).__next__))
from5 = tokenize.untokenize(tokens)
print("from5 ", from5, eval(from5))
tokens2 = [tok[:2] for tok in tokens]
from2 = tokenize.untokenize(tokens2)
print("from2 ", from2, eval(from2))>>> test(b't"{ {} }"')
from5 b't"{ {} }"' Template(strings=('', ''), interpolations=(Interpolation({}, ' {}', None, ''),))
from2 b't"{{}}"' Template(strings=('{}',), interpolations=())
>>> test(b'f"{ {} }"')
from5 b'f"{ {} }"' {}
from2 b'f"{ {} }"' {}
Bug report
Bug description:
Found when investigating test.test_tokenize.TestRoundtrip.test_random_files failing on #134577
From what I understand, untokenize should round-trip correctly even with only the type and string of tokens.
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Linked PRs