Turns out that showing warnings in the tokenizer is quite tricky because in the process of showing the warning we need to fetch the encoding which needs to tokenize the first two lines and if the warning is there that leads to an infinite loop. Check for instance with a file contaning:

0b1and x

Notice that this only happens when tokenizing files.

Linked PRs

• gh-99891: Fix infinite recursion in the tokenizer when showing warnings #99893
• [3.11] gh-99891: Fix infinite recursion in the tokenizer when showing warnings (GH-99893) #99896