PyREPL doesn't correctly color double braces in f-strings / t-strings #134158
Description
Bug report
Bug description:
Syntax highlighting in PyREPL (added in #131507) does not correctly color the double-braces used for escaping braces in f-strings or t-strings:
After investigation, this is caused by Python tokenizer producing only 1 brace in this case:
>>> from tokenize import tokenize
>>> from io import BytesIO
>>> from pprint import pp
>>>
>>> pp(list(tokenize(BytesIO(b'f"a{{b"').readline)))
[TokenInfo(type=68 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line=''),
TokenInfo(type=59 (FSTRING_START), string='f"', start=(1, 0), end=(1, 2), line='f"a{{b"'),
TokenInfo(type=60 (FSTRING_MIDDLE), string='a{', start=(1, 2), end=(1, 4), line='f"a{{b"'),
TokenInfo(type=60 (FSTRING_MIDDLE), string='b', start=(1, 5), end=(1, 6), line='f"a{{b"'),
TokenInfo(type=61 (FSTRING_END), string='"', start=(1, 6), end=(1, 7), line='f"a{{b"'),
TokenInfo(type=4 (NEWLINE), string='', start=(1, 7), end=(1, 8), line='f"a{{b"'),
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]So PyREPL think there is only one brace in the original string, and doesn't color the second one.
I have a fix for this issue, PR on the way!
CPython versions tested on:
3.14, CPython main branch
Operating systems tested on:
Windows, macOS