X Tutup
Skip to content

gh-63161: Fix tokenize.detect_encoding()#139446

Merged
serhiy-storchaka merged 2 commits intopython:mainfrom
serhiy-storchaka:tokenize-detect_encoding
Oct 20, 2025
Merged

gh-63161: Fix tokenize.detect_encoding()#139446
serhiy-storchaka merged 2 commits intopython:mainfrom
serhiy-storchaka:tokenize-detect_encoding

Conversation

@serhiy-storchaka
Copy link
Member

@serhiy-storchaka serhiy-storchaka commented Sep 30, 2025 •
edited by bedevere-app bot
Loading

  • Support non-UTF-8 shebang and comments if non-UTF-8 encoding is specified.
  • Detect decoding error for non-UTF-8 encoding.
  • Detect null bytes in source code.

@serhiy-storchaka
pythongh-63161: Fix tokenize.detect_encoding()
c8c0b20 
* Support non-UTF-8 shebang and comments if non-UTF-8 encoding is specified.
* Detect decoding error for non-UTF-8 encoding.
* Detect null bytes in source code.
@bedevere-app bedevere-app bot mentioned this pull request Sep 30, 2025
Closed
@serhiy-storchaka
Copy link
Member Author

serhiy-storchaka commented Sep 30, 2025

It is a draft until we fix the Python interpreter.

@serhiy-storchaka serhiy-storchaka marked this pull request as ready for review October 13, 2025 12:36
@serhiy-storchaka serhiy-storchaka merged commit 38d4b43 into python:main Oct 20, 2025
47 checks passed
@serhiy-storchaka serhiy-storchaka deleted the tokenize-detect_encoding branch October 20, 2025 17:08
@serhiy-storchaka serhiy-storchaka added the needs backport to 3.14 bugs and security fixes label Oct 20, 2025
@miss-islington-app
Copy link

miss-islington-app bot commented Oct 20, 2025

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 20, 2025
@serhiy-storchaka @miss-islington
pythongh-63161: Fix tokenize.detect_encoding() (pythonGH-139446)
afe4829 
* Support non-UTF-8 shebang and comments if non-UTF-8 encoding is specified.
* Detect decoding error for non-UTF-8 encoding.
* Detect null bytes in source code.
(cherry picked from commit 38d4b43)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-app
Copy link

bedevere-app bot commented Oct 20, 2025

GH-140378 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Oct 20, 2025
serhiy-storchaka added a commit that referenced this pull request Oct 20, 2025
@miss-islington @serhiy-storchaka
[3.14] gh-63161: Fix tokenize.detect_encoding() (GH-139446) (GH-140378)
8e93f6e 
* Support non-UTF-8 shebang and comments if non-UTF-8 encoding is specified.
* Detect decoding error for non-UTF-8 encoding.
* Detect null bytes in source code.
(cherry picked from commit 38d4b43)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
@bedevere-bot
Copy link

bedevere-bot commented Oct 20, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL Refleaks 3.x (tier-1) has failed when building commit 38d4b43.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1610/builds/2273) and take a look at the build logs.
  4. Check if the failure is related to this commit (38d4b43) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1610/builds/2273

Failed tests:

  • test_free_threading

Test leaking resources:

  • test_free_threading: file descriptors
  • test_free_threading: memory blocks

Summary of the results of the build (if available):

==

Click to see traceback logs 
remote: Enumerating objects: 11, done.        
remote: Counting objects:  11% (1/9)        
remote: Counting objects:  22% (2/9)        
remote: Counting objects:  33% (3/9)        
remote: Counting objects:  44% (4/9)        
remote: Counting objects:  55% (5/9)        
remote: Counting objects:  66% (6/9)        
remote: Counting objects:  77% (7/9)        
remote: Counting objects:  88% (8/9)        
remote: Counting objects: 100% (9/9)        
remote: Counting objects: 100% (9/9), done.        
remote: Compressing objects:  16% (1/6)        
remote: Compressing objects:  33% (2/6)        
remote: Compressing objects:  50% (3/6)        
remote: Compressing objects:  66% (4/6)        
remote: Compressing objects:  83% (5/6)        
remote: Compressing objects: 100% (6/6)        
remote: Compressing objects: 100% (6/6), done.        
remote: Total 11 (delta 3), reused 3 (delta 3), pack-reused 2 (from 2)        
From https://github.com/python/cpython
 * branch                    main       -> FETCH_HEAD
Note: switching to '38d4b436ca767351db834189b3a5379406cd52a8'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 38d4b436ca7 gh-63161: Fix tokenize.detect_encoding() (GH-139446)
Switched to and reset branch 'main'

configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module

make: *** [Makefile:2489: buildbottest] Error 2

@bedevere-bot
Copy link

bedevere-bot commented Oct 20, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD Refleaks 3.x (tier-3) has failed when building commit 38d4b43.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1613/builds/2204) and take a look at the build logs.
  4. Check if the failure is related to this commit (38d4b43) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1613/builds/2204

Test leaking resources:

  • test_events: references
  • test_events: memory blocks

Summary of the results of the build (if available):

==

Click to see traceback logs 
Traceback (most recent call last):
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/support/__init__.py", line 847, in gc_collect
    gc.collect()
    ~~~~~~~~~~^^
ResourceWarning: unclosed <socket.socket fd=9, family=2, type=1, proto=6, laddr=('127.0.0.1', 12865), raddr=('127.0.0.1', 12866)>
Task was destroyed but it is pending!
task: <Task pending name='Task-4087' coro=<BaseSelectorEventLoop._accept_connection2() done, defined at /buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/asyncio/selector_events.py:217> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Warning -- Unraisable exception
Exception ignored while calling deallocator <function _SelectorTransport.__del__ at 0x83ecdd310>:
Traceback (most recent call last):
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/asyncio/selector_events.py", line 873, in __del__
    _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ResourceWarning: unclosed transport <_SelectorSocketTransport closing fd=9>
k


Traceback (most recent call last):
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/support/__init__.py", line 847, in gc_collect
    gc.collect()
    ~~~~~~~~~~^^
ResourceWarning: unclosed <socket.socket fd=9, family=2, type=1, proto=6, laddr=('127.0.0.1', 20768), raddr=('127.0.0.1', 20769)>
Task was destroyed but it is pending!
task: <Task pending name='Task-1035' coro=<BaseSelectorEventLoop._accept_connection2() done, defined at /buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/asyncio/selector_events.py:217> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Warning -- Unraisable exception
Exception ignored while calling deallocator <function _SelectorTransport.__del__ at 0x83a6c6810>:
Traceback (most recent call last):
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/asyncio/selector_events.py", line 873, in __del__
    _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ResourceWarning: unclosed transport <_SelectorSocketTransport closing fd=9>
k

@asottile
Copy link
Contributor

asottile commented Oct 29, 2025

changing this regex to bytes appears to break code: asottile/pyupgrade#1038

@serhiy-storchaka
Copy link
Member Author

serhiy-storchaka commented Oct 29, 2025

This is not a public API.

StanFromIreland pushed a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025
@serhiy-storchaka @StanFromIreland
pythongh-63161: Fix tokenize.detect_encoding() (pythonGH-139446)
c98951b 
* Support non-UTF-8 shebang and comments if non-UTF-8 encoding is specified.
* Detect decoding error for non-UTF-8 encoding.
* Detect null bytes in source code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vstinner vstinner Awaiting requested review from vstinner

@pablogsal pablogsal Awaiting requested review from pablogsal pablogsal is a code owner

@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou lysnikolaou is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@serhiy-storchaka @bedevere-bot @asottile
X Tutup