X Tutup
The Wayback Machine - https://web.archive.org/web/20241202221656/https://github.com/python/cpython/issues/124043
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash when free-threaded Python is built with --with-trace-refs #124043

Closed
XuehaiPan opened this issue Sep 13, 2024 · 9 comments
Closed

Crash when free-threaded Python is built with --with-trace-refs #124043

XuehaiPan opened this issue Sep 13, 2024 · 9 comments
Labels
stdlib Python modules in the Lib dir topic-free-threading type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@XuehaiPan
Copy link
Contributor

XuehaiPan commented Sep 13, 2024 •
edited by bedevere-app bot
Loading

Crash report

What happened?

I'm using ThreadPoolExecutor to write tests for free-threading enabled C extensions.

When running concurrent threads larger than num_workers, CPython crashes. CPython crashes when num_futures >= 2 if the test function is complex enough.

The Python is installed via the following configuration with PyDebug enabled:

./configure --prefix="${PWD}/pydev" \
    --enable-ipv6 --with-openssl="$(brew --prefix openssl)" \
    --with-system-expat --with-system-libmpdec \
    --with-assertions --with-pydebug --with-trace-refs \
    --disable-gil
from concurrent.futures import ThreadPoolExecutor

NUM_WORKERS = 32
NUM_FUTURES = 1024

def concurrent_run(func):
    with ThreadPoolExecutor(max_workers=NUM_WORKERS) as executor:
        for _ in range(NUM_FUTURES):
            executor.submit(func)


concurrent_run(lambda : None)
Python 3.13.0rc2+ experimental free-threading build (heads/3.13:112b1704fa6, Sep 13 2024, 15:34:48) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from concurrent.futures import ThreadPoolExecutor
>>> NUM_WORKERS = 32
... NUM_FUTURES = 1024
... 
... def concurrent_run(func):
...     with ThreadPoolExecutor(max_workers=NUM_WORKERS) as executor:
...         for _ in range(NUM_FUTURES):
...             executor.submit(func)
...             
>>> concurrent_run(lambda : None)
Assertion failed: (value == REFCHAIN_VALUE), function _PyRefchain_Remove, file object.c, line 195.
Fatal Python error: Aborted

Thread 0x0000000173e2b000 (most recent call first):
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 304 in __enter__
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 528 in release
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/concurrent/futures/thread.py", line 87 in _worker
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 992 in run
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x0000000172e1f000 (most recent call first):
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/concurrent/futures/thread.py", line 89 in _worker
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 992 in run
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 1012 in _bootstrap

Thread 0x0000000171e13000 (most recent call first):
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/concurrent/futures/thread.py", line 89 in _worker
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 992 in run
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 1041 in _bootstrap_inner
  File "/Users/PanXuehai/Projects/cpython/pydev/lib/python3.13t/threading.py", line 1012 in _bootstrap

Current thread 0x0000000170e07000 (most recent call first):
Assertion failed: (PyCode_Check(f->f_executable)), function _PyFrame_GetCode, file pycore_frame.h, line 81.

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Output from running 'python -VV' on the command line:

Python 3.13.0rc2+ experimental free-threading build (heads/3.13:112b1704fa6, Sep 13 2024, 15:34:48) [Clang 15.0.0 (clang-1500.3.9.4)]

Linked PRs
@XuehaiPan XuehaiPan added the type-crash A hard crash of the interpreter, possibly with a core dump label Sep 13, 2024
@XuehaiPan XuehaiPan mentioned this issue Sep 13, 2024
Merged
12 tasks
@picnixz picnixz added the stdlib Python modules in the Lib dir label Sep 13, 2024
@ZeroIntensity
Copy link
Member

At a glance, looks like this could be a data race. I'll take a look.

@ZeroIntensity
Copy link
Member

Ah, --with-trace-refs is unsupported under free-threading, so this is a non-issue. As a double check, I confirmed that this doesn't occur with tracerefs disabled. I wonder if configure should disallow mixing --with-trace-refs and --disable-gil, as this isn't the first case of this happening. See #122921, for example.

@encukou, this needs pending.

@colesbury
Copy link
Contributor

Let's disable --with-trace-refs until it works so that more people don't run into this

@colesbury colesbury changed the title Crash for concurrent.futures.ThreadPoolExecutor when num_futures > num_workers on free-threaded Python Crash when building with --with-trace-refs in free-threaded Python Sep 13, 2024
@colesbury colesbury changed the title Crash when building with --with-trace-refs in free-threaded Python Crash when free-threaded Python is built with --with-trace-refs Sep 13, 2024
@bedevere-app bedevere-app bot mentioned this issue Sep 13, 2024
Merged
@ZeroIntensity
Copy link
Member

Let's disable --with-trace-refs until it works so that more people don't run into this

I've created #124078 for this.

colesbury pushed a commit that referenced this issue Sep 16, 2024
@ZeroIntensity
gh-124043: Disallow mixing --with-trace-refs and --disable-gil (#…
3b45df0 
…124078)

Tracing references is not currently thread-safe in the free-threaded build.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Sep 16, 2024
@ZeroIntensity @miss-islington
pythongh-124043: Disallow mixing --with-trace-refs and `--disable-g…
fd6cfdf 
…il` (pythonGH-124078)

Tracing references is not currently thread-safe in the free-threaded build.
(cherry picked from commit 3b45df0)

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
@bedevere-app bedevere-app bot mentioned this issue Sep 16, 2024
Merged
savannahostrowski pushed a commit to savannahostrowski/cpython that referenced this issue Sep 22, 2024
@ZeroIntensity @savannahostrowski
pythongh-124043: Disallow mixing --with-trace-refs and `--disable-g…
cc9d6b5 
…il` (python#124078)

Tracing references is not currently thread-safe in the free-threaded build.
@ZeroIntensity
Copy link
Member

Should this get closed, or are we planning on merging the backport before 3.13.0?

@encukou
Copy link
Member

encukou commented Sep 23, 2024

There's no free-threading before 1.13, so I don't think we should no backports there.
But, do we close issues before the PR is merged? (I assume that it'll be for 3.13.1, next week.)

@ZeroIntensity
Copy link
Member

But, do we close issues before the PR is merged?

I'm not totally sure, I've seen core devs close issues when the only remaining PR is for 3.13, because the branch is locked (but I've also seen other core devs leave issues open, so I don't know).

Yhg1s pushed a commit that referenced this issue Sep 24, 2024
@miss-islington @ZeroIntensity
[3.13] gh-124043: Disallow mixing --with-trace-refs and `--disable-…
8af2d18 
…gil` (GH-124078) (#124138)

gh-124043: Disallow mixing `--with-trace-refs` and `--disable-gil` (GH-124078)

Tracing references is not currently thread-safe in the free-threaded build.
(cherry picked from commit 3b45df0)

Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
@ZeroIntensity
Copy link
Member

I think everything is done here. Thank you for the report, @XuehaiPan!

@terryjreedy
Copy link
Member

I think issues should be left open until all backports done. The right sidebar of PRs have a Development button to make closing the (3.13) PR close the issue. I posted on Discord release-discussions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stdlib Python modules in the Lib dir topic-free-threading type-crash A hard crash of the interpreter, possibly with a core dump
Projects
Threading issues 🧵
Status: Done
Milestone
No milestone
Development

No branches or pull requests
6 participants
@encukou @colesbury @picnixz @XuehaiPan @terryjreedy @ZeroIntensity
X Tutup