Crash at finalization after fail to start new thread #109746

chgnrdv · 2023-09-22T15:48:49Z

Crash report

What happened?

Bisected to e11fc03, but I guess this issue exists longer, and assertion that is added to PyThreadState_Clear by this commit just made it visible.

import resource
import threading

# this isn't essential, but helps PyThread_start_new_thread() fail
resource.setrlimit(resource.RLIMIT_NPROC, (150, 150))

while True:
    t = threading.Thread()
    t.start()
    t.join()

Error message with backtrace:

Traceback (most recent call last):
  File "/home/radislav/projects/cpython/thread_repro.py", line 10, in <module>
    t.start()
  File "/home/radislav/projects/cpython/Lib/threading.py", line 978, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
python: Python/pystate.c:1484: PyThreadState_Clear: Assertion `tstate->_status.initialized && !tstate->_status.cleared' failed.

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7c87537 in __GI_abort () at abort.c:79
#2  0x00007ffff7c8740f in __assert_fail_base (fmt=0x7ffff7dfe688 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=0x5555559c7be0 "tstate->_status.initialized && !tstate->_status.cleared", file=0x5555559c6e2f "Python/pystate.c", line=1484, function=<optimized out>)
    at assert.c:92
#3  0x00007ffff7c96662 in __GI___assert_fail (assertion=assertion@entry=0x5555559c7be0 "tstate->_status.initialized && !tstate->_status.cleared", 
    file=file@entry=0x5555559c6e2f "Python/pystate.c", line=line@entry=1484, function=function@entry=0x5555559c83e0 <__PRETTY_FUNCTION__.47> "PyThreadState_Clear")
    at assert.c:101
#4  0x000055555587075f in PyThreadState_Clear (tstate=tstate@entry=0x555555c4be00) at Python/pystate.c:1484
#5  0x0000555555871483 in _PyThreadState_DeleteExcept (tstate=tstate@entry=0x555555c03338 <_PyRuntime+508728>) at Python/pystate.c:1680
#6  0x000055555586afa4 in Py_FinalizeEx () at Python/pylifecycle.c:1831
#7  0x000055555589f6fc in Py_RunMain () at Modules/main.c:691
#8  0x000055555589f74b in pymain_main (args=args@entry=0x7fffffffe160) at Modules/main.c:719
#9  0x000055555589f7c0 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:743
#10 0x00005555555cf74e in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15

When trying to start a new thread, Python creates new thread state by _PyThreadState_New call, adding this new state to list of thread states for current interpreter:

cpython/Modules/_threadmodule.c

Line 1191 in c32abf1

boot->tstate = _PyThreadState_New(interp);

If consequent call to PyThread_start_new_thread fails, this new state gets cleared, but remains in list:

cpython/Modules/_threadmodule.c

Lines 1203 to 1209 in c32abf1

    
           unsigned long ident = PyThread_start_new_thread(thread_run, (void*) boot); 
        
           if (ident == PYTHREAD_INVALID_THREAD_ID) { 
        
               PyErr_SetString(ThreadError, "can't start new thread"); 
        
               PyThreadState_Clear(boot->tstate); 
        
               thread_bootstate_free(boot, 1); 
        
               return NULL; 
        
           }

Then, at Python finalization, call to _PyThreadState_DeleteExcept attempts to clear this thread state again, which causes assertion failure:

cpython/Python/pystate.c

Lines 1674 to 1682 in 3e8fcb7

    
               /* Clear and deallocate all stale thread states.  Even if this 
        
                  executes Python code, we should be safe since it executes 
        
                  in the current thread, not one of the stale threads. */ 
        
               PyThreadState *p, *next; 
        
               for (p = list; p; p = next) { 
        
                   next = p->next; 
        
                   PyThreadState_Clear(p); 
        
                   free_threadstate(p); 
        
               }

cc @ericsnowcurrently

CPython versions tested on:

3.12, CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.13.0a0 (heads/main:d4cea794a7, Sep 22 2023, 18:42:05) [GCC 10.2.1 20210110]

Linked PRs

The text was updated successfully, but these errors were encountered:

chgnrdv · 2023-09-22T15:50:54Z

Possible fix (the tests are green):

diff --git a/Modules/_threadmodule.c b/Modules/_threadmodule.c
index 7692bacccc..071dc90a24 100644
--- a/Modules/_threadmodule.c
+++ b/Modules/_threadmodule.c
@@ -1204,6 +1204,7 @@ thread_PyThread_start_new_thread(PyObject *self, PyObject *fargs)
     if (ident == PYTHREAD_INVALID_THREAD_ID) {
         PyErr_SetString(ThreadError, "can't start new thread");
         PyThreadState_Clear(boot->tstate);
+        PyThreadState_Delete(boot->tstate);
         thread_bootstate_free(boot, 1);
         return NULL;
     }
diff --git a/Python/pystate.c b/Python/pystate.c
index dcc6c11221..983937202b 100644
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -1589,7 +1589,9 @@ tstate_delete_common(PyThreadState *tstate)
     if (tstate->_status.bound_gilstate) {
         unbind_gilstate_tstate(tstate);
     }
-    unbind_tstate(tstate);
+    if (tstate->_status.bound) {
+        unbind_tstate(tstate);
+    }
 
     // XXX Move to PyThreadState_Clear()?
     clear_datastack(tstate);

@ericsnowcurrently, is my analysis correct?

…n its startup failure (GH-109761) If Python fails to start newly created thread due to failure of underlying PyThread_start_new_thread() call, its state should be removed from interpreter' thread states list to avoid its double cleanup. Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

… new thread on its startup failure (pythonGH-109761) If Python fails to start newly created thread due to failure of underlying PyThread_start_new_thread() call, its state should be removed from interpreter' thread states list to avoid its double cleanup. (cherry picked from commit ca3ea9a) Co-authored-by: Radislav Chugunov <52372310+chgnrdv@users.noreply.github.com> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

serhiy-storchaka · 2024-11-22T19:34:32Z

Thank you for your contribution @chgnrdv.

…hread on its startup failure (GH-109761) (GH-127171) If Python fails to start newly created thread due to failure of underlying PyThread_start_new_thread() call, its state should be removed from interpreter' thread states list to avoid its double cleanup. (cherry picked from commit ca3ea9a) Co-authored-by: Radislav Chugunov <52372310+chgnrdv@users.noreply.github.com>

…hread on its startup failure (GH-109761) (GH-127173) If Python fails to start newly created thread due to failure of underlying PyThread_start_new_thread() call, its state should be removed from interpreter' thread states list to avoid its double cleanup. (cherry picked from commit ca3ea9a) Co-authored-by: Radislav Chugunov <52372310+chgnrdv@users.noreply.github.com>

encukou · 2024-11-25T10:59:26Z

Since this was merged to 3.13, the NoGIL buildbot is showing frequent failures in test_threading: https://buildbot.python.org/#/builders/1396

chgnrdv added the type-crash A hard crash of the interpreter, possibly with a core dump label Sep 22, 2023

AA-Turner added interpreter-core (Objects, Python, Grammar, and Parser dirs) 3.12 bugs and security fixes 3.13 bugs and security fixes extension-modules C modules in the Modules dir labels Sep 22, 2023

chgnrdv changed the title ~~Crash at finalization after fail to create new thread~~ Crash at finalization after fail to start new thread Sep 22, 2023

chgnrdv mentioned this issue Sep 22, 2023

gh-109746: Make _thread.start_new_thread delete state of new thread on its startup failure #109761

Merged

devdanzin mentioned this issue Nov 10, 2024

_interpreters is not thread safe on the free-threaded build #126644

Open

serhiy-storchaka closed this as completed in #109761 Nov 22, 2024

bedevere-app bot mentioned this issue Nov 22, 2024

[3.13] gh-109746: Make _thread.start_new_thread delete state of new thread on its startup failure (GH-109761) #127171

Merged

bedevere-app bot mentioned this issue Nov 22, 2024

[3.12] gh-109746: Make _thread.start_new_thread delete state of new thread on its startup failure (GH-109761) #127173

Merged

encukou reopened this Nov 25, 2024

serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Nov 26, 2024

pythongh-109746: Fix race condition in test_start_new_thread_failed

4e74699

bedevere-app bot mentioned this issue Nov 26, 2024

gh-109746: Fix race condition in test_start_new_thread_failed #127299

Open

Oct	NOV	Dec
	27
2023	2024	2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash at finalization after fail to start new thread #109746

Crash at finalization after fail to start new thread #109746

chgnrdv commented Sep 22, 2023 •

edited by bedevere-app bot

Loading

chgnrdv commented Sep 22, 2023

serhiy-storchaka commented Nov 22, 2024

encukou commented Nov 25, 2024

Crash at finalization after fail to start new thread #109746

Crash at finalization after fail to start new thread #109746

Comments

chgnrdv commented Sep 22, 2023 • edited by bedevere-app bot Loading

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

chgnrdv commented Sep 22, 2023

serhiy-storchaka commented Nov 22, 2024

encukou commented Nov 25, 2024

chgnrdv commented Sep 22, 2023 •

edited by bedevere-app bot

Loading