X Tutup
The Wayback Machine - https://web.archive.org/web/20240709205021/https://github.com/python/cpython/issues/121528
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[3.13] Py_INCREF(PyExc_TypeError) in stable ABI causes python3.13: ./Include/internal/pycore_object.h:284: _PyObject_Init: Assertion '_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE) || _Py_IsImmortal(typeobj)' failed. #121528

Open
mgorny opened this issue Jul 9, 2024 · 1 comment · May be fixed by #121550
Open

[3.13] Py_INCREF(PyExc_TypeError) in stable ABI causes python3.13: ./Include/internal/pycore_object.h:284: _PyObject_Init: Assertion '_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE) || _Py_IsImmortal(typeobj)' failed. #121528

mgorny opened this issue Jul 9, 2024 · 1 comment · May be fixed by #121550
Labels
type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@mgorny
Copy link
Contributor

mgorny commented Jul 9, 2024 •
edited by bedevere-app bot
Loading

Crash report

What happened?

Consider the following extension.

C code:

#include <Python.h>

static PyObject *
foo_bar(PyObject *self, PyObject *args)
{
	Py_INCREF(PyExc_TypeError);
	PyErr_SetString(PyExc_TypeError, "foo");
	return NULL;
}

static PyMethodDef foomethods[] = {
	{"bar", foo_bar, METH_VARARGS, ""},
	{NULL, NULL, 0, NULL},
};

static PyModuleDef foomodule = {
	PyModuleDef_HEAD_INIT,
	.m_name = "foo",
	.m_doc = "foo test module",
	.m_size = -1,
	.m_methods = foomethods,
};

PyMODINIT_FUNC
PyInit_foo(void)
{
	return PyModule_Create(&foomodule);
}

setup.py:

from setuptools import setup, Extension

setup(name='foo',
      version='0',
      ext_modules=[
          Extension('foo', ['foo.c'], py_limited_api='cp38'),
      ])

If I compile the extension using Python older than 3.12, and then run the method, Python 3.13.0b3 (built --with-assertions) crashes:

$ python3.11 setup.py build_ext -i
running build_ext
building 'foo' extension
creating build
creating build/temp.linux-x86_64-cpython-311
x86_64-pc-linux-gnu-gcc -Wsign-compare -fPIC -I/usr/include/python3.11 -c foo.c -o build/temp.linux-x86_64-cpython-311/foo.o
creating build/lib.linux-x86_64-cpython-311
x86_64-pc-linux-gnu-gcc -shared build/temp.linux-x86_64-cpython-311/foo.o -L/usr/lib64 -o build/lib.linux-x86_64-cpython-311/foo.abi3.so
copying build/lib.linux-x86_64-cpython-311/foo.abi3.so -> 
$ python3.13 -c 'import foo; foo.bar()'
python3.13: ./Include/internal/pycore_object.h:284: _PyObject_Init: Assertion `_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE) || _Py_IsImmortal(typeobj)' failed.
Aborted (core dumped)

I've been able to bisect it to c32dc47 (CC @markshannon). Originally hit it in extensions using PyO3, and reported to PyO3/pyo3#4311.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.13.0b3 (main, Jul 4 2024, 14:30:57) [GCC 14.1.1 20240622]

Linked PRs
@mgorny mgorny added the type-crash A hard crash of the interpreter, possibly with a core dump label Jul 9, 2024
@mgorny mgorny mentioned this issue Jul 9, 2024
Open
@Zheaoli
Copy link
Contributor

Zheaoli commented Jul 9, 2024

The issue is confirmed, I have found the reason, but I'm not sure this is a bug, I prefer its undefined behavior.

It's caused by the Py_INCREF different behavior between 3.11 and 3.13(or master)

TL;DR;

For the 3.11 version of the Py_INCREF, we add the reference count directly without check FYI https://github.com/python/cpython/blob/3.11/Include/object.h#L491-L504

For the 3.13 version of the Py_INCREF, we add the reference count and check if it exceeds. If it exceeds, this means that this is an immortal object, we do noting except return the function. FYI https://github.com/python/cpython/blob/3.13/Include/object.h#L796-L846

And the function is an inline function, which is means we will expand the function into the binary during the compile. So in 3.11 version, the PyExc_TypeError's reference count is change from 4294967295(the immortal flag) to 4294967296. And the process will fail here https://github.com/python/cpython/pull/116115/files#diff-2a12f738a77b362d74a65949b58c37f2affcd15ba8b1c979b63bd00223b8a456R268

We can compare the assemble code to prove it

For 3.11

0000000000001120 <foo_bar>:
    1120:	48 83 ec 08          	sub    $0x8,%rsp
    1124:	48 8b 05 9d 2e 00 00 	mov    0x2e9d(%rip),%rax        # 3fc8 <PyExc_TypeError@Base>
    112b:	48 8d 35 ce 0e 00 00 	lea    0xece(%rip),%rsi        # 2000 <_fini+0xe9c>
    1132:	48 8b 38             	mov    (%rax),%rdi
    1135:	48 83 07 01          	addq   $0x1,(%rdi)
    1139:	e8 f2 fe ff ff       	call   1030 <PyErr_SetString@plt>
    113e:	31 c0                	xor    %eax,%eax
    1140:	48 83 c4 08          	add    $0x8,%rsp
    1144:	c3                   	ret
    1145:	66 66 2e 0f 1f 84 00 	data16 cs nopw 0x0(%rax,%rax,1)
    114c:	00 00 00 00

For 3.13

0000000000001120 <foo_bar>:
    1120:	48 83 ec 08          	sub    $0x8,%rsp
    1124:	48 8b 05 9d 2e 00 00 	mov    0x2e9d(%rip),%rax        # 3fc8 <PyExc_TypeError@Base>
    112b:	48 8b 38             	mov    (%rax),%rdi
    112e:	8b 07                	mov    (%rdi),%eax
    1130:	83 c0 01             	add    $0x1,%eax
    1133:	74 02                	je     1137 <foo_bar+0x17>
    1135:	89 07                	mov    %eax,(%rdi)
    1137:	48 8d 35 c2 0e 00 00 	lea    0xec2(%rip),%rsi        # 2000 <_fini+0xe9c>
    113e:	e8 ed fe ff ff       	call   1030 <PyErr_SetString@plt>
    1143:	31 c0                	xor    %eax,%eax
    1145:	48 83 c4 08          	add    $0x8,%rsp
    1149:	c3                   	ret
    114a:	66 0f 1f 44 00 00    	nopw   0x0(%rax,%rax,1)

The patch to fix this issue is simple ,we just the code back

    if (_PyType_HasFeature(typeobj, Py_TPFLAGS_HEAPTYPE)  ||  _Py_IsImmortal(typeobj)) {
        Py_INCREF(typeobj);
    }

I'm not sure this patch has side effect or not. I may need @markshannon help

@Zheaoli Zheaoli linked a pull request Jul 9, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Milestone
No milestone
2 participants
@mgorny @Zheaoli
X Tutup