X Tutup
The Wayback Machine - https://web.archive.org/web/20220906174806/https://github.com/python/cpython/issues/96581
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python call C++ function has memory leak #96581

Open
qiuyang163 opened this issue Sep 5, 2022 · 5 comments
Open

python call C++ function has memory leak #96581

qiuyang163 opened this issue Sep 5, 2022 · 5 comments
Labels
pending The issue will be closed if no feedback is provided type-bug An unexpected behavior, bug, or error

Comments

@qiuyang163
Copy link

qiuyang163 commented Sep 5, 2022

Bug report

I write a c++ extension function below, in this code, I have a buffer from heap(buf),and it content will copy to python memory_view object, finally I called buf.release() to make python gc own this buf.
Then memory leak happend, but when I do not call buf.release(), python will get not correct content.What can I do in this situation? Tks.

struct ObjectClearer {
    void operator()(PyObject *obj) {
        Py_XDECREF(obj);
    }
};

typedef ::std::unique_ptr<PyObject, ObjectClearer> unique_object;


PyObject *ReadSharedMemory(PyObject *self, PyObject *args) {
    PyObject *agent_capsule = NULL;
    int worker_index;

    if (!PyArg_ParseTuple(args, "Oi", &agent_capsule, &worker_index)) {
        PyErr_BadArgument();
        return NULL;
    }

    SharedRingBufferAgent *agent = static_cast<SharedRingBufferAgent *>(
                        PyCapsule_GetPointer(agent_capsule, NULL));

    auto* buffer = agent->get_ring_buffer(static_cast<uint32_t>(worker_index));
    uint32_t msg_len = buffer->HeadMsgLen();

    std::unique_ptr<uint8_t[]> buf(new (std::nothrow) uint8_t[msg_len]);
    bool result = buffer->Read(buf.get());

    if (!result) {
        return Py_None;
    }

    struct TaskInfo* task_info = (struct TaskInfo*)(buf.get());
    unique_object msg(baidu::rpc::CreateMessage());

    uint32_t data_len = msg_len - sizeof(struct TaskInfo);
    uint8_t* data_ptr = buf.get() + sizeof(struct TaskInfo);

#if PY_MAJOR_VERSION >= 3
    unique_object req_buf(
        PyMemoryView_FromMemory(
                reinterpret_cast<char *>(data_ptr),
                static_cast<Py_ssize_t>(data_len),
                PyBUF_READ));
#else
    unique_object req_buf(
            PyBuffer_FromMemory(
                    data_ptr,
                    static_cast<Py_ssize_t>(data_len)));
#endif

    PyObject_SetAttrString(msg.get(), "_request_buf", req_buf.get());

    unique_object task_id(PyLong_FromLong(task_info->task_id));
    PyObject_SetAttrString(msg.get(), "_task_id", task_id.get());

    unique_object index(PyLong_FromLong(worker_index));
    PyObject_SetAttrString(msg.get(), "_worker_index", index.get());

    buf.release();
    return msg.release();
}

class Message(object):
    def __init__(self):
        self._request_buf = None
        self._response = None
        self._cntl = None
        self._closure = None
        self._method_index = -1
        self._task_id = -1
        self._worker_index = -1

Your environment

  • CPython versions tested on: python3.6.5
  • Operating system and architecture: centos OS 6.3 gcc 8.2
@qiuyang163 qiuyang163 added the type-bug An unexpected behavior, bug, or error label Sep 5, 2022
@qiuyang163 qiuyang163 changed the title C++ call python function has memory leak python call C++ function has memory leak Sep 5, 2022
@mrabarnett
Copy link

mrabarnett commented Sep 5, 2022

It might be due to incorrect reference counting, but I don't know what unique_object does, and I've been unable to find a declaration of it.

@qiuyang163
Copy link
Author

qiuyang163 commented Sep 6, 2022

It might be due to incorrect reference counting, but I don't know what unique_object does, and I've been unable to find a declaration of it.

struct ObjectClearer {
    void operator()(PyObject *obj) {
        Py_XDECREF(obj);
    }
};

typedef ::std::unique_ptr<PyObject, ObjectClearer> unique_object;

this is unique_object declaration

@qiuyang163
Copy link
Author

qiuyang163 commented Sep 6, 2022

and when I use

import sys
import _cpp
message = _cpp.ReadSharedMemory(shared_mem_agent, 0)
print(sys.getrefcount(message))

The refcount result is 2,I think it is normal.

@matthiasgoergens
Copy link
Contributor

matthiasgoergens commented Sep 6, 2022

Does your problem also occur on more recent versions of Python?

@kumaraditya303 kumaraditya303 added the pending The issue will be closed if no feedback is provided label Sep 6, 2022
@mrabarnett
Copy link

mrabarnett commented Sep 6, 2022

I think the problem is that you're creating and storing a memory view of a buffer that Python knows nothing about. It's your responsibility to free the buffer when it's no longer needed. buf.release() merely releases the ownership and returns the address of the pointer; Python still won't know anything about it!

You could try making a bytes object from the contents of the buffer and then store that instead of a memory view.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pending The issue will be closed if no feedback is provided type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

4 participants
X Tutup