Actually, I'm a bit worried about backporting the change to 3.13 and 3.14 because we are now using a real heap type.

🤖 New build scheduled with the buildbot fleet by @picnixz for commit 4dd0652 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F143245%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

Mmh yes, that would aloow me to backport this and avoid burdening the GC. I didn't consider this approach as I am more used to rely on existing mechanisms.

I think I will add some macros or functions for holding the refs when needed.

So I've tried, but I miserably failed. Each callback creates a shared callback context that is either freed when the connection dies or when it got replaced. The problem is when I replace a callback:

• Define cb(). This creates some conn->ctx.
• Execute cb. During the execution, remove cb.
• Removal of cb triggers removal of conn->ctx (it gets replaced by NULL) and conn->ctx is freed directly (deferring freeing the memory isn't possible because no one is tracking it after it's replaced in the connection's object)

Maybe I got it wrong or understood the mechanism wrongly, but relying on the GC is much more easier.

When the callback is added, set its refcount to 1. If it is replaced, decrement its refcount. When the callback is called, increment its refcount (so that it doesn't disappear from under our feet), after calling it, decrement its refcount. If after any decrement it becomes 0, free it. This is very similar to how refcounts work for PyObject, but We don't need PyTypeObject and other boilerplate.

Other way is just make local strong references to callback and module (if needed) and a weak reference to state (if needed) for the time of calling the callback, On this way you will only need to change the code that calls the callback, and use Python refcounts.

Thanks @picnixz for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

Sorry, @picnixz, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 7f6c16a956d598663d8c67071c492f197045d967 3.14

Sorry, @picnixz, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 7f6c16a956d598663d8c67071c492f197045d967 3.13

Of course backports have conflicts -_-

GH-143322 is a backport of this pull request to the 3.14 branch.

GH-143323 is a backport of this pull request to the 3.13 branch.

Earlier we relied on GC for these callbacks; IOW, we did not use the SQLite C API provided destructor machinery. Perhaps we should revert to that again. It is more verbose, but it's less hacky than what the current solution has grown to be.