gh-128182: Add per-object memory access synchronization to ctypes#128490
gh-128182: Add per-object memory access synchronization to ctypes#128490encukou merged 24 commits intopython:mainfrom
ctypes#128490Conversation
|
🤖 New build scheduled with the buildbot fleet by @ZeroIntensity for commit bdee2b2 🤖 If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
picnixz
left a comment
There was a problem hiding this comment.
For locked_deref, you might want to add an additional macro with an explicit return type instead instead of casting to a void **, say locked_deref_t(T, self). I didn't count how many usages of locked_deref need casting though so you might also inline those casts.
Now, AFAICT, most of the changes can be written within a critical section because, except perhaps some of them where the lock depends on something else? (I don't really use critical sections but AFAIK, critical sections are roughly equivalent to acquire the GIL and hence safe lock all current resources, right?)
Is a critical section an overkill compared to a per-object lock as well?
| Thread Safety Without The GIL | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| In Python 3.13, the :term:`GIL` may be disabled on :term:`experimental free threaded <free threading>` builds. |
There was a problem hiding this comment.
This is orthogonal but whenever I review FT PRs, we always use :term:`free threaded `. I believe we can add a new term for free threaded which points to free threading so that we can reduce the docs burden.
There was a problem hiding this comment.
A free threaded term sounds like a good idea, but that should probably be in a larger follow-up PR. I'll play around with the wording here though.
|
Thanks for the thorough review!
Acknowledged. I'll look at the cases.
Critical sections are just a per-object locking mechanism, but they're safe against lock-ordering and re-entrancy deadlocks, because a thread's active locks are released when the thread state is detached (generally in a I could try and do this with a critical section, but it significantly complicates the change, because nearly every function that touches |
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
This reverts commit 1c92b4b.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
I say "per-object" as in, using a single ctypes object is thread-safe if that's the only object accessing the memory. IMO, access to arbitrary memory across different pointer objects shouldn't (or can't?) be thread safe--ctypes should remain pretty low level, and that kind of synchronization might do bad things to applications that don't want it. I've clarified this in the docs. That said, I could see use in some sort of wrapper that ensures only one thread accesses memory using a table of addresses, but that should go to DPO first.
A few notes about the implementation here:
getfuncandsetfuncfunctions being non-reentrant. In practice, they really shouldn't be, but of course I'm not absolutely certain. We might want to go with a recursive mutex if that turns out to be an issue.PyFoo_FromBarfunctions don't have a chance to re-enter. The cases I could think of that could maybe cause re-entrancy are: an embedder could hijack the allocator and run the interpreter from it, or audit events. (Let me know if this is a deal-breaker!)ctypespointer writes are not thread safe #128182📚 Documentation preview 📚: https://cpython-previews--128490.org.readthedocs.build/en/128490/library/ctypes.html#thread-safety-without-the-gil