-
-
Notifications
You must be signed in to change notification settings - Fork 32.1k
gh-101444: Optimize bytearray slice assignment for bytes-like object #101445
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
bytes is equivalent to immutable bytearray, and therefore, we could safely avoid unnecessary copy, thereby giving 300% speedup or so.
In addition to bytearray, bytes-like object supporting buffer protocol could bypass unnecessary data copies, thereby giving 3 times speedup or so.
|
I leveraged GitHub actions to measure optimization results on various systems. Only differences between bytearray and bytes, memoryview are significant, since benchmarks might not be run on the identical hardware. Before: https://github.com/msoxzw/cpython/actions/runs/4120204201
After: https://github.com/msoxzw/cpython/actions/runs/4120296060
Therefore, this PR makes bytearray slice assignment for bytes-like object run as fast as that for bytearray. |
|
I manage to benchmark only on Windows on the same machine through GitHub actions. Buffer protocol would incur marginal ~10% overhead, if bytearray is regarded as buffer object.
If data copies are bypassed only for bytes objects, such performance penalty would be avoided accordingly.
So is it acceptable to such performance overhead? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How does it work for ba[:] = memoryview(b'abcd')[::2]?
Thanks for such wonderful review. It would work like byte or bytearray concatenation: This behavior is dictated by |
For memoryview slice, such as `ba[:] = memoryview(b'abcd')[::2]`
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ignoring arbitrary errors in PyObject_GetBuffer() is not good.
How does it work for ba[::-1] = memoryview(ba)? For ba[:0] = memoryview(ba), ba[:2] = memoryview(ba)[-2:]?
Please add tests for all cases in which your intermediate code failed.


In addition to bytearray, bytes-like object supporting buffer protocol could also bypass unnecessary data copies, thereby giving 3 ~ 4 times speedup.