bpo-24234: Implement bytes.bytes #27901

corona10 · 2021-08-23T01:37:28Z

https://bugs.python.org/issue24234

corona10 · 2021-08-23T01:39:15Z

Tools/clinic/clinic.py

@@ -2179,7 +2179,6 @@ def __repr__(self):
 __abs__
 __add__
 __and__
-__bytes__


https://docs.python.org/3/c-api/typeobj.html?highlight=nb_int#sub-slots

Same reason as __complex__
#27886

gvanrossum

Should we perhaps combine the two News items for this bpo issue?

corona10 · 2021-08-23T01:48:28Z

Should we perhaps combine the two News items for this bpo issue?

Yes, it looks reasonable to add them as News items :) cc @mdickinson

corona10 · 2021-08-23T02:05:33Z

doctest (doctest) ... 66 tests with zero failures
**********************************************************************
File "/home/runner/work/cpython/cpython/Lib/test/test_doctest.py", line 671, in test.test_doctest.test_DocTestFinder.non_Python_modules
Failed example:
    816 < len(tests) < 836 # approximate number of objects with docstrings
Expected:
    True
Got:
    False
**********

The broken test will be fixed after #27887 is merged and rebased.

mdickinson · 2021-08-23T08:08:33Z

Should we perhaps combine the two News items for this bpo issue?

SGTM

mdickinson

LGTM, modulo the combining news items and the test failure.

corona10 · 2021-08-23T08:43:06Z

Doc/whatsnew/3.11.rst

@@ -169,6 +169,14 @@ Other Language Changes
  (Contributed by Serhiy Storchaka in :issue:`12022`.)


+Other CPython Implementation Changes


@mdickinson Can you please take a look? Sorry, I am not a good English writer.

@corona10 What you have looks fine to me!

bedevere-bot · 2021-08-23T10:09:48Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora Clang 3.x has failed when building commit 24b63c6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/3/builds/617) and take a look at the build logs.
Check if the failure is related to this commit (24b63c6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/3/builds/617

Failed tests:

test_multiprocessing_spawn

Failed subtests:

test_process - test.test_multiprocessing_spawn.WithProcessesTestProcess

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

412 tests OK.

10 slowest tests:

test_peg_generator: 2 min 53 sec
test_concurrent_futures: 2 min 30 sec
test_multiprocessing_spawn: 1 min 11 sec
test_multiprocessing_forkserver: 1 min 2 sec
test_multiprocessing_fork: 53.1 sec
test_asyncio: 48.7 sec
test_signal: 46.7 sec
test_io: 36.2 sec
test_socket: 34.9 sec
test_tokenize: 32.8 sec

1 test failed:
test_multiprocessing_spawn

15 tests skipped:
test_devpoll test_ioctl test_kqueue test_msilib test_nis
test_ossaudiodev test_readline test_startfile test_tix test_tk
test_ttk_guionly test_winconsoleio test_winreg test_winsound
test_zipfile64

Total duration: 5 min

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/test/_test_multiprocessing.py", line 360, in test_process
    self.assertEqual(q.get(), current.authkey)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'y\x[24 chars]xf8\xcf\xda' != b'y\x[24 chars]xf8\xcf\xda\x00\xbd\xaf\xa3\xc3IV\xdde|\x8b\xb9\xc0yl\xa9*\x1f'


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.clang/build/Lib/test/_test_multiprocessing.py", line 360, in test_process
    self.assertEqual(q.get(), current.authkey)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'\xd[19 chars]x0f\xb2\xd1Z' != b'\xd[19 chars]x0f\xb2\xd1Z\x00Ly\x8d\r;\xc8c\xf6l}\xbf\x00\xfd\xcf\xc5Hd\x1d'

bedevere-bot · 2021-08-23T10:11:36Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Fedora Stable 3.x has failed when building commit 24b63c6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/125/builds/523) and take a look at the build logs.
Check if the failure is related to this commit (24b63c6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/125/builds/523

Failed tests:

test_concurrent_futures

Failed subtests:

test_ressources_gced_in_workers - test.test_concurrent_futures.ProcessPoolForkProcessPoolExecutorTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

414 tests OK.

10 slowest tests:

test_concurrent_futures: 2 min 37 sec
test_peg_generator: 1 min 42 sec
test_multiprocessing_spawn: 1 min 29 sec
test_asyncio: 1 min 24 sec
test_multiprocessing_forkserver: 1 min 9 sec
test_unparse: 1 min 7 sec
test_multiprocessing_fork: 1 min
test_capi: 1 min
test_tokenize: 56.9 sec
test_gdb: 54.6 sec

1 test failed:
test_concurrent_futures

13 tests skipped:
test_devpoll test_ioctl test_kqueue test_msilib test_ossaudiodev
test_startfile test_tix test_tk test_ttk_guionly test_winconsoleio
test_winreg test_winsound test_zipfile64

Total duration: 2 min 45 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_concurrent_futures.py", line 95, in __del__
    self.event.set()
    ^^^^^^^^^^
AttributeError: 'EventfulGCObj' object has no attribute 'event'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_concurrent_futures.py", line 95, in __del__
    self.event.set()
    ^^^^^^^^^^
AttributeError: 'EventfulGCObj' object has no attribute 'event'
Warning -- Unraisable exception
Exception ignored in: <function EventfulGCObj.__del__ at 0xffff8975fb70>
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_concurrent_futures.py", line 95, in __del__
    self.event.set()
Warning -- Unraisable exception
Exception ignored in: <function EventfulGCObj.__del__ at 0xffff8975fb70>
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_concurrent_futures.py", line 95, in __del__
    ^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 1089, in set
    self.event.set()
    ^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 1089, in set
    return self._callmethod('set')
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 814, in _callmethod
    return self._callmethod('set')
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 814, in _callmethod
    self._connect()
    ^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 801, in _connect
    self._connect()
    ^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 801, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/connection.py", line 513, in Client
    conn = self._Client(self._token.address, authkey=self._authkey)
     answer_challenge(c, authkey)
          ^^^^^^^^^^^^^^^^^^^^^^^^^ ^ ^ ^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_concurrent_futures.py", line 95, in __del__
    self.event.set()
    ^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 1089, in set
    return self._callmethod('set')
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 814, in _callmethod
    self._connect()
    ^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/managers.py", line 801, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/connection.py", line 507, in Client
    c = SocketClient(address)
        ^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/multiprocessing/connection.py", line 635, in SocketClient
    s.connect(address)
    ^^^^^^^^^^^^^^^^^^
ConnectionRefusedError: [Errno 111] Connection refused


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/support/hashlib_helper.py", line 36, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/test/test_concurrent_futures.py", line 978, in test_ressources_gced_in_workers
    future.result()
    ^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/concurrent/futures/_base.py", line 445, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64/build/Lib/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
    ^^^^^^^^^^^^^^^^^^^^^
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

bedevere-bot · 2021-08-23T10:14:55Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Fedora Stable LTO + PGO 3.x has failed when building commit 24b63c6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/29/builds/586) and take a look at the build logs.
Check if the failure is related to this commit (24b63c6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/29/builds/586

Failed tests:

test_multiprocessing_spawn

Failed subtests:

test_process - test.test_multiprocessing_spawn.WithProcessesTestProcess

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

413 tests OK.

10 slowest tests:

test_concurrent_futures: 2 min 26 sec
test_nntplib: 1 min 21 sec
test_multiprocessing_spawn: 1 min 5 sec
test_multiprocessing_forkserver: 57.1 sec
test_asyncio: 54.2 sec
test_multiprocessing_fork: 53.4 sec
test_signal: 47.0 sec
test_io: 37.6 sec
test_pydoc: 31.7 sec
test_xmlrpc: 27.6 sec

1 test failed:
test_multiprocessing_spawn

14 tests skipped:
test_devpoll test_gdb test_ioctl test_kqueue test_msilib
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

Total duration: 2 min 52 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/test/_test_multiprocessing.py", line 360, in test_process
    self.assertEqual(q.get(), current.authkey)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'\xd[27 chars]8d\xaf?<@]\x88o\xc2\xc7\x1aY\xc4' != b'\xd[27 chars]8d\xaf?<@]\x88o\xc2\xc7\x1aY\xc4\x00\xae<\x9d\x04\x0e\xf3\x99k'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-x86_64.lto-pgo/build/Lib/test/_test_multiprocessing.py", line 360, in test_process
    self.assertEqual(q.get(), current.authkey)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'\xf[45 chars]\xc0\x8a\x97\x14\xfe' != b'\xf[45 chars]\xc0\x8a\x97\x14\xfe\x00\xe1\xdd&\xad\xdcY\xccn\xbd_\x04\xfbTd'

bedevere-bot · 2021-08-23T10:15:20Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL7 3.x has failed when building commit 24b63c6.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/15/builds/668) and take a look at the build logs.
Check if the failure is related to this commit (24b63c6) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/15/builds/668

Failed tests:

test_multiprocessing_forkserver

Failed subtests:

test_process - test.test_multiprocessing_forkserver.WithProcessesTestProcess

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

411 tests OK.

10 slowest tests:

test_mailbox: 7 min 25 sec
test_shelve: 6 min 15 sec
test_concurrent_futures: 2 min 46 sec
test_peg_generator: 2 min 26 sec
test_unparse: 1 min 47 sec
test_multiprocessing_spawn: 1 min 47 sec
test_capi: 1 min 33 sec
test_tokenize: 1 min 24 sec
test_asyncio: 1 min 20 sec
test_multiprocessing_forkserver: 1 min 14 sec

1 test failed:
test_multiprocessing_forkserver

16 tests skipped:
test_devpoll test_gdb test_ioctl test_kqueue test_msilib
test_ossaudiodev test_smtpnet test_ssl test_startfile test_tix
test_tk test_ttk_guionly test_winconsoleio test_winreg
test_winsound test_zipfile64

Total duration: 10 min 14 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64/build/Lib/test/_test_multiprocessing.py", line 360, in test_process
    self.assertEqual(q.get(), current.authkey)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'\x1[34 chars]x86\x95\xc0o@\x89\x1ao\xa5W\xba\xbd' != b'\x1[34 chars]x86\x95\xc0o@\x89\x1ao\xa5W\xba\xbd\x00=\xb3\xfd\xf1\\\xaf\x90'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL7-x86_64/build/Lib/test/_test_multiprocessing.py", line 360, in test_process
    self.assertEqual(q.get(), current.authkey)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: b'\x1[30 chars]M\xb9\xc0\x86{\xbdq9|\xbbH\xed' != b'\x1[30 chars]M\xb9\xc0\x86{\xbdq9|\xbbH\xed\x00X \xe9\xc0\x84S\x99\xec\xaew'

mdickinson · 2021-08-23T10:19:26Z

Hmm. Unfortunately, I think those buildbot failures aren't false positives. By using PyBytes_FromString, we're truncating at the first zero byte. I think we need to use PyBytes_FromStringAndSize instead (and add a regression test).

>>> class B(bytes): pass
... 
>>> x = B(b"\x00\x01\x02")
>>> x.__bytes__()
b''

corona10 requested a review from gvanrossum as a code owner Aug 23, 2021

the-knights-who-say-ni added the CLA signed label Aug 23, 2021

bedevere-bot added the awaiting core review label Aug 23, 2021

corona10 requested a review from mdickinson Aug 23, 2021

corona10 reviewed Aug 23, 2021

View changes

corona10 force-pushed the corona10:bpo-24234 branch from 59e0a3a to 01cdf3e Aug 23, 2021

corona10 changed the title ~~bpo-34234: Implement bytes.__bytes__~~ bpo-24234: Implement bytes.__bytes__ Aug 23, 2021

gvanrossum approved these changes Aug 23, 2021

View changes

bedevere-bot added awaiting merge and removed awaiting core review labels Aug 23, 2021

corona10 force-pushed the corona10:bpo-24234 branch from fd8dbc2 to 22009fb Aug 23, 2021

mdickinson approved these changes Aug 23, 2021

View changes

corona10 added 2 commits Aug 23, 2021

bpo-24234: Implement bytes.__bytes__

b0638e2

bpo-24234: nit

8f283da

corona10 force-pushed the corona10:bpo-24234 branch from 22009fb to 8f283da Aug 23, 2021

corona10 reviewed Aug 23, 2021

View changes

bpo-24234: Add whatsnew 3.11

a6ba24a

corona10 force-pushed the corona10:bpo-24234 branch from 02d838a to a6ba24a Aug 23, 2021

bedevere-bot removed the awaiting merge label Aug 23, 2021

corona10 deleted the corona10:bpo-24234 branch Aug 23, 2021

mdickinson mentioned this pull request Aug 23, 2021

bpo-24234: fix bytes.__bytes__ to not truncate at a zero byte #27902

Merged

python / cpython

bpo-24234: Implement bytes.bytes #27901

bpo-24234: Implement bytes.bytes #27901

corona10 commented Aug 23, 2021 •

edited

This comment has been minimized.

gvanrossum left a comment

corona10 commented Aug 23, 2021

corona10 commented Aug 23, 2021

mdickinson commented Aug 23, 2021

mdickinson left a comment

This comment has been minimized.

This comment has been minimized.

bedevere-bot commented Aug 23, 2021

bedevere-bot commented Aug 23, 2021

bedevere-bot commented Aug 23, 2021

bedevere-bot commented Aug 23, 2021

mdickinson commented Aug 23, 2021

		@@ -169,6 +169,14 @@ Other Language Changes
		(Contributed by Serhiy Storchaka in :issue:`12022`.)


		Other CPython Implementation Changes

Aug	SEP	Oct
	01
2020	2021	2022

python / cpython

Sponsor python/cpython

bpo-24234: Implement bytes.__bytes__ #27901

bpo-24234: Implement bytes.__bytes__ #27901

Conversation

corona10 commented Aug 23, 2021 • edited

This comment has been minimized.

corona10 Aug 23, 2021 Author Member

gvanrossum left a comment

corona10 commented Aug 23, 2021

corona10 commented Aug 23, 2021

mdickinson commented Aug 23, 2021

mdickinson left a comment

This comment has been minimized.

corona10 Aug 23, 2021 Author Member

This comment has been minimized.

mdickinson Aug 23, 2021 Member

bedevere-bot commented Aug 23, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Aug 23, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Aug 23, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented Aug 23, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

mdickinson commented Aug 23, 2021

bpo-24234: Implement bytes.bytes #27901

bpo-24234: Implement bytes.bytes #27901

corona10 commented Aug 23, 2021 •

edited

corona10 Aug 23, 2021
Author Member

corona10 Aug 23, 2021
Author Member

mdickinson Aug 23, 2021
Member