X Tutup
The Wayback Machine - https://web.archive.org/web/20230420182417/https://github.com/python/cpython/issues/89240
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multiprocessing.Pool(64) throws on Windows #89240

Open
saschanaz mannequin opened this issue Sep 1, 2021 · 4 comments
Open

multiprocessing.Pool(64) throws on Windows #89240

saschanaz mannequin opened this issue Sep 1, 2021 · 4 comments
Labels
3.9 only security fixes 3.10 only security fixes 3.11 bug and security fixes OS-windows stdlib Python modules in the Lib dir topic-multiprocessing type-bug An unexpected behavior, bug, or error

Comments

@saschanaz
Copy link
Mannequin

saschanaz mannequin commented Sep 1, 2021 •
edited by bedevere-bot

BPO 45077
Nosy @pfmoore, @tjguk, @zware, @eryksun, @zooba, @corona10, @saschanaz

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-09-01.13:20:44.186>
labels = ['type-bug', '3.9', '3.10', '3.11', 'library', 'OS-windows']
title = 'multiprocessing.Pool(64) throws on Windows'
updated_at = <Date 2021-10-03.11:23:58.490>
user = 'https://github.com/saschanaz'

bugs.python.org fields:

activity = <Date 2021-10-03.11:23:58.490>
actor = 'saschanaz'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2021-09-01.13:20:44.186>
creator = 'saschanaz'
dependencies = []
files = []
hgrepos = []
issue_num = 45077
keywords = []
message_count = 3.0
messages = ['400832', '400834', '400987']
nosy_count = 7.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'corona10', 'saschanaz']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45077'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs
@saschanaz
Copy link
Mannequin Author

saschanaz mannequin commented Sep 1, 2021

Similar issue as the previous bpo-26903.

Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> multiprocessing.cpu_count()
64
>>> multiprocessing.Pool(multiprocessing.cpu_count())
Exception in thread Thread-1:
Traceback (most recent call last):
<multiprocessing.pool.Pool state=RUN pool_size=64>
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 973, in _bootstrap_inner
>>>     self.run()
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 519, in _handle_workers
    cls._wait_for_updates(current_sentinels, change_notifier)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 499, in _wait_for_updates
    wait(sentinels, timeout=timeout)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 884, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 66

@saschanaz saschanaz mannequin added 3.9 only security fixes OS-windows type-bug An unexpected behavior, bug, or error labels Sep 1, 2021
@saschanaz
Copy link
Mannequin Author

saschanaz mannequin commented Sep 1, 2021

The argument-less instantiation also fails, which is worse.

>>> multiprocessing.Pool()
<multiprocessing.pool.Pool state=RUN pool_size=64>
>>> Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 973, in _bootstrap_inner
    self.run()
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 519, in _handle_workers
    cls._wait_for_updates(current_sentinels, change_notifier)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 499, in _wait_for_updates
    wait(sentinels, timeout=timeout)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 884, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 66

@eryksun
Copy link
Contributor

eryksun commented Sep 3, 2021

See bpo-26903 for a similar problem in concurrent.futures.ProcessPoolExecutor. It was resolved by adding a limit constant, _MAX_WINDOWS_WORKERS == 61.

WaitForMultipleObjects() can wait on up to 64 object handles, but in this case 3 slots are already taken. The pool wait includes two events for its output and change-notifier queues (named pipes), plus the _winapi module always reserves a slot for the SIGINT event, even though this event is only used by waits on the main thread.

To avoid the need to limit the pool size, connection._exhaustive_wait() could be modified to combine simultaneous waits on up to 63 threads, for which each thread exhaustively populates a list of up to 64 signaled objects. I wouldn't want to modify _winapi.WaitForMultipleObjects, but the exhaustive wait should still be implemented in C, probably in the _multiprocessing extension module. A benefit of implementing _exhaustive_wait() in C is lightweight thread creation, directly with CreateThread() and a relatively small stack commit size.

@eryksun eryksun added stdlib Python modules in the Lib dir 3.10 only security fixes 3.11 bug and security fixes labels Sep 3, 2021
@saschanaz saschanaz mannequin changed the title multiprocessing.Pool(64) crashes on Windows multiprocessing.Pool(64) throws on Windows Oct 3, 2021
@saschanaz saschanaz mannequin changed the title multiprocessing.Pool(64) crashes on Windows multiprocessing.Pool(64) throws on Windows Oct 3, 2021
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
aisk added a commit to aisk/cpython that referenced this issue Mar 22, 2023
@aisk
pythongh-89240: limit multiprocessing.Pool to 61 workers on windows
8914b90
@bedevere-bot bedevere-bot mentioned this issue Mar 22, 2023
Open
aisk added a commit to aisk/cpython that referenced this issue Mar 22, 2023
@aisk
pythongh-89240: limit multiprocessing.Pool to 61 workers on windows
ff81b63
aisk added a commit to aisk/cpython that referenced this issue Mar 22, 2023
@aisk
pythongh-89240: limit multiprocessing.Pool to 61 workers on windows
441ada9
juj added a commit to juj/emscripten that referenced this issue Apr 20, 2023
@juj
Workaround Python multiprocessing pool bug python/cpython#89240 to fi…
6ea5d01 
…x parallel test suite on Windows.
@juj
Copy link

juj commented Apr 20, 2023 •
edited

Note that the bug title is not accurate: on Windows calling multiprocessing.Pool(62) (or higher) does not (just) throw, but it prints an exception callstack like an uncaught throw would, and additionally Python hangs afterwards.

E.g. the script

a.py

import multiprocessing
a = multiprocessing.Pool(62)

never returns back to the OS on Windows, when invoked with python a.py. Calling taskkill /f /im python.exe is needed to recover the command prompt.

What happens on macOS and Linux if one attempts to create a multiprocessing pool with 100, 1000 or 10000 processes? I.e. what is the failure behavior on non-Windows platforms when too many are requested?

I wonder if instead of throwing, it would make sense to either silently cap the number of workers in a pool to the max limit?

Otherwise, how will users know what the limit currently is so creating a Pool will not throw? E.g. should there be a function call to ask max_worker_count_that_will_not_throw, or should users linearly search or bisect for the limit themselves? (if so, how would they discover that they have this burden on them in order to correctly use the API?)

@juj juj mentioned this issue Apr 20, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.9 only security fixes 3.10 only security fixes 3.11 bug and security fixes OS-windows stdlib Python modules in the Lib dir topic-multiprocessing type-bug An unexpected behavior, bug, or error
Projects
Multiprocessing issues
Status: No status
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vstinner @juj @eryksun
X Tutup