bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals #9627
Conversation
…ization of worker processes
|
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA). Unfortunately our records indicate you have not signed the CLA. For legal reasons we need you to sign this before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue. You can check yourself to see if the CLA has been received. Thanks again for your contribution, we look forward to reviewing it! |
seanharr11
changed the title
|
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
|
I have made the requested changes; please review again |
|
Thanks for making the requested changes! @terryjreedy: please review the changes made to this pull request. |
There was a problem hiding this comment.
As I indicated on the mailing-list, I am still opposed to the general idea (though, congratulations for making a full-fledged PR). I am open to revise my opinion if there are enough core developers in favour of the idea, but that doesn't seem to be the case currently...
| @@ -264,7 +282,10 @@ def _maintain_pool(ctx, Process, processes, pool, inqueue, outqueue, | |||
| """Clean up any exited workers and start replacements for them. | |||
| """ | |||
| if Pool._join_exited_workers(pool): | |||
| Pool._repopulate_pool_static(ctx, Process, processes, pool, | |||
| Pool. | |||
|
|
|||
There was a problem hiding this comment.
Thanks for the PR. There seems to white spaces in this line and causes make patchcheck to fail in CI. Please remove them or run make patchcheck locally from the source directory that will fix this for you automatically.
Relevant make patchcheck output :
Getting base branch for PR ... upstream/master
Getting the list of files that have been added/changed ... 5 files
Fixing Python file whitespace ... 1 file:
Lib/multiprocessing/pool.py
Fixing C file whitespace ... 0 files
Fixing docs whitespace ... 0 files
Docs modified ... yes
Misc/ACKS updated ... NO
Misc/NEWS.d updated with `blurb` ... NO
configure regenerated ... not needed
pyconfig.h.in regenerated ... not needed
|
@seanharr11 I came across your blog about multiprocessing. Seems this PR is still in progress. What blockers do we have? I'm curious if the issue in your blog still exists in the latest version of python? |
|
To my knowledge, this issue still exists in the latest version of Python3. Per comments above, we were looking for consensus from core developers to merge this in. |
|
This PR is stale because it has been open for 30 days with no activity. |
Proposing a new kwarg in the
__init__()method ofmultiprocessing.Poolnamedexpect_initret.This kwarg defaults to
False. When set toTrue, the return value of theinitializerfunction is passed to the function we are applying (i.e.funcinPool.map(func, ls)), as a kwarg namedinitret.This PR includes thorough test coverage, and provides backwards compatibility (at least in Python3.x).
See this blog post for example use cases, as well as an initial defense for why this makes the
multiprocessing.PoolAPI more approachable, to more Python end-users, and augments the library.https://bugs.python.org/issue34837