Bug 205280 - Python 3: Multiprocessing doesn't do well when nested
Summary: Python 3: Multiprocessing doesn't do well when nested
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-12-16 09:59 PST by Jonathan Bedard
Modified: 2019-12-16 15:41 PST (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Bedard 2019-12-16 09:59:43 PST
Not quite clear why. We end up with some weird race conditions, I spent a few hours trying to get to the bottom of it with no luck. I can reproduce the problem except in unit testing, so I'm inclined to defer investigation since run-webkit-tests works fine in Python 3 despite this bug.
Comment 1 Alexey Proskuryakov 2019-12-16 11:06:40 PST
Which start method are we using with Python 3?
Comment 2 Jonathan Bedard 2019-12-16 11:13:04 PST
(In reply to Alexey Proskuryakov from comment #1)
> Which start method are we using with Python 3?

Not sure what you mean by this.

For testing, I actually changed the shebang, but the code to run enable run-webkit-tests in Python3 isn't up for review quite yet, polishing that patch now. I'll be referencing this bug in that patch.
Comment 3 Alexey Proskuryakov 2019-12-16 15:28:22 PST
In Python 3, one can choose the underlying method for how multiprocessing works, it's called "start method". See <https://docs.python.org/3/library/multiprocessing.html>.

As this bug is about multiprocessing having some strange behavior on tests with Python 3, it's relevant which of the methods you are using. I'm guessing that it is 'spawn' because that's the default. Since 'fork' was the only one that Python 2 supported, that would be the 1:1 replacement, but of course we shouldn't be using that on macOS.
Comment 4 Jonathan Bedard 2019-12-16 15:41:32 PST
(In reply to Alexey Proskuryakov from comment #3)
> In Python 3, one can choose the underlying method for how multiprocessing
> works, it's called "start method". See
> <https://docs.python.org/3/library/multiprocessing.html>.
> 
> As this bug is about multiprocessing having some strange behavior on tests
> with Python 3, it's relevant which of the methods you are using. I'm
> guessing that it is 'spawn' because that's the default. Since 'fork' was the
> only one that Python 2 supported, that would be the 1:1 replacement, but of
> course we shouldn't be using that on macOS.

I wasn't aware of this.

I haven't modified the start method used, so yes, it should be using spawn. Seems possible that the spawn/fork difference is the root of our problem, but I'm still at a loss as to why I saw a race condition.