Bug 170149 – EWS should kill processes before attempting to run tests

• Component: Tools / Tests
• Status: NEW

[Jonathan Bedard]

Recently, we saw an issue where Simulators remained booted between EWS test runs, greatly slowing them down.  To prevent this, all relevant processes should be killed before running any tests.

[Radar WebKit Bug Importer]

<rdar://problem/31287788>

[Jonathan Bedard]

Created attachment 305530 [details]
Patch

[Build Bot]

Comment on attachment 305530 [details]
Patch

Attachment 305530 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/3423708

New failing tests:
http/tests/incremental/stylesheet-body-incremental-rendering.html

[Build Bot]

Created attachment 305566 [details]
Archive of layout-test-results from ews124 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews124  Port: ios-simulator-wk2  Platform: Mac OS X 10.11.6

[Alexey Proskuryakov]

This needs to be considered in the context of the issue that we have on Mac EWS, where DumpRenderTree processes get stuck.

This patch would kill them, but it will not resolve the root cause that makes CG calls freeze, so it may make the behavior worse overall. Not sure how to determine that.

[Carlos Alberto Lopez Perez]

(In reply to Alexey Proskuryakov from comment #5)
> This needs to be considered in the context of the issue that we have on Mac
> EWS, where DumpRenderTree processes get stuck.
> 
> This patch would kill them, but it will not resolve the root cause that
> makes CG calls freeze, so it may make the behavior worse overall. Not sure
> how to determine that.

This mimics the behaviour of the bots. The script kill-old-processes is executed by the bots on each new run.

I think that having the behaviour on the bots and on the EWS is something good.

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #6)
> I think that having the behaviour on the bots and on the EWS is something
> good.

 ^^^^ I think that having the (same) behaviour on the bots and on the EWS is something good.

[Alexey Proskuryakov]

I would agree in theory, but in practice, we don't get the same behavior with stuck CG calls on regular bots.

[Carlos Alberto Lopez Perez]

(In reply to Alexey Proskuryakov from comment #8)
> I would agree in theory, but in practice, we don't get the same behavior
> with stuck CG calls on regular bots.

Not sure if this is an still an issue for you, but giving the timing of the issue I guess its not.

In any I think this patch will be useful anyway.

Since we enabled tests on the GTK EWS in bug 186559 I'm seeing several zombie process on our EWS bot after each run that should be killed to allow it continue without major issues.

I guess the same happens on our bots, but since there we rely on the kill-old-process step that runs always after each run, that is not an issue on the bots.


I can deploy some workaround locally to kill or restart the EWS more often to avoid this, but having support in the tooling itself will be much better.

[Carlos Alberto Lopez Perez]

Comment on attachment 305530 [details]
Patch

This makes the EWS work more similar to the bots (those always kill old process before starting a new run), so I think this is an overall improvement to make tests on the EWS more reproducible.

I will like to see this landed.

[Jonathan Bedard]

Just a few notes on this:

We aren't seeing these sorts of issues on Mac or iOS Simulator EWS any more. If memory serves, we resolved them by adding exit handlers to webkitpy Simulator management and investigating the root-cause of the zombie processes.

On that note, we had at least one instance in the last year where the zombie processes on EWS were a symptom of a bug in WebKit's multi-process code which was orphaning processes. It's probably worth root-causing why those processes are orphaned.

I believe that in case of EWS machines managed by Apple, we reboot them every 5-10 test runs, Aakash or Lucas would likely know the exact configuration.  I wonder if this might be another solution to the problems the GTK bots are experiencing.

That being said, I'm not opposed to this patch.