Bug 84189 – nrwt: restart the driver in every new directory

• Status: RESOLVED
• Resolution: WONTFIX
• Component: New Bugs

[Dirk Pranke]

nrwt: restart the driver in every new directory

[Dirk Pranke]

Created attachment 137596 [details]
Patch

[Ojan Vafai]

Comment on attachment 137596 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=137596&action=review

How do we evaluate whether this improves things? Just by looking subjectively at the flakiness dashboard?

> Tools/Scripts/webkitpy/layout_tests/controllers/worker.py:147
> +            if not self._options.batch_size and not self._options.fully_parallel and not self._last_test_dir == test_dir:

not self._last_test_dir == test_dir --> self._last_test_dir != test_dir ?

[Dirk Pranke]

(In reply to comment #2)
> (From update of attachment 137596 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=137596&action=review
> 
> How do we evaluate whether this improves things? Just by looking subjectively at the flakiness dashboard?
> 

This is a good question, and partially why I'm not sure if I want to land it at all yet. I currently have a bunch of other expectations derived locally that, in conjunction with this, get me to a point where I can reliably run with zero flakiness or unexpected results on both SL and Linux, but I don't yet know if this will be stable as the tree moves on.

I think we will need other changes before the usefulness of this can really be determined. At the moment I'm considering something breaking the layout test run in two on the bots, one step that runs the "stable" tests, and one step that runs the "unstable" tests (ones that timeout, crash, or are flaky). Then, any failure on the "stable" set would turn the step red (i.e., we'd have a zero-flakiness tolerance).

But I may need to spin up an experimental bot to see how well this works for a while at all ...

> > Tools/Scripts/webkitpy/layout_tests/controllers/worker.py:147
> > +            if not self._options.batch_size and not self._options.fully_parallel and not self._last_test_dir == test_dir:
> 
> not self._last_test_dir == test_dir --> self._last_test_dir != test_dir ?

Good catch.

[Dirk Pranke]

Comment on attachment 137596 [details]
Patch

closing this as wontfix for now; I'm not sure it actually improves anything.