Bug 227744 – [GTK] Add a new GTK layout tester bot to build.webkit.org that runs with --skip-failing-tests switch

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Carlos Alberto Lopez Perez]

We are trying to have a layout tests runner for the GTK port in the EWS, but we are experimenting there random flakiness and failures that are not reproducible on the GTK layout tester of build.webkit.org

I think this is related with the different way of running the layout tests:
- On build.webkit.org all known tests are scheduled for running (even those that are expected to crash or timeout).
- On the EWS, the switch --skip-failing-tests is passed and that changes the order in which the tests are run because the test that are expected to fail are not scheduled for running.

This different order of scheduling the test in the EWS I think it may cause different results sometimes or can be a source of undetected flakiness.

So I think is a good idea to add a new tester for the GTK port for build.webkit.org that runs with this flag like in the EWS.

I believe that will help us to easily discover and make easier doing gardening of this tests that fail or are flaky due to this different order of scheduling the tests because thanks to this bot we will get access to info about this runs in https://results.webkit.org and wktesthunter <https://github.com/clopez/webkit-testhunter>

[Carlos Alberto Lopez Perez]

Created attachment 433028 [details]
Patch

[Jonathan Bedard]

If I'm understanding the thought process here: You are adding a new queue so that the failures caused by test ordering are surfaced somewhere and can be investigated so that the tree can be kept clean enough for EWS to be effective.

That generally seems reasonable, just a few thoughts here because this is a problem Apple's port have encountered too:
- Such failures are usually indicative of a bug in either the test harness or the engine itself, tests *shouldn't* be able to effect the outcome of tests running after them, but this happens commonly enough that's it's something we explicitly check for any time we encounter a flakey test
- When we've noticed this happening with many tests, it's often just a few bugs and you can make a huge difference by focusing on a single random case of test interdependence and fixing it
- Test order is never consistent, so while this strategy is probably a good idea as a temporary measure to get EWS working, it's probably not going to be a great permanent solution.

[Aakash Jain]

Comment on attachment 433028 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=433028&action=review

> Tools/CISupport/build-webkit-org/config.json:438
> +                      "name": "GTK-Linux-64-bit-Release-Skip-Failing-EWS-Tests", "factory": "TestAllButJSCFactory", "builddir": "gtk-linux-64-release-skip-failing-ews-tests",

Nit: I don't think builder name should have EWS in it. All it does is pass an additional argument which is already in the name.

[Carlos Alberto Lopez Perez]

(In reply to Jonathan Bedard from comment #2)
> If I'm understanding the thought process here: You are adding a new queue so
> that the failures caused by test ordering are surfaced somewhere and can be
> investigated so that the tree can be kept clean enough for EWS to be
> effective.
> 
> That generally seems reasonable, just a few thoughts here because this is a
> problem Apple's port have encountered too:
> - Such failures are usually indicative of a bug in either the test harness
> or the engine itself, tests *shouldn't* be able to effect the outcome of
> tests running after them, but this happens commonly enough that's it's
> something we explicitly check for any time we encounter a flakey test
> - When we've noticed this happening with many tests, it's often just a few
> bugs and you can make a huge difference by focusing on a single random case
> of test interdependence and fixing it
> - Test order is never consistent, so while this strategy is probably a good
> idea as a temporary measure to get EWS working, it's probably not going to
> be a great permanent solution.

Thanks for the thoughts. I agree with them. The main idea here is to use this bot as a temporary measure to get the EWS working with the pass rate that is expected. I think it can be also helpful to identify better those cases of test interdependences and help fixing them.

[Carlos Alberto Lopez Perez]

Created attachment 433332 [details]
Patch

Remove EWS from the bot queue name

[Carlos Alberto Lopez Perez]

Comment on attachment 433332 [details]
Patch

Clearing flags on attachment: 433332

Committed r279844 (239600@main): <https://commits.webkit.org/239600@main>

[Carlos Alberto Lopez Perez]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/80474599>

[Aakash Jain]

Restarted buildbot few hours back to pick up this change. Queue is live at: https://build.webkit.org/#/builders/GTK-Linux-64-bit-Release-Skip-Failing-Tests