Summary: | [GTK] Add a new GTK layout tester bot to build.webkit.org that runs with --skip-failing-tests switch | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | WebKit | Reporter: | Carlos Alberto Lopez Perez <clopez> | ||||||
Component: | Tools / Tests | Assignee: | Carlos Alberto Lopez Perez <clopez> | ||||||
Status: | RESOLVED FIXED | ||||||||
Severity: | Normal | CC: | aakash_jain, bugs-noreply, dpino, jbedard, webkit-bug-importer | ||||||
Priority: | P2 | Keywords: | InRadar | ||||||
Version: | WebKit Nightly Build | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
See Also: |
https://bugs.webkit.org/show_bug.cgi?id=178472 https://bugs.webkit.org/show_bug.cgi?id=229270 |
||||||||
Attachments: |
|
Description
Carlos Alberto Lopez Perez
2021-07-07 07:46:53 PDT
Created attachment 433028 [details]
Patch
If I'm understanding the thought process here: You are adding a new queue so that the failures caused by test ordering are surfaced somewhere and can be investigated so that the tree can be kept clean enough for EWS to be effective. That generally seems reasonable, just a few thoughts here because this is a problem Apple's port have encountered too: - Such failures are usually indicative of a bug in either the test harness or the engine itself, tests *shouldn't* be able to effect the outcome of tests running after them, but this happens commonly enough that's it's something we explicitly check for any time we encounter a flakey test - When we've noticed this happening with many tests, it's often just a few bugs and you can make a huge difference by focusing on a single random case of test interdependence and fixing it - Test order is never consistent, so while this strategy is probably a good idea as a temporary measure to get EWS working, it's probably not going to be a great permanent solution. Comment on attachment 433028 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=433028&action=review > Tools/CISupport/build-webkit-org/config.json:438 > + "name": "GTK-Linux-64-bit-Release-Skip-Failing-EWS-Tests", "factory": "TestAllButJSCFactory", "builddir": "gtk-linux-64-release-skip-failing-ews-tests", Nit: I don't think builder name should have EWS in it. All it does is pass an additional argument which is already in the name. (In reply to Jonathan Bedard from comment #2) > If I'm understanding the thought process here: You are adding a new queue so > that the failures caused by test ordering are surfaced somewhere and can be > investigated so that the tree can be kept clean enough for EWS to be > effective. > > That generally seems reasonable, just a few thoughts here because this is a > problem Apple's port have encountered too: > - Such failures are usually indicative of a bug in either the test harness > or the engine itself, tests *shouldn't* be able to effect the outcome of > tests running after them, but this happens commonly enough that's it's > something we explicitly check for any time we encounter a flakey test > - When we've noticed this happening with many tests, it's often just a few > bugs and you can make a huge difference by focusing on a single random case > of test interdependence and fixing it > - Test order is never consistent, so while this strategy is probably a good > idea as a temporary measure to get EWS working, it's probably not going to > be a great permanent solution. Thanks for the thoughts. I agree with them. The main idea here is to use this bot as a temporary measure to get the EWS working with the pass rate that is expected. I think it can be also helpful to identify better those cases of test interdependences and help fixing them. Created attachment 433332 [details]
Patch
Remove EWS from the bot queue name
Comment on attachment 433332 [details] Patch Clearing flags on attachment: 433332 Committed r279844 (239600@main): <https://commits.webkit.org/239600@main> All reviewed patches have been landed. Closing bug. Restarted buildbot few hours back to pick up this change. Queue is live at: https://build.webkit.org/#/builders/GTK-Linux-64-bit-Release-Skip-Failing-Tests |