If the layout-test finds any test failure, we re-run the tests (to rule-out flakiness) and do clean-tree run (to check pre-existing failures). on iOS, recently the re-runs and clean-tree runs have frequently started failing with 30+ failures. e.g.: https://ews-build.webkit.org/#/builders/24/builds/5262 https://ews-build.webkit.org/#/builders/24/builds/5260 https://ews-build.webkit.org/#/builders/24/builds/5259 https://ews-build.webkit.org/#/builders/24/builds/5256 https://ews-build.webkit.org/#/builders/24/builds/5251 https://ews-build.webkit.org/#/builders/24/builds/5077 https://ews-build.webkit.org/#/builders/24/builds/4981 https://ews-build.webkit.org/#/builders/24/builds/4779 This behavior is somewhat flaky. For e.g. in https://ews-build.webkit.org/#/builders/24/builds/5261 run-layout-tests-without-patch worked, and in https://ews-build.webkit.org/#/builders/24/builds/5254 re-run-layout-tests worked.
This issue stalls iOS WK2 queue whenever there is any failure on trunk.
I don't understand how clean-tree results are somehow more flakey than the test runs with the patch.
This seems like another fallout from Catalina update on iOS bots. We updated the bots around Oct 30, and we started seeing 30+ layout-test failure in https://bugs.webkit.org/show_bug.cgi?id=203792 That issue was fixed on Nov 10. I believe this issue has been happening since then.
I bet that running kill-old-processes between runs would help. As Alexey pointed out, this is probably a bug of some sort, so I just landed <https://trac.webkit.org/r252691> to help us figure out what the problem is.
Created attachment 383974 [details] Patch
Comment on attachment 383974 [details] Patch Now that we have some logging from <https://trac.webkit.org/r252691>, it seems to be simulator and https. I'll work on fixes for those problems independently, but as I mention offline to Aakash: "I think the kill-old-processes stuff needs to land. If nothing else, it's possible that a patch which fails to clean up resources appropriately could cause failures in the clean run and cause us to mark the patch as 'good' even though it is demonstrably not"
Committed r252700: <https://trac.webkit.org/changeset/252700>
Comment on attachment 383974 [details] Patch Landed manually, removing flags.
<rdar://problem/57368402>
<rdar://problem/57368404>