Bug 245329 – EWS false positive when crashes didn't reproduce on run-layout-tests-without-change

• Status: NEW
• Component: Tools / Tests

[Alexey Proskuryakov]

https://github.com/WebKit/WebKit/pull/4441 got wrongly marked as failing. Buidbot link: https://ews-build.webkit.org/#/builders/73/builds/14733

These tests crashed on first try:

inspector/debugger/symbolic-breakpoint-intrinsic-js-regex-case-insensitive.html
inspector/debugger/symbolic-breakpoint-intrinsic-js-regex-case-sensitive.html

Retry:

inspector/debugger/symbolic-breakpoint-intrinsic-js-regex-case-sensitive.html

But nothing crashed on run-layout-tests-without-change, which only ran these two tests.

Was this just an unfortunate coincidence due to flakiness? Or did crashing depend on having run preceding tests? I'm feeling uneasy about only running tests that failed in run-layout-tests-without-change, we seem to be getting a lot of false positives lately.

[Carlos Alberto Lopez Perez]

> Was this just an unfortunate coincidence due to flakiness? Or did crashing depend on having run preceding tests?

In this case it looks to me it was an unfortunate coincidence due to flakiness.

See: https://results.webkit.org/?suite=layout-tests&test=inspector%2Fdebugger%2Fsymbolic-breakpoint-intrinsic-js-regex-case-sensitive.html

inspector/debugger/symbolic-breakpoint-intrinsic-js-regex-case-sensitive.html  looks flaky (crash/pass) on "Big Sur E wk1 Release on Macmini8,1 with x86_64" and other Apple platforms.

I suggest to mark this test as flaky on the TestExpectations for the Apple ports.

> I'm feeling uneasy about only running tests that failed in run-layout-tests-without-change, we seem to be getting a lot of false positives lately.

The EWS GTK or WPE bots use a new class added in bug 231999 (LayoutTestClass = RunWebKitTestsRedTree) which repeats failures at least 10 times to discard flakies. We couldn't make reliable in our ports the default layout test runner, because of the unpredictable random flakies, so we took this big hammer approach to discard flakies on the EWS itself by repeating them 10 times.

So If you wish, I don't have any issue at this moment with removing the optimizations added on bug 219500 and bug 231265 (the GTK/WPE bots don't use that code as said above). But removing that is going to cause the EWS bots that use the default layout test runner to do more work, so they will take longer to finish.

[Radar WebKit Bug Importer]

<rdar://problem/100370073>