Bug 270757 – Do not run the references when a test crashes or times out

• Component: Tools / Tests
• Status: RESOLVED DUPLICATE

[Ahmad Saleem]

Hi Team,

While going through Blink's commit, I came across this optimization for test suite but don't know whether we have done similar in one way or shape but just wanted to raise to get input.

Blink Commit - https://chromium.googlesource.com/chromium/blink/+/a7ba83f0e6c7c9c1ebcab9ea4f53e899b26c397e

WebKit Source: https://searchfox.org/wubkat/rev/93c3eb8ac99a6b6e3894deaf14e5d38304898b48/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py#354

and

https://searchfox.org/wubkat/rev/93c3eb8ac99a6b6e3894deaf14e5d38304898b48/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py#378

__

Blink Commit Details:

If the actual test in a reftest crashes or times out, we probably shouldn't
bother executing the reference as well, since we have nothing
to compare it to, and the test failed regardless. This can save
a bunch of execution time and avoid some confusion.

__

Just raising to get input.

Thanks!

[Radar WebKit Bug Importer]

<rdar://problem/124790780>

[Jonathan Bedard]

This doesn't seem like a bad idea, although in practice, it seems unlikely to save us much time because we tend to not leave CRASH and TIMEOUT tests running, because such expectations are of very low value.

[Jonathan Bedard]

I'm resolving this as a duplicate of https://bugs.webkit.org/show_bug.cgi?id=273212.

The rationale in that bug is quite different than in this one, but since crashes and timeouts also don't have test output, https://bugs.webkit.org/show_bug.cgi?id=273212 will end up with the same effect as the one requested here.

*** This bug has been marked as a duplicate of bug 273212 ***