Bug 229533

Summary: [EWS] compile-webkit-without-patch failed, but bot ran layout tests anyway and blamed the patch for a pre-existing failure
Product: WebKit Reporter: Ryan Haddad <ryanhaddad>
Component: Tools / TestsAssignee: Aakash Jain <aakash_jain>
Status: RESOLVED FIXED    
Severity: Normal CC: aakash_jain, ap, jbedard, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch none

Description Ryan Haddad 2021-08-25 20:27:32 PDT 
As you can see in https://ews-build.webkit.org/#/builders/60/builds/10553, the layout-tests and re-run-layout-tests both failed. The bot then proceeded to the compile-webkit-without-patch step and failed to compile. 

Surprisingly, the bot still proceeded to the run-layout-tests-without-patch step, which failed with the following:

16:30:10.005 10889 WebKitTestRunner was not found at /Volumes/Data/worker/macOS-AppleSilicon-Big-Sur-Debug-WK2-Tests-EWS/build/WebKitBuild/Debug/WebKitTestRunner

I would expect that the EWS run would be considered inconclusive or invalid, but instead it blamed the patch under test for introducing a new test failure that really should have been considered pre-existing.

Other examples from around the same time:
https://ews-build.webkit.org/#/builders/60/builds/10560
https://ews-build.webkit.org/#/builders/60/builds/10561
https://ews-build.webkit.org/#/builders/60/builds/10562
https://ews-build.webkit.org/#/builders/60/builds/10563
Comment 1 Radar WebKit Bug Importer 2021-08-25 20:28:02 PDT 
<rdar://problem/82371726>
Comment 2 Aakash Jain 2021-08-26 13:38:25 PDT 
Created attachment 436560 [details]
Patch
Comment 3 Aakash Jain 2021-08-26 13:39:53 PDT 
Tested on uat instance, e.g.: https://ews-build.webkit-uat.org/#/builders/34/builds/40260
Comment 4 EWS 2021-08-26 15:07:20 PDT 
Committed r281660 (241013@main): <https://commits.webkit.org/241013@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 436560 [details].
Comment 5 Aakash Jain 2021-08-27 03:34:32 PDT 
Restarted buildbot to pick up this change.
Comment 6 Aakash Jain 2021-08-27 04:20:14 PDT Comment hidden (obsolete)