Bug 177723 – EWS bot should fail if a new test is missing its result

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Sam Weinig]

I found it quite surprising that the EWS bot didn't fail when I updated a change with test missing. I would expect that to also be a failure condition.

[Radar WebKit Bug Importer]

<rdar://problem/34768404>

[Alexey Proskuryakov]

*** Bug 181940 has been marked as a duplicate of this bug. ***

[Aakash Jain]

Noticed this in: https://ews-build.webkit.org/#/builders/24/builds/28039. Patch 411263 added two tests with missing results. layout-tests step indicated '2 missing results' but did not mark the step/build as failed.

The fix is to simply mark the build step as failed, that would cause layout-tests to be retried, and then retry on clean build, and since clean build would pass, it would cause the build to be marked as failed.

[Aakash Jain]

Created attachment 411436 [details]
Patch

[Aakash Jain]

Comment on attachment 411436 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=411436&action=review

> Tools/BuildSlaveSupport/ews-build/steps.py:2000
> +                elif line.find('missing results') >= 0:

Added this elif explicitly for readability. Can remove this as well if reviewers feel strongly about it.

[Jonathan Bedard]

Comment on attachment 411436 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=411436&action=review

>> Tools/BuildSlaveSupport/ews-build/steps.py:2000
>> +                elif line.find('missing results') >= 0:
> 
> Added this elif explicitly for readability. Can remove this as well if reviewers feel strongly about it.

I'm fine with the elif, but it seems like the failure condition should be first. (since in the event a line contained both 'flakes' and 'missing results', we would want to mark the build as failed). I know this concern is more about purity than practicality at the moment, though.

[Aakash Jain]

(In reply to Jonathan Bedard from comment #6)
> since in the event a line contained both 'flakes' and 'missing results'
'flakes' and 'missing results' words aren't being checked literally, they store output of regular expressions matching from https://trac.webkit.org/browser/webkit/trunk/Tools/BuildSlaveSupport/ews-build/steps.py#L1949

[EWS]

Committed r268526: <https://trac.webkit.org/changeset/268526>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 411436 [details].

[Aakash Jain]

Restarted EWS server to pick up this change.