Bug 209912 – [ews] Display list of test failures along-with the layout-test build step

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

Display list of test failures along-with the layout-test build step in the buildbot page. It would make it easy to notice the test failures without opening the complete 'layout test results' in a new tab. It would also make it easier to compare test-failures from the first run, re-run and re-run-without-patch.

[Aakash Jain]

Created attachment 395273 [details]
Patch

[Aakash Jain]

e.g.: 
https://ews-build.webkit-uat.org/#/builders/34/builds/119
https://ews-build.webkit-uat.org/#/builders/34/builds/119/steps/10/logs/test-failures

[Aakash Jain]

Created attachment 395302 [details]
Screenshot

[John Wilander]

Nice! Would it be possible to get the type of failure in that list too? Like text diff, timeout, crash, and such?

[Jonathan Bedard]

(In reply to John Wilander from comment #4)
> Nice! Would it be possible to get the type of failure in that list too? Like
> text diff, timeout, crash, and such?

Just curious, are engineers not generally aware of the stdout output: https://ews-build.webkit-uat.org/#/builders/34/builds/119/steps/10/logs/stdio? That has what you're looking for.

[John Wilander]

(In reply to Jonathan Bedard from comment #5)
> (In reply to John Wilander from comment #4)
> > Nice! Would it be possible to get the type of failure in that list too? Like
> > text diff, timeout, crash, and such?
> 
> Just curious, are engineers not generally aware of the stdout output:
> https://ews-build.webkit-uat.org/#/builders/34/builds/119/steps/10/logs/
> stdio? That has what you're looking for.

That I don't know. I'm just generally in favor of minimal amounts of clicks, scrolling, and irrelevant information. If a patch failed, I want to see why immediately and I want to see as little other information as possible. The sooner I know something failed, the better.

Personally, I would also love a "Stop processing" feature so that when I see some issue and I realize I need to fix my patch, I can immediately free up resources for others. Aakash and I have discussed that. It's for another change but in the same vein.

Another thing we discussed was the ability to give a hint to EWS on which part of the test suite to test first, again striving for early failure. The quicker I know my patch is not ready for review or landing, the better for everyone.

[Aakash Jain]

(In reply to Jonathan Bedard from comment #5)
> Just curious, are engineers not generally aware of the stdout output
I personally find it confusing since it also list many tests which flaked (which I am not interested in).

[Aakash Jain]

(In reply to John Wilander from comment #4)
> Nice! Would it be possible to get the type of failure in that list too? Like text diff, timeout, crash, and such?
It's possible, but not straight-forward. I will attempt that in a separate patch.

[EWS]

Committed r259457: <https://trac.webkit.org/changeset/259457>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 395273 [details].

[Radar WebKit Bug Importer]

<rdar://problem/61258395>

[Aakash Jain]

Deployed on the server. Seems to be working fine. e.g.: https://ews-build.webkit.org/#/builders/32/builds/6728