[ews] Add build step to Analyze JSC Tests Results

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Aakash Jain]

Add build step to analyze JavaScriptCore tests results, including re-run and clean-tree-run. Determine if the failures were introduced by the patch, were pre-existing.

[Aakash Jain]

Created attachment 383502 [details]
Patch

[Aakash Jain]

Sample runs:
JSC stress test failure: https://ews-build.webkit-uat.org/#/builders/21/builds/430

binary and stress test failure: https://ews-build.webkit-uat.org/#/builders/21/builds/442

Flaky tests: https://ews-build.webkit-uat.org/#/builders/21/builds/435
Pre-existing failure: https://ews-build.webkit-uat.org/#/builders/21/builds/457

[Jonathan Bedard]

Comment on attachment 383502 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=383502&action=review

> Tools/BuildSlaveSupport/ews-build/steps.py:1081
> +        flaky_binary_failures = first_run_binary_failures.union(second_run_binary_failures) - first_run_binary_failures.intersection(second_run_binary_failures)

I'm not convinced that we should give a green bubble with flaky binary failures....From talking to Zhifei, I get the sense that the binaries don't have the same sort of flake problems that layout tests or jsc stress tests have.

[Zhifei Fang]

wait, if we have already submitted results to results database, why we need to rerun it without patch ?

[Jonathan Bedard]

(In reply to Zhifei Fang from comment #4)
> wait, if we have already submitted results to results database, why we need
> to rerun it without patch ?

We don't have EWS plugged in to the results database yet.

We totally could now, though. Ling recently sorted out the network access.

[Jonathan Bedard]

Comment on attachment 383502 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=383502&action=review

>> Tools/BuildSlaveSupport/ews-build/steps.py:1081
>> +        flaky_binary_failures = first_run_binary_failures.union(second_run_binary_failures) - first_run_binary_failures.intersection(second_run_binary_failures)
> 
> I'm not convinced that we should give a green bubble with flaky binary failures....From talking to Zhifei, I get the sense that the binaries don't have the same sort of flake problems that layout tests or jsc stress tests have.

Aakash, Zhifei and I talked about this a bit offline.

Binary tests probably aren't flakey, but this is what old EWS did and we don't actually have concrete data on if test binaries are flakey without https://bugs.webkit.org/show_bug.cgi?id=204091. We might come back and revisit this, but for the sake of converting JSC EWS to new EWS, this behavior is the same as what we were doing before.

[WebKit Commit Bot]

Comment on attachment 383502 [details]
Patch

Clearing flags on attachment: 383502

Committed r252446: <https://trac.webkit.org/changeset/252446>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/57177809>