Bug 204174 - [ews] Add build step to Analyze JSC Tests Results
Summary: [ews] Add build step to Analyze JSC Tests Results
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Aakash Jain
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2019-11-13 15:33 PST by Aakash Jain
Modified: 2019-11-13 19:08 PST (History)
7 users (show)

See Also:


Attachments
Patch (14.39 KB, patch)
2019-11-13 15:38 PST, Aakash Jain
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Aakash Jain 2019-11-13 15:33:08 PST
Add build step to analyze JavaScriptCore tests results, including re-run and clean-tree-run. Determine if the failures were introduced by the patch, were pre-existing.
Comment 1 Aakash Jain 2019-11-13 15:38:59 PST
Created attachment 383502 [details]
Patch
Comment 3 Jonathan Bedard 2019-11-13 17:21:37 PST
Comment on attachment 383502 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=383502&action=review

> Tools/BuildSlaveSupport/ews-build/steps.py:1081
> +        flaky_binary_failures = first_run_binary_failures.union(second_run_binary_failures) - first_run_binary_failures.intersection(second_run_binary_failures)

I'm not convinced that we should give a green bubble with flaky binary failures....From talking to Zhifei, I get the sense that the binaries don't have the same sort of flake problems that layout tests or jsc stress tests have.
Comment 4 Zhifei Fang 2019-11-13 17:24:39 PST
wait, if we have already submitted results to results database, why we need to rerun it without patch ?
Comment 5 Jonathan Bedard 2019-11-13 17:35:36 PST
(In reply to Zhifei Fang from comment #4)
> wait, if we have already submitted results to results database, why we need
> to rerun it without patch ?

We don't have EWS plugged in to the results database yet.

We totally could now, though. Ling recently sorted out the network access.
Comment 6 Jonathan Bedard 2019-11-13 17:37:47 PST
Comment on attachment 383502 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=383502&action=review

>> Tools/BuildSlaveSupport/ews-build/steps.py:1081
>> +        flaky_binary_failures = first_run_binary_failures.union(second_run_binary_failures) - first_run_binary_failures.intersection(second_run_binary_failures)
> 
> I'm not convinced that we should give a green bubble with flaky binary failures....From talking to Zhifei, I get the sense that the binaries don't have the same sort of flake problems that layout tests or jsc stress tests have.

Aakash, Zhifei and I talked about this a bit offline.

Binary tests probably aren't flakey, but this is what old EWS did and we don't actually have concrete data on if test binaries are flakey without https://bugs.webkit.org/show_bug.cgi?id=204091. We might come back and revisit this, but for the sake of converting JSC EWS to new EWS, this behavior is the same as what we were doing before.
Comment 7 WebKit Commit Bot 2019-11-13 19:07:41 PST
Comment on attachment 383502 [details]
Patch

Clearing flags on attachment: 383502

Committed r252446: <https://trac.webkit.org/changeset/252446>
Comment 8 WebKit Commit Bot 2019-11-13 19:07:43 PST
All reviewed patches have been landed.  Closing bug.
Comment 9 Radar WebKit Bug Importer 2019-11-13 19:08:19 PST
<rdar://problem/57177809>