Bug 187549 – [ews-build] EWS should unapply patch and re-run jsc tests when patch fails jsc tests

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

When a patch fails to build jsc, EWS should unapply the patch and build ToT in order to confirm that it is not a pre-existing issue (Similar to https://bugs.webkit.org/show_bug.cgi?id=187358).

Also, when a patch fails jsc tests, EWS should re-run the tests (since these tests are flaky), and if the re-run still fails, it should un-apply the patch and re-run tests on clean tree (similar to current EWS).

[Aakash Jain]

Created attachment 344755 [details]
Proposed patch

We should also analyze the test results from all the re-runs, that would be in a separate step.

[Aakash Jain]

Tested in: http://ews-build.webkit-uat.org/#/builders/17/builds/23

[EWS Watchlist]

Attachment 344755 [details] did not pass style-queue:


ERROR: Tools/BuildSlaveSupport/ews-build/steps.py:213:  [RunJavaScriptCoreTests.evaluateCommand] Use of super on an old style class  [pylint/E1002] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps.py:229:  [ReRunJavaScriptCoreTests.evaluateCommand] Use of super on an old style class  [pylint/E1002] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:548:  [TestCompileJSCOnlyToT.test_success] Passing unexpected keyword argument 'state_string' in function call  [pylint/E1123] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:548:  [TestCompileJSCOnlyToT.test_success] No value passed for parameter 'status_text' in function call  [pylint/E1120] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:563:  [TestCompileJSCOnlyToT.test_failure] Passing unexpected keyword argument 'state_string' in function call  [pylint/E1123] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:563:  [TestCompileJSCOnlyToT.test_failure] No value passed for parameter 'status_text' in function call  [pylint/E1120] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:571:  [TestCompileJSCOnlyToT.test_skip] Passing unexpected keyword argument 'state_string' in function call  [pylint/E1123] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:571:  [TestCompileJSCOnlyToT.test_skip] No value passed for parameter 'status_text' in function call  [pylint/E1120] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:635:  [TestReRunJavaScriptCoreTests.test_success] Passing unexpected keyword argument 'state_string' in function call  [pylint/E1123] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:635:  [TestReRunJavaScriptCoreTests.test_success] No value passed for parameter 'status_text' in function call  [pylint/E1120] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:651:  [TestReRunJavaScriptCoreTests.test_failure] Passing unexpected keyword argument 'state_string' in function call  [pylint/E1123] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:651:  [TestReRunJavaScriptCoreTests.test_failure] No value passed for parameter 'status_text' in function call  [pylint/E1120] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:659:  [TestReRunJavaScriptCoreTests.test_skip] Passing unexpected keyword argument 'state_string' in function call  [pylint/E1123] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:659:  [TestReRunJavaScriptCoreTests.test_skip] No value passed for parameter 'status_text' in function call  [pylint/E1120] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:684:  [TestRunJavaScriptCoreTestsToT.test_success] Passing unexpected keyword argument 'state_string' in function call  [pylint/E1123] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:684:  [TestRunJavaScriptCoreTestsToT.test_success] No value passed for parameter 'status_text' in function call  [pylint/E1120] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:700:  [TestRunJavaScriptCoreTestsToT.test_failure] Passing unexpected keyword argument 'state_string' in function call  [pylint/E1123] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:700:  [TestRunJavaScriptCoreTestsToT.test_failure] No value passed for parameter 'status_text' in function call  [pylint/E1120] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:708:  [TestRunJavaScriptCoreTestsToT.test_skip] Passing unexpected keyword argument 'state_string' in function call  [pylint/E1123] [5]
ERROR: Tools/BuildSlaveSupport/ews-build/steps_unittest.py:708:  [TestRunJavaScriptCoreTestsToT.test_skip] No value passed for parameter 'status_text' in function call  [pylint/E1120] [5]
Total errors found: 20 in 4 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Aakash Jain]

Style checker warnings are false positive.

[Aakash Jain]

> We should also analyze the test results from all the re-runs, that would be in a separate step.
I was exploring if Build Factory can access step results and do the analysis, but it seems like they can not be. See: 
https://lists.buildbot.net/pipermail/devel/2018-July/012543.html
The other option suggested here is to have one step dynamically add next step, but I think in this case that would result in confusing and unreadable code.

Adding a separate step to do the analysis of all the re-runs seems like the best option.

[WebKit Commit Bot]

Comment on attachment 344755 [details]
Proposed patch

Clearing flags on attachment: 344755

Committed r233757: <https://trac.webkit.org/changeset/233757>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/42102573>