Bug 226097

Summary: Run layout tests in stress mode for 10 iterations on regular EWS layout-test queues
Product: WebKit Reporter: Aakash Jain <aakash_jain>
Component: Tools / TestsAssignee: Aakash Jain <aakash_jain>
Status: RESOLVED FIXED    
Severity: Normal CC: ap, ddkilzer, hi, jbedard, ryanhaddad, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=226273
Attachments:
Description Flags
Patch none

Description Aakash Jain 2021-05-21 11:49:58 PDT 
Stress mode for layout-test was added as a stand-alone queue in https://lists.webkit.org/pipermail/webkit-dev/2021-April/031773.html

Running for 100 iterations sometimes takes long time for certain patches. However we should run the tests being added/modified for limited iterations (e.g.: 10) on regular EWS layout-test queues. This would provide following benefit: 

1) some coverage for stress test on multiple platforms
2) early indication of issues (since running 10 iterations of layout-tests being added/modified is usually quick), before running the complete layout-test suite.
Comment 1 Aakash Jain 2021-05-21 11:51:29 PDT 
Created attachment 429317 [details]
Patch
Comment 2 Aakash Jain 2021-05-21 12:45:01 PDT 
Sample runs:

https://ews-build.webkit-uat.org/#/builders/34/builds/31233 (on patch which modify layout-test)
https://ews-build.webkit-uat.org/#/builders/34/builds/31254 (on patch which doesn't modify layout-test)
Comment 3 Jonathan Bedard 2021-05-21 13:24:28 PDT 
Comment on attachment 429317 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=429317&action=review

> Tools/CISupport/ews-build/steps_unittest.py:1947
> +                                 '--iterations', 100, 'test1', 'test2'],

I'm a bit unclear why anything as changed here.
Comment 4 Jonathan Bedard 2021-05-21 13:27:06 PDT 
Comment on attachment 429317 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=429317&action=review

>> Tools/CISupport/ews-build/steps_unittest.py:1947
>> +                                 '--iterations', 100, 'test1', 'test2'],
> 
> I'm a bit unclear why anything as changed here.

oh! Because you added the "modified_tests" property, got it.
Comment 5 Aakash Jain 2021-05-21 13:28:12 PDT 
Yeah. This should have been there in the first place. But adding the 'doStepIf' in this patch (based on modified_tests) makes it required now.
Comment 6 EWS 2021-05-21 13:58:11 PDT 
Committed r277884 (238021@main): <https://commits.webkit.org/238021@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 429317 [details].
Comment 7 Radar WebKit Bug Importer 2021-05-21 13:59:19 PDT 
<rdar://problem/78327180>
Comment 8 Aakash Jain 2021-05-22 04:32:11 PDT 
Restarted EWS to pick up this change.
Comment 9 Aakash Jain 2021-05-25 10:39:47 PDT 
This change seems to be working fine. e.g.:

Patch which doesn't modify layout-test: https://ews-build.webkit.org/#/builders/51/builds/14126

Patch which add/modify layout-tests and passes: https://ews-build.webkit.org/#/builders/51/builds/14125

Patch which add/modify layout-tests and introduce failure: https://ews-build.webkit.org/#/builders/51/builds/14098