Bug 197563 - run-api-tests: Add --iterations and --repeat-each
Summary: run-api-tests: Add --iterations and --repeat-each
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Enhancement
Assignee: Jonathan Bedard
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2019-05-03 09:32 PDT by Shawn Roberts
Modified: 2019-05-31 10:24 PDT (History)
9 users (show)

See Also:


Attachments
Patch (2.27 KB, patch)
2019-05-03 09:56 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews210 for win-future (13.77 MB, application/zip)
2019-05-03 13:11 PDT, EWS Watchlist
no flags Details
Patch for landing (3.35 KB, patch)
2019-05-03 17:24 PDT, Jonathan Bedard
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Shawn Roberts 2019-05-03 09:32:48 PDT
Would be nice, if like with run-webkit-tests we could pass --iterations to test for flaky API behavior.
Comment 1 Radar WebKit Bug Importer 2019-05-03 09:38:12 PDT
<rdar://problem/50446491>
Comment 2 Jonathan Bedard 2019-05-03 09:56:34 PDT
Created attachment 368940 [details]
Patch
Comment 3 Jonathan Bedard 2019-05-03 09:57:09 PDT
I'm actually surprised we don't already do this.

In any case, it's an easy enough change.
Comment 4 EWS Watchlist 2019-05-03 13:11:53 PDT
Comment on attachment 368940 [details]
Patch

Attachment 368940 [details] did not pass win-ews (win):
Output: https://webkit-queues.webkit.org/results/12090443

New failing tests:
fast/shadow-dom/svg-text-path-href-change-in-shadow-tree.html
Comment 5 EWS Watchlist 2019-05-03 13:11:55 PDT
Created attachment 368972 [details]
Archive of layout-test-results from ews210 for win-future

The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews210  Port: win-future  Platform: CYGWIN_NT-10.0-17763-3.0.5-338.x86_64-x86_64-64bit
Comment 6 Aakash Jain 2019-05-03 14:45:38 PDT
Comment on attachment 368940 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=368940&action=review

> Tools/Scripts/webkitpy/api_tests/manager.py:185
> +        test_names = [test for test in test_names for _ in xrange(self._options.repeat_each)]

It feels little weird to duplicate the list of test_names. Maybe a cleaner approach would be to run a for loop, and also log a message with iteration number, so that it's easier for someone to find out when a iteration is complete.

> Tools/Scripts/webkitpy/api_tests/manager.py:186
> +        test_names *= self._options.iterations

Maybe worth adding a log indicating the number of iterations.
Comment 7 Jonathan Bedard 2019-05-03 17:24:17 PDT
Created attachment 369029 [details]
Patch for landing
Comment 8 WebKit Commit Bot 2019-05-03 18:03:26 PDT
Comment on attachment 369029 [details]
Patch for landing

Clearing flags on attachment: 369029

Committed r244942: <https://trac.webkit.org/changeset/244942>
Comment 9 WebKit Commit Bot 2019-05-03 18:03:29 PDT
All reviewed patches have been landed.  Closing bug.