Bug 197563

Summary:

run-api-tests: Add --iterations and --repeat-each

Product:

WebKit

Reporter:

Shawn Roberts <sroberts>

Component:

Tools / Tests

Assignee:

Jonathan Bedard <jbedard>

Status:

RESOLVED FIXED

Severity:

Enhancement

CC:

aakash_jain, commit-queue, ews-watchlist, glenn, jbedard, lforschler, ryanhaddad, webkit-bot-watchers-bugzilla, webkit-bug-importer

Priority:

Keywords:

InRadar

Version:

WebKit Nightly Build

Hardware:

Unspecified

OS:

Unspecified

See Also:

https://bugs.webkit.org/show_bug.cgi?id=198425

Attachments:

Description	Flags
Patch	none
Archive of layout-test-results from ews210 for win-future	none
Patch for landing	none

Shawn Roberts

Reported 2019-05-03 09:32:48 PDT

Would be nice, if like with run-webkit-tests we could pass --iterations to test for flaky API behavior.

Attachments
Patch (2.27 KB, patch) 2019-05-03 09:56 PDT, Jonathan Bedard	no flags	Details Formatted Diff Diff
Archive of layout-test-results from ews210 for win-future (13.77 MB, application/zip) 2019-05-03 13:11 PDT, EWS Watchlist	no flags	Details
Patch for landing (3.35 KB, patch) 2019-05-03 17:24 PDT, Jonathan Bedard	no flags	Details Formatted Diff Diff
Show Obsolete (1) View All Add attachment proposed patch, testcase, etc.

Radar WebKit Bug Importer

Comment 1 2019-05-03 09:38:12 PDT

<rdar://problem/50446491>

Jonathan Bedard

Comment 2 2019-05-03 09:56:34 PDT

Created attachment 368940 [details] Patch

Jonathan Bedard

Comment 3 2019-05-03 09:57:09 PDT

I'm actually surprised we don't already do this. In any case, it's an easy enough change.

EWS Watchlist

Comment 4 2019-05-03 13:11:53 PDT

Comment on attachment 368940 [details] Patch Attachment 368940 [details] did not pass win-ews (win): Output: https://webkit-queues.webkit.org/results/12090443 New failing tests: fast/shadow-dom/svg-text-path-href-change-in-shadow-tree.html

EWS Watchlist

Comment 5 2019-05-03 13:11:55 PDT

Created attachment 368972 [details] Archive of layout-test-results from ews210 for win-future The attached test failures were seen while running run-webkit-tests on the win-ews. Bot: ews210 Port: win-future Platform: CYGWIN_NT-10.0-17763-3.0.5-338.x86_64-x86_64-64bit

Aakash Jain

Comment 6 2019-05-03 14:45:38 PDT

Comment on attachment 368940 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=368940&action=review > Tools/Scripts/webkitpy/api_tests/manager.py:185 > + test_names = [test for test in test_names for _ in xrange(self._options.repeat_each)] It feels little weird to duplicate the list of test_names. Maybe a cleaner approach would be to run a for loop, and also log a message with iteration number, so that it's easier for someone to find out when a iteration is complete. > Tools/Scripts/webkitpy/api_tests/manager.py:186 > + test_names *= self._options.iterations Maybe worth adding a log indicating the number of iterations.

Jonathan Bedard

Comment 7 2019-05-03 17:24:17 PDT

Created attachment 369029 [details] Patch for landing

WebKit Commit Bot

Comment 8 2019-05-03 18:03:26 PDT

Comment on attachment 369029 [details] Patch for landing Clearing flags on attachment: 369029 Committed r244942: <https://trac.webkit.org/changeset/244942>

WebKit Commit Bot

Comment 9 2019-05-03 18:03:29 PDT

All reviewed patches have been landed. Closing bug.

Note You need to log in before you can comment on or make changes to this bug.