We should add support for running API tests in the new OpenSource EWS Buildbot, both for mac as well as ios-simulator. Note that these tests require building. So, a builder should trigger these tests appropriately. Also see: <rdar://problem/13690874> and <rdar://problem/30214936>.
Created attachment 346051 [details] Proposed patch Sample run: http://ews-build.webkit-uat.org/#/builders/20/builds/2
Comment on attachment 346051 [details] Proposed patch Attachment 346051 [details] did not pass win-ews (win): Output: https://webkit-queues.webkit.org/results/8697263 New failing tests: imported/blink/transitions/unprefixed-transform.html
Created attachment 346055 [details] Archive of layout-test-results from ews206 for win-future The attached test failures were seen while running run-webkit-tests on the win-ews. Bot: ews206 Port: win-future Platform: CYGWIN_NT-6.1-2.9.0-0.318-5-3-x86_64-64bit
Sample run for macOS API Tests: http://ews-build.webkit-uat.org/#/builders/19/builds/5
Committed r234397: <https://trac.webkit.org/changeset/234397>
<rdar://problem/42751658>
Note: Since the API tests are flaky. I would also need to add logic to: - retry these tests on failure - modify run_api_tests script to output test results in json format - parse and compare results and decide whether the failures are introduced by the patch
(In reply to Aakash Jain from comment #7) > Note: Since the API tests are flaky. I would also need to add logic to: > - retry these tests on failure > - modify run_api_tests script to output test results in json format > - parse and compare results and decide whether the failures are introduced > by the patch Wait, which API tests are flaky? The rest of our infrastructure generally assumes they aren't (this isn't totally true, but it's mostly true)