Bug 188156

Summary:

[ews-build] Add support for API tests in OpenSource EWS

Product:

WebKit

Reporter:

Aakash Jain <aakash_jain>

Component:

Tools / Tests

Assignee:

Aakash Jain <aakash_jain>

Status:

RESOLVED FIXED

Severity:

Normal

CC:

aakash_jain, ap, ddkilzer, ews-watchlist, jbedard, lforschler, webkit-bug-importer

Priority:

Keywords:

InRadar

Version:

Other

Hardware:

Unspecified

OS:

Unspecified

Attachments:

Description	Flags
Proposed patch	lforschler: review+
Archive of layout-test-results from ews206 for win-future	none

Aakash Jain

Reported 2018-07-29 23:33:39 PDT

We should add support for running API tests in the new OpenSource EWS Buildbot, both for mac as well as ios-simulator. Note that these tests require building. So, a builder should trigger these tests appropriately. Also see: <rdar://problem/13690874> and <rdar://problem/30214936>.

Attachments
Proposed patch (7.62 KB, patch) 2018-07-29 23:43 PDT, Aakash Jain	lforschler: review+	Details Formatted Diff Diff
Archive of layout-test-results from ews206 for win-future (13.01 MB, application/zip) 2018-07-30 01:58 PDT, EWS Watchlist	no flags	Details
Show Obsolete (1) View All Add attachment proposed patch, testcase, etc.

Aakash Jain

Comment 1 2018-07-29 23:43:03 PDT

Created attachment 346051 [details] Proposed patch Sample run: http://ews-build.webkit-uat.org/#/builders/20/builds/2

EWS Watchlist

Comment 2 2018-07-30 01:57:59 PDT Comment hidden (obsolete)

Comment on attachment 346051 [details] Proposed patch Attachment 346051 [details] did not pass win-ews (win): Output: https://webkit-queues.webkit.org/results/8697263 New failing tests: imported/blink/transitions/unprefixed-transform.html

EWS Watchlist

Comment 3 2018-07-30 01:58:11 PDT Comment hidden (obsolete)

Created attachment 346055 [details] Archive of layout-test-results from ews206 for win-future The attached test failures were seen while running run-webkit-tests on the win-ews. Bot: ews206 Port: win-future Platform: CYGWIN_NT-6.1-2.9.0-0.318-5-3-x86_64-64bit

Aakash Jain

Comment 4 2018-07-30 12:10:06 PDT

Sample run for macOS API Tests: http://ews-build.webkit-uat.org/#/builders/19/builds/5

Aakash Jain

Comment 5 2018-07-30 16:30:40 PDT

Committed r234397: <https://trac.webkit.org/changeset/234397>

Radar WebKit Bug Importer

Comment 6 2018-07-30 16:32:26 PDT

<rdar://problem/42751658>

Aakash Jain

Comment 7 2018-07-31 17:11:30 PDT

Note: Since the API tests are flaky. I would also need to add logic to: - retry these tests on failure - modify run_api_tests script to output test results in json format - parse and compare results and decide whether the failures are introduced by the patch

Jonathan Bedard

Comment 8 2018-08-22 14:31:34 PDT

(In reply to Aakash Jain from comment #7) > Note: Since the API tests are flaky. I would also need to add logic to: > - retry these tests on failure > - modify run_api_tests script to output test results in json format > - parse and compare results and decide whether the failures are introduced > by the patch Wait, which API tests are flaky? The rest of our infrastructure generally assumes they aren't (this isn't totally true, but it's mostly true)

Note You need to log in before you can comment on or make changes to this bug.