Bug 197473 - REGRESSION: Layout test imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub.html is frequently failing on EWS
Summary: REGRESSION: Layout test imported/w3c/web-platform-tests/resource-timing/resou...
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: youenn fablet
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2019-05-01 13:15 PDT by Ryan Haddad
Modified: 2019-06-16 20:51 PDT (History)
4 users (show)

See Also:


Attachments
Patch (6.49 KB, patch)
2019-05-01 19:42 PDT, youenn fablet
ews: commit-queue-
Details | Formatted Diff | Diff
Archive of layout-test-results from ews121 for ios-simulator-wk2 (2.49 MB, application/zip)
2019-05-01 21:44 PDT, Build Bot
no flags Details
Archive of layout-test-results from ews215 for win-future (13.48 MB, application/zip)
2019-05-02 11:11 PDT, Build Bot
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ryan Haddad 2019-05-01 13:15:53 PDT
Layout test imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub.html is frequently failing on mac-ews bots

--- /Volumes/Data/EWS/WebKit/WebKitBuild/Debug/layout-test-results/imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub-expected.txt
+++ /Volumes/Data/EWS/WebKit/WebKitBuild/Debug/layout-test-results/imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub-actual.txt
@@ -1,11 +1,27 @@
-
-Harness Error (TIMEOUT), message = null
 
 PASS No timeline entry for about:blank 
 PASS 'iframe (Populate cache): The initial request populates the cache (if appropriate). 
 PASS 'iframe (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). 
 PASS 'xmlhttprequest (Populate cache): The initial request populates the cache (if appropriate). 
 PASS 'xmlhttprequest (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). 
-TIMEOUT 'script (Populate cache): The initial request populates the cache (if appropriate). Test timed out
-TIMEOUT 'script (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). Test timed out
+PASS 'script (Populate cache): The initial request populates the cache (if appropriate). 
+PASS 'script (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). 
+PASS 'link (Populate cache): The initial request populates the cache (if appropriate). 
+PASS 'link (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). 
+PASS 'iframe: 250ms delay before 'responseStart', another 250ms delay before 'responseEnd'. 
+PASS 'xmlhttprequest: 250ms delay before 'responseStart', another 250ms delay before 'responseEnd'. 
+PASS 'script: 250ms delay before 'responseStart', another 250ms delay before 'responseEnd'. 
+PASS 'link: 250ms delay before 'responseStart', another 250ms delay before 'responseEnd'. 
+PASS 'iframe (Redirected): 250ms delay before 'redirectEnd', another 250ms delay before 'responseStart'. 
+PASS 'xmlhttprequest (Redirected): 250ms delay before 'redirectEnd', another 250ms delay before 'responseStart'. 
+PASS 'script (Redirected): 250ms delay before 'redirectEnd', another 250ms delay before 'responseStart'. 
+PASS 'link (Redirected): 250ms delay before 'redirectEnd', another 250ms delay before 'responseStart'. 
+FAIL 'iframe 250ms delay in headers does not affect responseStart' assert_greater_than_equal: Delay after HTTP/1.1 status should not affect 'responseStart'. expected a number greater than or equal to 9254 but got 9012
+FAIL 'xmlhttprequest 250ms delay in headers does not affect responseStart' assert_greater_than_equal: Delay after HTTP/1.1 status should not affect 'responseStart'. expected a number greater than or equal to 9776 but got 9529
+FAIL 'script 250ms delay in headers does not affect responseStart' assert_greater_than_equal: Delay after HTTP/1.1 status should not affect 'responseStart'. expected a number greater than or equal to 10276 but got 10029
+FAIL 'link 250ms delay in headers does not affect responseStart' assert_greater_than_equal: Delay after HTTP/1.1 status should not affect 'responseStart'. expected a number greater than or equal to 10772 but got 10525
+FAIL 'iframe responseStart uses 1XX (first) response timings' assert_greater_than_equal: HTTP/1.1 1XX (first) response should determine 'responseStart' timing. expected a number greater than or equal to 11291 but got 11049
+FAIL 'xmlhttprequest responseStart uses 1XX (first) response timings' assert_greater_than_equal: HTTP/1.1 1XX (first) response should determine 'responseStart' timing. expected a number greater than or equal to 11808 but got 11561
+FAIL 'script responseStart uses 1XX (first) response timings' assert_greater_than_equal: HTTP/1.1 1XX (first) response should determine 'responseStart' timing. expected a number greater than or equal to 12316 but got 12068
+FAIL 'link responseStart uses 1XX (first) response timings' assert_greater_than_equal: HTTP/1.1 1XX (first) response should determine 'responseStart' timing. expected a number greater than or equal to 12821 but got 12574
 Show details
Comment 1 Ryan Haddad 2019-05-01 13:17:19 PDT
This test was imported with https://trac.webkit.org/changeset/239959/webkit
Comment 2 Ryan Haddad 2019-05-01 13:18:08 PDT
Based on the diff, it looks like we expect the test to time out, but it is actually running more subtests than before which produces unexpected output.
Comment 4 Ryan Haddad 2019-05-01 13:36:10 PDT
Marked test as flaky for macOS in https://trac.webkit.org/changeset/244844/webkit to stop the retries on EWS bots.
Comment 5 youenn fablet 2019-05-01 19:42:16 PDT
Created attachment 368752 [details]
Patch
Comment 6 youenn fablet 2019-05-01 19:42:43 PDT
Let's try to rebase the expected.txt then.
Comment 7 Build Bot 2019-05-01 21:44:31 PDT
Comment on attachment 368752 [details]
Patch

Attachment 368752 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: https://webkit-queues.webkit.org/results/12058388

New failing tests:
imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub.html
Comment 8 Build Bot 2019-05-01 21:44:33 PDT
Created attachment 368759 [details]
Archive of layout-test-results from ews121 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews121  Port: ios-simulator-wk2  Platform: Mac OS X 10.14.4
Comment 9 Build Bot 2019-05-02 11:11:11 PDT
Comment on attachment 368752 [details]
Patch

Attachment 368752 [details] did not pass win-ews (win):
Output: https://webkit-queues.webkit.org/results/12063497

New failing tests:
legacy-animation-engine/fast/layers/no-clipping-overflow-hidden-hardware-acceleration.html
Comment 10 Build Bot 2019-05-02 11:11:24 PDT
Created attachment 368791 [details]
Archive of layout-test-results from ews215 for win-future

The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews215  Port: win-future  Platform: CYGWIN_NT-10.0-17763-3.0.5-338.x86_64-x86_64-64bit
Comment 11 Radar WebKit Bug Importer 2019-05-03 10:28:45 PDT
<rdar://problem/50448266>
Comment 12 Ryan Haddad 2019-05-06 13:45:27 PDT
The test is flaky on ios-sim-ews as well. Marked as flaky in https://trac.webkit.org/r244976