REGRESSION: Layout test imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub.html is frequently failing on EWS

• Component: Tools / Tests
• Status: NEW ---

[Ryan Haddad]

Layout test imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub.html is frequently failing on mac-ews bots

--- /Volumes/Data/EWS/WebKit/WebKitBuild/Debug/layout-test-results/imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub-expected.txt
+++ /Volumes/Data/EWS/WebKit/WebKitBuild/Debug/layout-test-results/imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub-actual.txt
@@ -1,11 +1,27 @@
-
-Harness Error (TIMEOUT), message = null
 
 PASS No timeline entry for about:blank 
 PASS 'iframe (Populate cache): The initial request populates the cache (if appropriate). 
 PASS 'iframe (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). 
 PASS 'xmlhttprequest (Populate cache): The initial request populates the cache (if appropriate). 
 PASS 'xmlhttprequest (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). 
-TIMEOUT 'script (Populate cache): The initial request populates the cache (if appropriate). Test timed out
-TIMEOUT 'script (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). Test timed out
+PASS 'script (Populate cache): The initial request populates the cache (if appropriate). 
+PASS 'script (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). 
+PASS 'link (Populate cache): The initial request populates the cache (if appropriate). 
+PASS 'link (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). 
+PASS 'iframe: 250ms delay before 'responseStart', another 250ms delay before 'responseEnd'. 
+PASS 'xmlhttprequest: 250ms delay before 'responseStart', another 250ms delay before 'responseEnd'. 
+PASS 'script: 250ms delay before 'responseStart', another 250ms delay before 'responseEnd'. 
+PASS 'link: 250ms delay before 'responseStart', another 250ms delay before 'responseEnd'. 
+PASS 'iframe (Redirected): 250ms delay before 'redirectEnd', another 250ms delay before 'responseStart'. 
+PASS 'xmlhttprequest (Redirected): 250ms delay before 'redirectEnd', another 250ms delay before 'responseStart'. 
+PASS 'script (Redirected): 250ms delay before 'redirectEnd', another 250ms delay before 'responseStart'. 
+PASS 'link (Redirected): 250ms delay before 'redirectEnd', another 250ms delay before 'responseStart'. 
+FAIL 'iframe 250ms delay in headers does not affect responseStart' assert_greater_than_equal: Delay after HTTP/1.1 status should not affect 'responseStart'. expected a number greater than or equal to 9254 but got 9012
+FAIL 'xmlhttprequest 250ms delay in headers does not affect responseStart' assert_greater_than_equal: Delay after HTTP/1.1 status should not affect 'responseStart'. expected a number greater than or equal to 9776 but got 9529
+FAIL 'script 250ms delay in headers does not affect responseStart' assert_greater_than_equal: Delay after HTTP/1.1 status should not affect 'responseStart'. expected a number greater than or equal to 10276 but got 10029
+FAIL 'link 250ms delay in headers does not affect responseStart' assert_greater_than_equal: Delay after HTTP/1.1 status should not affect 'responseStart'. expected a number greater than or equal to 10772 but got 10525
+FAIL 'iframe responseStart uses 1XX (first) response timings' assert_greater_than_equal: HTTP/1.1 1XX (first) response should determine 'responseStart' timing. expected a number greater than or equal to 11291 but got 11049
+FAIL 'xmlhttprequest responseStart uses 1XX (first) response timings' assert_greater_than_equal: HTTP/1.1 1XX (first) response should determine 'responseStart' timing. expected a number greater than or equal to 11808 but got 11561
+FAIL 'script responseStart uses 1XX (first) response timings' assert_greater_than_equal: HTTP/1.1 1XX (first) response should determine 'responseStart' timing. expected a number greater than or equal to 12316 but got 12068
+FAIL 'link responseStart uses 1XX (first) response timings' assert_greater_than_equal: HTTP/1.1 1XX (first) response should determine 'responseStart' timing. expected a number greater than or equal to 12821 but got 12574
 Show details

[Ryan Haddad]

This test was imported with https://trac.webkit.org/changeset/239959/webkit

[Ryan Haddad]

Based on the diff, it looks like we expect the test to time out, but it is actually running more subtests than before which produces unexpected output.

[Ryan Haddad]

The test does not fail on trunk bots according to https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#tests=imported%2Fw3c%2Fweb-platform-tests%2Fresource-timing%2Fresource-timing-level1.sub.html

[Ryan Haddad]

Marked test as flaky for macOS in https://trac.webkit.org/changeset/244844/webkit to stop the retries on EWS bots.

[youenn fablet]

Created attachment 368752 [details]
Patch

[youenn fablet]

Let's try to rebase the expected.txt then.

[EWS Watchlist]

Comment on attachment 368752 [details]
Patch

Attachment 368752 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: https://webkit-queues.webkit.org/results/12058388

New failing tests:
imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub.html

[EWS Watchlist]

Created attachment 368759 [details]
Archive of layout-test-results from ews121 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews121  Port: ios-simulator-wk2  Platform: Mac OS X 10.14.4

[EWS Watchlist]

Comment on attachment 368752 [details]
Patch

Attachment 368752 [details] did not pass win-ews (win):
Output: https://webkit-queues.webkit.org/results/12063497

New failing tests:
legacy-animation-engine/fast/layers/no-clipping-overflow-hidden-hardware-acceleration.html

[EWS Watchlist]

Created attachment 368791 [details]
Archive of layout-test-results from ews215 for win-future

The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews215  Port: win-future  Platform: CYGWIN_NT-10.0-17763-3.0.5-338.x86_64-x86_64-64bit

[Radar WebKit Bug Importer]

<rdar://problem/50448266>

[Ryan Haddad]

The test is flaky on ios-sim-ews as well. Marked as flaky in https://trac.webkit.org/r244976