Layout test imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub.html is frequently failing on mac-ews bots --- /Volumes/Data/EWS/WebKit/WebKitBuild/Debug/layout-test-results/imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub-expected.txt +++ /Volumes/Data/EWS/WebKit/WebKitBuild/Debug/layout-test-results/imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub-actual.txt @@ -1,11 +1,27 @@ - -Harness Error (TIMEOUT), message = null PASS No timeline entry for about:blank PASS 'iframe (Populate cache): The initial request populates the cache (if appropriate). PASS 'iframe (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). PASS 'xmlhttprequest (Populate cache): The initial request populates the cache (if appropriate). PASS 'xmlhttprequest (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). -TIMEOUT 'script (Populate cache): The initial request populates the cache (if appropriate). Test timed out -TIMEOUT 'script (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). Test timed out +PASS 'script (Populate cache): The initial request populates the cache (if appropriate). +PASS 'script (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). +PASS 'link (Populate cache): The initial request populates the cache (if appropriate). +PASS 'link (Potentially Cached): Immediately fetch the same URL, exercising the cache hit path (if any). +PASS 'iframe: 250ms delay before 'responseStart', another 250ms delay before 'responseEnd'. +PASS 'xmlhttprequest: 250ms delay before 'responseStart', another 250ms delay before 'responseEnd'. +PASS 'script: 250ms delay before 'responseStart', another 250ms delay before 'responseEnd'. +PASS 'link: 250ms delay before 'responseStart', another 250ms delay before 'responseEnd'. +PASS 'iframe (Redirected): 250ms delay before 'redirectEnd', another 250ms delay before 'responseStart'. +PASS 'xmlhttprequest (Redirected): 250ms delay before 'redirectEnd', another 250ms delay before 'responseStart'. +PASS 'script (Redirected): 250ms delay before 'redirectEnd', another 250ms delay before 'responseStart'. +PASS 'link (Redirected): 250ms delay before 'redirectEnd', another 250ms delay before 'responseStart'. +FAIL 'iframe 250ms delay in headers does not affect responseStart' assert_greater_than_equal: Delay after HTTP/1.1 status should not affect 'responseStart'. expected a number greater than or equal to 9254 but got 9012 +FAIL 'xmlhttprequest 250ms delay in headers does not affect responseStart' assert_greater_than_equal: Delay after HTTP/1.1 status should not affect 'responseStart'. expected a number greater than or equal to 9776 but got 9529 +FAIL 'script 250ms delay in headers does not affect responseStart' assert_greater_than_equal: Delay after HTTP/1.1 status should not affect 'responseStart'. expected a number greater than or equal to 10276 but got 10029 +FAIL 'link 250ms delay in headers does not affect responseStart' assert_greater_than_equal: Delay after HTTP/1.1 status should not affect 'responseStart'. expected a number greater than or equal to 10772 but got 10525 +FAIL 'iframe responseStart uses 1XX (first) response timings' assert_greater_than_equal: HTTP/1.1 1XX (first) response should determine 'responseStart' timing. expected a number greater than or equal to 11291 but got 11049 +FAIL 'xmlhttprequest responseStart uses 1XX (first) response timings' assert_greater_than_equal: HTTP/1.1 1XX (first) response should determine 'responseStart' timing. expected a number greater than or equal to 11808 but got 11561 +FAIL 'script responseStart uses 1XX (first) response timings' assert_greater_than_equal: HTTP/1.1 1XX (first) response should determine 'responseStart' timing. expected a number greater than or equal to 12316 but got 12068 +FAIL 'link responseStart uses 1XX (first) response timings' assert_greater_than_equal: HTTP/1.1 1XX (first) response should determine 'responseStart' timing. expected a number greater than or equal to 12821 but got 12574 Show details
This test was imported with https://trac.webkit.org/changeset/239959/webkit
Based on the diff, it looks like we expect the test to time out, but it is actually running more subtests than before which produces unexpected output.
The test does not fail on trunk bots according to https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#tests=imported%2Fw3c%2Fweb-platform-tests%2Fresource-timing%2Fresource-timing-level1.sub.html
Marked test as flaky for macOS in https://trac.webkit.org/changeset/244844/webkit to stop the retries on EWS bots.
Created attachment 368752 [details] Patch
Let's try to rebase the expected.txt then.
Comment on attachment 368752 [details] Patch Attachment 368752 [details] did not pass ios-sim-ews (ios-simulator-wk2): Output: https://webkit-queues.webkit.org/results/12058388 New failing tests: imported/w3c/web-platform-tests/resource-timing/resource-timing-level1.sub.html
Created attachment 368759 [details] Archive of layout-test-results from ews121 for ios-simulator-wk2 The attached test failures were seen while running run-webkit-tests on the ios-sim-ews. Bot: ews121 Port: ios-simulator-wk2 Platform: Mac OS X 10.14.4
Comment on attachment 368752 [details] Patch Attachment 368752 [details] did not pass win-ews (win): Output: https://webkit-queues.webkit.org/results/12063497 New failing tests: legacy-animation-engine/fast/layers/no-clipping-overflow-hidden-hardware-acceleration.html
Created attachment 368791 [details] Archive of layout-test-results from ews215 for win-future The attached test failures were seen while running run-webkit-tests on the win-ews. Bot: ews215 Port: win-future Platform: CYGWIN_NT-10.0-17763-3.0.5-338.x86_64-x86_64-64bit
<rdar://problem/50448266>
The test is flaky on ios-sim-ews as well. Marked as flaky in https://trac.webkit.org/r244976