Bug 177322 – [debug] LayoutTest imported/w3c/web-platform-tests/html/semantics/document-metadata/the-meta-element/pragma-directives/attr-meta-http-equiv-refresh/parsing.html is a flaky failure

• Component: New Bugs
• Status: NEW

[Ryan Haddad]

LayoutTest imported/w3c/web-platform-tests/html/semantics/document-metadata/the-meta-element/pragma-directives/attr-meta-http-equiv-refresh/parsing.html is a flaky failure

https://build.webkit.org/builders/Apple%20El%20Capitan%20Debug%20WK2%20(Tests)/builds/3205

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fsemantics%2Fdocument-metadata%2Fthe-meta-element%2Fpragma-directives%2Fattr-meta-http-equiv-refresh%2Fparsing.html

--- /Volumes/Data/slave/elcapitan-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/document-metadata/the-meta-element/pragma-directives/attr-meta-http-equiv-refresh/parsing-expected.txt
+++ /Volumes/Data/slave/elcapitan-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/document-metadata/the-meta-element/pragma-directives/attr-meta-http-equiv-refresh/parsing-actual.txt
@@ -85,17 +85,17 @@
 FAIL <meta>: "foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
 PASS Refresh header: "foo" 
 FAIL <meta>: "+1; url=foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
-FAIL Refresh header: "+1; url=foo" assert_equals: expected "refresh.py" but got "foo"
+PASS Refresh header: "+1; url=foo" 
 FAIL <meta>: "-1; url=foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
 PASS Refresh header: "-1; url=foo" 
 FAIL <meta>: "+0; url=foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
 FAIL Refresh header: "+0; url=foo" assert_equals: expected "refresh.py" but got "foo"
 FAIL <meta>: "-0; url=foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
-FAIL Refresh header: "-0; url=foo" assert_equals: expected "refresh.py" but got "foo"
+PASS Refresh header: "-0; url=foo" 
 TIMEOUT <meta>: "0; url=foo" Test timed out
 PASS Refresh header: "0; url=foo" 
 FAIL <meta>: "+1; foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
-FAIL Refresh header: "+1; foo" assert_equals: expected "refresh.py" but got "foo"
+PASS Refresh header: "+1; foo" 
 FAIL <meta>: "-1; foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
 PASS Refresh header: "-1; foo" 
 FAIL <meta>: "+0; foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
@@ -105,7 +105,7 @@
 TIMEOUT <meta>: "0; foo" Test timed out
 PASS Refresh header: "0; foo" 
 FAIL <meta>: "+1" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
-FAIL Refresh header: "+1" assert_unreached: Got > 1 load events Reached unreachable code
+PASS Refresh header: "+1" 
 FAIL <meta>: "-1" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
 PASS Refresh header: "-1" 
 FAIL <meta>: "+0" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"

[Ryan Haddad]

This appears to have started with the latest WPT refresh: https://trac.webkit.org/changeset/222307/webkit

[Ryan Haddad]

Marked as flaky in https://trac.webkit.org/r222341

[Matt Lewis]

This has started to be flaky on iOS as well:
https://build.webkit.org/results/Apple%20iOS%2011%20Simulator%20Debug%20WK2%20(Tests)/r223691%20(483)/results.html

[Ryan Haddad]

Marked test as flaky on iOS in https://trac.webkit.org/r224333

[Matt Lewis]

This test now consistently fails on Debug and has since become flaky on macOS Release.

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fsemantics%2Fdocument-metadata%2Fthe-meta-element%2Fpragma-directives%2Fattr-meta-http-equiv-refresh%2Fparsing.html

however it has a slightly different diff:

--- /Volumes/Data/slave/highsierra-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/document-metadata/the-meta-element/pragma-directives/attr-meta-http-equiv-refresh/parsing-expected.txt
+++ /Volumes/Data/slave/highsierra-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/document-metadata/the-meta-element/pragma-directives/attr-meta-http-equiv-refresh/parsing-actual.txt
@@ -67,7 +67,7 @@
 TIMEOUT <meta>: "1; url foo" Test timed out
 PASS Refresh header: "1; url foo" 
 TIMEOUT <meta>: "1; urlfoo" Test timed out
-PASS Refresh header: "1; urlfoo" 
+TIMEOUT Refresh header: "1; urlfoo" Test timed out
 TIMEOUT <meta>: "1; urfoo" Test timed out
 PASS Refresh header: "1; urfoo" 
 TIMEOUT <meta>: "1; ufoo" Test timed out

Updated Expectations:
https://trac.webkit.org/changeset/225910/webkit