Bug 177322

Summary:	[debug] LayoutTest imported/w3c/web-platform-tests/html/semantics/document-metadata/the-meta-element/pragma-directives/attr-meta-http-equiv-refresh/parsing.html is a flaky failure
Product:	WebKit	Reporter:	Ryan Haddad <ryanhaddad>
Component:	New Bugs	Assignee:	Nobody <webkit-unassigned>
Status:	NEW ---
Severity:	Normal	CC:	jlewis3, youennf
Priority:	P2
Version:	WebKit Nightly Build
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=201694

Description Ryan Haddad 2017-09-21 13:09:41 PDT

LayoutTest imported/w3c/web-platform-tests/html/semantics/document-metadata/the-meta-element/pragma-directives/attr-meta-http-equiv-refresh/parsing.html is a flaky failure

https://build.webkit.org/builders/Apple%20El%20Capitan%20Debug%20WK2%20(Tests)/builds/3205

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fsemantics%2Fdocument-metadata%2Fthe-meta-element%2Fpragma-directives%2Fattr-meta-http-equiv-refresh%2Fparsing.html

--- /Volumes/Data/slave/elcapitan-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/document-metadata/the-meta-element/pragma-directives/attr-meta-http-equiv-refresh/parsing-expected.txt
+++ /Volumes/Data/slave/elcapitan-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/document-metadata/the-meta-element/pragma-directives/attr-meta-http-equiv-refresh/parsing-actual.txt
@@ -85,17 +85,17 @@
 FAIL <meta>: "foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
 PASS Refresh header: "foo" 
 FAIL <meta>: "+1; url=foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
-FAIL Refresh header: "+1; url=foo" assert_equals: expected "refresh.py" but got "foo"
+PASS Refresh header: "+1; url=foo" 
 FAIL <meta>: "-1; url=foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
 PASS Refresh header: "-1; url=foo" 
 FAIL <meta>: "+0; url=foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
 FAIL Refresh header: "+0; url=foo" assert_equals: expected "refresh.py" but got "foo"
 FAIL <meta>: "-0; url=foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
-FAIL Refresh header: "-0; url=foo" assert_equals: expected "refresh.py" but got "foo"
+PASS Refresh header: "-0; url=foo" 
 TIMEOUT <meta>: "0; url=foo" Test timed out
 PASS Refresh header: "0; url=foo" 
 FAIL <meta>: "+1; foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
-FAIL Refresh header: "+1; foo" assert_equals: expected "refresh.py" but got "foo"
+PASS Refresh header: "+1; foo" 
 FAIL <meta>: "-1; foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
 PASS Refresh header: "-1; foo" 
 FAIL <meta>: "+0; foo" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
@@ -105,7 +105,7 @@
 TIMEOUT <meta>: "0; foo" Test timed out
 PASS Refresh header: "0; foo" 
 FAIL <meta>: "+1" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
-FAIL Refresh header: "+1" assert_unreached: Got > 1 load events Reached unreachable code
+PASS Refresh header: "+1" 
 FAIL <meta>: "-1" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"
 PASS Refresh header: "-1" 
 FAIL <meta>: "+0" assert_equals: expected "refresh.sub.html" but got "{\"error\": {\"message\": \"\", \"code\": 404}}"

Comment 1 Ryan Haddad 2017-09-21 13:11:40 PDT

This appears to have started with the latest WPT refresh: https://trac.webkit.org/changeset/222307/webkit

Comment 2 Ryan Haddad 2017-09-21 13:17:22 PDT

Marked as flaky in https://trac.webkit.org/r222341

Comment 3 Matt Lewis 2017-10-19 13:10:10 PDT

This has started to be flaky on iOS as well:
https://build.webkit.org/results/Apple%20iOS%2011%20Simulator%20Debug%20WK2%20(Tests)/r223691%20(483)/results.html

Comment 4 Ryan Haddad 2017-11-02 09:15:45 PDT

Marked test as flaky on iOS in https://trac.webkit.org/r224333

Comment 5 Matt Lewis 2017-12-14 10:31:51 PST

This test now consistently fails on Debug and has since become flaky on macOS Release.

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fsemantics%2Fdocument-metadata%2Fthe-meta-element%2Fpragma-directives%2Fattr-meta-http-equiv-refresh%2Fparsing.html

however it has a slightly different diff:

--- /Volumes/Data/slave/highsierra-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/document-metadata/the-meta-element/pragma-directives/attr-meta-http-equiv-refresh/parsing-expected.txt
+++ /Volumes/Data/slave/highsierra-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/document-metadata/the-meta-element/pragma-directives/attr-meta-http-equiv-refresh/parsing-actual.txt
@@ -67,7 +67,7 @@
 TIMEOUT <meta>: "1; url foo" Test timed out
 PASS Refresh header: "1; url foo" 
 TIMEOUT <meta>: "1; urlfoo" Test timed out
-PASS Refresh header: "1; urlfoo" 
+TIMEOUT Refresh header: "1; urlfoo" Test timed out
 TIMEOUT <meta>: "1; urfoo" Test timed out
 PASS Refresh header: "1; urfoo" 
 TIMEOUT <meta>: "1; ufoo" Test timed out

Updated Expectations:
https://trac.webkit.org/changeset/225910/webkit