Bug 175061 – [WK2][ElCapitan] Multiple imported/w3c/web-platform-tests/fetch/api/ are failing

• Status: RESOLVED
• Resolution: CONFIGURATION CHANGED
• Component: New Bugs

[Matt Lewis]

Multiple imported/w3c/web-platform-tests/fetch/api/ test have begun to fail consistently:
  imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors-worker.html [ Failure ]
  imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors.html [ Failure ]
  imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.html [ Failure ]
  imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.worker.html [ Failure ]
  imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.html [ Failure ]
  imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.worker.html [ Failure ]
  imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.html [ Failure ]
  imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.worker.html [ Failure ]


Build:
https://build.webkit.org/builders/Apple%20El%20Capitan%20Release%20WK2%20(Tests)/builds/3439
https://build.webkit.org/results/Apple%20El%20Capitan%20Release%20WK2%20(Tests)/r220113%20(3439)/results.html

With each test, what looked to be an expected failure to fetch an api looks now to pass.
Sample diff:
--- /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors-expected.txt
+++ /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors-actual.txt
@@ -1,6 +1,6 @@
 
 PASS Fetch ../resources/top.txt with no-cors mode 
 PASS Fetch http://localhost:8800/fetch/api/resources/top.txt with no-cors mode 
-FAIL Fetch https://localhost:9443/fetch/api/resources/top.txt with no-cors mode promise_test: Unhandled rejection with value: object "TypeError: Type error"
+PASS Fetch https://localhost:9443/fetch/api/resources/top.txt with no-cors mode 
 PASS Fetch http://localhost:8801/fetch/api/resources/top.txt with no-cors mode 


Could these be a progression needing a rebaseline?

[Matt Lewis]

Talked with Youenn, added in expectations for Mac El Capitan for now.

https://trac.webkit.org/changeset/220116/webkit

[Matt Lewis]

Adjusted expectation location as it cause failures on WK1:
https://trac.webkit.org/changeset/220134/webkit

[Matt Lewis]

After the new expectations, this cause unrelated failures to EWS.
Talked with Youenn, this is most likely a bigger issue than with these tests.

Removed the expectations and marked the tests as flaky in
https://trac.webkit.org/changeset/220164/webkit

[Chris Dumez]

With the patch for Bug 175076 applied, it looks like those tests are passing on:
- mac-wk1 EWS (mavericks)
- mac-wk1 locally (high sierra)
- mac-wk2 locally (high sierra)
- iOS-wk2 EWS

but still failing on:
- mac-wk2 EWS (mavericks?)

[Matt Lewis]

Mac-wk2 should be El Capitan

[Chris Dumez]

Created attachment 317248 [details]
Patch

[Chris Dumez]

Committed r220276: <http://trac.webkit.org/changeset/220276>

[Chris Dumez]

Reopening since the patch merely updated test expectation. The tests are still failing on Mac ElCapitan WK2.

[Chris Dumez]

(In reply to Chris Dumez from comment #8)
> Reopening since the patch merely updated test expectation. The tests are
> still failing on Mac ElCapitan WK2.

Looks like all bots are green after those more-restrictive test expectations. It looks like the tests are indeed only failing on Mac ElCapitan WK2 which could be indicative of a behavior difference in our legacy WK2/non NETWORK_SESSION code path.

[Chris Dumez]

Failures:
--- /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any-expected.txt
+++ /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any-actual.txt
@@ -6,8 +6,6 @@
 CONSOLE MESSAGE: Fetch API cannot load http://127.0.0.1:8800/fetch/api/resources/top.txt. Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
 CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
 CONSOLE MESSAGE: Fetch API cannot load http://127.0.0.1:8801/fetch/api/resources/top.txt. Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
-CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
-CONSOLE MESSAGE: Fetch API cannot load https://127.0.0.1:9443/fetch/api/resources/top.txt. Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
 
 PASS Same domain different port [no-cors mode] 
 PASS Same domain different port [server forbid CORS] 
@@ -21,7 +19,7 @@
 PASS Cross domain different port [no-cors mode] 
 PASS Cross domain different port [server forbid CORS] 
 PASS Cross domain different port [cors mode] 
-PASS Cross domain different protocol [no-cors mode] 
+FAIL Cross domain different protocol [no-cors mode] promise_test: Unhandled rejection with value: object "TypeError: Type error"
 PASS Cross domain different protocol [server forbid CORS] 
-PASS Cross domain different protocol [cors mode] 
+FAIL Cross domain different protocol [cors mode] promise_test: Unhandled rejection with value: object "TypeError: Type error"

--- /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.worker-expected.txt
+++ /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.worker-actual.txt
@@ -1,4 +1,3 @@
-CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
 CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
 CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
 CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
@@ -16,7 +15,7 @@
 PASS Cross domain different port [no-cors mode] 
 PASS Cross domain different port [server forbid CORS] 
 PASS Cross domain different port [cors mode] 
-PASS Cross domain different protocol [no-cors mode] 
+FAIL Cross domain different protocol [no-cors mode] promise_test: Unhandled rejection with value: object "TypeError: Type error"
 PASS Cross domain different protocol [server forbid CORS] 
-PASS Cross domain different protocol [cors mode] 
+FAIL Cross domain different protocol [cors mode] promise_test: Unhandled rejection with value: object "TypeError: Type error"

--- /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any-expected.txt
+++ /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any-actual.txt
@@ -2,7 +2,7 @@
 PASS Cross domain basic usage [GET] 
 PASS Same domain different port [GET] 
 PASS Cross domain different port [GET] 
-PASS Cross domain different protocol [GET] 
+FAIL Cross domain different protocol [GET] promise_test: Unhandled rejection with value: object "TypeError: Type error"
 PASS Same domain different protocol different port [GET] 
 PASS Cross domain [POST] 
 PASS Cross domain [HEAD] 

--- /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.worker-expected.txt
+++ /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.worker-actual.txt
@@ -2,7 +2,7 @@
 PASS Cross domain basic usage [GET] 
 PASS Same domain different port [GET] 
 PASS Cross domain different port [GET] 
-PASS Cross domain different protocol [GET] 
+FAIL Cross domain different protocol [GET] promise_test: Unhandled rejection with value: object "TypeError: Type error"
 PASS Same domain different protocol different port [GET] 
 PASS Cross domain [POST] 
 PASS Cross domain [HEAD] 

--- /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any-expected.txt
+++ /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any-actual.txt
@@ -1,3 +1,18 @@
+CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Fetch API cannot load http://127.0.0.1:8800/fetch/api/resources/preflight.py?token=265b525d-acb4-4739-8aa1-a8a3836cc57b&max_age=0&origin=http%3A%2F%2F127.0.0.1%3A8800&allow_methods=GET. Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Fetch API cannot load http://localhost:8801/fetch/api/resources/preflight.py?token=56c2bcca-7020-41e5-a72c-c7c5fae6bd17&max_age=0&origin=http%3A%2F%2Flocalhost%3A8801&allow_methods=GET. Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Fetch API cannot load http://127.0.0.1:8801/fetch/api/resources/preflight.py?token=45c16c29-33de-4ebf-8f02-194886a759c9&max_age=0&origin=http%3A%2F%2F127.0.0.1%3A8801&allow_methods=GET. Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Fetch API cannot load https://localhost:9443/fetch/api/resources/preflight.py?token=ce2472db-ea82-4023-8d90-fa0ebb0494b2&max_age=0&origin=https%3A%2F%2Flocalhost%3A9443&allow_methods=GET. Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Fetch API cannot load http://127.0.0.1:8800/fetch/api/resources/preflight.py?token=8d964b53-9dbc-4cd8-bdc9-16c4218d88f3&max_age=0&origin=http%3A%2F%2F127.0.0.1%3A8800&allow_methods=POST. Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Fetch API cannot load http://127.0.0.1:8800/fetch/api/resources/preflight.py?token=9570d8e7-b8d3-4132-be35-d310df97e50e&max_age=0&origin=http%3A%2F%2F127.0.0.1%3A8800&allow_methods=HEAD. Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Fetch API cannot load http://127.0.0.1:8800/fetch/api/resources/preflight.py?token=1e8a0d66-48c6-4938-917b-81618eb06380&max_age=0&origin=http%3A%2F%2F127.0.0.1%3A8800&allow_methods=PUT. Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
+CONSOLE MESSAGE: Fetch API cannot load http://127.0.0.1:8800/fetch/api/resources/preflight.py?token=06e613bd-f021-457e-ae9f-54e41854ca4c&max_age=0&origin=http%3A%2F%2F127.0.0.1%3A8800&allow_methods=GET. Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
 
 PASS Cross domain different subdomain [origin OK] 
 PASS Cross domain different subdomain [origin KO] 
@@ -5,7 +20,7 @@
 PASS Same domain different port [origin KO] 
 PASS Cross domain different port [origin OK] 
 PASS Cross domain different port [origin KO] 
-PASS Cross domain different protocol [origin OK] 
+FAIL Cross domain different protocol [origin OK] promise_test: Unhandled rejection with value: object "TypeError: Type error"
 PASS Cross domain different protocol [origin KO] 
 PASS Same domain different protocol different port [origin OK] 
 PASS Same domain different protocol different port [origin KO] 

--- /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.worker-expected.txt
+++ /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.worker-actual.txt
@@ -1,4 +1,3 @@
-CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
 CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
 CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
 CONSOLE MESSAGE: Origin http://localhost:8800 is not allowed by Access-Control-Allow-Origin.
@@ -13,7 +12,7 @@
 PASS Same domain different port [origin KO] 
 PASS Cross domain different port [origin OK] 
 PASS Cross domain different port [origin KO] 
-PASS Cross domain different protocol [origin OK] 
+FAIL Cross domain different protocol [origin OK] promise_test: Unhandled rejection with value: object "TypeError: Type error"
 PASS Cross domain different protocol [origin KO] 
 PASS Same domain different protocol different port [origin OK] 
 PASS Same domain different protocol different port [origin KO]

[Chris Dumez]

It may be a good idea to go though the "return Exception { TypeError };" statements in Fetch code and pass a second parameter with a useful exception message. It would tell us why those tests are failing. Currently, the message is the generic "type error" which is not useful.

[Joseph Pecoraro]

Closing. El Capitan expectations have been removed and this was an El Capitan specific failure / expectation.

[Radar WebKit Bug Importer]

<rdar://problem/43118308>