Bug 175076 – Multiple Layout tests from web-platform-tests/beacon/ are timing out.

• Component: New Bugs
• Status: RESOLVED FIXED

[Matt Lewis]

Multiple Layout tests from web-platform-tests/beacon/ are timing out on Sierra and iOS 10 Simulators according to the dashboard:
  imported/w3c/web-platform-tests/beacon/headers/header-referrer-no-referrer-when-downgrade.https.html [ Timeout ]
  imported/w3c/web-platform-tests/beacon/headers/header-referrer-strict-origin-when-cross-origin.https.html [ Timeout ]
  imported/w3c/web-platform-tests/beacon/headers/header-referrer-strict-origin.https.html [ Timeout ]
  imported/w3c/web-platform-tests/beacon/headers/header-referrer-unsafe-url.https.html [ Timeout ]

They have been timing out since being added in: https://trac.webkit.org/changeset/220121/webkit

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fbeacon%2Fheaders%2Fheader-referrer-strict-origin-when-cross-origin.https.html

History example:
https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fbeacon%2Fheaders%2Fheader-referrer-strict-origin-when-cross-origin.https.html

Build and results:
https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20%28Tests%29/builds/3307
https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r220121%20(3307)/results.html

Sample Diff:
--- /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/beacon/headers/header-referrer-strict-origin-when-cross-origin.https-expected.txt
+++ /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/beacon/headers/header-referrer-strict-origin-when-cross-origin.https-actual.txt
@@ -1,5 +1,7 @@
 CONSOLE MESSAGE: line 8: Failed to set referrer policy: The value 'strict-origin' is not one of 'no-referrer', 'origin', 'no-referrer-when-downgrade', or 'unsafe-url'. Defaulting to 'no-referrer'.
+CONSOLE MESSAGE: Unhandled Promise Rejection: TypeError: Type error
+#PID UNRESPONSIVE - com.apple.WebKit.WebContent.Development (pid 85515)
+FAIL: Timed out waiting for notifyDone to be called
 
-FAIL Test referer header https://localhost:9443/beacon/resources/ assert_equals: Correct referrer header result expected "https://localhost:9443/" but got ""
-PASS Test referer header http://localhost:8800/beacon/resources/ 
-
+#EOF
+#EOF

[Chris Dumez]

It is interesting that it is only the HTTPS ones. They were timing out for me locally until Youenn's fix in bug 174992.

[Chris Dumez]

Temporarily skipped in <http://trac.webkit.org/changeset/220137>.

[Chris Dumez]

They pass locally but time out on the bots. The stderr output shows the following error:
22:29:17.380 81095   2017-08-01 22:29:17.352 com.apple.WebKit.Networking.Development[82010:100058563] NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL, -9814)
2

[Chris Dumez]

I see DRT / WKTR do:
    [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"localhost"];
    [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"127.0.0.1"];

Maybe we need to whitelist another host of web-platform-tests?

[Chris Dumez]

(In reply to Chris Dumez from comment #4)
> I see DRT / WKTR do:
>     [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"localhost"];
>     [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"127.0.0.1"];
> 
> Maybe we need to whitelist another host of web-platform-tests?

Nah, looks like WPT uses localhost:
Starting web-platform-tests server on <http://localhost:8800/> and <https://localhost:9443/>
No handlers could be found for logger "webkitpy.layout_tests.servers.web_platform_test_server"

[Chris Dumez]

Interestingly, the tests only seem to time out on WK2, not WK1.

I am wondering if this really does the right thing for WKTR:
    [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"localhost"];
    [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"127.0.0.1"];

This is in the injected bundle, therefore in the WebProcess. Will this really impact our NetworkProcess loads?

[Chris Dumez]

Hmm, interestingly, the ElCapitan WK2 but seems fine:
https://build.webkit.org/results/Apple%20El%20Capitan%20Release%20WK2%20(Tests)/r220121%20(3446)/results.html

The Sierra WK2 one experiences the error though.

[Chris Dumez]

(In reply to Chris Dumez from comment #7)
> Hmm, interestingly, the ElCapitan WK2 but seems fine:
> https://build.webkit.org/results/
> Apple%20El%20Capitan%20Release%20WK2%20(Tests)/r220121%20(3446)/results.html
> 
> The Sierra WK2 one experiences the error though.

One difference I think between ElCapitan and Sierra is that Sierra is using the newer NETWORK_SESSION code path. So maybe the [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"localhost"] is not working properly with NetworkSession?

[Chris Dumez]

(In reply to Chris Dumez from comment #3)
> They pass locally but time out on the bots. The stderr output shows the
> following error:
> 22:29:17.380 81095   2017-08-01 22:29:17.352
> com.apple.WebKit.Networking.Development[82010:100058563]
> NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL,
> -9814)
> 2

This error is:
NSOSStatusErrorDomain errSSLCertExpired -9814

So the certificate used by WPT tests seems to have expired?

[Chris Dumez]

(In reply to Chris Dumez from comment #9)
> (In reply to Chris Dumez from comment #3)
> > They pass locally but time out on the bots. The stderr output shows the
> > following error:
> > 22:29:17.380 81095   2017-08-01 22:29:17.352
> > com.apple.WebKit.Networking.Development[82010:100058563]
> > NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL,
> > -9814)
> > 2
> 
> This error is:
> NSOSStatusErrorDomain errSSLCertExpired -9814
> 
> So the certificate used by WPT tests seems to have expired?

If I launch the wpt server locally:
$ Tools/Scripts/run-webkit-httpd

and go to https://localhost:9443 in Safari

I indeed get a certificate error. However, what I see locally is not that the certificate has expired, but rather that it is signed by an unknown authority.

[Chris Dumez]

(In reply to Chris Dumez from comment #10)
> (In reply to Chris Dumez from comment #9)
> > (In reply to Chris Dumez from comment #3)
> > > They pass locally but time out on the bots. The stderr output shows the
> > > following error:
> > > 22:29:17.380 81095   2017-08-01 22:29:17.352
> > > com.apple.WebKit.Networking.Development[82010:100058563]
> > > NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL,
> > > -9814)
> > > 2
> > 
> > This error is:
> > NSOSStatusErrorDomain errSSLCertExpired -9814
> > 
> > So the certificate used by WPT tests seems to have expired?
> 
> If I launch the wpt server locally:
> $ Tools/Scripts/run-webkit-httpd
> 
> and go to https://localhost:9443 in Safari
> 
> I indeed get a certificate error. However, what I see locally is not that
> the certificate has expired, but rather that it is signed by an unknown
> authority.

I looked at another bot (iOS) and the error was different:
NSOSStatusErrorDomain errSSLNoRootCert -9813

So it looks like WKTR is not correctly ignoring invalid certificates for localhost / 127.0.0.1 when using NETWORK_SESSION code path.

[Chris Dumez]

Tests skipped on WK2 only via <http://trac.webkit.org/changeset/220149>.

[Alexey Proskuryakov]

> I am wondering if this really does the right thing for WKTR:
>    [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"localhost"];
>    [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"127.0.0.1"];

This code is for in-process loading (such as media). What should work in the general case is the code in TestController::canAuthenticateAgainstProtectionSpace(). We override server trust evaluation.

[Chris Dumez]

(In reply to Alexey Proskuryakov from comment #13)
> > I am wondering if this really does the right thing for WKTR:
> >    [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"localhost"];
> >    [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"127.0.0.1"];
> 
> This code is for in-process loading (such as media). What should work in the
> general case is the code in
> TestController::canAuthenticateAgainstProtectionSpace(). We override server
> trust evaluation.

Thanks for the info. It is odd, I added logging and I see canAuthenticateAgainstProtectionSpace(localhost) getting called. I see us returning true, as expected. However, I then get a:
2017-08-03 09:02:31.055 com.apple.WebKit.Networking.Development[25431:3791340] NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL, -9813)

[Chris Dumez]

If I tweak NetworkLoad::didReceiveChallenge() to completionHandler(AuthenticationChallengeDisposition::UseCredential, serverTrustCredential(challenge)); whenever challenge.protectionSpace().authenticationScheme() == ProtectionSpaceAuthenticationSchemeServerTrustEvaluationRequested, then the tests no longer time out.

[Chris Dumez]

(In reply to Chris Dumez from comment #15)
> If I tweak NetworkLoad::didReceiveChallenge() to
> completionHandler(AuthenticationChallengeDisposition::UseCredential,
> serverTrustCredential(challenge)); whenever
> challenge.protectionSpace().authenticationScheme() ==
> ProtectionSpaceAuthenticationSchemeServerTrustEvaluationRequested, then the
> tests no longer time out.

The problem seems to be happen when NetworkLoad::completeAuthenticationChallenge() is called and m_parameters.clientCredentialPolicy == ClientCredentialPolicy::CannotAskClientForCredentials.

In such case, we call:
completionHandler(AuthenticationChallengeDisposition::UseCredential, { });

If I replace with:
completionHandler(AuthenticationChallengeDisposition::UseCredential, serverTrustCredential(*m_challenge));

Then the tests no longer time out.

[Chris Dumez]

(In reply to Chris Dumez from comment #16)
> (In reply to Chris Dumez from comment #15)
> > If I tweak NetworkLoad::didReceiveChallenge() to
> > completionHandler(AuthenticationChallengeDisposition::UseCredential,
> > serverTrustCredential(challenge)); whenever
> > challenge.protectionSpace().authenticationScheme() ==
> > ProtectionSpaceAuthenticationSchemeServerTrustEvaluationRequested, then the
> > tests no longer time out.
> 
> The problem seems to be happen when
> NetworkLoad::completeAuthenticationChallenge() is called and
> m_parameters.clientCredentialPolicy ==
> ClientCredentialPolicy::CannotAskClientForCredentials.
> 
> In such case, we call:
> completionHandler(AuthenticationChallengeDisposition::UseCredential, { });
> 
> If I replace with:
> completionHandler(AuthenticationChallengeDisposition::UseCredential,
> serverTrustCredential(*m_challenge));
> 
> Then the tests no longer time out.

So the problem is that some requests like CORS preflight use CannotAskClientForCredentials policy. As a result, the WebKitTestRunner is not asked about server trust evaluation for those and we end up doing the default behavior (which fails because certificate is not trusted).

[Chris Dumez]

Created attachment 317133 [details]
Patch

[Radar WebKit Bug Importer]

<rdar://problem/33704752>

[Alexey Proskuryakov]

Comment on attachment 317133 [details]
Patch

Amazing.

Please let EWS settle down before landing.

[Build Bot]

Comment on attachment 317133 [details]
Patch

Attachment 317133 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/4248260

New failing tests:
imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.worker.html
imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.html
imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.worker.html
imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors-worker.html
imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.worker.html
imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors.html
imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.html
imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.html

[Build Bot]

Created attachment 317153 [details]
Archive of layout-test-results from ews125 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews125  Port: ios-simulator-wk2  Platform: Mac OS X 10.12.5

[youenn fablet]

(In reply to Build Bot from comment #21)
> Comment on attachment 317133 [details]
> Patch
> 
> Attachment 317133 [details] did not pass ios-sim-ews (ios-simulator-wk2):
> Output: http://webkit-queues.webkit.org/results/4248260
> 
> New failing tests:
> imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.worker.
> html
> imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.html
> imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.worker.html
> imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors-worker.html
> imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.worker.html
> imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors.html
> imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.html
> imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.html

This patch fixes issues with all these tests that now need to be rebased.

[Chris Dumez]

(In reply to youenn fablet from comment #23)
> (In reply to Build Bot from comment #21)
> > Comment on attachment 317133 [details]
> > Patch
> > 
> > Attachment 317133 [details] did not pass ios-sim-ews (ios-simulator-wk2):
> > Output: http://webkit-queues.webkit.org/results/4248260
> > 
> > New failing tests:
> > imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.worker.
> > html
> > imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.html
> > imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.worker.html
> > imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors-worker.html
> > imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.worker.html
> > imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors.html
> > imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.html
> > imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.html
> 
> This patch fixes issues with all these tests that now need to be rebased.

Will rebaseline them and unsnap them on Mac.

[Chris Dumez]

Created attachment 317162 [details]
Patch

[Chris Dumez]

Created attachment 317164 [details]
Patch

[Chris Dumez]

Created attachment 317176 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 317176 [details]
Patch

Clearing flags on attachment: 317176

Committed r220245: <http://trac.webkit.org/changeset/220245>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.