Multiple Layout tests from web-platform-tests/beacon/ are timing out on Sierra and iOS 10 Simulators according to the dashboard: imported/w3c/web-platform-tests/beacon/headers/header-referrer-no-referrer-when-downgrade.https.html [ Timeout ] imported/w3c/web-platform-tests/beacon/headers/header-referrer-strict-origin-when-cross-origin.https.html [ Timeout ] imported/w3c/web-platform-tests/beacon/headers/header-referrer-strict-origin.https.html [ Timeout ] imported/w3c/web-platform-tests/beacon/headers/header-referrer-unsafe-url.https.html [ Timeout ] They have been timing out since being added in: https://trac.webkit.org/changeset/220121/webkit https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fbeacon%2Fheaders%2Fheader-referrer-strict-origin-when-cross-origin.https.html History example: https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fbeacon%2Fheaders%2Fheader-referrer-strict-origin-when-cross-origin.https.html Build and results: https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20%28Tests%29/builds/3307 https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r220121%20(3307)/results.html Sample Diff: --- /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/beacon/headers/header-referrer-strict-origin-when-cross-origin.https-expected.txt +++ /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/beacon/headers/header-referrer-strict-origin-when-cross-origin.https-actual.txt @@ -1,5 +1,7 @@ CONSOLE MESSAGE: line 8: Failed to set referrer policy: The value 'strict-origin' is not one of 'no-referrer', 'origin', 'no-referrer-when-downgrade', or 'unsafe-url'. Defaulting to 'no-referrer'. +CONSOLE MESSAGE: Unhandled Promise Rejection: TypeError: Type error +#PID UNRESPONSIVE - com.apple.WebKit.WebContent.Development (pid 85515) +FAIL: Timed out waiting for notifyDone to be called -FAIL Test referer header https://localhost:9443/beacon/resources/ assert_equals: Correct referrer header result expected "https://localhost:9443/" but got "" -PASS Test referer header http://localhost:8800/beacon/resources/ - +#EOF +#EOF
It is interesting that it is only the HTTPS ones. They were timing out for me locally until Youenn's fix in bug 174992.
Temporarily skipped in <http://trac.webkit.org/changeset/220137>.
They pass locally but time out on the bots. The stderr output shows the following error: 22:29:17.380 81095 2017-08-01 22:29:17.352 com.apple.WebKit.Networking.Development[82010:100058563] NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL, -9814) 2
I see DRT / WKTR do: [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"localhost"]; [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"127.0.0.1"]; Maybe we need to whitelist another host of web-platform-tests?
(In reply to Chris Dumez from comment #4) > I see DRT / WKTR do: > [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"localhost"]; > [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"127.0.0.1"]; > > Maybe we need to whitelist another host of web-platform-tests? Nah, looks like WPT uses localhost: Starting web-platform-tests server on <http://localhost:8800/> and <https://localhost:9443/> No handlers could be found for logger "webkitpy.layout_tests.servers.web_platform_test_server"
Interestingly, the tests only seem to time out on WK2, not WK1. I am wondering if this really does the right thing for WKTR: [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"localhost"]; [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"127.0.0.1"]; This is in the injected bundle, therefore in the WebProcess. Will this really impact our NetworkProcess loads?
Hmm, interestingly, the ElCapitan WK2 but seems fine: https://build.webkit.org/results/Apple%20El%20Capitan%20Release%20WK2%20(Tests)/r220121%20(3446)/results.html The Sierra WK2 one experiences the error though.
(In reply to Chris Dumez from comment #7) > Hmm, interestingly, the ElCapitan WK2 but seems fine: > https://build.webkit.org/results/ > Apple%20El%20Capitan%20Release%20WK2%20(Tests)/r220121%20(3446)/results.html > > The Sierra WK2 one experiences the error though. One difference I think between ElCapitan and Sierra is that Sierra is using the newer NETWORK_SESSION code path. So maybe the [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"localhost"] is not working properly with NetworkSession?
(In reply to Chris Dumez from comment #3) > They pass locally but time out on the bots. The stderr output shows the > following error: > 22:29:17.380 81095 2017-08-01 22:29:17.352 > com.apple.WebKit.Networking.Development[82010:100058563] > NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL, > -9814) > 2 This error is: NSOSStatusErrorDomain errSSLCertExpired -9814 So the certificate used by WPT tests seems to have expired?
(In reply to Chris Dumez from comment #9) > (In reply to Chris Dumez from comment #3) > > They pass locally but time out on the bots. The stderr output shows the > > following error: > > 22:29:17.380 81095 2017-08-01 22:29:17.352 > > com.apple.WebKit.Networking.Development[82010:100058563] > > NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL, > > -9814) > > 2 > > This error is: > NSOSStatusErrorDomain errSSLCertExpired -9814 > > So the certificate used by WPT tests seems to have expired? If I launch the wpt server locally: $ Tools/Scripts/run-webkit-httpd and go to https://localhost:9443 in Safari I indeed get a certificate error. However, what I see locally is not that the certificate has expired, but rather that it is signed by an unknown authority.
(In reply to Chris Dumez from comment #10) > (In reply to Chris Dumez from comment #9) > > (In reply to Chris Dumez from comment #3) > > > They pass locally but time out on the bots. The stderr output shows the > > > following error: > > > 22:29:17.380 81095 2017-08-01 22:29:17.352 > > > com.apple.WebKit.Networking.Development[82010:100058563] > > > NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL, > > > -9814) > > > 2 > > > > This error is: > > NSOSStatusErrorDomain errSSLCertExpired -9814 > > > > So the certificate used by WPT tests seems to have expired? > > If I launch the wpt server locally: > $ Tools/Scripts/run-webkit-httpd > > and go to https://localhost:9443 in Safari > > I indeed get a certificate error. However, what I see locally is not that > the certificate has expired, but rather that it is signed by an unknown > authority. I looked at another bot (iOS) and the error was different: NSOSStatusErrorDomain errSSLNoRootCert -9813 So it looks like WKTR is not correctly ignoring invalid certificates for localhost / 127.0.0.1 when using NETWORK_SESSION code path.
Tests skipped on WK2 only via <http://trac.webkit.org/changeset/220149>.
> I am wondering if this really does the right thing for WKTR: > [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"localhost"]; > [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"127.0.0.1"]; This code is for in-process loading (such as media). What should work in the general case is the code in TestController::canAuthenticateAgainstProtectionSpace(). We override server trust evaluation.
(In reply to Alexey Proskuryakov from comment #13) > > I am wondering if this really does the right thing for WKTR: > > [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"localhost"]; > > [NSURLRequest setAllowsAnyHTTPSCertificate:YES forHost:@"127.0.0.1"]; > > This code is for in-process loading (such as media). What should work in the > general case is the code in > TestController::canAuthenticateAgainstProtectionSpace(). We override server > trust evaluation. Thanks for the info. It is odd, I added logging and I see canAuthenticateAgainstProtectionSpace(localhost) getting called. I see us returning true, as expected. However, I then get a: 2017-08-03 09:02:31.055 com.apple.WebKit.Networking.Development[25431:3791340] NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL, -9813)
If I tweak NetworkLoad::didReceiveChallenge() to completionHandler(AuthenticationChallengeDisposition::UseCredential, serverTrustCredential(challenge)); whenever challenge.protectionSpace().authenticationScheme() == ProtectionSpaceAuthenticationSchemeServerTrustEvaluationRequested, then the tests no longer time out.
(In reply to Chris Dumez from comment #15) > If I tweak NetworkLoad::didReceiveChallenge() to > completionHandler(AuthenticationChallengeDisposition::UseCredential, > serverTrustCredential(challenge)); whenever > challenge.protectionSpace().authenticationScheme() == > ProtectionSpaceAuthenticationSchemeServerTrustEvaluationRequested, then the > tests no longer time out. The problem seems to be happen when NetworkLoad::completeAuthenticationChallenge() is called and m_parameters.clientCredentialPolicy == ClientCredentialPolicy::CannotAskClientForCredentials. In such case, we call: completionHandler(AuthenticationChallengeDisposition::UseCredential, { }); If I replace with: completionHandler(AuthenticationChallengeDisposition::UseCredential, serverTrustCredential(*m_challenge)); Then the tests no longer time out.
(In reply to Chris Dumez from comment #16) > (In reply to Chris Dumez from comment #15) > > If I tweak NetworkLoad::didReceiveChallenge() to > > completionHandler(AuthenticationChallengeDisposition::UseCredential, > > serverTrustCredential(challenge)); whenever > > challenge.protectionSpace().authenticationScheme() == > > ProtectionSpaceAuthenticationSchemeServerTrustEvaluationRequested, then the > > tests no longer time out. > > The problem seems to be happen when > NetworkLoad::completeAuthenticationChallenge() is called and > m_parameters.clientCredentialPolicy == > ClientCredentialPolicy::CannotAskClientForCredentials. > > In such case, we call: > completionHandler(AuthenticationChallengeDisposition::UseCredential, { }); > > If I replace with: > completionHandler(AuthenticationChallengeDisposition::UseCredential, > serverTrustCredential(*m_challenge)); > > Then the tests no longer time out. So the problem is that some requests like CORS preflight use CannotAskClientForCredentials policy. As a result, the WebKitTestRunner is not asked about server trust evaluation for those and we end up doing the default behavior (which fails because certificate is not trusted).
Created attachment 317133 [details] Patch
<rdar://problem/33704752>
Comment on attachment 317133 [details] Patch Amazing. Please let EWS settle down before landing.
Comment on attachment 317133 [details] Patch Attachment 317133 [details] did not pass ios-sim-ews (ios-simulator-wk2): Output: http://webkit-queues.webkit.org/results/4248260 New failing tests: imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.worker.html imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.html imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.worker.html imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors-worker.html imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.worker.html imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors.html imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.html imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.html
Created attachment 317153 [details] Archive of layout-test-results from ews125 for ios-simulator-wk2 The attached test failures were seen while running run-webkit-tests on the ios-sim-ews. Bot: ews125 Port: ios-simulator-wk2 Platform: Mac OS X 10.12.5
(In reply to Build Bot from comment #21) > Comment on attachment 317133 [details] > Patch > > Attachment 317133 [details] did not pass ios-sim-ews (ios-simulator-wk2): > Output: http://webkit-queues.webkit.org/results/4248260 > > New failing tests: > imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.worker. > html > imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.html > imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.worker.html > imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors-worker.html > imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.worker.html > imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors.html > imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.html > imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.html This patch fixes issues with all these tests that now need to be rebased.
(In reply to youenn fablet from comment #23) > (In reply to Build Bot from comment #21) > > Comment on attachment 317133 [details] > > Patch > > > > Attachment 317133 [details] did not pass ios-sim-ews (ios-simulator-wk2): > > Output: http://webkit-queues.webkit.org/results/4248260 > > > > New failing tests: > > imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.worker. > > html > > imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.html > > imported/w3c/web-platform-tests/fetch/api/cors/cors-basic.any.worker.html > > imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors-worker.html > > imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.worker.html > > imported/w3c/web-platform-tests/fetch/api/basic/mode-no-cors.html > > imported/w3c/web-platform-tests/fetch/api/cors/cors-origin.any.html > > imported/w3c/web-platform-tests/fetch/api/cors/cors-no-preflight.any.html > > This patch fixes issues with all these tests that now need to be rebased. Will rebaseline them and unsnap them on Mac.
Created attachment 317162 [details] Patch
Created attachment 317164 [details] Patch
Created attachment 317176 [details] Patch
Comment on attachment 317176 [details] Patch Clearing flags on attachment: 317176 Committed r220245: <http://trac.webkit.org/changeset/220245>
All reviewed patches have been landed. Closing bug.