Bug 198676 – Import Content Security Policy Web Platform Tests

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Daniel Bates]

Import Content Security Policy Web Platform Tests.

[Radar WebKit Bug Importer]

<rdar://problem/51533785>

[Daniel Bates]

Created attachment 371616 [details]
A bunch of tests

[youenn fablet]

Comment on attachment 371616 [details]
A bunch of tests

r=me once bots are happy.
Please skip tests that are timing out, at least the ones that have no PASS.

It would be really good to remove the www1/www2 wherever possible.
Instead, you can rely on hosts[alt].
This can be done as a follow-up and should be upstreamed in WPT.

View in context: https://bugs.webkit.org/attachment.cgi?id=371616&action=review

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/base-uri/base-uri-deny.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to change the document base URL to http://www2.localhost:8800/ because it does not appear in the base-uri directive of the Content Security Policy.

Usually, we prefer to use non www1/www2 URLS if possible.
If we can, it would be nice to update the test upstream to use hosts[alt], or different port or protocol.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/base-uri/base-uri_iframe_sandbox.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to change the document base URL to http://www2.localhost:8800/base/ because it does not appear in the base-uri directive of the Content Security Policy.

Ditto.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/base-uri/report-uri-does-not-respect-base-uri.sub-expected.txt:8
> +TIMEOUT Event is fired Test timed out

Test is timing out, we may want to skip it.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/blob/blob-urls-do-not-match-self.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to load blob:http://localhost:8800/d820d6f6-f018-4285-b815-0c92b963e921 because it does not appear in the script-src directive of the Content Security Policy.

Test will probably be flaky, use DumpJSConsoleLogInStdErr if needed.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/blob/self-doesnt-match-blob.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to load blob:http://localhost:8800/f7ffe13f-be1a-482e-b2ab-ccc3114b5110 because it does not appear in the child-src directive of the Content Security Policy.

Will probably be flaky. You can use DumpJSConsoleLogInStdErr.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/blob/star-doesnt-match-blob.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to load blob:http://localhost:8800/9636972c-adc4-43d0-a730-fa5498d6bb99 because it does not appear in the child-src directive of the Content Security Policy.

Ditto.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/child-src/child-src-cross-origin-load.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to load http://www2.localhost:8800/content-security-policy/support/postmessage-fail.html because it does not appear in the child-src directive of the Content Security Policy.

www2 issue here as well.
It might be that the test is timing out due to that.
Might be best to skip it and fix it as a follow-up.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/connect-src/connect-src-beacon-blocked.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to connect to http://www1.localhost:8800/security/contentSecurityPolicy/echo-report.php because it does not appear in the connect-src directive of the Content Security Policy.

www1 potential issue, here and below.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/securitypolicyviolation/img-src-redirect-upgrade-reporting.https-expected.txt:6
> +TIMEOUT Image that redirects to http:// URL prohibited by Report-Only must generate a violation report, even with upgrade-insecure-requests Test timed out

Please skip this one and others that are timing out.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/securitypolicyviolation/inside-dedicated-worker-expected.txt:8
> +TIMEOUT SecurityPolicyViolation event fired on global with the correct blockedURI. Test timed out

Ditto here.

[EWS Watchlist]

Comment on attachment 371616 [details]
A bunch of tests

Attachment 371616 [details] did not pass mac-ews (mac):
Output: https://webkit-queues.webkit.org/results/12413055

Number of test failures exceeded the failure limit.

[EWS Watchlist]

Created attachment 371621 [details]
Archive of layout-test-results from ews100 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews100  Port: mac-highsierra  Platform: Mac OS X 10.13.6

[Daniel Bates]

Created attachment 371625 [details]
To land

[EWS Watchlist]

Comment on attachment 371625 [details]
To land

Attachment 371625 [details] did not pass mac-wk2-ews (mac-wk2):
Output: https://webkit-queues.webkit.org/results/12414054

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes-inherit-self.sub.html
imported/w3c/web-platform-tests/content-security-policy/svg/scripted.svg
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes.sub.html
imported/w3c/web-platform-tests/content-security-policy/script-src/script-src-multiple-policies-one-using-hashing-algorithms.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/unsandboxed-blob-scheme.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/sandboxed-blob-scheme.html

[EWS Watchlist]

Created attachment 371635 [details]
Archive of layout-test-results from ews105 for mac-highsierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews105  Port: mac-highsierra-wk2  Platform: Mac OS X 10.13.6

[EWS Watchlist]

Comment on attachment 371625 [details]
To land

Attachment 371625 [details] did not pass mac-ews (mac):
Output: https://webkit-queues.webkit.org/results/12414191

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-blocked.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes-inherit-self.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-self.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/svg/scripted.svg
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes.sub.html
imported/w3c/web-platform-tests/content-security-policy/script-src/script-src-multiple-policies-one-using-hashing-algorithms.html
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-allowed.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-child.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-list.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-none.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/unsandboxed-blob-scheme.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/sandboxed-blob-scheme.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-fallback.https.sub.html

[EWS Watchlist]

Created attachment 371637 [details]
Archive of layout-test-results from ews103 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-highsierra  Platform: Mac OS X 10.13.6

[EWS Watchlist]

Comment on attachment 371625 [details]
To land

Attachment 371625 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: https://webkit-queues.webkit.org/results/12414020

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes-inherit-self.sub.html
imported/w3c/web-platform-tests/content-security-policy/svg/scripted.svg
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes.sub.html
imported/w3c/web-platform-tests/content-security-policy/script-src/script-src-multiple-policies-one-using-hashing-algorithms.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/unsandboxed-blob-scheme.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/sandboxed-blob-scheme.html

[EWS Watchlist]

Created attachment 371639 [details]
Archive of layout-test-results from ews126 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews126  Port: ios-simulator-wk2  Platform: Mac OS X 10.14.5

[EWS Watchlist]

Comment on attachment 371625 [details]
To land

Attachment 371625 [details] did not pass mac-debug-ews (mac):
Output: https://webkit-queues.webkit.org/results/12414044

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-blocked.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes-inherit-self.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-self.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/svg/scripted.svg
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes.sub.html
imported/w3c/web-platform-tests/content-security-policy/script-src/script-src-multiple-policies-one-using-hashing-algorithms.html
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-allowed.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-child.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-list.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-none.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/unsandboxed-blob-scheme.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/sandboxed-blob-scheme.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-fallback.https.sub.html

[EWS Watchlist]

Created attachment 371640 [details]
Archive of layout-test-results from ews113 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews113  Port: mac-highsierra  Platform: Mac OS X 10.13.6

[Daniel Bates]

Created attachment 371793 [details]
Patch

[EWS Watchlist]

Comment on attachment 371793 [details]
Patch

Attachment 371793 [details] did not pass mac-ews (mac):
Output: https://webkit-queues.webkit.org/results/12439217

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-blocked.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-self.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-allowed.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-child.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-list.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-none.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-fallback.https.sub.html

[EWS Watchlist]

Created attachment 371802 [details]
Archive of layout-test-results from ews101 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews101  Port: mac-highsierra  Platform: Mac OS X 10.13.6

[EWS Watchlist]

Comment on attachment 371793 [details]
Patch

Attachment 371793 [details] did not pass mac-debug-ews (mac):
Output: https://webkit-queues.webkit.org/results/12440053

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-blocked.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-self.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-allowed.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-child.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-list.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-none.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-fallback.https.sub.html

[EWS Watchlist]

Created attachment 371817 [details]
Archive of layout-test-results from ews117 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews117  Port: mac-highsierra  Platform: Mac OS X 10.13.6

[Daniel Bates]

Created attachment 371855 [details]
Patch

[EWS Watchlist]

Comment on attachment 371855 [details]
Patch

Attachment 371855 [details] did not pass mac-ews (mac):
Output: https://webkit-queues.webkit.org/results/12445791

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg

[EWS Watchlist]

Created attachment 371863 [details]
Archive of layout-test-results from ews103 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-highsierra  Platform: Mac OS X 10.13.6

[EWS Watchlist]

Comment on attachment 371855 [details]
Patch

Attachment 371855 [details] did not pass mac-debug-ews (mac):
Output: https://webkit-queues.webkit.org/results/12445755

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg

[EWS Watchlist]

Created attachment 371865 [details]
Archive of layout-test-results from ews115 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews115  Port: mac-highsierra  Platform: Mac OS X 10.13.6

[Daniel Bates]

Created attachment 371868 [details]
To land

[Daniel Bates]

Comment on attachment 371868 [details]
To land

Clearing flags on attachment: 371868

Committed r246330: <https://trac.webkit.org/changeset/246330>

[Daniel Bates]

All reviewed patches have been landed.  Closing bug.

[Russell Epstein]

It looks like the tests imported in https://trac.webkit.org/changeset/246330/webkit

Have 1 flakey failure.

imported/w3c/web-platform-tests/content-security-policy/reporting/report-only-in-meta.sub.html

History 
https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-security-policy%2Freporting%2Freport-only-in-meta.sub.html

Diff
--- /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting/report-only-in-meta.sub-expected.txt
+++ /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting/report-only-in-meta.sub-actual.txt
@@ -1,4 +1,5 @@
 
 
 PASS Image should load 
+PASS Violation report status OK.

[Daniel Bates]

(In reply to Russell Epstein from comment #28)
> It looks like the tests imported in
> https://trac.webkit.org/changeset/246330/webkit
> 
> Have 1 flakey failure.
> 
> imported/w3c/web-platform-tests/content-security-policy/reporting/report-
> only-in-meta.sub.html
> 
> History 
> https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.
> html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-
> security-policy%2Freporting%2Freport-only-in-meta.sub.html
> 
> Diff
> ---
> /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/
> imported/w3c/web-platform-tests/content-security-policy/reporting/report-
> only-in-meta.sub-expected.txt
> +++
> /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/
> imported/w3c/web-platform-tests/content-security-policy/reporting/report-
> only-in-meta.sub-actual.txt
> @@ -1,4 +1,5 @@
>  
>  
>  PASS Image should load 
> +PASS Violation report status OK.

I'll take a look tomorrow. There are some known flakiness in the test set. This import was a first pass. So, in my opinion, I would just skip the test for now if that is being weighed with a rollout.

[Russell Epstein]

Marked test as flakey in https://bugs.webkit.org/show_bug.cgi?id=198977