Bug 198676 - Import Content Security Policy Web Platform Tests
Summary: Import Content Security Policy Web Platform Tests
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Local Build
Hardware: All All
: P2 Normal
Assignee: Daniel Bates
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2019-06-07 13:44 PDT by Daniel Bates
Modified: 2019-06-18 15:38 PDT (History)
6 users (show)

See Also:


Attachments
A bunch of tests (1.58 MB, patch)
2019-06-07 15:11 PDT, Daniel Bates
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews100 for mac-highsierra (911.40 KB, application/zip)
2019-06-07 16:08 PDT, Build Bot
no flags Details
To land (1.56 MB, patch)
2019-06-07 16:14 PDT, Daniel Bates
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews105 for mac-highsierra-wk2 (3.46 MB, application/zip)
2019-06-07 17:41 PDT, Build Bot
no flags Details
Archive of layout-test-results from ews103 for mac-highsierra (3.28 MB, application/zip)
2019-06-07 17:52 PDT, Build Bot
no flags Details
Archive of layout-test-results from ews126 for ios-simulator-wk2 (3.61 MB, application/zip)
2019-06-07 18:03 PDT, Build Bot
no flags Details
Archive of layout-test-results from ews113 for mac-highsierra (3.31 MB, application/zip)
2019-06-07 18:07 PDT, Build Bot
no flags Details
Patch (1.56 MB, patch)
2019-06-10 16:30 PDT, Daniel Bates
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews101 for mac-highsierra (3.35 MB, application/zip)
2019-06-10 18:20 PDT, Build Bot
no flags Details
Archive of layout-test-results from ews117 for mac-highsierra (3.14 MB, application/zip)
2019-06-10 21:08 PDT, Build Bot
no flags Details
Patch (1.56 MB, patch)
2019-06-11 10:45 PDT, Daniel Bates
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews103 for mac-highsierra (3.29 MB, application/zip)
2019-06-11 12:25 PDT, Build Bot
no flags Details
Archive of layout-test-results from ews115 for mac-highsierra (2.98 MB, application/zip)
2019-06-11 12:52 PDT, Build Bot
no flags Details
To land (1.56 MB, patch)
2019-06-11 12:58 PDT, Daniel Bates
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Bates 2019-06-07 13:44:39 PDT
Import Content Security Policy Web Platform Tests.
Comment 1 Radar WebKit Bug Importer 2019-06-07 13:45:16 PDT
<rdar://problem/51533785>
Comment 2 Daniel Bates 2019-06-07 15:11:50 PDT
Created attachment 371616 [details]
A bunch of tests
Comment 3 youenn fablet 2019-06-07 15:22:32 PDT
Comment on attachment 371616 [details]
A bunch of tests

r=me once bots are happy.
Please skip tests that are timing out, at least the ones that have no PASS.

It would be really good to remove the www1/www2 wherever possible.
Instead, you can rely on hosts[alt].
This can be done as a follow-up and should be upstreamed in WPT.

View in context: https://bugs.webkit.org/attachment.cgi?id=371616&action=review

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/base-uri/base-uri-deny.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to change the document base URL to http://www2.localhost:8800/ because it does not appear in the base-uri directive of the Content Security Policy.

Usually, we prefer to use non www1/www2 URLS if possible.
If we can, it would be nice to update the test upstream to use hosts[alt], or different port or protocol.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/base-uri/base-uri_iframe_sandbox.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to change the document base URL to http://www2.localhost:8800/base/ because it does not appear in the base-uri directive of the Content Security Policy.

Ditto.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/base-uri/report-uri-does-not-respect-base-uri.sub-expected.txt:8
> +TIMEOUT Event is fired Test timed out

Test is timing out, we may want to skip it.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/blob/blob-urls-do-not-match-self.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to load blob:http://localhost:8800/d820d6f6-f018-4285-b815-0c92b963e921 because it does not appear in the script-src directive of the Content Security Policy.

Test will probably be flaky, use DumpJSConsoleLogInStdErr if needed.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/blob/self-doesnt-match-blob.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to load blob:http://localhost:8800/f7ffe13f-be1a-482e-b2ab-ccc3114b5110 because it does not appear in the child-src directive of the Content Security Policy.

Will probably be flaky. You can use DumpJSConsoleLogInStdErr.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/blob/star-doesnt-match-blob.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to load blob:http://localhost:8800/9636972c-adc4-43d0-a730-fa5498d6bb99 because it does not appear in the child-src directive of the Content Security Policy.

Ditto.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/child-src/child-src-cross-origin-load.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to load http://www2.localhost:8800/content-security-policy/support/postmessage-fail.html because it does not appear in the child-src directive of the Content Security Policy.

www2 issue here as well.
It might be that the test is timing out due to that.
Might be best to skip it and fix it as a follow-up.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/connect-src/connect-src-beacon-blocked.sub-expected.txt:1
> +CONSOLE MESSAGE: Refused to connect to http://www1.localhost:8800/security/contentSecurityPolicy/echo-report.php because it does not appear in the connect-src directive of the Content Security Policy.

www1 potential issue, here and below.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/securitypolicyviolation/img-src-redirect-upgrade-reporting.https-expected.txt:6
> +TIMEOUT Image that redirects to http:// URL prohibited by Report-Only must generate a violation report, even with upgrade-insecure-requests Test timed out

Please skip this one and others that are timing out.

> LayoutTests/imported/w3c/web-platform-tests/content-security-policy/securitypolicyviolation/inside-dedicated-worker-expected.txt:8
> +TIMEOUT SecurityPolicyViolation event fired on global with the correct blockedURI. Test timed out

Ditto here.
Comment 4 Build Bot 2019-06-07 16:08:00 PDT
Comment on attachment 371616 [details]
A bunch of tests

Attachment 371616 [details] did not pass mac-ews (mac):
Output: https://webkit-queues.webkit.org/results/12413055

Number of test failures exceeded the failure limit.
Comment 5 Build Bot 2019-06-07 16:08:01 PDT
Created attachment 371621 [details]
Archive of layout-test-results from ews100 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews100  Port: mac-highsierra  Platform: Mac OS X 10.13.6
Comment 6 Daniel Bates 2019-06-07 16:14:52 PDT
Created attachment 371625 [details]
To land
Comment 7 Build Bot 2019-06-07 17:41:51 PDT
Comment on attachment 371625 [details]
To land

Attachment 371625 [details] did not pass mac-wk2-ews (mac-wk2):
Output: https://webkit-queues.webkit.org/results/12414054

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes-inherit-self.sub.html
imported/w3c/web-platform-tests/content-security-policy/svg/scripted.svg
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes.sub.html
imported/w3c/web-platform-tests/content-security-policy/script-src/script-src-multiple-policies-one-using-hashing-algorithms.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/unsandboxed-blob-scheme.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/sandboxed-blob-scheme.html
Comment 8 Build Bot 2019-06-07 17:41:52 PDT
Created attachment 371635 [details]
Archive of layout-test-results from ews105 for mac-highsierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews105  Port: mac-highsierra-wk2  Platform: Mac OS X 10.13.6
Comment 9 Build Bot 2019-06-07 17:52:19 PDT
Comment on attachment 371625 [details]
To land

Attachment 371625 [details] did not pass mac-ews (mac):
Output: https://webkit-queues.webkit.org/results/12414191

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-blocked.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes-inherit-self.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-self.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/svg/scripted.svg
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes.sub.html
imported/w3c/web-platform-tests/content-security-policy/script-src/script-src-multiple-policies-one-using-hashing-algorithms.html
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-allowed.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-child.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-list.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-none.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/unsandboxed-blob-scheme.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/sandboxed-blob-scheme.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-fallback.https.sub.html
Comment 10 Build Bot 2019-06-07 17:52:21 PDT
Created attachment 371637 [details]
Archive of layout-test-results from ews103 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-highsierra  Platform: Mac OS X 10.13.6
Comment 11 Build Bot 2019-06-07 18:03:52 PDT
Comment on attachment 371625 [details]
To land

Attachment 371625 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: https://webkit-queues.webkit.org/results/12414020

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes-inherit-self.sub.html
imported/w3c/web-platform-tests/content-security-policy/svg/scripted.svg
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes.sub.html
imported/w3c/web-platform-tests/content-security-policy/script-src/script-src-multiple-policies-one-using-hashing-algorithms.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/unsandboxed-blob-scheme.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/sandboxed-blob-scheme.html
Comment 12 Build Bot 2019-06-07 18:03:54 PDT
Created attachment 371639 [details]
Archive of layout-test-results from ews126 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews126  Port: ios-simulator-wk2  Platform: Mac OS X 10.14.5
Comment 13 Build Bot 2019-06-07 18:07:33 PDT
Comment on attachment 371625 [details]
To land

Attachment 371625 [details] did not pass mac-debug-ews (mac):
Output: https://webkit-queues.webkit.org/results/12414044

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-blocked.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes-inherit-self.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-self.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/svg/scripted.svg
imported/w3c/web-platform-tests/content-security-policy/inheritance/iframe-all-local-schemes.sub.html
imported/w3c/web-platform-tests/content-security-policy/script-src/script-src-multiple-policies-one-using-hashing-algorithms.html
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-allowed.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-child.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-list.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-none.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/unsandboxed-blob-scheme.html
imported/w3c/web-platform-tests/content-security-policy/inheritance/sandboxed-blob-scheme.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-fallback.https.sub.html
Comment 14 Build Bot 2019-06-07 18:07:37 PDT
Created attachment 371640 [details]
Archive of layout-test-results from ews113 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews113  Port: mac-highsierra  Platform: Mac OS X 10.13.6
Comment 15 Daniel Bates 2019-06-10 16:30:23 PDT
Created attachment 371793 [details]
Patch
Comment 16 Build Bot 2019-06-10 18:20:50 PDT
Comment on attachment 371793 [details]
Patch

Attachment 371793 [details] did not pass mac-ews (mac):
Output: https://webkit-queues.webkit.org/results/12439217

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-blocked.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-self.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-allowed.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-child.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-list.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-none.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-fallback.https.sub.html
Comment 17 Build Bot 2019-06-10 18:20:52 PDT
Created attachment 371802 [details]
Archive of layout-test-results from ews101 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews101  Port: mac-highsierra  Platform: Mac OS X 10.13.6
Comment 18 Build Bot 2019-06-10 21:08:57 PDT
Comment on attachment 371793 [details]
Patch

Attachment 371793 [details] did not pass mac-debug-ews (mac):
Output: https://webkit-queues.webkit.org/results/12440053

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-blocked.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-self.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/prefetch-src/prefetch-allowed.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-child.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-list.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-none.https.sub.html
imported/w3c/web-platform-tests/content-security-policy/worker-src/service-fallback.https.sub.html
Comment 19 Build Bot 2019-06-10 21:08:59 PDT
Created attachment 371817 [details]
Archive of layout-test-results from ews117 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews117  Port: mac-highsierra  Platform: Mac OS X 10.13.6
Comment 20 Daniel Bates 2019-06-11 10:45:48 PDT
Created attachment 371855 [details]
Patch
Comment 21 Build Bot 2019-06-11 12:25:10 PDT
Comment on attachment 371855 [details]
Patch

Attachment 371855 [details] did not pass mac-ews (mac):
Output: https://webkit-queues.webkit.org/results/12445791

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
Comment 22 Build Bot 2019-06-11 12:25:11 PDT
Created attachment 371863 [details]
Archive of layout-test-results from ews103 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-highsierra  Platform: Mac OS X 10.13.6
Comment 23 Build Bot 2019-06-11 12:52:42 PDT
Comment on attachment 371855 [details]
Patch

Attachment 371855 [details] did not pass mac-debug-ews (mac):
Output: https://webkit-queues.webkit.org/results/12445755

New failing tests:
imported/w3c/web-platform-tests/content-security-policy/svg/including.sub.svg
Comment 24 Build Bot 2019-06-11 12:52:44 PDT
Created attachment 371865 [details]
Archive of layout-test-results from ews115 for mac-highsierra

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews115  Port: mac-highsierra  Platform: Mac OS X 10.13.6
Comment 25 Daniel Bates 2019-06-11 12:58:52 PDT
Created attachment 371868 [details]
To land
Comment 26 Daniel Bates 2019-06-11 13:14:53 PDT
Comment on attachment 371868 [details]
To land

Clearing flags on attachment: 371868

Committed r246330: <https://trac.webkit.org/changeset/246330>
Comment 27 Daniel Bates 2019-06-11 13:14:55 PDT
All reviewed patches have been landed.  Closing bug.
Comment 28 Russell Epstein 2019-06-11 15:22:06 PDT
It looks like the tests imported in https://trac.webkit.org/changeset/246330/webkit

Have 1 flakey failure.

imported/w3c/web-platform-tests/content-security-policy/reporting/report-only-in-meta.sub.html

History 
https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-security-policy%2Freporting%2Freport-only-in-meta.sub.html

Diff
--- /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting/report-only-in-meta.sub-expected.txt
+++ /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/imported/w3c/web-platform-tests/content-security-policy/reporting/report-only-in-meta.sub-actual.txt
@@ -1,4 +1,5 @@
 
 
 PASS Image should load 
+PASS Violation report status OK.
Comment 29 Daniel Bates 2019-06-11 19:24:14 PDT
(In reply to Russell Epstein from comment #28)
> It looks like the tests imported in
> https://trac.webkit.org/changeset/246330/webkit
> 
> Have 1 flakey failure.
> 
> imported/w3c/web-platform-tests/content-security-policy/reporting/report-
> only-in-meta.sub.html
> 
> History 
> https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.
> html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2Fcontent-
> security-policy%2Freporting%2Freport-only-in-meta.sub.html
> 
> Diff
> ---
> /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/
> imported/w3c/web-platform-tests/content-security-policy/reporting/report-
> only-in-meta.sub-expected.txt
> +++
> /Volumes/Data/slave/mojave-release-tests-wk1/build/layout-test-results/
> imported/w3c/web-platform-tests/content-security-policy/reporting/report-
> only-in-meta.sub-actual.txt
> @@ -1,4 +1,5 @@
>  
>  
>  PASS Image should load 
> +PASS Violation report status OK.

I'll take a look tomorrow. There are some known flakiness in the test set. This import was a first pass. So, in my opinion, I would just skip the test for now if that is being weighed with a rollout.
Comment 30 Russell Epstein 2019-06-18 15:38:15 PDT
Marked test as flakey in https://bugs.webkit.org/show_bug.cgi?id=198977