Bug 172930 – Teach run-webkit-tests how to run HTTPS Web Platform Tests

• Component: Tools / Tests
• Status: RESOLVED DUPLICATE

[Daniel Bates]

Some Web Platform Tests need to be accessed from an HTTPS server. One example of such a test is LayoutTests/imported/w3c/web-platform-tests/WebCryptoAPI/secure_context/crypto-subtle-secure-context-available.https.sub.html. We should support running Web Platform Tests from an HTTPS server.

[Radar WebKit Bug Importer]

<rdar://problem/32570201>

[Daniel Bates]

Created attachment 312023 [details]
Patch

[youenn fablet]

Comment on attachment 312023 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=312023&action=review

> Tools/ChangeLog:13
> +        Web Platform Tests that need be run from an HTTPS server have a filename that contains ".https.".

I thought this case was handled by the wptserver directly. But it seems you are right that this currently needs to be handled by the test runner.
Is there some documentation about this "https" rule somewhere?

[Daniel Bates]

(In reply to youenn fablet from comment #3)
> Comment on attachment 312023 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=312023&action=review
> 
> > Tools/ChangeLog:13
> > +        Web Platform Tests that need be run from an HTTPS server have a filename that contains ".https.".
> 
> I thought this case was handled by the wptserver directly. But it seems you
> are right that this currently needs to be handled by the test runner.
> Is there some documentation about this "https" rule somewhere?

I have not had success finding any documentation about the "https" rule. There seems to be some unwritten convention of adding ".https." to the suffix of a filename to indicate that the test is to be run from an HTTPS server. The Blink project first adopted the convention of running tests that contain ".https." using an HTTPS server in <https://chromium.googlesource.com/chromium/src/+/21b3fb22fb360dc6205bb8585d0a45046e245aec>. Specifically, see the change to <https://chromium.googlesource.com/chromium/src/+/21b3fb22fb360dc6205bb8585d0a45046e245aec/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/driver.py#242>.

[Daniel Bates]

(In reply to Daniel Bates from comment #4)
> (In reply to youenn fablet from comment #3)
> > Comment on attachment 312023 [details]
> > Patch
> > 
> > View in context:
> > https://bugs.webkit.org/attachment.cgi?id=312023&action=review
> > 
> > > Tools/ChangeLog:13
> > > +        Web Platform Tests that need be run from an HTTPS server have a filename that contains ".https.".
> > 
> > I thought this case was handled by the wptserver directly. But it seems you
> > are right that this currently needs to be handled by the test runner.
> > Is there some documentation about this "https" rule somewhere?
> 
> I have not had success finding any documentation about the "https" rule.
> There seems to be some unwritten convention of adding ".https." to the
> suffix of a filename to indicate that the test is to be run from an HTTPS
> server. The Blink project first adopted the convention of running tests that
> contain ".https." using an HTTPS server in
> <https://chromium.googlesource.com/chromium/src/+/
> 21b3fb22fb360dc6205bb8585d0a45046e245aec>. Specifically, see the change to
> <https://chromium.googlesource.com/chromium/src/+/
> 21b3fb22fb360dc6205bb8585d0a45046e245aec/third_party/WebKit/Tools/Scripts/
> webkitpy/layout_tests/port/driver.py#242>.

From briefly reading the wptrunner source (https://github.com/w3c/wptrunner), it seems that looking for ".https." in the name of a test is a Blink convention. The tool wptrunner does not support this and looks for a manifest file to determine the port to use to run a test.

[Daniel Bates]

jgraham mentioned that wptrunner is responsible for running a web platform test using an HTTPS server: "https://github.com/w3c/web-platform-tests/blob/master/tools/manifest/item.py#L41 sets the https flag in the manifest from the filename, and wptrunner uses that to load the tests in the correct way" (https://github.com/w3c/web-platform-tests/issues/6168#issuecomment-306615240).

At the time of writing, we use run-webkit-tests as our test driver. That is, we do not make use of wptrunner.

[Daniel Bates]

Committed r217902: <http://trac.webkit.org/changeset/217902>

[Ryan Haddad]

Reverted r217902 for reason:

This change appears to have caused imported/w3c/web-platform-tests/fetch/api/cors tests to fail on El Capitan.

Committed r217913: <http://trac.webkit.org/changeset/217913>

[Ryan Haddad]

(In reply to Ryan Haddad from comment #8)
> Reverted r217902 for reason:
> 
> This change appears to have caused
> imported/w3c/web-platform-tests/fetch/api/cors tests to fail on El Capitan.
> 
> Committed r217913: <http://trac.webkit.org/changeset/217913>
See:
https://build.webkit.org/builders/Apple%20El%20Capitan%20Release%20WK2%20%28Tests%29/builds/2135

These tests have been flaky on other platforms in the past, but since this change they have been consistently failing on El Capitan WK2. Rolled out because this affects EWS.

[Daniel Bates]

*** This bug has been marked as a duplicate of bug 174992 ***