Bug 177615 – REGRESSION (High Sierra, iOS 11): LayoutTest imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/successes_RSA-PSS.worker.html is very slow, and a flaky failure

• Component: New Bugs
• Status: RESOLVED FIXED

[Ryan Haddad]

LayoutTest imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/successes_RSA-PSS.worker.html is a flaky failure

https://build.webkit.org/results/Apple%20High%20Sierra%20Debug%20WK2%20(Tests)/r222613%20(35)/results.html

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2FWebCryptoAPI%2FgenerateKey%2Fsuccesses_RSA-PSS.worker.html

--- /Volumes/Data/slave/highsierra-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/successes_RSA-PSS.worker-expected.txt
+++ /Volumes/Data/slave/highsierra-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/successes_RSA-PSS.worker-actual.txt
@@ -1,3 +1,5 @@
+
+Harness Error (TIMEOUT), message = null
 
 PASS Success: generateKey({hash: SHA-1, modulusLength: 2048, name: RSA-PSS, publicExponent: {0: 1, 1: 0, 2: 1}}, false, [sign]) 
 PASS Success: generateKey({hash: SHA-1, modulusLength: 2048, name: RSA-PSS, publicExponent: {0: 1, 1: 0, 2: 1}}, true, [sign]) 
@@ -25,14 +27,14 @@
 PASS Success: generateKey({hash: SHA-256, modulusLength: 2048, name: rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, true, [sign, verify, sign, sign, verify]) 
 PASS Success: generateKey({hash: SHA-1, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, false, [sign]) 
 PASS Success: generateKey({hash: SHA-1, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, true, [sign]) 
-PASS Success: generateKey({hash: SHA-1, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, false, [verify, sign]) 
-PASS Success: generateKey({hash: SHA-1, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, true, [verify, sign]) 
-PASS Success: generateKey({hash: SHA-1, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, false, [sign, verify, sign, sign, verify]) 
-PASS Success: generateKey({hash: SHA-1, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, true, [sign, verify, sign, sign, verify]) 
-PASS Success: generateKey({hash: SHA-256, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, false, [sign]) 
-PASS Success: generateKey({hash: SHA-256, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, true, [sign]) 
-PASS Success: generateKey({hash: SHA-256, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, false, [verify, sign]) 
-PASS Success: generateKey({hash: SHA-256, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, true, [verify, sign]) 
-PASS Success: generateKey({hash: SHA-256, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, false, [sign, verify, sign, sign, verify]) 
-PASS Success: generateKey({hash: SHA-256, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, true, [sign, verify, sign, sign, verify]) 
+TIMEOUT Success: generateKey({hash: SHA-1, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, false, [verify, sign]) Test timed out
+NOTRUN Success: generateKey({hash: SHA-1, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, true, [verify, sign]) 
+NOTRUN Success: generateKey({hash: SHA-1, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, false, [sign, verify, sign, sign, verify]) 
+NOTRUN Success: generateKey({hash: SHA-1, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, true, [sign, verify, sign, sign, verify]) 
+NOTRUN Success: generateKey({hash: SHA-256, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, false, [sign]) 
+NOTRUN Success: generateKey({hash: SHA-256, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, true, [sign]) 
+NOTRUN Success: generateKey({hash: SHA-256, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, false, [verify, sign]) 
+NOTRUN Success: generateKey({hash: SHA-256, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, true, [verify, sign]) 
+NOTRUN Success: generateKey({hash: SHA-256, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, false, [sign, verify, sign, sign, verify]) 
+NOTRUN Success: generateKey({hash: SHA-256, modulusLength: 2048, name: Rsa-pss, publicExponent: {0: 1, 1: 0, 2: 1}}, true, [sign, verify, sign, sign, verify])

[Ryan Haddad]

This test fails more often than it passes on High Sierra Debug WK2.

[Ryan Haddad]

Marked test as flaky in https://trac.webkit.org/changeset/222629/webkit

[Alexey Proskuryakov]

This is a High Sierra regression - both debug and release are always very slow, taking 15-30 seconds. This test takes 1-2 seconds on Sierra. iOS 11 is slow too.

[Radar WebKit Bug Importer]

<rdar://problem/34760820>

[Alexey Proskuryakov]

Youenn, is there a way to run this test in browser? There is some WPT magic at play, as this file only contains a comment.

[youenn fablet]

Use open-layout-tests script passing the path to the test from LayiutTests.
It will start the wpt server and open the test at the right http url.
The html is generated by the server. This is a wrapper around the corresponding .worker.js file.

[Jiewen Tan]

(In reply to Alexey Proskuryakov from comment #3)
> This is a High Sierra regression - both debug and release are always very
> slow, taking 15-30 seconds. This test takes 1-2 seconds on Sierra. iOS 11 is
> slow too.

The reason why this test is faster in Sierra is that RSA-PSS is not enabled in Sierra. Hence, all the tests will hit the NotSupport exceptions and thus return early, which makes the test faster.

[Jiewen Tan]

We should mark it slow as all the other slow RSA key generation tests.

[Jiewen Tan]

Committed r222734: <http://trac.webkit.org/changeset/222734>

[youenn fablet]

Usually wpt tests that are slow have some metadata attached to them and the importer detects that the test is slow and mark it as such.
It might be worth fixing the test and/or raising on issue on GitHub.

[youenn fablet]

Or maybe the importer is missing the metadata somehow.

[Jiewen Tan]

(In reply to youenn fablet from comment #10)
> Usually wpt tests that are slow have some metadata attached to them and the
> importer detects that the test is slow and mark it as such.
> It might be worth fixing the test and/or raising on issue on GitHub.

I think it will be likely these tests haven't been added with corresponding metadata.