Bug 159638 - Improve test coverage for imported/w3c/WebCryptoAPI/
Summary: Improve test coverage for imported/w3c/WebCryptoAPI/
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebCore Misc. (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on: 159636 159979 163953 165629 165680
Blocks: 160880
  Show dependency treegraph
 
Reported: 2016-07-11 12:35 PDT by Jiewen Tan
Modified: 2016-12-10 02:12 PST (History)
7 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jiewen Tan 2016-07-11 12:35:03 PDT
Improve test coverage for imported/w3c/WebCryptoAPI/
Comment 1 Jiewen Tan 2016-07-13 17:10:39 PDT
imported/w3c/WebCryptoAPI/encrypt_decrypt/test_aes_cbc.html [ Skip ]
imported/w3c/WebCryptoAPI/encrypt_decrypt/test_aes_ctr.html [ Skip ]
imported/w3c/WebCryptoAPI/encrypt_decrypt/test_aes_gcm.html [ Skip ]
imported/w3c/WebCryptoAPI/encrypt_decrypt/test_rsa_oaep.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_aes-cbc.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_aes-ctr.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_failures_AES-CBC.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_failures_AES-CTR.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_failures_AES-GCM.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_failures_AES-KW.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_failures_ECDH.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_failures_ECDSA.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_failures_HMAC.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_failures_RSA-OAEP.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_failures_RSA-PSS.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_failures_RSASSA-PKCS1-v1_5.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_failures.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_successes_AES-CBC.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_successes_AES-CTR.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_successes_AES-GCM.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_successes_AES-KW.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_successes_ECDH.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_successes_ECDSA.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_successes_HMAC.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_successes_RSA-OAEP.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_successes_RSA-PSS.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_successes_RSASSA-PKCS1-v1_5.html [ Skip ]
imported/w3c/WebCryptoAPI/generateKey/test_successes.html [ Skip ]
imported/w3c/WebCryptoAPI/idlharness.html [ Skip ]
Comment 2 Jiewen Tan 2016-07-20 16:26:53 PDT
Fix timeout of imported/w3c/WebCryptoAPI/idlharness.html.

Committed r203476: <http://trac.webkit.org/changeset/203476>
Comment 3 Radar WebKit Bug Importer 2016-08-16 00:43:31 PDT
<rdar://problem/27862255>
Comment 4 Jiewen Tan 2016-10-25 10:46:14 PDT
The following tests are enabled in Committed r207809: <http://trac.webkit.org/changeset/207809>
imported/w3c/WebCryptoAPI/generateKey/test_aes-cbc.html
imported/w3c/WebCryptoAPI/generateKey/test_aes-ctr.html
imported/w3c/WebCryptoAPI/generateKey/test_failures_AES-CBC.html
imported/w3c/WebCryptoAPI/generateKey/test_failures_AES-CTR.html
imported/w3c/WebCryptoAPI/generateKey/test_failures_AES-GCM.html
imported/w3c/WebCryptoAPI/generateKey/test_failures_AES-KW.html
imported/w3c/WebCryptoAPI/generateKey/test_failures_ECDH.html
imported/w3c/WebCryptoAPI/generateKey/test_failures_ECDSA.html
imported/w3c/WebCryptoAPI/generateKey/test_failures_HMAC.html
imported/w3c/WebCryptoAPI/generateKey/test_failures_RSA-OAEP.html
imported/w3c/WebCryptoAPI/generateKey/test_failures_RSA-PSS.html
imported/w3c/WebCryptoAPI/generateKey/test_failures_RSASSA-PKCS1-v1_5.html
imported/w3c/WebCryptoAPI/generateKey/test_failures.html
imported/w3c/WebCryptoAPI/generateKey/test_successes_AES-CBC.html
imported/w3c/WebCryptoAPI/generateKey/test_successes_AES-CTR.html
imported/w3c/WebCryptoAPI/generateKey/test_successes_AES-GCM.html
imported/w3c/WebCryptoAPI/generateKey/test_successes_AES-KW.html
imported/w3c/WebCryptoAPI/generateKey/test_successes_ECDH.html
imported/w3c/WebCryptoAPI/generateKey/test_successes_ECDSA.html
imported/w3c/WebCryptoAPI/generateKey/test_successes_HMAC.html
imported/w3c/WebCryptoAPI/generateKey/test_successes_RSA-OAEP.html
imported/w3c/WebCryptoAPI/generateKey/test_successes_RSA-PSS.html
imported/w3c/WebCryptoAPI/generateKey/test_successes_RSASSA-PKCS1-v1_5.html
imported/w3c/WebCryptoAPI/generateKey/test_successes.html
Comment 5 Ryan Haddad 2016-10-25 10:47:44 PDT
The following tests are also flaky timeouts:

imported/w3c/WebCryptoAPI/generateKey/test_successes.html
imported/w3c/WebCryptoAPI/generateKey/test_failures.html
imported/w3c/WebCryptoAPI/generateKey/test_successes_RSA-OAEP.html
imported/w3c/WebCryptoAPI/generateKey/test_successes_RSASSA-PKCS1-v1_5.html

https://build.webkit.org/results/Apple%20Yosemite%20Debug%20WK2%20(Tests)/r207818%20(15854)/results.html
Comment 6 Jiewen Tan 2016-11-18 14:55:43 PST
The following tests are enabled in Committed r208891: <http://trac.webkit.org/changeset/208891>:
imported/w3c/WebCryptoAPI/encrypt_decrypt/test_aes_cbc.html
imported/w3c/WebCryptoAPI/encrypt_decrypt/test_aes_ctr.html
imported/w3c/WebCryptoAPI/encrypt_decrypt/test_aes_gcm.html
imported/w3c/WebCryptoAPI/encrypt_decrypt/test_rsa_oaep.html
Comment 7 Jiewen Tan 2016-12-07 17:31:27 PST
Current status of all tests:
1. idlharness.html [46/64]
2. test_getRandomValues.html [4/4]
3. digest/test_digest.html [80/80]
4. encrypt_decrypt/test_aes_cbc.html [30/42]
5. encrypt_decrypt/test_aes_ctr.html [0/33]*
6. encrypt_decrypt/test_aes_gcm.html [0/330]*
7. encrypt_decrypt/test_rsa_oaep.html [96/108]
8. generateKey/test_aes-cbc.html [288/288]
9. generateKey/test_aes-ctr.html [0/288]*
10. generateKey/test_failures_AES-CBC.html [738/738]
11. generateKey/test_failures_AES-CBC.html [324/738]*
12. generateKey/test_failures_AES-GCM.html [324/738]*
13. generateKey/test_failures_AES-KW.html [480/480]
14. generateKey/test_failures_ECDH.html [344/440]*
15. generateKey/test_failures_ECDSA.htm [340/418]*
16. generateKey/test_failures_HMAC.html [452/452]
17. generateKey/test_failures_RSA-OAEP.html [496/496]
18. generateKey/test_failures_RSA-PSS.html [324/392]*
19. generateKey/test_failures_RSASSA-PKCS1-v1_5.html [392/392]
20. generateKey/test_failures.html [1298/2368]**
21. generateKey/test_successes_AES-CBC.html [288/288]
22. generateKey/test_successes_AES-CTR.html [0/288]*
23. generateKey/test_successes_AES-GCM.html [0/288]*
24. generateKey/test_successes_AES-KW.html [72/72]
25. generateKey/test_successes_ECDH.html [0/72]*
26. generateKey/test_successes_ECDSA.html [0/54]*
27. generateKey/test_successes_HMAC.html [96/96]
28. generateKey/test_successes_RSA-OAEP.html [156/156]
29. generateKey/test_successes_RSA-PSS.html [0/36]*
30. generateKey/test_successes_RSASSA-PKCS1-v1_5.html [36/36]
31. generateKey/test_successes.html [648/738]**
* are algorithms that we haven't supported yet.
** are superset tests.

I will start investigating why 1, 4 and 7 don't pass all tests.
Comment 8 Jiewen Tan 2016-12-07 17:50:25 PST
Committed r209508: <http://trac.webkit.org/changeset/209508>
Comment 9 Jiewen Tan 2016-12-08 15:15:45 PST
idlharness.html is tracked in Bug 165629.
Comment 10 Jiewen Tan 2016-12-08 17:27:10 PST
Failing tests of encrypt_decrypt/test_aes_cbc.html are:
FAIL AES-CBC 128-bit key with mismatched key and algorithm assert_equals: Mismatch should cause InvalidAccessError instead of The operation is not supported. expected "InvalidAccessError" but got "NotSupportedError"
FAIL AES-CBC 192-bit key with mismatched key and algorithm assert_equals: Mismatch should cause InvalidAccessError instead of The operation is not supported. expected "InvalidAccessError" but got "NotSupportedError"
FAIL AES-CBC 256-bit key with mismatched key and algorithm assert_equals: Mismatch should cause InvalidAccessError instead of The operation is not supported. expected "InvalidAccessError" but got "NotSupportedError"

FAIL AES-CBC 128-bit key, zeroPadChar assert_unreached: should have thrown exception for test AES-CBC 128-bit key, zeroPadChar Reached unreachable code
FAIL AES-CBC 128-bit key, bigPadChar assert_unreached: should have thrown exception for test AES-CBC 128-bit key, bigPadChar Reached unreachable code
FAIL AES-CBC 128-bit key, inconsistentPadChars assert_unreached: should have thrown exception for test AES-CBC 128-bit key, inconsistentPadChars Reached unreachable code
FAIL AES-CBC 192-bit key, zeroPadChar assert_unreached: should have thrown exception for test AES-CBC 192-bit key, zeroPadChar Reached unreachable code
FAIL AES-CBC 192-bit key, bigPadChar assert_unreached: should have thrown exception for test AES-CBC 192-bit key, bigPadChar Reached unreachable code
FAIL AES-CBC 192-bit key, inconsistentPadChars assert_unreached: should have thrown exception for test AES-CBC 192-bit key, inconsistentPadChars Reached unreachable code
FAIL AES-CBC 256-bit key, zeroPadChar assert_unreached: should have thrown exception for test AES-CBC 256-bit key, zeroPadChar Reached unreachable code
FAIL AES-CBC 256-bit key, bigPadChar assert_unreached: should have thrown exception for test AES-CBC 256-bit key, bigPadChar Reached unreachable code
FAIL AES-CBC 256-bit key, inconsistentPadChars assert_unreached: should have thrown exception for test AES-CBC 256-bit key, inconsistentPadChars Reached unreachable code

For the first group, the test case use "AES-GCM" as the mismatched algorithm name, which we haven't support yet. Therefore, we return NotSupportedError instead of InvalidAccessError.

For the second group, our underlying SecLibrary silently bypass the failures. Actually I don't think there is a failure in our SecLibrary as adding a wrong padding to the plain text will just result in another new plain text. Since we hand over all the padding jobs to the underlying SecLibrary, we should not follow the spec to remove bad padding. Otherwise, it will produce corrupted decrypted text. Mark as NTF.
Comment 11 Jiewen Tan 2016-12-09 13:39:28 PST
encrypt_decrypt/test_rsa_oaep.html is tracked in Bug 165680.
Comment 12 Jiewen Tan 2016-12-10 02:09:42 PST
Close this umbrella bug as all of its depending bugs are fixed.

Final Results:
1. idlharness.html [64/64]
2. test_getRandomValues.html [4/4]
3. digest/test_digest.html [80/80]
4. encrypt_decrypt/test_aes_cbc.html [30/42]***
5. encrypt_decrypt/test_aes_ctr.html [0/33]*
6. encrypt_decrypt/test_aes_gcm.html [0/330]*
7. encrypt_decrypt/test_rsa_oaep.html [108/108]
8. generateKey/test_aes-cbc.html [288/288]
9. generateKey/test_aes-ctr.html [0/288]*
10. generateKey/test_failures_AES-CBC.html [738/738]
11. generateKey/test_failures_AES-CBC.html [324/738]*
12. generateKey/test_failures_AES-GCM.html [324/738]*
13. generateKey/test_failures_AES-KW.html [480/480]
14. generateKey/test_failures_ECDH.html [344/440]*
15. generateKey/test_failures_ECDSA.htm [340/418]*
16. generateKey/test_failures_HMAC.html [452/452]
17. generateKey/test_failures_RSA-OAEP.html [496/496]
18. generateKey/test_failures_RSA-PSS.html [324/392]*
19. generateKey/test_failures_RSASSA-PKCS1-v1_5.html [392/392]
20. generateKey/test_failures.html [1298/2368]**
21. generateKey/test_successes_AES-CBC.html [288/288]
22. generateKey/test_successes_AES-CTR.html [0/288]*
23. generateKey/test_successes_AES-GCM.html [0/288]*
24. generateKey/test_successes_AES-KW.html [72/72]
25. generateKey/test_successes_ECDH.html [0/72]*
26. generateKey/test_successes_ECDSA.html [0/54]*
27. generateKey/test_successes_HMAC.html [96/96]
28. generateKey/test_successes_RSA-OAEP.html [156/156]
29. generateKey/test_successes_RSA-PSS.html [0/36]*
30. generateKey/test_successes_RSASSA-PKCS1-v1_5.html [36/36]
31. generateKey/test_successes.html [648/738]**
* are algorithms that we haven't supported yet.
** are superset tests.
*** are marked as NTBF. See comments above.