Bug 185482 - Update Test262 tests through the new import script - 20180509
Summary: Update Test262 tests through the new import script - 20180509
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Leo Balter
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2018-05-09 11:47 PDT by Leo Balter
Modified: 2018-05-29 12:51 PDT (History)
6 users (show)

See Also:


Attachments
Patch (978.76 KB, patch)
2018-05-09 11:50 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews122 for ios-simulator-wk2 (2.39 MB, application/zip)
2018-05-09 13:23 PDT, Build Bot
no flags Details
Patch (692.19 KB, patch)
2018-05-09 14:25 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (691.88 KB, patch)
2018-05-09 14:28 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews100 for mac-sierra (2.35 MB, application/zip)
2018-05-09 15:19 PDT, Build Bot
no flags Details
Patch (691.41 KB, patch)
2018-05-10 12:27 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (691.40 KB, patch)
2018-05-10 13:45 PDT, Leo Balter
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Leo Balter 2018-05-09 11:47:33 PDT
Update Test262 tests through the new import script - 20180509
Comment 1 Leo Balter 2018-05-09 11:50:37 PDT
Created attachment 339997 [details]
Patch
Comment 2 Leo Balter 2018-05-09 12:00:51 PDT
This patch updates the Test262 files and the respective JSTests/test262/expectations.yaml file with the most recent passes.

This patch helps the report from the Test262 Runner since several tests files started passing from a recent change on WebKit.
Comment 3 Leo Balter 2018-05-09 13:14:17 PDT
There is an error in the expectations file I'm trying to fix right now
Comment 4 Build Bot 2018-05-09 13:23:17 PDT
Comment on attachment 339997 [details]
Patch

Attachment 339997 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/7626712

New failing tests:
http/tests/appcache/abort-cache-onchecking-manifest-404.html
Comment 5 Build Bot 2018-05-09 13:23:18 PDT
Created attachment 340014 [details]
Archive of layout-test-results from ews122 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews122  Port: ios-simulator-wk2  Platform: Mac OS X 10.13.4
Comment 6 Leo Balter 2018-05-09 14:25:08 PDT
Created attachment 340024 [details]
Patch
Comment 7 Leo Balter 2018-05-09 14:28:35 PDT
Created attachment 340026 [details]
Patch
Comment 8 Michael Saboff 2018-05-09 14:50:31 PDT
Comment on attachment 340026 [details]
Patch

r=me
Comment 9 Build Bot 2018-05-09 15:19:38 PDT
Comment on attachment 340026 [details]
Patch

Attachment 340026 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/7628875

New failing tests:
http/tests/appcache/abort-cache-onchecking-manifest-404.html
Comment 10 Build Bot 2018-05-09 15:19:40 PDT
Created attachment 340040 [details]
Archive of layout-test-results from ews100 for mac-sierra

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews100  Port: mac-sierra  Platform: Mac OS X 10.12.6
Comment 11 Leo Balter 2018-05-09 15:34:11 PDT
seems like something else is breaking. Should I rebase this?
Comment 12 Michael Saboff 2018-05-10 08:36:13 PDT
(In reply to Leo Balter from comment #11)
> seems like something else is breaking. Should I rebase this?

These are unrelated failures.  I just cq+ the patch again.  Let's see if it goes through.  If not, I'll land it manually.
Comment 13 WebKit Commit Bot 2018-05-10 08:43:53 PDT
Comment on attachment 340026 [details]
Patch

Rejecting attachment 340026 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.webkit.org', '--bot-id=webkit-cq-01', 'apply-attachment', '--no-update', '--non-interactive', 340026, '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit

Last 500 characters of output:
t/language/statements/try/early-catch-duplicates.js
patching file JSTests/test262/test/language/statements/try/early-catch-function.js
patching file JSTests/test262/test/language/statements/try/early-catch-lex.js
patching file JSTests/test262/test/language/statements/try/early-catch-var.js
patching file JSTests/test262/test262-Revision.txt

Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force', '--reviewer', u'Michael Saboff']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

Full output: http://webkit-queues.webkit.org/results/7639088
Comment 14 Leo Balter 2018-05-10 12:27:39 PDT
Created attachment 340117 [details]
Patch
Comment 15 Michael Saboff 2018-05-10 13:00:09 PDT
Comment on attachment 340117 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=340117&action=review

r-

> JSTests/ChangeLog:7
> +        Update Test262 tests through the new import script - 20180509
> +
> +        Also update the test262/expecatations.yaml with the recent
> +        imported files.
> +        https://bugs.webkit.org/show_bug.cgi?id=185482

The format of a ChangeLog should be:
    Title
    Bug URL

    Reviewed by ...

    Description

Please move the "Also update..." to after Reviewed By ...
Comment 16 Leo Balter 2018-05-10 13:45:54 PDT
Created attachment 340124 [details]
Patch
Comment 17 Michael Saboff 2018-05-10 14:30:31 PDT
Comment on attachment 340124 [details]
Patch

r=me
Comment 18 WebKit Commit Bot 2018-05-10 15:29:42 PDT
Comment on attachment 340124 [details]
Patch

Clearing flags on attachment: 340124

Committed r231666: <https://trac.webkit.org/changeset/231666>
Comment 19 WebKit Commit Bot 2018-05-10 15:29:43 PDT
All reviewed patches have been landed.  Closing bug.
Comment 20 Radar WebKit Bug Importer 2018-05-10 15:30:41 PDT
<rdar://problem/40144395>
Comment 21 Caitlin Potter (:caitp) 2018-05-28 17:11:07 PDT
In the future, it would be good to update JSTests/test262.yaml also. I'm unable to run the test262 suite the usual way (run-jsc-stress-tests JSTests/test262.yaml) to update expectations when fixing bugs, due to the large number of deleted files which are still referenced (which cause the python script to crash).
Comment 22 Leo Balter 2018-05-29 12:51:18 PDT
> In the future, it would be good to update JSTests/test262.yaml also. I'm unable to run the test262 suite the usual way (run-jsc-stress-tests JSTests/test262.yaml) to update expectations when fixing bugs, due to the large number of deleted files which are still referenced (which cause the python script to crash).

We introduced a new test262 runner on WebKit already being used by the build bot where the old Test262 runner is still there for a switch back if ever necessary.

Would it be fine for you to use the `test262-runner` instead? It doesn't require any maintenance burden for this legacy test262.yaml file.

You can run it with --help to see some of the extra options and it also generates a results folder with summaries and a full report.

If you have anything else blocking you from using it, please let me understand it and I'll dedicate some time to update the yaml file.