Bug 184934 - Detach the old Test262 Runner from the stress tests
Summary: Detach the old Test262 Runner from the stress tests
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: JavaScriptCore (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-04-24 13:44 PDT by Leo Balter
Modified: 2018-06-14 11:34 PDT (History)
3 users (show)

See Also:


Attachments
Patch (11.92 MB, patch)
2018-04-25 14:33 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (16.80 KB, patch)
2018-05-04 08:37 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Remove JSTests/test262.yaml (11.90 MB, patch)
2018-05-04 09:21 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews106 for mac-sierra-wk2 (2.95 MB, application/zip)
2018-05-04 09:41 PDT, Build Bot
no flags Details
Patch (16.73 KB, patch)
2018-05-04 10:59 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Remove JSTests/test262.yaml (11.90 MB, patch)
2018-05-04 11:01 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (13.43 KB, patch)
2018-05-09 11:00 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Remove JSTests/test262.yaml (11.90 MB, patch)
2018-05-09 11:15 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews104 for mac-sierra-wk2 (2.80 MB, application/zip)
2018-05-09 12:46 PDT, Build Bot
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Leo Balter 2018-04-24 13:44:14 PDT
With the new Test262 Runner and import scripts (Tools/Scripts/test262-runner and Tools/Script/test262-import), the old runner should be all set to be removed from the stress tests script.
Comment 1 Leo Balter 2018-04-25 14:33:33 PDT
Created attachment 338795 [details]
Patch
Comment 2 Leo Balter 2018-05-04 07:29:51 PDT
I'm rebasing this one with the most recent changes in the Test262 Runner
Comment 3 Leo Balter 2018-05-04 08:37:20 PDT
Created attachment 339547 [details]
Patch
Comment 4 Leo Balter 2018-05-04 08:39:07 PDT
I uploaded a new patch without removing the JSTests/test262.yaml to allow an easier review
Comment 5 Leo Balter 2018-05-04 09:21:45 PDT
Created attachment 339551 [details]
Remove JSTests/test262.yaml

This change is in a separate to simplify the review process.
Comment 6 Build Bot 2018-05-04 09:41:40 PDT
Comment on attachment 339547 [details]
Patch

Attachment 339547 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/7564930

New failing tests:
fast/mediastream/delayed-permission-allowed.html
Comment 7 Build Bot 2018-05-04 09:41:41 PDT
Created attachment 339554 [details]
Archive of layout-test-results from ews106 for mac-sierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews106  Port: mac-sierra-wk2  Platform: Mac OS X 10.12.6
Comment 8 Leo Balter 2018-05-04 09:53:57 PDT
It feels like the failures are unrelated to the actual patch, I'll confirm this.
Comment 9 Leo Balter 2018-05-04 10:59:10 PDT
Created attachment 339566 [details]
Patch
Comment 10 Leo Balter 2018-05-04 11:01:49 PDT
Created attachment 339567 [details]
Remove JSTests/test262.yaml
Comment 11 Aakash Jain 2018-05-07 21:42:33 PDT
Comment on attachment 339566 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=339566&action=review

> Tools/ChangeLog:9
> +        (RunTest262Tests):

Can you please write the detailed ChangeLog?
Comment 12 Aakash Jain 2018-05-07 21:43:56 PDT
(In reply to Leo Balter from comment #10)
> Created attachment 339567 [details]
> Remove JSTests/test262.yaml

Can the removal of old test-runner related files be done in a separate patch after the switch is complete?
Comment 13 Aakash Jain 2018-05-07 21:44:04 PDT
Can you please mention why are we removing old test runner and using new one?
Comment 14 Leo Balter 2018-05-08 09:01:06 PDT
> Can you please mention why are we removing old test runner and using new one?

I'm not sure if should mention this here or through the changelog. In any case, I'll try both, one should be right.

I can summarize some of the goals for the new Test262 Runner:

- It's configurable, with mechanisms to handled new imported files, failing tests, etc.
- It eases the process of importing new Test262 content, where skipped and known failing tests are kept and flagged without a hassle.
- It's compatible with any target for running the Test262 tests locally through the CI bots.
- It's also detached from the stress tests as Test262 has a different content and purpose (specs conformance, mostly).

> Can the removal of old test-runner related files be done in a separate patch after the switch is complete?

I this case I should use the bug 184933 for the switch only and then we follow up on this one. I'm happy to fix this and I'll ping you as soon I have a patch up there.

https://bugs.webkit.org/show_bug.cgi?id=184933

This current bug purpose is to have the actual removal of the old runner.

> Can you please write the detailed ChangeLog?

sure!

Thanks for the feedback!
Comment 15 Leo Balter 2018-05-09 11:00:45 PDT
Created attachment 339984 [details]
Patch
Comment 16 Aakash Jain 2018-05-09 11:05:51 PDT
Let's make sure that the new test-runner finishes at-least few successful runs on build.wekbit.org before landing this.
Comment 17 Leo Balter 2018-05-09 11:15:48 PDT
Created attachment 339989 [details]
Remove JSTests/test262.yaml
Comment 18 Leo Balter 2018-05-09 11:16:38 PDT
> Let's make sure that the new test-runner finishes at-least few successful runs on build.wekbit.org before landing this.

Of course, I was just rebasing the changes to reflect the Bug 184933.
Comment 19 Build Bot 2018-05-09 12:46:22 PDT
Comment on attachment 339984 [details]
Patch

Attachment 339984 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/7626540

New failing tests:
http/tests/security/XFrameOptions/x-frame-options-deny-multiple-clients.html
Comment 20 Build Bot 2018-05-09 12:46:24 PDT
Created attachment 340006 [details]
Archive of layout-test-results from ews104 for mac-sierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews104  Port: mac-sierra-wk2  Platform: Mac OS X 10.12.6