Bug 184934 – Detach the old Test262 Runner from the stress tests

• Status: NEW
• Component: JavaScriptCore

[Leo Balter]

With the new Test262 Runner and import scripts (Tools/Scripts/test262-runner and Tools/Script/test262-import), the old runner should be all set to be removed from the stress tests script.

[Leo Balter]

Created attachment 338795 [details]
Patch

[Leo Balter]

I'm rebasing this one with the most recent changes in the Test262 Runner

[Leo Balter]

Created attachment 339547 [details]
Patch

[Leo Balter]

I uploaded a new patch without removing the JSTests/test262.yaml to allow an easier review

[Leo Balter]

Created attachment 339551 [details]
Remove JSTests/test262.yaml

This change is in a separate to simplify the review process.

[EWS Watchlist]

Comment on attachment 339547 [details]
Patch

Attachment 339547 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/7564930

New failing tests:
fast/mediastream/delayed-permission-allowed.html

[EWS Watchlist]

Created attachment 339554 [details]
Archive of layout-test-results from ews106 for mac-sierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews106  Port: mac-sierra-wk2  Platform: Mac OS X 10.12.6

[Leo Balter]

It feels like the failures are unrelated to the actual patch, I'll confirm this.

[Leo Balter]

Created attachment 339566 [details]
Patch

[Leo Balter]

Created attachment 339567 [details]
Remove JSTests/test262.yaml

[Aakash Jain]

Comment on attachment 339566 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=339566&action=review

> Tools/ChangeLog:9
> +        (RunTest262Tests):

Can you please write the detailed ChangeLog?

[Aakash Jain]

(In reply to Leo Balter from comment #10)
> Created attachment 339567 [details]
> Remove JSTests/test262.yaml

Can the removal of old test-runner related files be done in a separate patch after the switch is complete?

[Aakash Jain]

Can you please mention why are we removing old test runner and using new one?

[Leo Balter]

> Can you please mention why are we removing old test runner and using new one?

I'm not sure if should mention this here or through the changelog. In any case, I'll try both, one should be right.

I can summarize some of the goals for the new Test262 Runner:

- It's configurable, with mechanisms to handled new imported files, failing tests, etc.
- It eases the process of importing new Test262 content, where skipped and known failing tests are kept and flagged without a hassle.
- It's compatible with any target for running the Test262 tests locally through the CI bots.
- It's also detached from the stress tests as Test262 has a different content and purpose (specs conformance, mostly).

> Can the removal of old test-runner related files be done in a separate patch after the switch is complete?

I this case I should use the bug 184933 for the switch only and then we follow up on this one. I'm happy to fix this and I'll ping you as soon I have a patch up there.

https://bugs.webkit.org/show_bug.cgi?id=184933

This current bug purpose is to have the actual removal of the old runner.

> Can you please write the detailed ChangeLog?

sure!

Thanks for the feedback!

[Leo Balter]

Created attachment 339984 [details]
Patch

[Aakash Jain]

Let's make sure that the new test-runner finishes at-least few successful runs on build.wekbit.org before landing this.

[Leo Balter]

Created attachment 339989 [details]
Remove JSTests/test262.yaml

[Leo Balter]

> Let's make sure that the new test-runner finishes at-least few successful runs on build.wekbit.org before landing this.

Of course, I was just rebasing the changes to reflect the Bug 184933.

[EWS Watchlist]

Comment on attachment 339984 [details]
Patch

Attachment 339984 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/7626540

New failing tests:
http/tests/security/XFrameOptions/x-frame-options-deny-multiple-clients.html

[EWS Watchlist]

Created attachment 340006 [details]
Archive of layout-test-results from ews104 for mac-sierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews104  Port: mac-sierra-wk2  Platform: Mac OS X 10.12.6