Bug 171578 - run-javascriptcore-tests misses tests JSTests/asyncFunctionTests.yaml JSTests/heapProfiler.yaml and JSTests/test262.yaml
Summary: run-javascriptcore-tests misses tests JSTests/asyncFunctionTests.yaml JSTests...
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-05-02 15:25 PDT by Carlos Alberto Lopez Perez
Modified: 2018-02-07 09:29 PST (History)
9 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Carlos Alberto Lopez Perez 2017-05-02 15:25:01 PDT
The script run-javascript-tests has a list of tests that it should run:
https://trac.webkit.org/browser/trunk/Tools/Scripts/run-javascriptcore-tests?rev=215108#L268

On this list, currently there are 3 tests missing:

JSTests/asyncFunctionTests.yaml
JSTests/heapProfiler.yaml
JSTests/test262.yaml

I have checked this tests, and all 3 work fine. So this seems an overlook.

In bug 170523, new bots were added to just run the JSTests/test262.yaml alone for the Mac port instead of adding JSTests/test262.yaml to the list of tests that the JSC bots run with the script run-javascript-tests.

The other two tests (JSTests/asyncFunctionTests.yaml and JSTests/heapProfiler.yaml) are currently not tested by any bot.


In order to fix this, my proposal is the following:
 1) Remove the list of tests to run from run-javascriptcore-tests
 2) Add instead a list of tests to be skipped
 3) Make the script auto-discover current tests by looking at *.yaml files on the test directories
    (shell example: find JSTests/ LayoutTests/ PerformanceTests/ -type f -name \*.yaml )

So the tool will be automatically running all the discovered tests except the ones skipped.

This will ensure we don't hit this situation in the future of JavaScriptCore tests that are not ran by any bot.
This is similar to how we run the layout tests (tests to run are auto-discovered from the filesystem).

The skipped list should contain JSTests/test262.yaml (this test is currently run in another step on the Mac bots).
 - The skipped list could be empty if the Mac port maintainers think that it would be a good idea to run JSTests/test262.yaml as part of run-javascript-tests, and migrate the curent test262 bots to general JSC testers (A build slave can have several workers). In any case this should be done in another bug.
Comment 1 Alexey Proskuryakov 2017-05-02 22:51:53 PDT
We do want tests to run fast, so adding more tests suites to the same bots should not be the norm.
Comment 2 Joseph Pecoraro 2017-05-03 12:25:55 PDT
(In reply to Alexey Proskuryakov from comment #1)
> We do want tests to run fast, so adding more tests suites to the same bots
> should not be the norm.

I don't think that applies here, these are very small suites.

> JSTests/asyncFunctionTests.yaml

On my machine this takes 4 seconds to run.

> JSTests/heapProfiler.yaml

On my machine this takes 2 seconds to run.