NEW 171578
run-javascriptcore-tests misses tests JSTests/asyncFunctionTests.yaml JSTests/heapProfiler.yaml and JSTests/test262.yaml 
https://bugs.webkit.org/show_bug.cgi?id=171578
Summary run-javascriptcore-tests misses tests JSTests/asyncFunctionTests.yaml JSTests...
Carlos Alberto Lopez Perez
Reported 2017-05-02 15:25:01 PDT
The script run-javascript-tests has a list of tests that it should run: https://trac.webkit.org/browser/trunk/Tools/Scripts/run-javascriptcore-tests?rev=215108#L268 On this list, currently there are 3 tests missing: JSTests/asyncFunctionTests.yaml JSTests/heapProfiler.yaml JSTests/test262.yaml I have checked this tests, and all 3 work fine. So this seems an overlook. In bug 170523, new bots were added to just run the JSTests/test262.yaml alone for the Mac port instead of adding JSTests/test262.yaml to the list of tests that the JSC bots run with the script run-javascript-tests. The other two tests (JSTests/asyncFunctionTests.yaml and JSTests/heapProfiler.yaml) are currently not tested by any bot. In order to fix this, my proposal is the following: 1) Remove the list of tests to run from run-javascriptcore-tests 2) Add instead a list of tests to be skipped 3) Make the script auto-discover current tests by looking at *.yaml files on the test directories (shell example: find JSTests/ LayoutTests/ PerformanceTests/ -type f -name \*.yaml ) So the tool will be automatically running all the discovered tests except the ones skipped. This will ensure we don't hit this situation in the future of JavaScriptCore tests that are not ran by any bot. This is similar to how we run the layout tests (tests to run are auto-discovered from the filesystem). The skipped list should contain JSTests/test262.yaml (this test is currently run in another step on the Mac bots). - The skipped list could be empty if the Mac port maintainers think that it would be a good idea to run JSTests/test262.yaml as part of run-javascript-tests, and migrate the curent test262 bots to general JSC testers (A build slave can have several workers). In any case this should be done in another bug.
Attachments
Add attachment proposed patch, testcase, etc.
Alexey Proskuryakov
Comment 1 2017-05-02 22:51:53 PDT
We do want tests to run fast, so adding more tests suites to the same bots should not be the norm.
Joseph Pecoraro
Comment 2 2017-05-03 12:25:55 PDT
(In reply to Alexey Proskuryakov from comment #1) > We do want tests to run fast, so adding more tests suites to the same bots > should not be the norm. I don't think that applies here, these are very small suites. > JSTests/asyncFunctionTests.yaml On my machine this takes 4 seconds to run. > JSTests/heapProfiler.yaml On my machine this takes 2 seconds to run.
Note You need to log in before you can comment on or make changes to this bug.