Opening this issue to track the development of a new Perl-based Test262 runner to be integrated with the JSC test tools.
Created attachment 335026 [details] Patch
The current patch represent an initial work in progress and it should not be considered complete. I should send incremental patches for the development of this tool.
Created attachment 335149 [details] Patch
Created attachment 335150 [details] Patch
Created attachment 335349 [details] Patch
When I follow the instructions in Tools/Scripts/test262-helpers/README.md, I get a build failure running perlbrew install perl-5.8.8. I have tried this on two different systems. The last line of ~/perl5/perlbrew/build.perl-5.8.8.log are: pod/emptycmd................................ok pod/find....................................ok pod/for.....................................ok pod/headings................................ok pod/include.................................ok pod/included................................ok pod/lref....................................ok pod/multiline_items.........................ok pod/nested_items............................ok pod/nested_seqs.............................ok pod/oneline_cmds............................ok pod/plainer.................................ok pod/pod2usage...............................ok pod/pod2usage2..............................ok pod/poderrs.................................ok pod/podselect...............................ok pod/special_seqs............................ok x2p/s2p.....................................ok Failed Test Stat Wstat Total Fail Failed List of Failed ------------------------------------------------------------------------------- ../lib/locale.t 117 1 0.85% 99 op/pwent.t 2 1 50.00% 1 (1 subtest UNEXPECTEDLY SUCCEEDED), 58 tests and 228 subtests skipped. Failed 2/996 test scripts, 99.80% okay. 2/117454 subtests failed, 100.00% okay. make[2]: *** [_test_tty] Error 25 make[1]: *** [_test] Error 2 make: *** [test_harness] Error 2 ##### Brew Failed ##### ------ I see these failure notices in the log file: ... op/pwent....................................FAILED test 1 Failed 1/2 tests, 50.00% okay ... -- and -- ... # None of your locales were broken. FAILED test 99 Failed 1/117 tests, 99.15% okay ../lib/Math/BigInt/t/bare_mbf...............ok ...
This is a known issue from Perlbrew at least since 2017. It's observable on many versions of Perl you might want to install. There are some workarounds for this but none are completely satisfying until I have something tested. One of them is skipping the tests after compiling the installed version as in `perlbrew install --notests 5.8.8`, this is far from good but quick for getting things working. I've been using this for some versions I have installed in my machines. The other alternative is locally installing `berkeley-db` before installing Perl through Perlbrew, if you have Homebrew https://brew.sh/ available it should be fast as `brew install berkeley-db`. While I have this Berkeley DB installed in another personal machine, I'm reinstalling everything in my work laptop to confirm it works for me before updating the docs to reflect Berkeley DB as a dependency. I'm also assuming we should only need Perlbrew for development environments, as 5.8.8 would be available for the target environments, right? References: http://www.perlmonks.org/?node_id=1204276 https://origin-discussions-us.apple.com/message/32430391#32430391
I also found this article from 2007 with similar failures you found. https://wincent.com/wiki/Building_Perl_5.8.8_on_Mac_OS_X_Leopard
Created attachment 335735 [details] Patch
Created attachment 335791 [details] Patch
I followed the README.md steps, using `perlbrew install --notests 5.8.8` to install 5.8.8. I go to execute the test runner and it complains that it can't find YAML.pm. $ carton exec Tools/Scripts/test262-helpers/test262-runner.pl Can't locate YAML.pm in @INC (@INC contains: /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/local/lib/perl5/darwin-2level /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/local/lib/perl5 /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/lib . /Volumes/Data/src/webkit/local/lib/perl5 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/5.8.8/darwin-2level /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/5.8.8 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl/5.8.8/darwin-2level /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl/5.8.8 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl .) at Tools/Scripts/test262-helpers/test262-runner.pl line 58. BEGIN failed--compilation aborted at Tools/Scripts/test262-helpers/test262-runner.pl line 58. How do I install YAML.pm? Or is there something else I need to do? I have a greater fundamental question and that is what is the reason for this patch? We already have a valid means for running test262 tests using run-jsc-stress-tests. Is this intended for you or others to use s common perl based harness to run test262? Given the difficulty to set this up, I doubt that many JavaScriptCore developers will use this harness and therefore it will be subject to bit rot.
Created attachment 336366 [details] Patch
I just changed the readme file a bit to clarify this step. After you install 5.8.8 with Perlbrew and the Carton module, you need to install the dependencies from cpanfile: `carton install` is the missing step before the `carton exec test262-runner.pl` call. I'll check how to bundle the dependencies with this patch, so we could skip the dependencies part. In fact, this code should also run with any version of Perl <= 5.8.8, so I should also inform the perlbrew parts can be skipped. We can't skip the modules part as we still need a parser for the Yaml and the Parallel Fork manager.
Created attachment 336431 [details] Patch
I ran `carton install` before trying to exec the runner and still got the same error: $ carton install Installing modules using /Volumes/Data/src/webkit/cpanfile Complete! Modules were installed into /Volumes/Data/src/webkit/local $ carton exec Tools/Scripts/test262-helpers/test262-runner.pl Can't locate YAML.pm in @INC (@INC contains: /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/local/lib/perl5/darwin-2level /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/local/lib/perl5 /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/lib . /Volumes/Data/src/webkit/local/lib/perl5 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/5.8.8/darwin-2level /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/5.8.8 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl/5.8.8/darwin-2level /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl/5.8.8 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl .) at Tools/Scripts/test262-helpers/test262-runner.pl line 58. BEGIN failed--compilation aborted at Tools/Scripts/test262-helpers/test262-runner.pl line 58.
I believe it's the best to prioritize running this script without installing any extras. We might still use carton and Perlbrew to develop this script. We assert Perl 5.8.8 is properly used to match requirements. Carton can assist us if we ever need to install new dependencies. Other than that, the script should run from an single call. It's an advantage for us that any newer version of Perl 5.x is retro compatible, so - in the best theory - we don't need to worry running the script with any Perl we have available. Michael Saboff, thanks for the feedback, I'll get back to you here when I have this script running without requirements to install dependencies or running from a specific folder. btw, the current reported bug was a quite silent error due to calling `carton install` and `carton exec` from the project's root folder and not from the folder where the script is located. The reason for this is because it's still a WIP and I didn't have this part in my priority list before. I appreciate the patience, thou.
Created attachment 336854 [details] Patch
The current patch is working without a need to install Perlbrew or Carton. It also doesn't require switching to Perl 5.8.8 if you're using any newer version. It's still in progress and there are parts to be finished but it provides an idea of it running already. As we're bundling the dependencies together, we also need to fix the respective files to match the linter for whitespace rules. Mostly changing tabs to spaces.
Created attachment 336860 [details] Patch
Created attachment 337110 [details] Patch
We should have a new patch update tomorrow with an improved usage documentation.
I tried the current patch with the command line: $ Tools/Scripts/test262-helpers/test262-runner.pl and it appears that the runner is not properly setting DYLD_FRAMEWORK_PATH as we get the following dld errors: Using the following jsc path: /Volumes/Data/src/webkit/WebKitBuild/Release/jsc dyld: Symbol not found: __ZN3JSCdyld: Symbol not found: __ZN3JSC24DOMAttributeGetterSetter6s_inf24DOMAttributeGetterSetter6s_infoE Referenced from: /Volumes/DoE Referenced from: /Volumes/Data/src/webkit/WebKitBuild/Releaata/src/webkit/WebKitBuild/Release/jsc Expected in: /System/Lise/jsc Expected in: /System/Library/Frameworks/JavaScriptCore.brary/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCframework/Versions/A/JavaScriptCore in /Volumes/Data/src/webkitore in /Volumes/Data/src/webkit/WebKitBuild/Release/jsc /WebKitBuild/Release/jsc dyld: Symbol not found: __ZN3JSC24DOMAttributeGetterSetter6s_infoE Referenced from: /Volumes/Data/src/webkit/WebKitBuild/Release/jsc Expected in: /System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore in /Volumes/Data/src/webkit/WebKitBuild/Release/jsc ... The runner does recognize the local build, but not the framework location. If I start the runner with the -j option and specify the jsc in the framework, the runner appears to work. $ Tools/Scripts/test262-helpers/test262-runner.pl -j /Volumes/Data/src/webkit/WebKitBuild/Release/JavaScriptCore.framework/Resources/jsc
AFAIK, setting DYLD_FRAMEWORK_PATH is a configuration necessary for running JSC itself, not really the runner. I'm happy to set it in the runner when it's not defined. --- I'm also uploading a patch with an updated readme file.
Created attachment 337208 [details] Patch
Created attachment 337219 [details] Patch
Thanks for the feedback on the DYLD_FRAMEWORK_PATH! It's now set and it should fix the error you found previously.
Created attachment 337301 [details] Patch
Created attachment 337390 [details] Patch
Created attachment 337548 [details] Patch
(In reply to Leo Balter from comment #0) > Opening this issue to track the development of a new Perl-based Test262 > runner to be integrated with the JSC test tools. Can you please elaborate on why we want to write a new Test262 runner in Perl. What is wrong with the current runner?
Created attachment 337713 [details] Patch
Created attachment 337921 [details] Patch
I can't get to the end of the patch to select the review or commit flags. I think the problem is due to Tools/Scripts/test262/test262-results.yaml being over 7MB. Could you pull this results file out of the patch so I can review and land. We'll then check in the results manually. If would be good to shrink the results.yaml, maybe into separate files per test sections, e.g. test/built-ins/Array, test/built-ins/ArrayIteratorPrototype, ...
Created attachment 338026 [details] Patch
> I can't get to the end of the patch to select the review or commit flags. I think the problem is due to Tools/Scripts/test262/test262-results.yaml being over 7MB. Could you pull this results file out of the patch so I can review and land. We'll then check in the results manually. I just pushed a new patch without the results yaml file. It took me a bit more than I expected as I had to manually flag the previous patch. > If would be good to shrink the results.yaml, maybe into separate files per test sections, e.g. test/built-ins/Array, test/built-ins/ArrayIteratorPrototype, ... Yes, the next steps we have ahead are to verify optimizations like these and this is indeed a very large file that should be really hard to read in any diff.
Comment on attachment 338026 [details] Patch Needed to delete extra ChangeLog. Will land after that edit.
Committed r230680: <https://trac.webkit.org/changeset/230680>
<rdar://problem/39468619>
Reopening to attach new patch.
Created attachment 338040 [details] Patch
Oops, I sent another patch through the CLI with a fix for the ChangeLog, at least I got no other changes it this should be fine
Comment on attachment 338040 [details] Patch This has been requesting review for more than one year. If this is still needed, please rebase and re-request review.