Bug 183343 – Perl-based Test262 runner

• Status: RESOLVED
• Resolution: FIXED
• Component: JavaScriptCore

[Leo Balter]

Opening this issue to track the development of a new Perl-based Test262 runner to be integrated with the JSC test tools.

[Leo Balter]

Created attachment 335026 [details]
Patch

[Leo Balter]

The current patch represent an initial work in progress and it should not be considered complete. I should send incremental patches for the development of this tool.

[Leo Balter]

Created attachment 335149 [details]
Patch

[Leo Balter]

Created attachment 335150 [details]
Patch

[Leo Balter]

Created attachment 335349 [details]
Patch

[Michael Saboff]

When I follow the instructions in Tools/Scripts/test262-helpers/README.md, I get a build failure running perlbrew install perl-5.8.8.  I have tried this on two different systems.

The last line of ~/perl5/perlbrew/build.perl-5.8.8.log are:
pod/emptycmd................................ok
pod/find....................................ok
pod/for.....................................ok
pod/headings................................ok
pod/include.................................ok
pod/included................................ok
pod/lref....................................ok
pod/multiline_items.........................ok
pod/nested_items............................ok
pod/nested_seqs.............................ok
pod/oneline_cmds............................ok
pod/plainer.................................ok
pod/pod2usage...............................ok
pod/pod2usage2..............................ok
pod/poderrs.................................ok
pod/podselect...............................ok
pod/special_seqs............................ok
x2p/s2p.....................................ok
Failed Test     Stat Wstat Total Fail  Failed  List of Failed
-------------------------------------------------------------------------------
../lib/locale.t              117    1   0.85%  99
op/pwent.t                     2    1  50.00%  1
 (1 subtest UNEXPECTEDLY SUCCEEDED), 58 tests and 228 subtests skipped.
Failed 2/996 test scripts, 99.80% okay. 2/117454 subtests failed, 100.00% okay.
make[2]: *** [_test_tty] Error 25
make[1]: *** [_test] Error 2
make: *** [test_harness] Error 2
##### Brew Failed #####

------
I see these failure notices in the log file:
...
op/pwent....................................FAILED test 1
        Failed 1/2 tests, 50.00% okay
...
-- and --
...
# None of your locales were broken.
FAILED test 99
        Failed 1/117 tests, 99.15% okay
../lib/Math/BigInt/t/bare_mbf...............ok
...

[Leo Balter]

This is a known issue from Perlbrew at least since 2017. It's observable on many versions of Perl you might want to install.

There are some workarounds for this but none are completely satisfying until I have something tested.

One of them is skipping the tests after compiling the installed version as in `perlbrew install --notests 5.8.8`, this is far from good but quick for getting things working. I've been using this for some versions I have installed in my machines.

The other alternative is locally installing `berkeley-db` before installing Perl through Perlbrew, if you have Homebrew https://brew.sh/ available it should be fast as `brew install berkeley-db`. 

While I have this Berkeley DB installed in another personal machine, I'm reinstalling everything in my work laptop to confirm it works for me before updating the docs to reflect Berkeley DB as a dependency.

I'm also assuming we should only need Perlbrew for development environments, as 5.8.8 would be available for the target environments, right?

References: 

http://www.perlmonks.org/?node_id=1204276

https://origin-discussions-us.apple.com/message/32430391#32430391

[Leo Balter]

I also found this article from 2007 with similar failures you found.

https://wincent.com/wiki/Building_Perl_5.8.8_on_Mac_OS_X_Leopard

[Leo Balter]

Created attachment 335735 [details]
Patch

[Leo Balter]

Created attachment 335791 [details]
Patch

[Michael Saboff]

I followed the README.md steps, using `perlbrew install --notests 5.8.8` to install 5.8.8.  I go to execute the test runner and it complains that it can't find YAML.pm.

$ carton exec Tools/Scripts/test262-helpers/test262-runner.pl
Can't locate YAML.pm in @INC (@INC contains: /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/local/lib/perl5/darwin-2level /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/local/lib/perl5 /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/lib . /Volumes/Data/src/webkit/local/lib/perl5 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/5.8.8/darwin-2level /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/5.8.8 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl/5.8.8/darwin-2level /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl/5.8.8 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl .) at Tools/Scripts/test262-helpers/test262-runner.pl line 58.
BEGIN failed--compilation aborted at Tools/Scripts/test262-helpers/test262-runner.pl line 58.

How do I install YAML.pm?  Or is there something else I need to do?

I have a greater fundamental question and that is what is the reason for this patch?  We already have a valid means for running test262 tests using run-jsc-stress-tests.  Is this intended for you or others to use s common perl based harness to run test262?  Given the difficulty to set this up, I doubt that many JavaScriptCore developers will use this harness and therefore it will be subject to bit rot.

[Leo Balter]

Created attachment 336366 [details]
Patch

[Leo Balter]

I just changed the readme file a bit to clarify this step.

After you install 5.8.8 with Perlbrew and the Carton module, you need to install the dependencies from cpanfile:

`carton install` is the missing step before the `carton exec test262-runner.pl` call.

I'll check how to bundle the dependencies with this patch, so we could skip the dependencies part.

In fact, this code should also run with any version of Perl <= 5.8.8, so I should also inform the perlbrew parts can be skipped. We can't skip the modules part as we still need a parser for the Yaml and the Parallel Fork manager.

[Leo Balter]

Created attachment 336431 [details]
Patch

[Michael Saboff]

I ran `carton install` before trying to exec the runner and still got the same error:

$ carton install
Installing modules using /Volumes/Data/src/webkit/cpanfile
Complete! Modules were installed into /Volumes/Data/src/webkit/local
$ carton exec Tools/Scripts/test262-helpers/test262-runner.pl
Can't locate YAML.pm in @INC (@INC contains: /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/local/lib/perl5/darwin-2level /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/local/lib/perl5 /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/lib . /Volumes/Data/src/webkit/local/lib/perl5 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/5.8.8/darwin-2level /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/5.8.8 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl/5.8.8/darwin-2level /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl/5.8.8 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl .) at Tools/Scripts/test262-helpers/test262-runner.pl line 58.
BEGIN failed--compilation aborted at Tools/Scripts/test262-helpers/test262-runner.pl line 58.

[Leo Balter]

I believe it's the best to prioritize running this script without installing any extras.

We might still use carton and Perlbrew to develop this script. We assert Perl 5.8.8 is properly used to match requirements. Carton can assist us if we ever need to install new dependencies. Other than that, the script should run from an single call.

It's an advantage for us that any newer version of Perl 5.x is retro compatible, so - in the best theory - we don't need to worry running the script with any Perl we have available.

Michael Saboff, thanks for the feedback, I'll get back to you here when I have this script running without requirements to install dependencies or running from a specific folder.

btw, the current reported bug was a quite silent error due to calling `carton install` and `carton exec` from the project's root folder and not from the folder where the script is located. The reason for this is because it's still a WIP and I didn't have this part in my priority list before. I appreciate the patience, thou.

[Leo Balter]

Created attachment 336854 [details]
Patch

[Leo Balter]

The current patch is working without a need to install Perlbrew or Carton. It also doesn't require switching to Perl 5.8.8 if you're using any newer version.

It's still in progress and there are parts to be finished but it provides an idea of it running already.

As we're bundling the dependencies together, we also need to fix the respective files to match the linter for whitespace rules. Mostly changing tabs to spaces.

[Leo Balter]

Created attachment 336860 [details]
Patch

[Leo Balter]

Created attachment 337110 [details]
Patch

[Leo Balter]

We should have a new patch update tomorrow with an improved usage documentation.

[Michael Saboff]

I tried the current patch with the command line:
$ Tools/Scripts/test262-helpers/test262-runner.pl 

and it appears that the runner is not properly setting DYLD_FRAMEWORK_PATH as we get the following dld errors:

Using the following jsc path: /Volumes/Data/src/webkit/WebKitBuild/Release/jsc
dyld: Symbol not found: __ZN3JSCdyld: Symbol not found: __ZN3JSC24DOMAttributeGetterSetter6s_inf24DOMAttributeGetterSetter6s_infoE
  Referenced from: /Volumes/DoE
  Referenced from: /Volumes/Data/src/webkit/WebKitBuild/Releaata/src/webkit/WebKitBuild/Release/jsc
  Expected in: /System/Lise/jsc
  Expected in: /System/Library/Frameworks/JavaScriptCore.brary/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCframework/Versions/A/JavaScriptCore
 in /Volumes/Data/src/webkitore
 in /Volumes/Data/src/webkit/WebKitBuild/Release/jsc
/WebKitBuild/Release/jsc
dyld: Symbol not found: __ZN3JSC24DOMAttributeGetterSetter6s_infoE
  Referenced from: /Volumes/Data/src/webkit/WebKitBuild/Release/jsc
  Expected in: /System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
 in /Volumes/Data/src/webkit/WebKitBuild/Release/jsc

...

The runner does recognize the local build, but not the framework location.  If I start the runner with the -j option and specify the jsc in the framework, the runner appears to work.
$ Tools/Scripts/test262-helpers/test262-runner.pl -j /Volumes/Data/src/webkit/WebKitBuild/Release/JavaScriptCore.framework/Resources/jsc

[Leo Balter]

AFAIK, setting DYLD_FRAMEWORK_PATH is a configuration necessary for running JSC itself, not really the runner.

I'm happy to set it in the runner when it's not defined.

---

I'm also uploading a patch with an updated readme file.

[Leo Balter]

Created attachment 337208 [details]
Patch

[Leo Balter]

Created attachment 337219 [details]
Patch

[Leo Balter]

Thanks for the feedback on the DYLD_FRAMEWORK_PATH! It's now set and it should fix the error you found previously.

[Leo Balter]

Created attachment 337301 [details]
Patch

[Leo Balter]

Created attachment 337390 [details]
Patch

[Leo Balter]

Created attachment 337548 [details]
Patch

[Daniel Bates]

(In reply to Leo Balter from comment #0)
> Opening this issue to track the development of a new Perl-based Test262
> runner to be integrated with the JSC test tools.

Can you please elaborate on why we want to write a new Test262 runner in Perl. What is wrong with the current runner?

[Leo Balter]

Created attachment 337713 [details]
Patch

[Leo Balter]

Created attachment 337921 [details]
Patch

[Michael Saboff]

I can't get to the end of the patch to select the review or commit flags.  I think the problem is due to Tools/Scripts/test262/test262-results.yaml being over 7MB.  Could you pull this results file out of the patch so I can review and land.  We'll then check in the results manually.

If would be good to shrink the results.yaml, maybe into separate files per test sections, e.g. test/built-ins/Array,  test/built-ins/ArrayIteratorPrototype, ...

[Leo Balter]

Created attachment 338026 [details]
Patch

[Leo Balter]

> I can't get to the end of the patch to select the review or commit flags.  I think the problem is due to Tools/Scripts/test262/test262-results.yaml being over 7MB.  Could you pull this results file out of the patch so I can review and land.  We'll then check in the results manually.

I just pushed a new patch without the results yaml file. It took me a bit more than I expected as I had to manually flag the previous patch.

> If would be good to shrink the results.yaml, maybe into separate files per test sections, e.g. test/built-ins/Array,  test/built-ins/ArrayIteratorPrototype, ...

Yes, the next steps we have ahead are to verify optimizations like these and this is indeed a very large file that should be really hard to read in any diff.

[Michael Saboff]

Comment on attachment 338026 [details]
Patch

Needed to delete extra ChangeLog.  Will land after that edit.

[Michael Saboff]

Committed r230680: <https://trac.webkit.org/changeset/230680>

[Radar WebKit Bug Importer]

<rdar://problem/39468619>

[Leo Balter]

Reopening to attach new patch.

[Leo Balter]

Created attachment 338040 [details]
Patch

[Leo Balter]

Oops, I sent another patch through the CLI with a fix for the ChangeLog, at least I got no other changes it this should be fine

[Alex Christensen]

Comment on attachment 338040 [details]
Patch

This has been requesting review for more than one year.  If this is still needed, please rebase and re-request review.