Bug 183343 - Perl-based Test262 runner
Summary: Perl-based Test262 runner
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: JavaScriptCore (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks: 184667
  Show dependency treegraph
 
Reported: 2018-03-05 13:53 PST by Leo Balter
Modified: 2018-04-16 14:08 PDT (History)
5 users (show)

See Also:


Attachments
Patch (7.67 KB, patch)
2018-03-05 14:22 PST, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (10.30 KB, patch)
2018-03-06 15:23 PST, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (10.32 KB, patch)
2018-03-06 15:25 PST, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (12.02 KB, patch)
2018-03-08 14:46 PST, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (34.09 KB, patch)
2018-03-13 15:01 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (49.97 KB, patch)
2018-03-14 13:22 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (48.07 KB, patch)
2018-03-23 06:05 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (60.62 KB, patch)
2018-03-23 15:08 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (3.41 MB, patch)
2018-03-30 08:04 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (3.42 MB, patch)
2018-03-30 09:22 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (1.50 MB, patch)
2018-04-03 13:47 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (1.50 MB, patch)
2018-04-04 12:41 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (1.50 MB, patch)
2018-04-04 13:39 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (1.51 MB, patch)
2018-04-05 14:30 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (1.51 MB, patch)
2018-04-06 14:21 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (1.52 MB, patch)
2018-04-09 15:09 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (1.93 MB, patch)
2018-04-11 10:43 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (9.42 MB, text/plain)
2018-04-13 13:38 PDT, Leo Balter
no flags Details
Patch (1.88 MB, patch)
2018-04-16 12:34 PDT, Leo Balter
no flags Details | Formatted Diff | Diff
Patch (1.88 MB, patch)
2018-04-16 14:07 PDT, Leo Balter
leo: review?
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Leo Balter 2018-03-05 13:53:15 PST
Opening this issue to track the development of a new Perl-based Test262 runner to be integrated with the JSC test tools.
Comment 1 Leo Balter 2018-03-05 14:22:39 PST
Created attachment 335026 [details]
Patch
Comment 2 Leo Balter 2018-03-05 14:24:56 PST
The current patch represent an initial work in progress and it should not be considered complete. I should send incremental patches for the development of this tool.
Comment 3 Leo Balter 2018-03-06 15:23:00 PST
Created attachment 335149 [details]
Patch
Comment 4 Leo Balter 2018-03-06 15:25:53 PST
Created attachment 335150 [details]
Patch
Comment 5 Leo Balter 2018-03-08 14:46:20 PST
Created attachment 335349 [details]
Patch
Comment 6 Michael Saboff 2018-03-09 17:43:25 PST
When I follow the instructions in Tools/Scripts/test262-helpers/README.md, I get a build failure running perlbrew install perl-5.8.8.  I have tried this on two different systems.

The last line of ~/perl5/perlbrew/build.perl-5.8.8.log are:
pod/emptycmd................................ok
pod/find....................................ok
pod/for.....................................ok
pod/headings................................ok
pod/include.................................ok
pod/included................................ok
pod/lref....................................ok
pod/multiline_items.........................ok
pod/nested_items............................ok
pod/nested_seqs.............................ok
pod/oneline_cmds............................ok
pod/plainer.................................ok
pod/pod2usage...............................ok
pod/pod2usage2..............................ok
pod/poderrs.................................ok
pod/podselect...............................ok
pod/special_seqs............................ok
x2p/s2p.....................................ok
Failed Test     Stat Wstat Total Fail  Failed  List of Failed
-------------------------------------------------------------------------------
../lib/locale.t              117    1   0.85%  99
op/pwent.t                     2    1  50.00%  1
 (1 subtest UNEXPECTEDLY SUCCEEDED), 58 tests and 228 subtests skipped.
Failed 2/996 test scripts, 99.80% okay. 2/117454 subtests failed, 100.00% okay.
make[2]: *** [_test_tty] Error 25
make[1]: *** [_test] Error 2
make: *** [test_harness] Error 2
##### Brew Failed #####

------
I see these failure notices in the log file:
...
op/pwent....................................FAILED test 1
        Failed 1/2 tests, 50.00% okay
...
-- and --
...
# None of your locales were broken.
FAILED test 99
        Failed 1/117 tests, 99.15% okay
../lib/Math/BigInt/t/bare_mbf...............ok
...
Comment 7 Leo Balter 2018-03-10 12:17:28 PST
This is a known issue from Perlbrew at least since 2017. It's observable on many versions of Perl you might want to install.

There are some workarounds for this but none are completely satisfying until I have something tested.

One of them is skipping the tests after compiling the installed version as in `perlbrew install --notests 5.8.8`, this is far from good but quick for getting things working. I've been using this for some versions I have installed in my machines.

The other alternative is locally installing `berkeley-db` before installing Perl through Perlbrew, if you have Homebrew https://brew.sh/ available it should be fast as `brew install berkeley-db`. 

While I have this Berkeley DB installed in another personal machine, I'm reinstalling everything in my work laptop to confirm it works for me before updating the docs to reflect Berkeley DB as a dependency.

I'm also assuming we should only need Perlbrew for development environments, as 5.8.8 would be available for the target environments, right?

References: 

http://www.perlmonks.org/?node_id=1204276

https://origin-discussions-us.apple.com/message/32430391#32430391
Comment 8 Leo Balter 2018-03-12 09:13:26 PDT
I also found this article from 2007 with similar failures you found.

https://wincent.com/wiki/Building_Perl_5.8.8_on_Mac_OS_X_Leopard
Comment 9 Leo Balter 2018-03-13 15:01:35 PDT
Created attachment 335735 [details]
Patch
Comment 10 Leo Balter 2018-03-14 13:22:48 PDT
Created attachment 335791 [details]
Patch
Comment 11 Michael Saboff 2018-03-23 03:09:45 PDT
I followed the README.md steps, using `perlbrew install --notests 5.8.8` to install 5.8.8.  I go to execute the test runner and it complains that it can't find YAML.pm.

$ carton exec Tools/Scripts/test262-helpers/test262-runner.pl
Can't locate YAML.pm in @INC (@INC contains: /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/local/lib/perl5/darwin-2level /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/local/lib/perl5 /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/lib . /Volumes/Data/src/webkit/local/lib/perl5 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/5.8.8/darwin-2level /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/5.8.8 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl/5.8.8/darwin-2level /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl/5.8.8 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl .) at Tools/Scripts/test262-helpers/test262-runner.pl line 58.
BEGIN failed--compilation aborted at Tools/Scripts/test262-helpers/test262-runner.pl line 58.

How do I install YAML.pm?  Or is there something else I need to do?

I have a greater fundamental question and that is what is the reason for this patch?  We already have a valid means for running test262 tests using run-jsc-stress-tests.  Is this intended for you or others to use s common perl based harness to run test262?  Given the difficulty to set this up, I doubt that many JavaScriptCore developers will use this harness and therefore it will be subject to bit rot.
Comment 12 Leo Balter 2018-03-23 06:05:57 PDT
Created attachment 336366 [details]
Patch
Comment 13 Leo Balter 2018-03-23 06:11:13 PDT
I just changed the readme file a bit to clarify this step.

After you install 5.8.8 with Perlbrew and the Carton module, you need to install the dependencies from cpanfile:

`carton install` is the missing step before the `carton exec test262-runner.pl` call.

I'll check how to bundle the dependencies with this patch, so we could skip the dependencies part.

In fact, this code should also run with any version of Perl <= 5.8.8, so I should also inform the perlbrew parts can be skipped. We can't skip the modules part as we still need a parser for the Yaml and the Parallel Fork manager.
Comment 14 Leo Balter 2018-03-23 15:08:46 PDT
Created attachment 336431 [details]
Patch
Comment 15 Michael Saboff 2018-03-26 01:53:07 PDT
I ran `carton install` before trying to exec the runner and still got the same error:

$ carton install
Installing modules using /Volumes/Data/src/webkit/cpanfile
Complete! Modules were installed into /Volumes/Data/src/webkit/local
$ carton exec Tools/Scripts/test262-helpers/test262-runner.pl
Can't locate YAML.pm in @INC (@INC contains: /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/local/lib/perl5/darwin-2level /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/local/lib/perl5 /Volumes/Data/src/webkit/Tools/Scripts/test262-helpers/lib . /Volumes/Data/src/webkit/local/lib/perl5 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/5.8.8/darwin-2level /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/5.8.8 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl/5.8.8/darwin-2level /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl/5.8.8 /Users/msaboff/perl5/perlbrew/perls/perl-5.8.8/lib/site_perl .) at Tools/Scripts/test262-helpers/test262-runner.pl line 58.
BEGIN failed--compilation aborted at Tools/Scripts/test262-helpers/test262-runner.pl line 58.
Comment 16 Leo Balter 2018-03-26 11:54:59 PDT
I believe it's the best to prioritize running this script without installing any extras.

We might still use carton and Perlbrew to develop this script. We assert Perl 5.8.8 is properly used to match requirements. Carton can assist us if we ever need to install new dependencies. Other than that, the script should run from an single call.

It's an advantage for us that any newer version of Perl 5.x is retro compatible, so - in the best theory - we don't need to worry running the script with any Perl we have available.

Michael Saboff, thanks for the feedback, I'll get back to you here when I have this script running without requirements to install dependencies or running from a specific folder.

btw, the current reported bug was a quite silent error due to calling `carton install` and `carton exec` from the project's root folder and not from the folder where the script is located. The reason for this is because it's still a WIP and I didn't have this part in my priority list before. I appreciate the patience, thou.
Comment 17 Leo Balter 2018-03-30 08:04:54 PDT
Created attachment 336854 [details]
Patch
Comment 18 Leo Balter 2018-03-30 08:08:28 PDT
The current patch is working without a need to install Perlbrew or Carton. It also doesn't require switching to Perl 5.8.8 if you're using any newer version.

It's still in progress and there are parts to be finished but it provides an idea of it running already.

As we're bundling the dependencies together, we also need to fix the respective files to match the linter for whitespace rules. Mostly changing tabs to spaces.
Comment 19 Leo Balter 2018-03-30 09:22:43 PDT
Created attachment 336860 [details]
Patch
Comment 20 Leo Balter 2018-04-03 13:47:40 PDT
Created attachment 337110 [details]
Patch
Comment 21 Leo Balter 2018-04-03 13:48:19 PDT
We should have a new patch update tomorrow with an improved usage documentation.
Comment 22 Michael Saboff 2018-04-04 11:55:20 PDT
I tried the current patch with the command line:
$ Tools/Scripts/test262-helpers/test262-runner.pl 

and it appears that the runner is not properly setting DYLD_FRAMEWORK_PATH as we get the following dld errors:

Using the following jsc path: /Volumes/Data/src/webkit/WebKitBuild/Release/jsc
dyld: Symbol not found: __ZN3JSCdyld: Symbol not found: __ZN3JSC24DOMAttributeGetterSetter6s_inf24DOMAttributeGetterSetter6s_infoE
  Referenced from: /Volumes/DoE
  Referenced from: /Volumes/Data/src/webkit/WebKitBuild/Releaata/src/webkit/WebKitBuild/Release/jsc
  Expected in: /System/Lise/jsc
  Expected in: /System/Library/Frameworks/JavaScriptCore.brary/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCframework/Versions/A/JavaScriptCore
 in /Volumes/Data/src/webkitore
 in /Volumes/Data/src/webkit/WebKitBuild/Release/jsc
/WebKitBuild/Release/jsc
dyld: Symbol not found: __ZN3JSC24DOMAttributeGetterSetter6s_infoE
  Referenced from: /Volumes/Data/src/webkit/WebKitBuild/Release/jsc
  Expected in: /System/Library/Frameworks/JavaScriptCore.framework/Versions/A/JavaScriptCore
 in /Volumes/Data/src/webkit/WebKitBuild/Release/jsc

...

The runner does recognize the local build, but not the framework location.  If I start the runner with the -j option and specify the jsc in the framework, the runner appears to work.
$ Tools/Scripts/test262-helpers/test262-runner.pl -j /Volumes/Data/src/webkit/WebKitBuild/Release/JavaScriptCore.framework/Resources/jsc
Comment 23 Leo Balter 2018-04-04 12:40:30 PDT
AFAIK, setting DYLD_FRAMEWORK_PATH is a configuration necessary for running JSC itself, not really the runner.

I'm happy to set it in the runner when it's not defined.

---

I'm also uploading a patch with an updated readme file.
Comment 24 Leo Balter 2018-04-04 12:41:49 PDT
Created attachment 337208 [details]
Patch
Comment 25 Leo Balter 2018-04-04 13:39:53 PDT
Created attachment 337219 [details]
Patch
Comment 26 Leo Balter 2018-04-04 13:41:04 PDT
Thanks for the feedback on the DYLD_FRAMEWORK_PATH! It's now set and it should fix the error you found previously.
Comment 27 Leo Balter 2018-04-05 14:30:47 PDT
Created attachment 337301 [details]
Patch
Comment 28 Leo Balter 2018-04-06 14:21:56 PDT
Created attachment 337390 [details]
Patch
Comment 29 Leo Balter 2018-04-09 15:09:25 PDT
Created attachment 337548 [details]
Patch
Comment 30 Daniel Bates 2018-04-09 22:14:19 PDT
(In reply to Leo Balter from comment #0)
> Opening this issue to track the development of a new Perl-based Test262
> runner to be integrated with the JSC test tools.

Can you please elaborate on why we want to write a new Test262 runner in Perl. What is wrong with the current runner?
Comment 31 Leo Balter 2018-04-11 10:43:34 PDT
Created attachment 337713 [details]
Patch
Comment 32 Leo Balter 2018-04-13 13:38:18 PDT
Created attachment 337921 [details]
Patch
Comment 33 Michael Saboff 2018-04-16 11:35:20 PDT
I can't get to the end of the patch to select the review or commit flags.  I think the problem is due to Tools/Scripts/test262/test262-results.yaml being over 7MB.  Could you pull this results file out of the patch so I can review and land.  We'll then check in the results manually.

If would be good to shrink the results.yaml, maybe into separate files per test sections, e.g. test/built-ins/Array,  test/built-ins/ArrayIteratorPrototype, ...
Comment 34 Leo Balter 2018-04-16 12:34:49 PDT
Created attachment 338026 [details]
Patch
Comment 35 Leo Balter 2018-04-16 12:40:51 PDT
> I can't get to the end of the patch to select the review or commit flags.  I think the problem is due to Tools/Scripts/test262/test262-results.yaml being over 7MB.  Could you pull this results file out of the patch so I can review and land.  We'll then check in the results manually.

I just pushed a new patch without the results yaml file. It took me a bit more than I expected as I had to manually flag the previous patch.

> If would be good to shrink the results.yaml, maybe into separate files per test sections, e.g. test/built-ins/Array,  test/built-ins/ArrayIteratorPrototype, ...

Yes, the next steps we have ahead are to verify optimizations like these and this is indeed a very large file that should be really hard to read in any diff.
Comment 36 Michael Saboff 2018-04-16 13:50:37 PDT
Comment on attachment 338026 [details]
Patch

Needed to delete extra ChangeLog.  Will land after that edit.
Comment 37 Michael Saboff 2018-04-16 14:04:01 PDT
Committed r230680: <https://trac.webkit.org/changeset/230680>
Comment 38 Radar WebKit Bug Importer 2018-04-16 14:06:30 PDT
<rdar://problem/39468619>
Comment 39 Leo Balter 2018-04-16 14:07:06 PDT
Reopening to attach new patch.
Comment 40 Leo Balter 2018-04-16 14:07:09 PDT
Created attachment 338040 [details]
Patch
Comment 41 Leo Balter 2018-04-16 14:08:23 PDT
Oops, I sent another patch through the CLI with a fix for the ChangeLog, at least I got no other changes it this should be fine