Bug 125111 – [Win] run-jsc-stress-tests has a great number of failures (2026/7606) on Windows

• Status: RESOLVED
• Resolution: FIXED
• Component: JavaScriptCore

[Brent Fulgham]

When I run the jsc stress tests, I get a very large number of failures.

1. Profiling output seems to be unavailable.
2. DFG tests all fail; need a way to disable them for non-DFG builds.
3. LINT-specific tests won't work.

[Brent Fulgham]

Created attachment 218233 [details]
List of failed tests

Unfortunately, the error report doesn't show much information about why tests are fail; only the list of failed test files is shown.

[Brent Fulgham]

It looks like 828 of the failures are due to missing DFG support.

[Brent Fulgham]

(In reply to comment #2)
> It looks like 828 of the failures are due to missing DFG support.

Landing Bug 123615 might clear these up.

[Geoffrey Garen]

> Landing Bug 123615 might clear these up.

I think that would muddy things up. The best way to clear up DFG-only tests failing when you run a non-DFG build is to mark the tests as DFG-only. Hiding the problem by turning on the DFG in one configuration will not help.

[Brent Fulgham]

(In reply to comment #4)
> > Landing Bug 123615 might clear these up.
> 
> I think that would muddy things up. The best way to clear up DFG-only tests failing when you run a non-DFG build is to mark the tests as DFG-only. Hiding the problem by turning on the DFG in one configuration will not help.

For what it's worth, I tried this on my local machine and didn't see any improvement in results. So either I didn't actually activate DFG, or these failures are not actually caused by DFG being turned off. They may be meant to exercise the DFG implementation, but might still work using one of the other backends.

I'm getting similar results on WinCairo with DFG enabled (7552 tests):

Results for JSC stress tests:
    2004 failures found.

I'm also getting these warnings at startup:

Warning: did not find json or highline; some features will be disabled.
Error: #<LoadError: cannot load such file -- highline>
Warning: cannot identify JSC framework, doing generic VM copy.

Should these warnings be fixed?

[Filip Pizlo]

(In reply to comment #4)
> > Landing Bug 123615 might clear these up.
> 
> I think that would muddy things up. The best way to clear up DFG-only tests failing when you run a non-DFG build is to mark the tests as DFG-only. Hiding the problem by turning on the DFG in one configuration will not help.

We shouldn't have DFG-only tests.  Can someone clarify what is meant by "DFG tests all fail"?

Created attachment 218342 [details]
Patch

Most of the errors seems to be caused by differences in line endings :)

7552/7552 (failed 12)

** The following JSC stress test failures have been introduced:
        mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla
        mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla-llint
        mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla-baseline
        mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla-dfg-eager-no-cjit-validate-phases
        jsc-layout-tests.yaml/js/script-tests/math.js.layout
        jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-llint
        jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-cjit
        jsc-layout-tests.yaml/js/script-tests/math.js.layout-dfg-eager-no-cjit
        jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout
        jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-llint
        jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-cjit
        jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-dfg-eager-no-cjit

Results for JSC stress tests:
    12 failures found.

[Brent Fulgham]

(In reply to comment #7)
> (In reply to comment #4)
> > > Landing Bug 123615 might clear these up.
> > 
> > I think that would muddy things up. The best way to clear up DFG-only tests failing when you run a non-DFG build is to mark the tests as DFG-only. Hiding the problem by turning on the DFG in one configuration will not help.
> 
> We shouldn't have DFG-only tests.  Can someone clarify what is meant by "DFG tests all fail"?

Just to follow-up on our hallway conversation, I just flagged them because they had names of the form "dfg-x-y-z".

To clarify: These tests are meant to exercise the DFG, but will still run on whatever JSC backend you are using.

[Brent Fulgham]

(In reply to comment #9)
> Most of the errors seems to be caused by differences in line endings :)

Doh!  I didn't even look into the tests yet.  Great!

> ** The following JSC stress test failures have been introduced:
>         mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla
>         mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla-llint
>         mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla-baseline
>         mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla-dfg-eager-no-cjit-validate-phases
>         jsc-layout-tests.yaml/js/script-tests/math.js.layout
>         jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-llint
>         jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-cjit
>         jsc-layout-tests.yaml/js/script-tests/math.js.layout-dfg-eager-no-cjit
>         jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout
>         jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-llint
>         jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-cjit
>         jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-dfg-eager-no-cjit
> 
> Results for JSC stress tests:
>     12 failures found.

Wonderful -- so we are only dealing with 12 actual failures.

(In reply to comment #11)
> > 
> > Results for JSC stress tests:
> >     12 failures found.
> 
> Wonderful -- so we are only dealing with 12 actual failures.

Yep, that's not so bad :) This test was with WinCairo 32-bit, DFG enabled.

[Brent Fulgham]

Comment on attachment 218342 [details]
Patch

This looks good to me, but I'll ping Filip to confirm.

[Brent Fulgham]

It looks like we do a bit worse without DFG enabled:

** The following JSC stress test failures have been introduced:
        profiler-test.yaml/tests/sunspider-1.0/3d-cube.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/3d-morph.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/3d-raytrace.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/access-binary-trees.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/access-fannkuch.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/access-nbody.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/access-nsieve.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/bitops-3bit-bits-in-byte.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/bitops-bits-in-byte.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/bitops-bitwise-and.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/bitops-nsieve-bits.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/controlflow-recursive.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/crypto-aes.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/crypto-md5.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/crypto-sha1.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/date-format-tofte.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/date-format-xparb.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/math-cordic.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/math-partial-sums.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/math-spectral-norm.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/regexp-dna.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/string-base64.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/string-fasta.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/string-tagcloud.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/string-unpack-code.js.profiler
        profiler-test.yaml/tests/sunspider-1.0/string-validate-input.js.profiler
        jsc-layout-tests.yaml/js/script-tests/math.js.layout
        jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-llint
        jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-cjit
        jsc-layout-tests.yaml/js/script-tests/math.js.layout-dfg-eager-no-cjit
        jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout
        jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-llint
        jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-cjit
        jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-dfg-eager-no-cjit

Results for JSC stress tests:
    34 failures found.

Maybe the profiler tests aren't running on your system, or my system is missing something needed by the profiling step.  Checking....

It's funny that the mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js* tests are failing on WinCairo, since they are marked as "skipped" in run-javascriptcore-tests.  Are they not getting ignored on WinCairo?

[Filip Pizlo]

(In reply to comment #14)
> It looks like we do a bit worse without DFG enabled:
> 
> ** The following JSC stress test failures have been introduced:
>         profiler-test.yaml/tests/sunspider-1.0/3d-cube.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/3d-morph.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/3d-raytrace.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/access-binary-trees.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/access-fannkuch.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/access-nbody.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/access-nsieve.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/bitops-3bit-bits-in-byte.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/bitops-bits-in-byte.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/bitops-bitwise-and.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/bitops-nsieve-bits.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/controlflow-recursive.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/crypto-aes.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/crypto-md5.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/crypto-sha1.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/date-format-tofte.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/date-format-xparb.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/math-cordic.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/math-partial-sums.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/math-spectral-norm.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/regexp-dna.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/string-base64.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/string-fasta.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/string-tagcloud.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/string-unpack-code.js.profiler
>         profiler-test.yaml/tests/sunspider-1.0/string-validate-input.js.profiler
>         jsc-layout-tests.yaml/js/script-tests/math.js.layout
>         jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-llint
>         jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-cjit
>         jsc-layout-tests.yaml/js/script-tests/math.js.layout-dfg-eager-no-cjit
>         jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout
>         jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-llint
>         jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-cjit
>         jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-dfg-eager-no-cjit
> 
> Results for JSC stress tests:
>     34 failures found.
> 
> Maybe the profiler tests aren't running on your system, or my system is missing something needed by the profiling step.  Checking....
> 
> It's funny that the mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js* tests are failing on WinCairo, since they are marked as "skipped" in run-javascriptcore-tests.  Are they not getting ignored on WinCairo?

Hmmm, they're not skipped in Tools/Scripts/create-mozilla-js-test-list, so that means that run-javascriptcore-tests will run them.

[WebKit Commit Bot]

Comment on attachment 218342 [details]
Patch

Clearing flags on attachment: 218342

Committed r160048: <http://trac.webkit.org/changeset/160048>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

(In reply to comment #14)
> It looks like we do a bit worse without DFG enabled:
> 
> ** The following JSC stress test failures have been introduced:
>         profiler-test.yaml/tests/sunspider-1.0/3d-cube.js.profiler

...
...

> Results for JSC stress tests:
>     34 failures found.
> 
> Maybe the profiler tests aren't running on your system, or my system is missing something needed by the profiling step.  Checking....
> 

I'm getting this message about the profiler tests:

Running simple version of profiler-test.yaml/tests/sunspider-1.0/3d-cube.js because some required Ruby features are unavailable.

[Filip Pizlo]

(In reply to comment #18)
> (In reply to comment #14)
> > It looks like we do a bit worse without DFG enabled:
> > 
> > ** The following JSC stress test failures have been introduced:
> >         profiler-test.yaml/tests/sunspider-1.0/3d-cube.js.profiler
> 
> ...
> ...
> 
> > Results for JSC stress tests:
> >     34 failures found.
> > 
> > Maybe the profiler tests aren't running on your system, or my system is missing something needed by the profiling step.  Checking....
> > 
> 
> I'm getting this message about the profiler tests:
> 
> Running simple version of profiler-test.yaml/tests/sunspider-1.0/3d-cube.js because some required Ruby features are unavailable.

That message is harmless. It's not an error. You will only not get this message if you have all of the Ruby features to use the profiler.