When I run the jsc stress tests, I get a very large number of failures. 1. Profiling output seems to be unavailable. 2. DFG tests all fail; need a way to disable them for non-DFG builds. 3. LINT-specific tests won't work.
Created attachment 218233 [details] List of failed tests Unfortunately, the error report doesn't show much information about why tests are fail; only the list of failed test files is shown.
It looks like 828 of the failures are due to missing DFG support.
(In reply to comment #2) > It looks like 828 of the failures are due to missing DFG support. Landing Bug 123615 might clear these up.
> Landing Bug 123615 might clear these up. I think that would muddy things up. The best way to clear up DFG-only tests failing when you run a non-DFG build is to mark the tests as DFG-only. Hiding the problem by turning on the DFG in one configuration will not help.
(In reply to comment #4) > > Landing Bug 123615 might clear these up. > > I think that would muddy things up. The best way to clear up DFG-only tests failing when you run a non-DFG build is to mark the tests as DFG-only. Hiding the problem by turning on the DFG in one configuration will not help. For what it's worth, I tried this on my local machine and didn't see any improvement in results. So either I didn't actually activate DFG, or these failures are not actually caused by DFG being turned off. They may be meant to exercise the DFG implementation, but might still work using one of the other backends.
I'm getting similar results on WinCairo with DFG enabled (7552 tests): Results for JSC stress tests: 2004 failures found. I'm also getting these warnings at startup: Warning: did not find json or highline; some features will be disabled. Error: #<LoadError: cannot load such file -- highline> Warning: cannot identify JSC framework, doing generic VM copy. Should these warnings be fixed?
(In reply to comment #4) > > Landing Bug 123615 might clear these up. > > I think that would muddy things up. The best way to clear up DFG-only tests failing when you run a non-DFG build is to mark the tests as DFG-only. Hiding the problem by turning on the DFG in one configuration will not help. We shouldn't have DFG-only tests. Can someone clarify what is meant by "DFG tests all fail"?
Created attachment 218342 [details] Patch
Most of the errors seems to be caused by differences in line endings :) 7552/7552 (failed 12) ** The following JSC stress test failures have been introduced: mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla-llint mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla-baseline mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla-dfg-eager-no-cjit-validate-phases jsc-layout-tests.yaml/js/script-tests/math.js.layout jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-llint jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-cjit jsc-layout-tests.yaml/js/script-tests/math.js.layout-dfg-eager-no-cjit jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-llint jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-cjit jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-dfg-eager-no-cjit Results for JSC stress tests: 12 failures found.
(In reply to comment #7) > (In reply to comment #4) > > > Landing Bug 123615 might clear these up. > > > > I think that would muddy things up. The best way to clear up DFG-only tests failing when you run a non-DFG build is to mark the tests as DFG-only. Hiding the problem by turning on the DFG in one configuration will not help. > > We shouldn't have DFG-only tests. Can someone clarify what is meant by "DFG tests all fail"? Just to follow-up on our hallway conversation, I just flagged them because they had names of the form "dfg-x-y-z". To clarify: These tests are meant to exercise the DFG, but will still run on whatever JSC backend you are using.
(In reply to comment #9) > Most of the errors seems to be caused by differences in line endings :) Doh! I didn't even look into the tests yet. Great! > ** The following JSC stress test failures have been introduced: > mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla > mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla-llint > mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla-baseline > mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js.mozilla-dfg-eager-no-cjit-validate-phases > jsc-layout-tests.yaml/js/script-tests/math.js.layout > jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-llint > jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-cjit > jsc-layout-tests.yaml/js/script-tests/math.js.layout-dfg-eager-no-cjit > jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout > jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-llint > jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-cjit > jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-dfg-eager-no-cjit > > Results for JSC stress tests: > 12 failures found. Wonderful -- so we are only dealing with 12 actual failures.
(In reply to comment #11) > > > > Results for JSC stress tests: > > 12 failures found. > > Wonderful -- so we are only dealing with 12 actual failures. Yep, that's not so bad :) This test was with WinCairo 32-bit, DFG enabled.
Comment on attachment 218342 [details] Patch This looks good to me, but I'll ping Filip to confirm.
It looks like we do a bit worse without DFG enabled: ** The following JSC stress test failures have been introduced: profiler-test.yaml/tests/sunspider-1.0/3d-cube.js.profiler profiler-test.yaml/tests/sunspider-1.0/3d-morph.js.profiler profiler-test.yaml/tests/sunspider-1.0/3d-raytrace.js.profiler profiler-test.yaml/tests/sunspider-1.0/access-binary-trees.js.profiler profiler-test.yaml/tests/sunspider-1.0/access-fannkuch.js.profiler profiler-test.yaml/tests/sunspider-1.0/access-nbody.js.profiler profiler-test.yaml/tests/sunspider-1.0/access-nsieve.js.profiler profiler-test.yaml/tests/sunspider-1.0/bitops-3bit-bits-in-byte.js.profiler profiler-test.yaml/tests/sunspider-1.0/bitops-bits-in-byte.js.profiler profiler-test.yaml/tests/sunspider-1.0/bitops-bitwise-and.js.profiler profiler-test.yaml/tests/sunspider-1.0/bitops-nsieve-bits.js.profiler profiler-test.yaml/tests/sunspider-1.0/controlflow-recursive.js.profiler profiler-test.yaml/tests/sunspider-1.0/crypto-aes.js.profiler profiler-test.yaml/tests/sunspider-1.0/crypto-md5.js.profiler profiler-test.yaml/tests/sunspider-1.0/crypto-sha1.js.profiler profiler-test.yaml/tests/sunspider-1.0/date-format-tofte.js.profiler profiler-test.yaml/tests/sunspider-1.0/date-format-xparb.js.profiler profiler-test.yaml/tests/sunspider-1.0/math-cordic.js.profiler profiler-test.yaml/tests/sunspider-1.0/math-partial-sums.js.profiler profiler-test.yaml/tests/sunspider-1.0/math-spectral-norm.js.profiler profiler-test.yaml/tests/sunspider-1.0/regexp-dna.js.profiler profiler-test.yaml/tests/sunspider-1.0/string-base64.js.profiler profiler-test.yaml/tests/sunspider-1.0/string-fasta.js.profiler profiler-test.yaml/tests/sunspider-1.0/string-tagcloud.js.profiler profiler-test.yaml/tests/sunspider-1.0/string-unpack-code.js.profiler profiler-test.yaml/tests/sunspider-1.0/string-validate-input.js.profiler jsc-layout-tests.yaml/js/script-tests/math.js.layout jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-llint jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-cjit jsc-layout-tests.yaml/js/script-tests/math.js.layout-dfg-eager-no-cjit jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-llint jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-cjit jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-dfg-eager-no-cjit Results for JSC stress tests: 34 failures found. Maybe the profiler tests aren't running on your system, or my system is missing something needed by the profiling step. Checking.... It's funny that the mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js* tests are failing on WinCairo, since they are marked as "skipped" in run-javascriptcore-tests. Are they not getting ignored on WinCairo?
(In reply to comment #14) > It looks like we do a bit worse without DFG enabled: > > ** The following JSC stress test failures have been introduced: > profiler-test.yaml/tests/sunspider-1.0/3d-cube.js.profiler > profiler-test.yaml/tests/sunspider-1.0/3d-morph.js.profiler > profiler-test.yaml/tests/sunspider-1.0/3d-raytrace.js.profiler > profiler-test.yaml/tests/sunspider-1.0/access-binary-trees.js.profiler > profiler-test.yaml/tests/sunspider-1.0/access-fannkuch.js.profiler > profiler-test.yaml/tests/sunspider-1.0/access-nbody.js.profiler > profiler-test.yaml/tests/sunspider-1.0/access-nsieve.js.profiler > profiler-test.yaml/tests/sunspider-1.0/bitops-3bit-bits-in-byte.js.profiler > profiler-test.yaml/tests/sunspider-1.0/bitops-bits-in-byte.js.profiler > profiler-test.yaml/tests/sunspider-1.0/bitops-bitwise-and.js.profiler > profiler-test.yaml/tests/sunspider-1.0/bitops-nsieve-bits.js.profiler > profiler-test.yaml/tests/sunspider-1.0/controlflow-recursive.js.profiler > profiler-test.yaml/tests/sunspider-1.0/crypto-aes.js.profiler > profiler-test.yaml/tests/sunspider-1.0/crypto-md5.js.profiler > profiler-test.yaml/tests/sunspider-1.0/crypto-sha1.js.profiler > profiler-test.yaml/tests/sunspider-1.0/date-format-tofte.js.profiler > profiler-test.yaml/tests/sunspider-1.0/date-format-xparb.js.profiler > profiler-test.yaml/tests/sunspider-1.0/math-cordic.js.profiler > profiler-test.yaml/tests/sunspider-1.0/math-partial-sums.js.profiler > profiler-test.yaml/tests/sunspider-1.0/math-spectral-norm.js.profiler > profiler-test.yaml/tests/sunspider-1.0/regexp-dna.js.profiler > profiler-test.yaml/tests/sunspider-1.0/string-base64.js.profiler > profiler-test.yaml/tests/sunspider-1.0/string-fasta.js.profiler > profiler-test.yaml/tests/sunspider-1.0/string-tagcloud.js.profiler > profiler-test.yaml/tests/sunspider-1.0/string-unpack-code.js.profiler > profiler-test.yaml/tests/sunspider-1.0/string-validate-input.js.profiler > jsc-layout-tests.yaml/js/script-tests/math.js.layout > jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-llint > jsc-layout-tests.yaml/js/script-tests/math.js.layout-no-cjit > jsc-layout-tests.yaml/js/script-tests/math.js.layout-dfg-eager-no-cjit > jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout > jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-llint > jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-no-cjit > jsc-layout-tests.yaml/js/script-tests/sort-stability.js.layout-dfg-eager-no-cjit > > Results for JSC stress tests: > 34 failures found. > > Maybe the profiler tests aren't running on your system, or my system is missing something needed by the profiling step. Checking.... > > It's funny that the mozilla-tests.yaml/ecma_3/Date/15.9.5.6.js* tests are failing on WinCairo, since they are marked as "skipped" in run-javascriptcore-tests. Are they not getting ignored on WinCairo? Hmmm, they're not skipped in Tools/Scripts/create-mozilla-js-test-list, so that means that run-javascriptcore-tests will run them.
Comment on attachment 218342 [details] Patch Clearing flags on attachment: 218342 Committed r160048: <http://trac.webkit.org/changeset/160048>
All reviewed patches have been landed. Closing bug.
(In reply to comment #14) > It looks like we do a bit worse without DFG enabled: > > ** The following JSC stress test failures have been introduced: > profiler-test.yaml/tests/sunspider-1.0/3d-cube.js.profiler ... ... > Results for JSC stress tests: > 34 failures found. > > Maybe the profiler tests aren't running on your system, or my system is missing something needed by the profiling step. Checking.... > I'm getting this message about the profiler tests: Running simple version of profiler-test.yaml/tests/sunspider-1.0/3d-cube.js because some required Ruby features are unavailable.
(In reply to comment #18) > (In reply to comment #14) > > It looks like we do a bit worse without DFG enabled: > > > > ** The following JSC stress test failures have been introduced: > > profiler-test.yaml/tests/sunspider-1.0/3d-cube.js.profiler > > ... > ... > > > Results for JSC stress tests: > > 34 failures found. > > > > Maybe the profiler tests aren't running on your system, or my system is missing something needed by the profiling step. Checking.... > > > > I'm getting this message about the profiler tests: > > Running simple version of profiler-test.yaml/tests/sunspider-1.0/3d-cube.js because some required Ruby features are unavailable. That message is harmless. It's not an error. You will only not get this message if you have all of the Ruby features to use the profiler.