Bug 18631 – SQUIRRELFISH: pass all WebKit regression tests

• Component: JavaScriptCore
• Status: RESOLVED FIXED

[Maciej Stachowiak]

Currently we pass only 8342 (92%) of the 9034 WebKit regression tests on the squirrelfish branch. There are 392 crashes and 666 other failues We need to pass them all to be able to land on trunk.

8342 test cases (92%) succeeded
298 test cases (3%) had incorrect layout
2 test cases (<1%) timed out
392 test cases (4%) crashed
366 test cases (4%) had stderr output


Individual issues that block passing the WebKit regression tests should be filed as blockers of this bug.

[Maciej Stachowiak]

Now down to:

8702 test cases (96%) succeeded
301 test cases (3%) had incorrect layout
2 test cases (<1%) timed out
29 test cases (<1%) crashed
36 test cases (<1%) had stderr output

[Maciej Stachowiak]

Now down to:

8834 test cases (97%) succeeded
183 test cases (2%) had incorrect layout
1 test case (<1%) timed out
16 test cases (<1%) crashed
30 test cases (<1%) had stderr output

At least 115 of the failures are due to a format change in exception strings.

[Cameron Zwarich (cpst)]

We might as well focus on the fast/js tests first. These are the failures on my machine:

fast/js/caller-property.html
fast/js/const.html - we don't support const, so no surprise
fast/js/deep-recursion-test.html - recursion limit
fast/js/encode-URI-test.html - throws exceptions for everything
fast/js/eval-keyword-vs-function.html - I expected this to fail, I think it is due to some missing code in globalFuncEval that didn't get copied over in its entirety
fast/js/for-in-exeception.html - should be an easy fix
fast/js/lexical-lookup-in-function-constructor.html
fast/js/read-modify-eval.html - the pre/post inc/dec operators don't work right inside of eval
fast/js/string-replace-2.html
fast/js/toString-elision-trailing-comma.html
fast/js/toString-prefix-postfix-preserve-parens.html
fast/js/kde/assignments.html - var i doesn't reinitialize i to undefined
fast/js/kde/const.html - we don't support const
fast/js/kde/eval.html
fast/js/kde/lval-exceptions.html - not printing function bodies
fast/js/kde/scope.html - possibly related to the other eval problems, but I am not sure

A few more tests crashed. Some crashes are expected, due to unimplemented features, but a few are possibly troubling:

fast/js/exec-state-marking.html
fast/regex/early-acid3-86.html - crashes in codegen

I'll probably try to work on the eval issues myself, but I'll also make separate bugs for specific failures.

[Maciej Stachowiak]

The following 147 tests fail for well-understood reasons:

exception messages are missing source snippet (122) - bug 18774
  dom/* (116 of them)
  editing/selection/contenteditable-click-inside.html
  editing/selection/contenteditable-click-outside.html
  fast/forms/selected-index-assert.html
  fast/xpath/nsresolver-exception.xhtml
  platform/mac/fast/AppleScript/001.html
  tables/mozilla_expected_failures/core/captions1.html

const broken (2) - bug 18749
  fast/js/const.html
  fast/js/kde/const.html

f.arguments / f.caller (5) - bug 18633
  fast/events/caller-access-from-event-listener.html
  fast/js/caller-property.html
  fast/js/vardecl-preserve-arguments.html
  fast/js/kde/function.html
  fast/js/kde/function_arguments.html

ObjC bindings are broken (5) - bug 
  editing/pasteboard/paste-RTFD.html
  editing/pasteboard/paste-TIFF.html
  platform/mac/editing/pasteboard/5583362.html
  platform/mac/plugins/bindings-test-objc.html
  plugins/jsobjc-simple.html

NPRuntime bindings are broken (1):
  plugins/bindings-test.html

window.this shows p as a property, shouldn't (1):
  fast/dom/Window/window-properties.html

function toString broken after calling (2):
  fast/js/toString-elision-trailing-comma.html
  fast/js/toString-prefix-postfix-preserve-parens.html

exception side effects (1):
  fast/js/kde/lval-exceptions.html

security check is wrong (global object issues?) (6):
  http/tests/security/cross-frame-access-callback-explicit-domain-ALLOW.htm
  http/tests/security/listener/xss-JSTargetNode-onclick-shortcut.html
  http/tests/security/listener/xss-XMLHttpRequest-addEventListener.html
  http/tests/security/listener/xss-XMLHttpRequest-shortcut.html
  http/tests/security/listener/xss-window-onclick-addEventListener.html
  http/tests/security/listener/xss-window-onclick-shortcut.html


The following 15 are not completely understood yet:


scope chain assertion ASSERTION FAILED: scopeChain->object == activation
 media/video-dom-loopstart.html

unknown timeout
 fast/canvas/patternfill-repeat.html
 fast/dom/SelectorAPI/bug-17313.html

unknown crash
 fast/regex/early-acid3-86.html
 http/tests/misc/acid3.html
 svg/carto.net/button.svg

weirdshit with toString / exceptions
 fast/canvas/canvas-pattern-behaviour.html

blank line missing? wtf?
 fast/canvas/toDataURL-supportedTypes.html

getter/setter conflicts with var?
 fast/dom/getter-on-window-object2.html

unknown:
 fast/dom/HTMLElement/set-inner-outer-optimization.html
 fast/frames/viewsource-empty-attribute-value.html
 fast/js/eval-var-decl.html
 fast/js/lexical-lookup-in-function-constructor.html

something undefined that shouldn't be:
 fast/dom/Window/btoa-pnglet.html

fails on different line. new results probably ok
 fast/js/recursion-limit-equal.html

[Maciej Stachowiak]

fast/js/kde/lval-exceptions is actually the same function.toString after calling issue.

[Geoffrey Garen]

I'm removing 18774 from the blocker list because we can land on trunk without full exception messages.

[Geoffrey Garen]

(In reply to comment #6)
What I meant to say was, I revised all the tests that were failing in that way: the pass now.

[Maciej Stachowiak]

fast/dom/javascript-url-crash-function.html crrently crashes and would be fixed by merging the rest of the split window work, most notably <http://trac.webkit.org/changeset/32585>, from trunk.

[Maciej Stachowiak]

fast/dynamic/flash-replacement-test.html currently crashes because we retrieve the offsetWidth property on an h1, forcing layout, which instantiates a plugin, which is Flash content that runs some JS, and re-entering the VM causes the register file to be re-alloated. But op_get_by_id is not prepared for the register file to move.

[Geoffrey Garen]

Here's the latest test breakdown:

rebasing will fix [mark]:
fast/dom/javascript-url-crash-function.html [don't clear global object]
fast/canvas/toDataURL-supportedTypes.html [disabled on trunk due to whitespace shenanigans]

maciej is performance-tuning a fix for [maciej]:
fast/canvas/patternfill-repeat.html
fast/dom/SelectorAPI/bug-17313.html

unkown [nobody]:
fast/dom/HTMLElement/set-inner-outer-optimization.html
fast/frames/viewsource-empty-attribute-value.html
platform/mac/plugins/bindings-test-objc.html

exception side effects [oliver]:
fast/js/kde/lval-exceptions.html

function.toString [maciej]:
fast/js/toString-elision-trailing-comma.html
fast/js/toString-prefix-postfix-preserve-parens.html

function.caller and function.arguments in nested call frames [geoff]:
fast/js/caller-property.html
fast/js/kde/function.html
fast/js/kde/function_arguments.html

native getter indirectly reallocates register file by causing a layout / plugin-in load / js execution [oliver]:
fast/dynamic/flash-replacement-test.html

[Geoffrey Garen]

OK, all failures are now classified:

rebasing will fix [mark]:
fast/dom/javascript-url-crash-function.html [don't clear global object]
fast/canvas/toDataURL-supportedTypes.html [disabled on trunk due to whitespace
shenanigans]

function.toString [maciej]:
fast/js/toString-elision-trailing-comma.html
fast/js/toString-prefix-postfix-preserve-parens.html
fast/js/kde/lval-exceptions.html

function.caller and function.arguments in nested call frames [geoff]:
fast/js/caller-property.html
fast/js/kde/function.html
fast/js/kde/function_arguments.html

native getter indirectly reallocates register file [oliver]:
fast/dom/HTMLElement/set-inner-outer-optimization.html
fast/dynamic/flash-replacement-test.html

[Cameron Zwarich (cpst)]

I ran run-webkit-tests --singly, and I got the following results, which is 20 failures with 11 crashes, and some more with stderr output:

Tests where results did not match expected results:

dom/html/level2/html/HTMLDocument12.html
dom/xhtml/level2/html/HTMLDocument12.xhtml
fast/block/basic/001.html
fast/canvas/toDataURL-supportedTypes.html
fast/cookies/local-file-can-set-cookies.html
fast/dom/setPrimitiveValue.html
fast/dom/HTMLElement/set-inner-outer-optimization.html
fast/forms/password-placeholder.html
fast/invalid/003.html
fast/js/caller-property.html
fast/js/toString-elision-trailing-comma.html
fast/js/toString-prefix-postfix-preserve-parens.html
fast/js/kde/function.html
fast/js/kde/function_arguments.html
fast/js/kde/lval-exceptions.html
fast/layers/removed-by-scroll-handler.html
svg/custom/frame-getSVGDocument.html
tables/mozilla/bugs/bug113235-1.html
tables/mozilla/other/wa_table_tr_align.html
tables/mozilla_expected_failures/bugs/bug56024.html

Tests that caused the DumpRenderTree tool to crash:

fast/css-generated-content/inline-display-types.html
fast/dom/javascript-url-crash-function.html
fast/dynamic/flash-replacement-test.html
fast/events/nested-event-remove-node-crash.html
fast/events/overflow-events.html
fast/loader/unloadable-script.html
fast/overflow/onscroll-layer-self-destruct.html
media/video-dom-loopstart.html
media/video-loopstart.html
storage/domstorage/localstorage/index-get-and-set.html
storage/domstorage/sessionstorage/index-get-and-set.html

Tests that had stderr output:

dom/xhtml/level3/core/entitygetinputencoding03.xhtml
dom/xhtml/level3/core/entitygetinputencoding04.xhtml
dom/xhtml/level3/core/entitygetxmlencoding02.xhtml
dom/xhtml/level3/core/entitygetxmlencoding03.xhtml
dom/xhtml/level3/core/entitygetxmlencoding04.xhtml
dom/xhtml/level3/core/entitygetxmlversion03.xhtml
dom/xhtml/level3/core/entitygetxmlversion04.xhtml
dom/xhtml/level3/core/nodegetbaseuri16.xhtml
dom/xhtml/level3/core/nodegetbaseuri19.xhtml
dom/xhtml/level3/core/nodegetbaseuri20.xhtml
fast/canvas/gradient-empty-path.html
fast/css-generated-content/inline-display-types.html
fast/forms/001.html
fast/forms/menulist-narrow-width.html
fast/loader/unloadable-script.html
fast/regex/slow.html
fast/regex/test1.html
fast/xsl/transform-xhr-doc.xhtml
http/tests/multipart/invalid-image-data.html
media/video-dom-loopstart.html
media/video-loopstart.html
plugins/return-error-from-new-stream-callback-in-full-frame-plugin.html
security/block-test.html

[Cameron Zwarich (cpst)]

Here's my attempt at diagnosing the new crashes I saw. The first two tests fail for me without --singly, and they might be fixed if I clear my cookies, and the last is also a cookies machine that might be isolated to my machine:

dom/html/level2/html/HTMLDocument12.html
dom/xhtml/level2/html/HTMLDocument12.xhtml
fast/cookies/local-file-can-set-cookies.html

These are just small changes in rendering, but I am not sure why they occur:

fast/block/basic/001.html
fast/forms/password-placeholder.html
fast/invalid/003.html
tables/mozilla/bugs/bug113235-1.html
tables/mozilla/other/wa_table_tr_align.html
tables/mozilla_expected_failures/bugs/bug56024.html

This one now works for me:

fast/layers/removed-by-scroll-handler.html

This one doesn't set successfullyParsed, but it works otherwise:

svg/custom/frame-getSVGDocument.html

The strange thing about it is that it works fine in the browser.

I am not sure if any of these failures are SquirrelFish's fault. Before we land, we should test them on trunk individually against the branch. The crashes are more interesting, because some of them occur in JavaScriptCore. I'll make another post about them.

[Cameron Zwarich (cpst)]

The first crash occurs in rendering with nothing JavaScript in its stack trace, so it is probably not our fault:

fast/css-generated-content/inline-display-types.html

We should check it against trunk before we land. The rest of the failures were fixed since I posted the original results:

fast/events/nested-event-remove-node-crash.html
fast/events/overflow-events.html
fast/loader/unloadable-script.html
fast/overflow/onscroll-layer-self-destruct.html
media/video-dom-loopstart.html
media/video-loopstart.html
storage/domstorage/localstorage/index-get-and-set.html
storage/domstorage/sessionstorage/index-get-and-set.html

All of these fixes are probably due to Oliver's register file tainting patch. The same is true of all the new tests with stderr output. So, the moral of the story is that we have nothing to worry about.

[Geoffrey Garen]

All tests pass as of r33383.