Currently we pass only 8342 (92%) of the 9034 WebKit regression tests on the squirrelfish branch. There are 392 crashes and 666 other failues We need to pass them all to be able to land on trunk. 8342 test cases (92%) succeeded 298 test cases (3%) had incorrect layout 2 test cases (<1%) timed out 392 test cases (4%) crashed 366 test cases (4%) had stderr output Individual issues that block passing the WebKit regression tests should be filed as blockers of this bug.
Now down to: 8702 test cases (96%) succeeded 301 test cases (3%) had incorrect layout 2 test cases (<1%) timed out 29 test cases (<1%) crashed 36 test cases (<1%) had stderr output
Now down to: 8834 test cases (97%) succeeded 183 test cases (2%) had incorrect layout 1 test case (<1%) timed out 16 test cases (<1%) crashed 30 test cases (<1%) had stderr output At least 115 of the failures are due to a format change in exception strings.
We might as well focus on the fast/js tests first. These are the failures on my machine: fast/js/caller-property.html fast/js/const.html - we don't support const, so no surprise fast/js/deep-recursion-test.html - recursion limit fast/js/encode-URI-test.html - throws exceptions for everything fast/js/eval-keyword-vs-function.html - I expected this to fail, I think it is due to some missing code in globalFuncEval that didn't get copied over in its entirety fast/js/for-in-exeception.html - should be an easy fix fast/js/lexical-lookup-in-function-constructor.html fast/js/read-modify-eval.html - the pre/post inc/dec operators don't work right inside of eval fast/js/string-replace-2.html fast/js/toString-elision-trailing-comma.html fast/js/toString-prefix-postfix-preserve-parens.html fast/js/kde/assignments.html - var i doesn't reinitialize i to undefined fast/js/kde/const.html - we don't support const fast/js/kde/eval.html fast/js/kde/lval-exceptions.html - not printing function bodies fast/js/kde/scope.html - possibly related to the other eval problems, but I am not sure A few more tests crashed. Some crashes are expected, due to unimplemented features, but a few are possibly troubling: fast/js/exec-state-marking.html fast/regex/early-acid3-86.html - crashes in codegen I'll probably try to work on the eval issues myself, but I'll also make separate bugs for specific failures.
The following 147 tests fail for well-understood reasons: exception messages are missing source snippet (122) - bug 18774 dom/* (116 of them) editing/selection/contenteditable-click-inside.html editing/selection/contenteditable-click-outside.html fast/forms/selected-index-assert.html fast/xpath/nsresolver-exception.xhtml platform/mac/fast/AppleScript/001.html tables/mozilla_expected_failures/core/captions1.html const broken (2) - bug 18749 fast/js/const.html fast/js/kde/const.html f.arguments / f.caller (5) - bug 18633 fast/events/caller-access-from-event-listener.html fast/js/caller-property.html fast/js/vardecl-preserve-arguments.html fast/js/kde/function.html fast/js/kde/function_arguments.html ObjC bindings are broken (5) - bug editing/pasteboard/paste-RTFD.html editing/pasteboard/paste-TIFF.html platform/mac/editing/pasteboard/5583362.html platform/mac/plugins/bindings-test-objc.html plugins/jsobjc-simple.html NPRuntime bindings are broken (1): plugins/bindings-test.html window.this shows p as a property, shouldn't (1): fast/dom/Window/window-properties.html function toString broken after calling (2): fast/js/toString-elision-trailing-comma.html fast/js/toString-prefix-postfix-preserve-parens.html exception side effects (1): fast/js/kde/lval-exceptions.html security check is wrong (global object issues?) (6): http/tests/security/cross-frame-access-callback-explicit-domain-ALLOW.htm http/tests/security/listener/xss-JSTargetNode-onclick-shortcut.html http/tests/security/listener/xss-XMLHttpRequest-addEventListener.html http/tests/security/listener/xss-XMLHttpRequest-shortcut.html http/tests/security/listener/xss-window-onclick-addEventListener.html http/tests/security/listener/xss-window-onclick-shortcut.html The following 15 are not completely understood yet: scope chain assertion ASSERTION FAILED: scopeChain->object == activation media/video-dom-loopstart.html unknown timeout fast/canvas/patternfill-repeat.html fast/dom/SelectorAPI/bug-17313.html unknown crash fast/regex/early-acid3-86.html http/tests/misc/acid3.html svg/carto.net/button.svg weirdshit with toString / exceptions fast/canvas/canvas-pattern-behaviour.html blank line missing? wtf? fast/canvas/toDataURL-supportedTypes.html getter/setter conflicts with var? fast/dom/getter-on-window-object2.html unknown: fast/dom/HTMLElement/set-inner-outer-optimization.html fast/frames/viewsource-empty-attribute-value.html fast/js/eval-var-decl.html fast/js/lexical-lookup-in-function-constructor.html something undefined that shouldn't be: fast/dom/Window/btoa-pnglet.html fails on different line. new results probably ok fast/js/recursion-limit-equal.html
fast/js/kde/lval-exceptions is actually the same function.toString after calling issue.
I'm removing 18774 from the blocker list because we can land on trunk without full exception messages.
(In reply to comment #6) What I meant to say was, I revised all the tests that were failing in that way: the pass now.
fast/dom/javascript-url-crash-function.html crrently crashes and would be fixed by merging the rest of the split window work, most notably <http://trac.webkit.org/changeset/32585>, from trunk.
fast/dynamic/flash-replacement-test.html currently crashes because we retrieve the offsetWidth property on an h1, forcing layout, which instantiates a plugin, which is Flash content that runs some JS, and re-entering the VM causes the register file to be re-alloated. But op_get_by_id is not prepared for the register file to move.
Here's the latest test breakdown: rebasing will fix [mark]: fast/dom/javascript-url-crash-function.html [don't clear global object] fast/canvas/toDataURL-supportedTypes.html [disabled on trunk due to whitespace shenanigans] maciej is performance-tuning a fix for [maciej]: fast/canvas/patternfill-repeat.html fast/dom/SelectorAPI/bug-17313.html unkown [nobody]: fast/dom/HTMLElement/set-inner-outer-optimization.html fast/frames/viewsource-empty-attribute-value.html platform/mac/plugins/bindings-test-objc.html exception side effects [oliver]: fast/js/kde/lval-exceptions.html function.toString [maciej]: fast/js/toString-elision-trailing-comma.html fast/js/toString-prefix-postfix-preserve-parens.html function.caller and function.arguments in nested call frames [geoff]: fast/js/caller-property.html fast/js/kde/function.html fast/js/kde/function_arguments.html native getter indirectly reallocates register file by causing a layout / plugin-in load / js execution [oliver]: fast/dynamic/flash-replacement-test.html
OK, all failures are now classified: rebasing will fix [mark]: fast/dom/javascript-url-crash-function.html [don't clear global object] fast/canvas/toDataURL-supportedTypes.html [disabled on trunk due to whitespace shenanigans] function.toString [maciej]: fast/js/toString-elision-trailing-comma.html fast/js/toString-prefix-postfix-preserve-parens.html fast/js/kde/lval-exceptions.html function.caller and function.arguments in nested call frames [geoff]: fast/js/caller-property.html fast/js/kde/function.html fast/js/kde/function_arguments.html native getter indirectly reallocates register file [oliver]: fast/dom/HTMLElement/set-inner-outer-optimization.html fast/dynamic/flash-replacement-test.html
I ran run-webkit-tests --singly, and I got the following results, which is 20 failures with 11 crashes, and some more with stderr output: Tests where results did not match expected results: dom/html/level2/html/HTMLDocument12.html dom/xhtml/level2/html/HTMLDocument12.xhtml fast/block/basic/001.html fast/canvas/toDataURL-supportedTypes.html fast/cookies/local-file-can-set-cookies.html fast/dom/setPrimitiveValue.html fast/dom/HTMLElement/set-inner-outer-optimization.html fast/forms/password-placeholder.html fast/invalid/003.html fast/js/caller-property.html fast/js/toString-elision-trailing-comma.html fast/js/toString-prefix-postfix-preserve-parens.html fast/js/kde/function.html fast/js/kde/function_arguments.html fast/js/kde/lval-exceptions.html fast/layers/removed-by-scroll-handler.html svg/custom/frame-getSVGDocument.html tables/mozilla/bugs/bug113235-1.html tables/mozilla/other/wa_table_tr_align.html tables/mozilla_expected_failures/bugs/bug56024.html Tests that caused the DumpRenderTree tool to crash: fast/css-generated-content/inline-display-types.html fast/dom/javascript-url-crash-function.html fast/dynamic/flash-replacement-test.html fast/events/nested-event-remove-node-crash.html fast/events/overflow-events.html fast/loader/unloadable-script.html fast/overflow/onscroll-layer-self-destruct.html media/video-dom-loopstart.html media/video-loopstart.html storage/domstorage/localstorage/index-get-and-set.html storage/domstorage/sessionstorage/index-get-and-set.html Tests that had stderr output: dom/xhtml/level3/core/entitygetinputencoding03.xhtml dom/xhtml/level3/core/entitygetinputencoding04.xhtml dom/xhtml/level3/core/entitygetxmlencoding02.xhtml dom/xhtml/level3/core/entitygetxmlencoding03.xhtml dom/xhtml/level3/core/entitygetxmlencoding04.xhtml dom/xhtml/level3/core/entitygetxmlversion03.xhtml dom/xhtml/level3/core/entitygetxmlversion04.xhtml dom/xhtml/level3/core/nodegetbaseuri16.xhtml dom/xhtml/level3/core/nodegetbaseuri19.xhtml dom/xhtml/level3/core/nodegetbaseuri20.xhtml fast/canvas/gradient-empty-path.html fast/css-generated-content/inline-display-types.html fast/forms/001.html fast/forms/menulist-narrow-width.html fast/loader/unloadable-script.html fast/regex/slow.html fast/regex/test1.html fast/xsl/transform-xhr-doc.xhtml http/tests/multipart/invalid-image-data.html media/video-dom-loopstart.html media/video-loopstart.html plugins/return-error-from-new-stream-callback-in-full-frame-plugin.html security/block-test.html
Here's my attempt at diagnosing the new crashes I saw. The first two tests fail for me without --singly, and they might be fixed if I clear my cookies, and the last is also a cookies machine that might be isolated to my machine: dom/html/level2/html/HTMLDocument12.html dom/xhtml/level2/html/HTMLDocument12.xhtml fast/cookies/local-file-can-set-cookies.html These are just small changes in rendering, but I am not sure why they occur: fast/block/basic/001.html fast/forms/password-placeholder.html fast/invalid/003.html tables/mozilla/bugs/bug113235-1.html tables/mozilla/other/wa_table_tr_align.html tables/mozilla_expected_failures/bugs/bug56024.html This one now works for me: fast/layers/removed-by-scroll-handler.html This one doesn't set successfullyParsed, but it works otherwise: svg/custom/frame-getSVGDocument.html The strange thing about it is that it works fine in the browser. I am not sure if any of these failures are SquirrelFish's fault. Before we land, we should test them on trunk individually against the branch. The crashes are more interesting, because some of them occur in JavaScriptCore. I'll make another post about them.
The first crash occurs in rendering with nothing JavaScript in its stack trace, so it is probably not our fault: fast/css-generated-content/inline-display-types.html We should check it against trunk before we land. The rest of the failures were fixed since I posted the original results: fast/events/nested-event-remove-node-crash.html fast/events/overflow-events.html fast/loader/unloadable-script.html fast/overflow/onscroll-layer-self-destruct.html media/video-dom-loopstart.html media/video-loopstart.html storage/domstorage/localstorage/index-get-and-set.html storage/domstorage/sessionstorage/index-get-and-set.html All of these fixes are probably due to Oliver's register file tainting patch. The same is true of all the new tests with stderr output. So, the moral of the story is that we have nothing to worry about.
All tests pass as of r33383.