Bug 167537 – Migrate 97 *-disabled tests to be skipped in TestExpectations

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[David Kilzer (:ddkilzer)]

A long time ago in a galaxy not too far away, the WebKit project used to skip layout tests by appending "-disabled" to the end of the test file.

Then support for TestExpectation files was added to WebKit, giving more control over how the tests were run (or not run), and providing a way to document why they were skipped (other than by reviewing the commit log message for the rename).

We should rename the remaining 99 *-disabled layout tests, and then add them into TestExpectations files if they don't currently pass.

If the person moving them back into the TestExpectations file has extra time on their hands, it would be nice to group them by the reason they were disabled in the first place, and maybe reference old bugs already filed about why they were disabled.  This will take a lot of time and research via git/svn log, although it could be done at a later time after moving them in bulk into LayoutTests/TestExpectations.

Count:

$ find LayoutTests -name \*-disabled | wc -l
      99

Rename all the *-disabled files in preparation for checking in (remove the second "echo" to actually move them):

$ for F in `find LayoutTests -name \*-disabled`; do G=`echo $F | sed -e 's/-disabled//'`; echo git mv $F $G; done

List of files:

$ find LayoutTests -name \*-disabled | tee list.txt
LayoutTests/animations/font-size-using-ems.html-disabled
LayoutTests/compositing/objects/composited-object-alignment.html-disabled
LayoutTests/compositing/tiling/huge-layer-resize.html-disabled
LayoutTests/compositing/tiling/huge-layer.html-disabled
LayoutTests/css2.1/t1202-counter-10-b.html-disabled
LayoutTests/css2.1/t1202-counters-10-b.html-disabled
LayoutTests/css2.1/t1204-increment-00-c-o.html-disabled
LayoutTests/css2.1/t1204-increment-01-c-o.html-disabled
LayoutTests/css2.1/t1204-increment-02-c-o.html-disabled
LayoutTests/css2.1/t1204-reset-00-c-o.html-disabled
LayoutTests/css2.1/t1204-reset-01-c-o.html-disabled
LayoutTests/css2.1/t1204-reset-02-c-o.html-disabled
LayoutTests/dom/xhtml/level2/html/HTMLFrameElement09.xhtml-disabled
LayoutTests/dom/xhtml/level3/core/documentadoptnode22.xhtml-disabled
LayoutTests/dom/xhtml/level3/core/documentnormalizedocument06.xhtml-disabled
LayoutTests/dom/xhtml/level3/core/documentsetdocumenturi01.xhtml-disabled
LayoutTests/dom/xhtml/level3/core/documentsetdocumenturi02.xhtml-disabled
LayoutTests/dom/xhtml/level3/core/domimplementationregistry12.xhtml-disabled
LayoutTests/dom/xhtml/level3/core/domimplementationregistry23.xhtml-disabled
LayoutTests/dom/xhtml/level3/core/nodecomparedocumentposition14.xhtml-disabled
LayoutTests/dom/xhtml/level3/core/nodecomparedocumentposition15.xhtml-disabled
LayoutTests/dom/xhtml/level3/core/noderemovechild03.xhtml-disabled
LayoutTests/dom/xhtml/level3/core/nodereplacechild06.xhtml-disabled
LayoutTests/dom/xhtml/level3/core/nodereplacechild07.xhtml-disabled
LayoutTests/dom/xhtml/level3/core/nodereplacechild08.xhtml-disabled
LayoutTests/editing/execCommand/create-list-1.html-disabled
LayoutTests/editing/input/attributed-substring-from-range-lines.html-disabled
LayoutTests/editing/pasteboard/paste-empty-startcontainer.html-disabled
LayoutTests/editing/selection/inconsistent-in-removeChildNode.html-disabled
LayoutTests/editing/style/5091898.html-disabled
LayoutTests/fast/css/css2-system-color.html-disabled
LayoutTests/fast/css/font-face-in-shadow-DOM.html-disabled
LayoutTests/fast/css/limited-vendor-prefix-behavior.html-disabled
LayoutTests/fast/dom/HTMLDataGridElement/DataGridColumns-basic.html-disabled
LayoutTests/fast/dom/HTMLDataGridElement/DataGridColumns-dom-attributes.html-disabled
LayoutTests/fast/dom/HTMLDataGridElement/DataGridColumns-dom.html-disabled
LayoutTests/fast/dom/HTMLDataGridElement/DataGridDataSource-basic.html-disabled
LayoutTests/fast/dom/Window/timeout-released-on-close.html-disabled
LayoutTests/fast/dom/Window/window-resize-nan.html-disabled
LayoutTests/fast/dom/gc-8.html-disabled
LayoutTests/fast/dynamic/crash-paint-no-documentElement-renderer.html-disabled
LayoutTests/fast/events/destroyed-atomic-string.html-disabled
LayoutTests/fast/events/key-events-in-frame.html-disabled
LayoutTests/fast/frames/iframe-scroll-page-up-down.html-disabled
LayoutTests/fast/html/marquee-alternate.html-disabled
LayoutTests/fast/leaks/003.html-disabled
LayoutTests/fast/loader/api-test-go-to-current-back-forward-item.html-disabled
LayoutTests/fast/loader/api-test-new-window-data-load-base-url.html-disabled
LayoutTests/fast/loader/form-events-back-forward.html-disabled
LayoutTests/fast/notifications/notifications-event-stop-propagation.html-disabled
LayoutTests/fast/notifications/notifications-multi-events.html-disabled
LayoutTests/fast/ruby/after-block-doesnt-crash.html-disabled
LayoutTests/fast/ruby/after-table-doesnt-crash.html-disabled
LayoutTests/fast/ruby/generated-after-counter-doesnt-crash.html-disabled
LayoutTests/fast/ruby/generated-before-and-after-counter-doesnt-crash.html-disabled
LayoutTests/fast/shadow-dom/copy-shadow-tree.html-disabled
LayoutTests/fast/table/double-height-table-no-tbody.html-disabled
LayoutTests/fast/text/large-text-composed-char-dos.html-disabled
LayoutTests/http/tests/appcache/dynamic-entries-no-cache.html-disabled
LayoutTests/http/tests/multipart/win-boundary-crash.html-disabled
LayoutTests/http/tests/navigation/post-goback-repost-policy.html-disabled
LayoutTests/http/tests/navigation/success200-frames-goback.html-disabled
LayoutTests/http/tests/navigation/success200-frames-reload.html-disabled
LayoutTests/http/tests/navigation/success200-subframeload-goback.html-disabled
LayoutTests/java/lc3/ArrayMethods/object-001.html-disabled
LayoutTests/java/lc3/forin/array-001.html-disabled
LayoutTests/jquery/effects.html-disabled
LayoutTests/js/garbage-collect-after-string-appends.html-disabled
LayoutTests/js/kde/Date.html-disabled
LayoutTests/js/resources/garbage-collect-after-string-appends.js-disabled
LayoutTests/js/string-concatenate-outofmemory.html-disabled
LayoutTests/media/video-canvas.html-disabled
LayoutTests/platform/mac/plugins/pluginDocumentView-deallocated-dataSource.html-disabled
LayoutTests/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.1_decodeURI/S15.1.3.1_A2.5_T1.html-disabled
LayoutTests/sputnik/Conformance/15_Native_Objects/15.1_The_Global_Object/15.1.3/15.1.3.2_decodeURIComponent/S15.1.3.2_A2.5_T1.html-disabled
LayoutTests/sputnik/Conformance/15_Native_Objects/15.9_Date/15.9.3/S15.9.3.1_A5_T1.html-disabled
LayoutTests/sputnik/Conformance/15_Native_Objects/15.9_Date/15.9.3/S15.9.3.1_A5_T2.html-disabled
LayoutTests/sputnik/Conformance/15_Native_Objects/15.9_Date/15.9.3/S15.9.3.1_A5_T3.html-disabled
LayoutTests/sputnik/Conformance/15_Native_Objects/15.9_Date/15.9.3/S15.9.3.1_A5_T4.html-disabled
LayoutTests/sputnik/Conformance/15_Native_Objects/15.9_Date/15.9.3/S15.9.3.1_A5_T5.html-disabled
LayoutTests/sputnik/Conformance/15_Native_Objects/15.9_Date/15.9.3/S15.9.3.1_A5_T6.html-disabled
LayoutTests/sputnik/Unicode/Unicode_218/S7.6_A1.1_T5.html-disabled
LayoutTests/sputnik/Unicode/Unicode_218/S7.6_A3.1.html-disabled
LayoutTests/sputnik/Unicode/Unicode_218/S7.6_A3.2.html-disabled
LayoutTests/sputnik/Unicode/Unicode_218/S7.6_A5.2_T5.html-disabled
LayoutTests/sputnik/Unicode/Unicode_320/S7.6_A1.1_T5.html-disabled
LayoutTests/sputnik/Unicode/Unicode_320/S7.6_A5.2_T5.html-disabled
LayoutTests/sputnik/Unicode/Unicode_410/S7.6_A1.1_T5.html-disabled
LayoutTests/sputnik/Unicode/Unicode_410/S7.6_A5.2_T5.html-disabled
LayoutTests/sputnik/Unicode/Unicode_500/S7.6_A1.1_T5.html-disabled
LayoutTests/sputnik/Unicode/Unicode_500/S7.6_A5.2_T5.html-disabled
LayoutTests/sputnik/Unicode/Unicode_510/S7.6_A1.1_T5.html-disabled
LayoutTests/sputnik/Unicode/Unicode_510/S7.6_A5.2_T5.html-disabled
LayoutTests/svg/W3C-SVG-1.1/resources/filters-comptran-01-f.svg-disabled
LayoutTests/svg/batik/text/textBiDi.svg-disabled
LayoutTests/svg/custom/filter-source-alpha.svg-disabled
LayoutTests/svg/custom/font-face-fallback.svg-disabled
LayoutTests/svg/custom/js-font-test.svg-disabled
LayoutTests/svg/zoom/page/zoom-svg-through-object-with-text.xhtml-disabled

[David Kilzer (:ddkilzer)]

Ryan/Alexey:  I'm happy to do a bulk rename of the files and add them to LayoutTests/TestExpectations, but I don't have time to categorize/sort/retest them.

Is it okay just to do the rename/addition, or do you really want them categorized as they go into LayoutTests/TestExpectations?

[David Kilzer (:ddkilzer)]

Heh, there's also two resource files:

LayoutTests/js/resources/garbage-collect-after-string-appends.js-disabled
LayoutTests/svg/W3C-SVG-1.1/resources/filters-comptran-01-f.svg-disabled

Not sure if those were renamed with corresponding test files or not.

[Alexey Proskuryakov]

I think that it's fine to just move these to Skipped list; one thing I'd do is add a comment saying that the reasons for disabling these tests should be found in Trac or svn log.

[David Kilzer (:ddkilzer)]

Output list of renamed files to a file:

$ find LayoutTests -name \*-disabled | sed -e 's/-disabled//' | grep -v '/resources/' > renamed.txt

[David Kilzer (:ddkilzer)]

Created attachment 300048 [details]
Patch v1

[David Kilzer (:ddkilzer)]

(In reply to comment #1)
> Ryan/Alexey:  I'm happy to do a bulk rename of the files and add them to
> LayoutTests/TestExpectations, but I don't have time to
> categorize/sort/retest them.

I categorized them with bug numbers or comments on when/why they were disabled.

I ran them a couple times locally with ASan enabled, but there were no interesting crashes.  I think roughly half can probably be re-enabled (some marked as slow or flaky), but I don't have time to watch the bots after doing so right now.

[David Kilzer (:ddkilzer)]

Created attachment 300050 [details]
List of renamed tests (useful for running just these tests)

$ ./Tools/Scripts/run-webkit-tests --child-processes=4 --fully-parallel --release `cat renamed.txt`

[WebKit Commit Bot]

Comment on attachment 300048 [details]
Patch v1

Clearing flags on attachment: 300048

Committed r211349: <http://trac.webkit.org/changeset/211349>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.