It's a big list - here's the flakiness dashboard: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#group=%40ToT%20-%20chromium.org&tests=editing%2Fdeleting%2Fdelete-3928305-fix.html%2Cediting%2Fdeleting%2Fdelete-3959464-fix.html%2Cediting%2Fdeleting%2Fdelete-and-undo.html%2Cediting%2Fdeleting%2Fdelete-block-contents-003.html%2Cediting%2Fdeleting%2Fdelete-contiguous-ws-001.html%2Cediting%2Fdeleting%2Fdelete-tab-002.html%2Cediting%2Fdeleting%2Fdelete-tab-003.html%2Cediting%2Fdeleting%2Fdelete-to-select-table.html%2Cediting%2Fdeleting%2Fdelete-trailing-ws-002.html%2Cediting%2Fdeleting%2Fdelete-ws-fixup-001.html%2Cediting%2Fdeleting%2Fdelete-ws-fixup-002.html%2Cediting%2FexecCommand%2F5569741.html%2Cediting%2FexecCommand%2Finsert-list-and-stitch.html%2Cediting%2FexecCommand%2FinsertHorizontalRule.html%2Cediting%2FexecCommand%2Foutdent-selection.html%2Cediting%2Finserting%2F5418891.html%2Cediting%2Finserting%2Fediting-empty-divs.html%2Cediting%2Finserting%2Finsert-3907422-fix.html%2Cediting%2Finserting%2Finsert-br-004.html%2Cediting%2Finserting%2Finsert-br-005.html%2Cediting%2Finserting%2Finsert-br-007.html%2Cediting%2Finserting%2Finsert-div-007.html%2Cediting%2Finserting%2Finsert-div-012.html%2Cediting%2Finserting%2Finsert-div-013.html%2Cediting%2Finserting%2Finsert-div-014.html%2Cediting%2Finserting%2Finsert-div-015.html%2Cediting%2Finserting%2Finsert-div-016.html%2Cediting%2Finserting%2Finsert-div-017.html%2Cediting%2Finserting%2Finsert-div-018.html%2Cediting%2Finserting%2Finsert-div-019.html%2Cediting%2Finserting%2Finsert-div-020.html%2Cediting%2Finserting%2Finsert-div-021.html%2Cediting%2Finserting%2Finsert-div-022.html%2Cediting%2Finserting%2Finsert-div-025.html%2Cediting%2Finserting%2Finsert-div-027.html%2Cediting%2Finserting%2Finsert-paragraph-02.html%2Cediting%2Finserting%2Finsert-paragraph-03.html%2Cediting%2Finserting%2Finsert-tab-002.html%2Cediting%2Finserting%2Finsert-tab-003.html%2Cediting%2Finserting%2Finsert-text-with-newlines.html%2Cediting%2Finserting%2Fparagraph-separator-01.html%2Cediting%2Finserting%2Fparagraph-separator-02.html%2Cediting%2Finserting%2Fparagraph-separator-03.html%2Cediting%2Finserting%2Freturn-key-with-selection-001.html%2Cediting%2Finserting%2Freturn-key-with-selection-002.html%2Cediting%2Finserting%2Freturn-key-with-selection-003.html%2Cediting%2Finserting%2Ftyping-002.html%2Cediting%2Finserting%2Ftyping-around-br-001.html%2Cediting%2Fpasteboard%2F4242293.html%2Cediting%2Fpasteboard%2F4944770-1.html%2Cediting%2Fpasteboard%2F8145-2.html%2Cediting%2Fpasteboard%2Fcopy-in-password-field.html%2Cediting%2Fpasteboard%2Fcut-text-001.html%2Cediting%2Fpasteboard%2Finterchange-newline-1.html%2Cediting%2Fpasteboard%2Fmerge-after-delete-1.html%2Cediting%2Fpasteboard%2Fmerge-after-delete-2.html%2Cediting%2Fpasteboard%2Fmerge-after-delete.html%2Cediting%2Fpasteboard%2Fmerge-end-4.html%2Cediting%2Fpasteboard%2Fmerge-end-blockquote.html%2Cediting%2Fpasteboard%2Fmerge-end-list.html%2Cediting%2Fpasteboard%2Fmerge-end-table.html%2Cediting%2Fpasteboard%2Fpaste-blockquote-3.html%2Cediting%2Fpasteboard%2Fpaste-text-001.html%2Cediting%2Fpasteboard%2Fpaste-text-012.html%2Cediting%2Fpasteboard%2Fpaste-text-013.html%2Cediting%2Fpasteboard%2Fpaste-text-014.html%2Cediting%2Fpasteboard%2Fpaste-text-019.html%2Cediting%2Fpasteboard%2Fpaste-xml.xhtml%2Cediting%2Fpasteboard%2Fundoable-fragment-removes.html%2Cediting%2Fselection%2F13804.html%2Cediting%2Fselection%2F5234383-1.html%2Cediting%2Fselection%2F5234383-2.html%2Cediting%2Fselection%2Fextend-by-word-001.html%2Cediting%2Fselection%2Fmove-backwords-by-word-001.html%2Cediting%2Fselection%2Fmove-by-character-001.html%2Cediting%2Fselection%2Fmove-by-line-001.html%2Cediting%2Fselection%2Fmove-by-word-001.html%2Cediting%2Fselection%2Fselect-from-textfield-outwards.html%2Cediting%2Fselection%2Funrendered-002.html%2Cediting%2Fspelling%2Finline_spelling_markers.html%2Cediting%2Fspelling%2Fspelling.html%2Cediting%2Fstyle%2F5046875-2.html%2Cediting%2Fstyle%2Fblock-styles-007.html%2Cediting%2Fstyle%2Fcreate-block-for-style-002.html%2Cediting%2Fstyle%2Fcreate-block-for-style-003.html%2Cediting%2Fstyle%2Fcreate-block-for-style-004.html%2Cediting%2Fstyle%2Fcreate-block-for-style-006.html%2Cediting%2Fstyle%2Fcreate-block-for-style-008.html%2Cediting%2Fstyle%2Fcreate-block-for-style-009.html%2Cediting%2Fstyle%2Fcreate-block-for-style-011.html%2Cediting%2Fstyle%2Fcreate-block-for-style-012.html%2Cediting%2Fstyle%2Fcreate-block-for-style-013.html%2Cfast%2Fdom%2Fblur-contenteditable.html%2Cfast%2Fdom%2Fobject-embed-plugin-scripting.html%2Cfast%2Fdom%2Fobject-plugin-hides-properties.html%2Cfast%2Fevents%2Fonchange-passwordfield.html%2Cfast%2Fevents%2Ftabindex-focus-blur-all.html%2Cfast%2Fforms%2Ffocus-control-to-page.html%2Cfast%2Fforms%2Ffocus2.html%2Cfast%2Fforms%2Fpassword-doubleclick-selection.html%2Cfast%2Fforms%2Fpassword-placeholder-text-security.html%2Cfast%2Fimages%2Fembed-image.html%2Cfast%2Fimages%2Fobject-image.html%2Cplugins%2Fembed-attributes-setting.html%2Cplugins%2Fembed-attributes-style.html%2Cplugins%2Fiframe-shims.html%2Cuserscripts%2Fuser-script-video-document.html Interestingly, it doesn't seem to affect the main chrome build tree, since we've already rolled past this revision, so it's probably test-infrastructure related. Not sure how 79035 could be related to this and I'm reluctant to roll it out because it includes a fix for the rebaselining tool, so I'm just going to mark these tests as TIMEOUT for now.
Interesting that a lot of these tests seem to be editing-related. I wonder if there's something taking out some sort of system-wide global lock that is keeping the tests from running in parallel with other DRT tests. Something having to do with focus, maybe? This is wild speculation ...
Seems much more likely to have been triggered by 79047/79062 (me enabling multiple processes).
investigating ... I will also note that while I would expect additional flakiness on SL bots, if anything, I would've expected the Leopard bots to become *more* stable, because we stopped using threads completely.
I am reverting the behavior on 10.5 (see bug 54925), so we'll see if this goes away ...
Tests pass on the Debug build, but timed-out on the Release.
Editing tests are still timing out on WebKit Mac 10.5 (release). We've lost significant test coverage due to this bug :(
It seems unlikely to me that its a NRWT thing at this point. Is it possible that something else changed in that timeframe that's causing this regression?
Maybe http://trac.webkit.org/changeset/79038/ ?
It's possible that my change in http://trac.webkit.org/changeset/79038/ caused this breakage. With this change , if the minimum DOM timer interval isn't manually set then it may be set to a high value. I'll try to run these tests locally to reproduce.
I'm having difficulty reproducing these test timeouts locally. I've built DRT Release mode out of a Chromium checkout, reversed the timeout expected results in r79228 and ran the following: cat tests.txt | xargs src/webkit/tools/layout_tests/run_webkit_tests.sh --chromium with the attached tests.txt. None of the tests timed out. This is on Snow Leopard. Is anyone else able to reproduce the timeouts?
Created attachment 83425 [details] Test list
Created attachment 83426 [details] Revert of code changes from r79038
After talking with dpranke, let's try reverting the code changes from r79038 to see whether they clean up the flakiness dashboard.
Comment on attachment 83426 [details] Revert of code changes from r79038 Worth a try. Feel free to land reverts unreviewed in the future if you prefer.
Committed r79387: <http://trac.webkit.org/changeset/79387>
(In reply to comment #15) > Committed r79387: <http://trac.webkit.org/changeset/79387> It seems like this wasn't the cause :( http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5/builds/1482
I sent out an email internally at Goole to look for a Leopard machine.
Since the code changes in r79038 weren't the root cause, I'm going to reapply them.
Committed r79456: <http://trac.webkit.org/changeset/79456>
Oops, accidentally closed the bug while landing that last patch. Reopening.
I was not able to reproduce this on a Leopard machine (doing a release build with build-webkit --chromium). James and I also noticed that: - non-editing tests were strangely affected too (e.g. userscripts/user-script-video-document.html) - the debug canary bots were not affected: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#group=%40ToT%20-%20chromium.org&tests=editing%2Fdeleting%2Fdelete-3928305-fix.html&showAllRuns=true - the DEPS bots are not affected: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=editing%2Fdeleting%2Fdelete-3928305-fix.html&showAllRuns=true We therefore suspect that the bot is in a weird state. When I ssh-ed into it, I noticed a 3 DRT processes that were hung (since February 6/7). I also couldn't start Terminal.app (it would hang with 100% CPU before showing me the shell). Nicolas restarted the bot, we'll see if this goes away.
(In reply to comment #21) > Nicolas restarted the bot, we'll see if this goes away. Yes, the tests are passing now. From http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5/builds/1505/steps/webkit_tests/logs/stdio: 2011-02-23 13:11:14,603 706 worker_mixin.py:154 DEBUG worker-0 editing/deleting/delete-3928305-fix.html passed 2011-02-23 13:11:14,605 706 printing.py:541 INFO editing/deleting/delete-3928305-fix.html -> unexpected pass 2011-02-23 13:11:14,676 706 worker_mixin.py:154 DEBUG worker-0 editing/deleting/delete-3959464-fix.html passed 2011-02-23 13:11:14,686 706 printing.py:541 INFO editing/deleting/delete-3959464-fix.html -> unexpected pass
Tests are now passing, TIMEOUT expectations removed with http://trac.webkit.org/changeset/79480.
(In reply to comment #23) > Tests are now passing, TIMEOUT expectations removed with http://trac.webkit.org/changeset/79480. Great! Thanks for fixing this bug.