Bug 54885 – Many tests started timing out on the chrome mac canaries around r79035

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Andrew Wilson]

It's a big list - here's the flakiness dashboard:

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#group=%40ToT%20-%20chromium.org&tests=editing%2Fdeleting%2Fdelete-3928305-fix.html%2Cediting%2Fdeleting%2Fdelete-3959464-fix.html%2Cediting%2Fdeleting%2Fdelete-and-undo.html%2Cediting%2Fdeleting%2Fdelete-block-contents-003.html%2Cediting%2Fdeleting%2Fdelete-contiguous-ws-001.html%2Cediting%2Fdeleting%2Fdelete-tab-002.html%2Cediting%2Fdeleting%2Fdelete-tab-003.html%2Cediting%2Fdeleting%2Fdelete-to-select-table.html%2Cediting%2Fdeleting%2Fdelete-trailing-ws-002.html%2Cediting%2Fdeleting%2Fdelete-ws-fixup-001.html%2Cediting%2Fdeleting%2Fdelete-ws-fixup-002.html%2Cediting%2FexecCommand%2F5569741.html%2Cediting%2FexecCommand%2Finsert-list-and-stitch.html%2Cediting%2FexecCommand%2FinsertHorizontalRule.html%2Cediting%2FexecCommand%2Foutdent-selection.html%2Cediting%2Finserting%2F5418891.html%2Cediting%2Finserting%2Fediting-empty-divs.html%2Cediting%2Finserting%2Finsert-3907422-fix.html%2Cediting%2Finserting%2Finsert-br-004.html%2Cediting%2Finserting%2Finsert-br-005.html%2Cediting%2Finserting%2Finsert-br-007.html%2Cediting%2Finserting%2Finsert-div-007.html%2Cediting%2Finserting%2Finsert-div-012.html%2Cediting%2Finserting%2Finsert-div-013.html%2Cediting%2Finserting%2Finsert-div-014.html%2Cediting%2Finserting%2Finsert-div-015.html%2Cediting%2Finserting%2Finsert-div-016.html%2Cediting%2Finserting%2Finsert-div-017.html%2Cediting%2Finserting%2Finsert-div-018.html%2Cediting%2Finserting%2Finsert-div-019.html%2Cediting%2Finserting%2Finsert-div-020.html%2Cediting%2Finserting%2Finsert-div-021.html%2Cediting%2Finserting%2Finsert-div-022.html%2Cediting%2Finserting%2Finsert-div-025.html%2Cediting%2Finserting%2Finsert-div-027.html%2Cediting%2Finserting%2Finsert-paragraph-02.html%2Cediting%2Finserting%2Finsert-paragraph-03.html%2Cediting%2Finserting%2Finsert-tab-002.html%2Cediting%2Finserting%2Finsert-tab-003.html%2Cediting%2Finserting%2Finsert-text-with-newlines.html%2Cediting%2Finserting%2Fparagraph-separator-01.html%2Cediting%2Finserting%2Fparagraph-separator-02.html%2Cediting%2Finserting%2Fparagraph-separator-03.html%2Cediting%2Finserting%2Freturn-key-with-selection-001.html%2Cediting%2Finserting%2Freturn-key-with-selection-002.html%2Cediting%2Finserting%2Freturn-key-with-selection-003.html%2Cediting%2Finserting%2Ftyping-002.html%2Cediting%2Finserting%2Ftyping-around-br-001.html%2Cediting%2Fpasteboard%2F4242293.html%2Cediting%2Fpasteboard%2F4944770-1.html%2Cediting%2Fpasteboard%2F8145-2.html%2Cediting%2Fpasteboard%2Fcopy-in-password-field.html%2Cediting%2Fpasteboard%2Fcut-text-001.html%2Cediting%2Fpasteboard%2Finterchange-newline-1.html%2Cediting%2Fpasteboard%2Fmerge-after-delete-1.html%2Cediting%2Fpasteboard%2Fmerge-after-delete-2.html%2Cediting%2Fpasteboard%2Fmerge-after-delete.html%2Cediting%2Fpasteboard%2Fmerge-end-4.html%2Cediting%2Fpasteboard%2Fmerge-end-blockquote.html%2Cediting%2Fpasteboard%2Fmerge-end-list.html%2Cediting%2Fpasteboard%2Fmerge-end-table.html%2Cediting%2Fpasteboard%2Fpaste-blockquote-3.html%2Cediting%2Fpasteboard%2Fpaste-text-001.html%2Cediting%2Fpasteboard%2Fpaste-text-012.html%2Cediting%2Fpasteboard%2Fpaste-text-013.html%2Cediting%2Fpasteboard%2Fpaste-text-014.html%2Cediting%2Fpasteboard%2Fpaste-text-019.html%2Cediting%2Fpasteboard%2Fpaste-xml.xhtml%2Cediting%2Fpasteboard%2Fundoable-fragment-removes.html%2Cediting%2Fselection%2F13804.html%2Cediting%2Fselection%2F5234383-1.html%2Cediting%2Fselection%2F5234383-2.html%2Cediting%2Fselection%2Fextend-by-word-001.html%2Cediting%2Fselection%2Fmove-backwords-by-word-001.html%2Cediting%2Fselection%2Fmove-by-character-001.html%2Cediting%2Fselection%2Fmove-by-line-001.html%2Cediting%2Fselection%2Fmove-by-word-001.html%2Cediting%2Fselection%2Fselect-from-textfield-outwards.html%2Cediting%2Fselection%2Funrendered-002.html%2Cediting%2Fspelling%2Finline_spelling_markers.html%2Cediting%2Fspelling%2Fspelling.html%2Cediting%2Fstyle%2F5046875-2.html%2Cediting%2Fstyle%2Fblock-styles-007.html%2Cediting%2Fstyle%2Fcreate-block-for-style-002.html%2Cediting%2Fstyle%2Fcreate-block-for-style-003.html%2Cediting%2Fstyle%2Fcreate-block-for-style-004.html%2Cediting%2Fstyle%2Fcreate-block-for-style-006.html%2Cediting%2Fstyle%2Fcreate-block-for-style-008.html%2Cediting%2Fstyle%2Fcreate-block-for-style-009.html%2Cediting%2Fstyle%2Fcreate-block-for-style-011.html%2Cediting%2Fstyle%2Fcreate-block-for-style-012.html%2Cediting%2Fstyle%2Fcreate-block-for-style-013.html%2Cfast%2Fdom%2Fblur-contenteditable.html%2Cfast%2Fdom%2Fobject-embed-plugin-scripting.html%2Cfast%2Fdom%2Fobject-plugin-hides-properties.html%2Cfast%2Fevents%2Fonchange-passwordfield.html%2Cfast%2Fevents%2Ftabindex-focus-blur-all.html%2Cfast%2Fforms%2Ffocus-control-to-page.html%2Cfast%2Fforms%2Ffocus2.html%2Cfast%2Fforms%2Fpassword-doubleclick-selection.html%2Cfast%2Fforms%2Fpassword-placeholder-text-security.html%2Cfast%2Fimages%2Fembed-image.html%2Cfast%2Fimages%2Fobject-image.html%2Cplugins%2Fembed-attributes-setting.html%2Cplugins%2Fembed-attributes-style.html%2Cplugins%2Fiframe-shims.html%2Cuserscripts%2Fuser-script-video-document.html

Interestingly, it doesn't seem to affect the main chrome build tree, since we've already rolled past this revision, so it's probably test-infrastructure related. Not sure how 79035 could be related to this and I'm reluctant to roll it out because it includes a fix for the rebaselining tool, so I'm just going to mark these tests as TIMEOUT for now.

[Dirk Pranke]

Interesting that a lot of these tests seem to be editing-related. I wonder if there's something taking out some sort of system-wide global lock that is keeping the tests from running in parallel with other DRT tests. Something having to do with focus, maybe? This is wild speculation ...

[Dirk Pranke]

Seems much more likely to have been triggered by 79047/79062 (me enabling multiple processes).

[Dirk Pranke]

investigating ...

I will also note that while I would expect additional flakiness on SL bots, if anything, I would've expected the Leopard bots to become *more* stable, because we stopped using threads completely.

[Dirk Pranke]

I am reverting the behavior on 10.5 (see bug 54925), so we'll see if this goes away ...

[Hajime Morrita]

Tests pass on the Debug build, but timed-out on the Release.

[Ryosuke Niwa]

Editing tests are still timing out on WebKit Mac 10.5 (release).  We've lost significant test coverage due to this bug :(

[Dirk Pranke]

It seems unlikely to me that its a NRWT thing at this point. Is it possible that something else changed in that timeframe that's causing this regression?

[Ryosuke Niwa]

Maybe http://trac.webkit.org/changeset/79038/ ?

[Kenneth Russell]

It's possible that my change in http://trac.webkit.org/changeset/79038/ caused this breakage. With this change , if the minimum DOM timer interval isn't manually set then it may be set to a high value. I'll try to run these tests locally to reproduce.

[Kenneth Russell]

I'm having difficulty reproducing these test timeouts locally. I've built DRT Release mode out of a Chromium checkout, reversed the timeout expected results in r79228 and ran the following:

cat tests.txt | xargs src/webkit/tools/layout_tests/run_webkit_tests.sh --chromium

with the attached tests.txt. None of the tests timed out. This is on Snow Leopard. Is anyone else able to reproduce the timeouts?

[Kenneth Russell]

Created attachment 83425 [details]
Test list

[Kenneth Russell]

Created attachment 83426 [details]
Revert of code changes from r79038

[Kenneth Russell]

After talking with dpranke, let's try reverting the code changes from r79038 to see whether they clean up the flakiness dashboard.

[James Robinson]

Comment on attachment 83426 [details]
Revert of code changes from r79038

Worth a try.  Feel free to land reverts unreviewed in the future if you prefer.

[Kenneth Russell]

Committed r79387: <http://trac.webkit.org/changeset/79387>

[Ryosuke Niwa]

(In reply to comment #15)
> Committed r79387: <http://trac.webkit.org/changeset/79387>

It seems like this wasn't the cause :( http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5/builds/1482

[Ryosuke Niwa]

I sent out an email internally at Goole to look for a Leopard machine.

[Kenneth Russell]

Since the code changes in r79038 weren't the root cause, I'm going to reapply them.

[Kenneth Russell]

Committed r79456: <http://trac.webkit.org/changeset/79456>

[Kenneth Russell]

Oops, accidentally closed the bug while landing that last patch. Reopening.

[Mihai Parparita]

I was not able to reproduce this on a Leopard machine (doing a release build with build-webkit --chromium). James and I also noticed that:
- non-editing tests were strangely affected too (e.g. userscripts/user-script-video-document.html)
- the debug canary bots were not affected: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#group=%40ToT%20-%20chromium.org&tests=editing%2Fdeleting%2Fdelete-3928305-fix.html&showAllRuns=true
- the DEPS bots are not affected: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=editing%2Fdeleting%2Fdelete-3928305-fix.html&showAllRuns=true

We therefore suspect that the bot is in a weird state. When I ssh-ed into it, I noticed a 3 DRT processes that were hung (since February 6/7). I also couldn't start Terminal.app (it would hang with 100% CPU before showing me the shell).

Nicolas restarted the bot, we'll see if this goes away.

[Mihai Parparita]

(In reply to comment #21)
> Nicolas restarted the bot, we'll see if this goes away.

Yes, the tests are passing now. From http://build.chromium.org/p/chromium.webkit/builders/Webkit%20Mac10.5/builds/1505/steps/webkit_tests/logs/stdio:

2011-02-23 13:11:14,603 706 worker_mixin.py:154 DEBUG worker-0 editing/deleting/delete-3928305-fix.html passed
2011-02-23 13:11:14,605 706 printing.py:541 INFO   editing/deleting/delete-3928305-fix.html -> unexpected pass
2011-02-23 13:11:14,676 706 worker_mixin.py:154 DEBUG worker-0 editing/deleting/delete-3959464-fix.html passed
2011-02-23 13:11:14,686 706 printing.py:541 INFO   editing/deleting/delete-3959464-fix.html -> unexpected pass

[Mihai Parparita]

Tests are now passing, TIMEOUT expectations removed with http://trac.webkit.org/changeset/79480.

[Ryosuke Niwa]

(In reply to comment #23)
> Tests are now passing, TIMEOUT expectations removed with http://trac.webkit.org/changeset/79480.

Great!  Thanks for fixing this bug.