Bug 35905 – REGRESSION(55699?): media/video-no-autoplay.html times out on Leopard Commit Bot

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Eric Seidel (no email)]

Created attachment 50279 [details]
Hang report from Leopard Commit Bot (media/video-no-autoplay.html)

REGRESSION: media/video-no-autoplay.html times out on Leopard Commit Bot

Log:
2 patches in commit-queue [50251, 50263]
Cleaning working directory
Updating working directory
Building WebKit
Running Python unit tests
Running Perl unit tests
Running JavaScriptCore tests
Running run-webkit-tests
Failed to run "['WebKitTools/Scripts/run-webkit-tests', '--no-launch-safari', '--exit-after-n-failures=1', '--quiet']" exit_code: 1
.................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 289 tests in 0.920s

OK
/Users/eseidel/Projects/CommitQueue/WebKitTools/Scripts/webkitperl/VCSUtils_unittest/fixChangeLogPatch.......ok
/Users/eseidel/Projects/CommitQueue/WebKitTools/Scripts/webkitperl/VCSUtils_unittest/generatePatchCommand....ok
/Users/eseidel/Projects/CommitQueue/WebKitTools/Scripts/webkitperl/VCSUtils_unittest/gitdiff2svndiff.........ok
/Users/eseidel/Projects/CommitQueue/WebKitTools/Scripts/webkitperl/VCSUtils_unittest/parseDiff...............ok
/Users/eseidel/Projects/CommitQueue/WebKitTools/Scripts/webkitperl/VCSUtils_unittest/parseDiffHeader.........ok
/Users/eseidel/Projects/CommitQueue/WebKitTools/Scripts/webkitperl/VCSUtils_unittest/parsePatch..............ok
/Users/eseidel/Projects/CommitQueue/WebKitTools/Scripts/webkitperl/VCSUtils_unittest/runPatchCommand.........ok
All tests successful.
Files=7, Tests=133,  1 wallclock secs ( 0.24 cusr +  0.07 csys =  0.31 CPU)
Running build-dumprendertree
Compiling Java tests
make: Nothing to be done for `default'.
Running tests from /Users/eseidel/Projects/CommitQueue/LayoutTests
Testing 12455 test cases.
media/video-no-autoplay.html -> timed out
Sampling process 40023 for 10 seconds with 10 milliseconds of run time between samples
Sampling completed, processing symbols...
Sample analysis of process 40023 written to file /Users/eseidel/Library/Logs/DumpRenderTree/HangReport.txt

Exiting early after 1 failures. 9185 tests run.
368.49s total testing time

9184 test cases (99%) succeeded
1 test case (<1%) timed out
9 test cases (<1%) had stderr output
Unabled to successfully build and test
Not proceeding with work item. Sleeping until 2010-03-08 20:51:28 (5 mins).

That was the first time the commit-queue saw the failure, so the regression must have been before 20:46 PST this evening.

[Eric Seidel (no email)]

http://trac.webkit.org/changeset/55699 seems related.

I'll make the commit-bot do a full rebuild of WebKit and see if that fixes the issue.

[Eric Seidel (no email)]

The commit-queue was blocked on a red tree most of the afternoon. The last time it cycled without blocking was:
Empty queue
No work item. Sleeping until 2010-03-08 15:37:12 (5 mins).

The last commit it made (and thus ran the tests successfully) before going empty was http://trac.webkit.org/changeset/55680 which was 03/08/10 13:06:08

[Eric Seidel (no email)]

I'm technically on vacation all week, so I will not be checking the queue regularly.  However, others can view its status using http://webkit-commit-queue.appspot.com/

Right now it says "Unabled to successfully build and test" due to this test timeout.

[Eric Seidel (no email)]

This seems to have resolved itself for now.  Closing.

[Eric Seidel (no email)]

I did a clean build, and then hit a build error in XPathGrammar.h.  Looking at that file history it seemed it hadn't changed in weeks, so I did another clean build and now the commit-queue is running fine.  Not sure what caused either the consistent timeout problem or the spurious build error.

[Eric Seidel (no email)]

Now media/video-play-empty-events.html -> timed out
 is failing.  I think somehitng is wrong still.

[Eric Seidel (no email)]

I've seen the no-autoplay test fail again too. I think 55699 may be a real regression.

[Eric Seidel (no email)]

This timeout is intermittent.

The bots seem to be seieing timeouts as well, possibly related to this.

This regression (along with is currently blocking 19 patches in the commit-queue along with intermittent timeouts related to bug 35824.

[Brady Eidson]

(In reply to comment #8)
> This timeout is intermittent.
> 
> The bots seem to be seieing timeouts as well, possibly related to this.
> 
> This regression (along with is currently blocking 19 patches in the
> commit-queue along with intermittent timeouts related to bug 35824.


What "timeout" did the fix for 35824 cause, out of curiosity?

I'm aware of the test failures, that we're cleaning up on various bots by removing cookies.plists.  But this is the first I've heard of any timeouts.

[Eric Seidel (no email)]

(In reply to comment #9)
> What "timeout" did the fix for 35824 cause, out of curiosity?
> 
> I'm aware of the test failures, that we're cleaning up on various bots by
> removing cookies.plists.  But this is the first I've heard of any timeouts.

As you correctly read, this bug is not related to cookies.  This bug is about http://trac.webkit.org/changeset/55699 which seems to have caused an intermittent media/video-no-autoplay.html on the commit bot, and may be responsible for other media timeouts on the bots:
http://build.webkit.org/results/Leopard%20Intel%20Release%20(Tests)/r55768%20(11425)/results.html
http://build.webkit.org/results/Leopard%20Intel%20Release%20(Tests)/r55760%20(11417)/results.html

[Darin Adler]

(In reply to comment #10)
> As you correctly read, this bug is not related to cookies.  This bug is about
> http://trac.webkit.org/changeset/55699 which seems to have caused an
> intermittent media/video-no-autoplay.html on the commit bot, and may be
> responsible for other media timeouts on the bots:
> http://build.webkit.org/results/Leopard%20Intel%20Release%20(Tests)/r55768%20(11425)/results.html
> http://build.webkit.org/results/Leopard%20Intel%20Release%20(Tests)/r55760%20(11417)/results.html

So you've said it's my check-in causing this; that gets my attention. Does anyone have any other data? For example, is this Leopard-only? Is it bot-only or is someone seeing it on their own computer too?

[Eric Seidel (no email)]

The Leopard Commit Bot has also seen:
media/video-played-collapse.html -> timed out
since this regression.

[Eric Seidel (no email)]

Created attachment 50428 [details]
Hang report from Leopard Commit Bot (media/video-played-collapse.html)

[Eric Seidel (no email)]

I checked the logs.  The first failure the commit queue saw was of "media/video-no-autoplay" and was shortly after 2010-03-08 20:30:27

[Eric Seidel (no email)]

The bots have seen http/tests/media/video-play-stall.html timeout, but we don't have hang reports from the bots.  I don't have data on when the bots first saw http/tests/media/video-play-stall.html timeout.

[Eric Seidel (no email)]

This looks like a hang down in CG/AppKit/LayerKit.  Perhaps I just need to restart the commit bot?  I've CC'd Eric Carlson and Simon Fraser in case either of them have theories.

[Simon Fraser (smfr)]

We disable accelerated compositing on Leopard now (until some future QuickTime version revives it). I don't understand how we're getting into compositing mode.

[Darin Adler]

(In reply to comment #17)
> We disable accelerated compositing on Leopard now (until some future QuickTime
> version revives it). I don't understand how we're getting into compositing
> mode.

WebGL was turned on for the buildbots; I think Oliver did it. Since <http://trac.webkit.org/changeset/55685> and <http://trac.webkit.org/changeset/55697>, that means that accelerated compositing is on for the Leopard buildbots.

[Simon Fraser (smfr)]

I didn't know that a) cmarrin put the webgl forcing pref back in, and b) that webgl was turned on in DRT.

So maybe this is the CoreVideo crash?

[Darin Adler]

(In reply to comment #19)
> I didn't know that a) cmarrin put the webgl forcing pref back in, and b) that
> webgl was turned on in DRT.
> 
> So maybe this is the CoreVideo crash?

Seems likely.

[Eric Seidel (no email)]

I agree, this is very reminiscent of the historical CV issues (at least from what I saw of them).

I have two CV-related crash dumps since 3/8/10 on the commit bot.  I'm happy to attach both if that would be helpful.

[Eric Seidel (no email)]

This is still the primary cause of build.webkit.org redness, and I believe to be the reason why the commit queue is still stuck around 19 patches (every so often it actually succeeded in landing one when this test happens to have passed on the bots and passes twice in a row).

Is someone still investigating or should we roll 55699 out?

[Simon Fraser (smfr)]

If anything gets rolled out, it should be 55697.

[Eric Seidel (no email)]

I agree.  Lets roll out r55697 until we have a work-around for this evil CV bug.

[Simon Fraser (smfr)]

Or we just turn off webgl on the bots. I think Oliver Hunt turned it on.

[Simon Fraser (smfr)]

I think this is really just a DRT bug: bug 36033.

[Simon Fraser (smfr)]

Created attachment 50546 [details]
Patch

[Oliver Hunt]

Comment on attachment 50546 [details]
Patch

r=me

[Eric Seidel (no email)]

Attachment 50546 [details] was posted by a committer and has review+, assigning to Simon Fraser for commit.

[Simon Fraser (smfr)]

Landed in http://trac.webkit.org/changeset/55857