Bug 28624

Summary: REGRESSION: media/video-pause-empty-events.html is occasionally timing out on bots
Product: WebKit Reporter: Eric Seidel (no email) <eric>
Component: New BugsAssignee: Nobody <webkit-unassigned>
Status: RESOLVED FIXED    
Severity: Normal CC: commit-queue, eric.carlson, simon.fraser
Priority: P1    
Version: 528+ (Nightly build)   
Hardware: PC   
OS: OS X 10.5   
Bug Depends on: 28845, 29745    
Bug Blocks: 38912    
Attachments:
Description Flags
spin report from most recent hang from landing bug 26700. This looks like these hangs are caused by bug 28624.
none
Possibly unrelated crashlog seen with HW compositing disabled (during http/tests/messaging/cross-domain-message-event-dispatch.html)
none
Proposed solution for disabling hardware compositing on Leopard
none
Set setAcceleratedCompositingEnabled:YES to work around bug 29751 simon.fraser: review+, commit-queue: commit-queue-

Description Eric Seidel (no email) 2009-08-21 14:33:13 PDT
REGRESSION: media/video-pause-empty-events.html is occasionally timing out on bots

http://build.webkit.org/results/Leopard%20Intel%20Release%20(Tests)/r47639%20(4313)/results.html

Seems to have been for just the last few hours.  Keeps popping up every other build or so.  I've not yet determined which revision started the problem.
Comment 1 Eric Seidel (no email) 2009-08-21 14:34:14 PDT
http://build.webkit.org/builders/Leopard%20Intel%20Release%20%28Tests%29/builds/4306

is the first build which seems to have seen the timeout.
Comment 2 Eric Seidel (no email) 2009-08-21 14:36:04 PDT
Actually:
http://build.webkit.org/builders/Leopard%20Intel%20Release%20%28Tests%29/builds/4295
is an earlier build from Thu Aug 20 21:27:41 2009 which saw the timeout.
Comment 3 Eric Carlson 2009-08-21 15:14:10 PDT
That build was triggered by ollie's JSC variable count change: http://trac.webkit.org/changeset/47620. Don't know how that would make this test timeout.
Comment 4 Eric Seidel (no email) 2009-08-21 15:42:02 PDT
I don't think it's related to Oliver's change at all.  I think this test is just flakey.  What caused the flake, I don't know, but something before Oliver's change.
Comment 5 Eric Seidel (no email) 2009-09-01 16:38:50 PDT
This could have the same root cause as bug 28845.
Comment 6 Eric Seidel (no email) 2009-09-08 10:50:22 PDT
Just saw media/video-source-error.html timeout.
http://build.webkit.org/results/Leopard%20Intel%20Release%20(Tests)/r48163%20(4792)/results.html
Possibly the same cause.
Comment 7 Eric Seidel (no email) 2009-09-09 09:33:29 PDT
Seen again landing bug 29038.media/video-source-error.html seems to be the most common timeout seen by the commit-bot.
Comment 8 Eric Seidel (no email) 2009-09-09 15:11:50 PDT
Another timeout of media/video-source-error.html seen while landing bug 26700.  I'll attach the timeout file.
Comment 9 Eric Seidel (no email) 2009-09-09 15:12:37 PDT
Created attachment 39306 [details]
spin report from most recent hang from landing bug 26700.  This looks like these hangs are caused by bug 28624.
Comment 10 Eric Seidel (no email) 2009-09-09 18:32:43 PDT
Yet another timeout when landing bug 29001. :(
Comment 11 Eric Seidel (no email) 2009-09-09 18:45:28 PDT
Yet another timeout landing bug 29065.
Comment 12 Simon Fraser (smfr) 2009-09-14 17:43:19 PDT
I'm running the media and compositing tests hundreds of times (via the patch in bug 29263), and haven't get crashed, on 10.5.8.
Comment 13 Eric Seidel (no email) 2009-09-23 14:01:23 PDT
media/video-source-type.html is a new one, probably the same root cause:
http://build.webkit.org/results/Leopard%20Intel%20Release%20(Tests)/r48683%20(5277)/results.html
Comment 14 Eric Seidel (no email) 2009-09-23 16:44:36 PDT
video-loop and video-source again just now:
http://build.webkit.org/results/Leopard%20Intel%20Release%20(Tests)/r48687%20(5281)/results.html

Looks like we have a few more media tests to skip. :(
Comment 15 Eric Seidel (no email) 2009-09-24 17:39:35 PDT
Simon suggests that we disable hardware compositiing until this bug can be fixed in QuickTime:

defaults write .... WebKitAcceleratedCompositingEnabled -bool NO

We will need to make a patch to DRT to set that default on leopard builds (at least until a new version of quicktime is available).

Eric, perhaps you could help me with the quicktime version check?  I'd like to make it only turn it off for bad versions of QuickTime.
Comment 16 Eric Seidel (no email) 2009-09-24 17:39:56 PDT
By eric I meant simon of course. :)
Comment 17 Eric Seidel (no email) 2009-09-25 11:54:32 PDT
Wow, this disaster never ends. :(

So I have a working patch which disables HW Compositing and makes the media tests reliably pass.

However, now I just saw this test fail:
http/tests/messaging/cross-domain-message-event-dispatch.html -> crashed

With a crash under:
Thread 8 Crashed:
0   ...uickTimeStreaming.component	0x1d4a939e QTSInetCachedDownload_SetProperty + 315
1   ...uickTimeStreaming.component	0x1d4af2ca QTSInetDownload_TransactionNotification + 324
2   ...uickTimeStreaming.component	0x1d4b28ca QTSInetTransaction_ProcessResponseHeader + 1659
3   ...uickTimeStreaming.component	0x1d4b39ea QTSInetTransaction_ReadStreamCallback + 699
4   com.apple.CoreFoundation      	0x955cab1b _signalEventSync + 91
5   com.apple.CoreFoundation      	0x955cabbe _cfstream_solo_signalEventSync + 126
6   com.apple.CoreFoundation      	0x955cd057 CFReadStreamSignalEvent + 39
7   com.apple.CFNetwork           	0x91d00e00 HTTPReadStream::streamEvent(unsigned long) + 68

I'll attach the crash log to the bug.  It may be completely unrelated.
Comment 18 Eric Seidel (no email) 2009-09-25 11:55:20 PDT
Created attachment 40129 [details]
Possibly unrelated crashlog seen with HW compositing disabled (during http/tests/messaging/cross-domain-message-event-dispatch.html)
Comment 19 Eric Seidel (no email) 2009-09-25 11:56:28 PDT
My "disaster" comment was intentional hyperbole, but I realize now it might not read that way. ;)  All and all this mess has been annoying, but certainly not a "disaster".
Comment 20 Eric Seidel (no email) 2009-09-25 12:00:25 PDT
Also, these two tests fail with HW compositing disabled:
fast/media/mq-transform-02.html
fast/media/mq-transform-03.html

I guess I could add special "passing" test results for Leopard since I assume these failures just need to be tracked as new bugs.
Comment 21 Eric Seidel (no email) 2009-09-25 13:31:06 PDT
Created attachment 40140 [details]
Proposed solution for disabling hardware compositing on Leopard
Comment 22 Eric Seidel (no email) 2009-09-25 13:32:57 PDT
Simon: I'm very interested in your comments as to this approach.

1.  If you think this is a good way to go about stopping the crashes.
2.  If I did my OS version check correctly.
3.  If I did my QT version check correctly (and if 7.6.5 is a reasonable revision to assume this might get fixed in).

Thanks!
Comment 23 Eric Seidel (no email) 2009-09-25 13:34:31 PDT
Furthermore, I only saw the crash seen in https://bugs.webkit.org/attachment.cgi?id=40129 once.  I've run the layout tests 3 times with this patch.  I expect we might be trading one bout of flakiness for another by disabling this if the crash from https://bugs.webkit.org/attachment.cgi?id=40129 reproduces again.  But hopefully we're moving one step further towards stability.
Comment 24 Eric Seidel (no email) 2009-09-25 13:38:53 PDT
4.  If you would rather I skip fast/media/mq-transform-02.html and fast/media/mq-transform-03.html on mac-leopard instead of checking in new results, let me know.
Comment 25 Eric Seidel (no email) 2009-09-25 15:13:39 PDT
I will need to slightly amend my change or fix bug 29751 or others may be bitten by bug 29751.
Comment 26 Eric Seidel (no email) 2009-09-25 15:24:13 PDT
Created attachment 40151 [details]
Set setAcceleratedCompositingEnabled:YES to work around bug 29751
Comment 27 Eric Seidel (no email) 2009-09-25 16:37:28 PDT
Comment on attachment 40151 [details]
Set setAcceleratedCompositingEnabled:YES to work around bug 29751

Thanks for the review.  I'll close out the rest of the bugs after we verify these crashes are gone for a couple days.
Comment 28 WebKit Commit Bot 2009-09-25 19:27:32 PDT
Comment on attachment 40151 [details]
Set setAcceleratedCompositingEnabled:YES to work around bug 29751

Rejecting patch 40151 from commit-queue.

Patch https://bugs.webkit.org/attachment.cgi?id=40151 from bug 28624 failed to download and apply.
Comment 29 WebKit Commit Bot 2009-09-28 14:22:58 PDT
Committed r48833: <http://trac.webkit.org/changeset/48833>
Comment 30 Eric Seidel (no email) 2009-10-01 12:14:31 PDT
*** Bug 29624 has been marked as a duplicate of this bug. ***