Bug 57983 - [Qt] media tests are flakey
Summary: [Qt] media tests are flakey
Status: RESOLVED INVALID
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: PC All
: P2 Normal
Assignee: Nancy Piedra
URL:
Keywords: Qt, QtTriaged
Depends on:
Blocks:
 
Reported: 2011-04-06 14:29 PDT by Csaba Osztrogonác
Modified: 2013-12-08 05:53 PST (History)
6 users (show)

See Also:


Attachments
Skip only failing media tests (2.74 KB, patch)
2011-06-22 05:36 PDT, Nancy Piedra
ossy: review-
ossy: commit-queue-
Details | Formatted Diff | Diff
Skip only failing media tests (2nd try) (3.05 KB, patch)
2011-06-23 10:42 PDT, Nancy Piedra
ossy: review-
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Csaba Osztrogonác 2011-04-06 14:29:00 PDT
Nowadays regulary one or two random media tests crash on the Qt bot.

The bot must be always green to be able to catch regression, 
so flakey tests are absolutely unacceptable.

I'm going to disable all media tests until we find what cause this flakiness.
Comment 1 Csaba Osztrogonác 2011-04-06 14:32:43 PDT
I disabled all media tests: http://trac.webkit.org/changeset/83100
Comment 2 Csaba Osztrogonác 2011-04-06 14:37:09 PDT
Guys, first we should try to find what cause these random crashes.
If we found and fixed it, we can enable media tests.

But we shouldn't kill the reputation and the trustiness 
of the Qt bot with these crazy crashy flakey tests.
Comment 3 Benjamin Poulain 2011-04-06 15:29:33 PDT
The flakey I could find in today's history: 

media/remove-from-document.html -> crashed
media/video-preload.html -> crashed
media/video-plays-past-end-of-test.html -> crashed
media/video-src.html -> crashed
media/video-volume.html -> crashed

That looks pretty random :(
Comment 4 Nancy Piedra 2011-06-22 05:36:17 PDT
Created attachment 98158 [details]
Skip only failing media tests

I downloaded the Qt Webkit VM to analyze the flakey media tests.

I modified the Skipped file to skip only those tests that failed outright on the VM.

I then ran the layout tests many times and did not see any of the media tests fail.  

With this set of tests I did not see any flakiness.  I'm not sure if this bug was logged before or after the buildbot was upgraded to Qt Mobility 1.2 so it is not clear to me what version of Qt Mobility was in place when the flakiness was observerd.

In other words, I could not reproduce the flakiness in order to find a fix for it but I could find a stable set of tests.
Comment 5 WebKit Review Bot 2011-06-22 05:36:30 PDT
Comment on attachment 98158 [details]
Skip only failing media tests

Rejecting attachment 98158 [details] from review queue.

nancy.piedra@nokia.com does not have reviewer permissions according to http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/config/committers.py.

- If you do not have reviewer rights please read http://webkit.org/coding/contributing.html for instructions on how to use bugzilla flags.

- If you have reviewer rights please correct the error in Tools/Scripts/webkitpy/common/config/committers.py by adding yourself to the file (no review needed).  The commit-queue restarts itself every 2 hours.  After restart the commit-queue will correctly respect your reviewer rights.
Comment 6 WebKit Review Bot 2011-06-22 05:37:14 PDT
Comment on attachment 98158 [details]
Skip only failing media tests

Rejecting attachment 98158 [details] from commit-queue.

nancy.piedra@nokia.com does not have committer permissions according to http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/config/committers.py.

- If you do not have committer rights please read http://webkit.org/coding/contributing.html for instructions on how to use bugzilla flags.

- If you have committer rights please correct the error in Tools/Scripts/webkitpy/common/config/committers.py by adding yourself to the file (no review needed).  The commit-queue restarts itself every 2 hours.  After restart the commit-queue will correctly respect your committer rights.
Comment 7 Nancy Piedra 2011-06-22 05:41:12 PDT
Sorry, meant to do review? commit-queue?  but hit + instead.
Comment 8 Laszlo Gombos 2011-06-23 06:32:45 PDT
I talked to Nancy and she really did her best to filter out flaky tests. I think we should try to enable the proposed list.
Comment 9 Alexis Menard (darktears) 2011-06-23 06:54:19 PDT
(In reply to comment #8)
> I talked to Nancy and she really did her best to filter out flaky tests. I think we should try to enable the proposed list.

Yep I think we should give a try.
Comment 10 Alexis Menard (darktears) 2011-06-23 06:55:06 PDT
Comment on attachment 98158 [details]
Skip only failing media tests

View in context: https://bugs.webkit.org/attachment.cgi?id=98158&action=review

> LayoutTests/ChangeLog:6
> +        Skipped the Qt media tests that are failing on the Qt Webkit VM

Extra line here.
Comment 11 Csaba Osztrogonác 2011-06-23 06:58:19 PDT
Comment on attachment 98158 [details]
Skip only failing media tests

$ Tools/Scripts/run-webkit-tests media 

media/video-preload.html -> failed
media/video-seek-no-src-exception.html -> timed out
media/video-seekable.html -> failed
media/video-source-inserted.html -> timed out
media/video-source-moved.html -> timed out
media/video-src-change.html -> timed out
media/video-src-remove.html -> failed
media/video-src-set.html -> timed out
media/video-src.html -> timed out
media/video-volume.html -> failed

and then run-webkit-tests waits forever ... :(

r- now, it needs more investigation
Comment 12 Csaba Osztrogonác 2011-06-23 07:06:20 PDT
media/adopt-node-crash.html -> timed out
Comment 13 Csaba Osztrogonác 2011-06-23 07:10:57 PDT
media/video-source-none-supported.html -> timed out
media/video-src-invalid-remove.html -> failed
Comment 14 Nancy Piedra 2011-06-23 07:13:31 PDT
Ossy - Are you running the tests on the VM?  If not, what is the configuration of the system you are running the tests on.
Comment 15 Csaba Osztrogonác 2011-06-23 07:14:35 PDT
media/video-source-removed.html -> crashed
media/before-load-member-access.html -> timed out
media/video-src-source.html -> timed out

Media tests are still flakey. :-S Unskipping flakey tests won't solve 
the bug, we shouldn't enable them until we find and fix the bug.
Comment 16 Csaba Osztrogonác 2011-06-23 07:15:21 PDT
(In reply to comment #14)
> Ossy - Are you running the tests on the VM?  If not, what is the configuration of the system you are running the tests on.

I ran tests on the buildbot, not on the VM. But they have same configuration.
Comment 17 Csaba Osztrogonác 2011-06-23 07:16:07 PDT
I tried to find flakey tests with:
$ run-tests-in-xvfb.sh media --iterations 100
Comment 18 Nancy Piedra 2011-06-23 07:20:33 PDT
OK.  I will try what you have done and see if I get similar results.  If I do, I will continue to debug.
Comment 19 Nancy Piedra 2011-06-23 07:43:37 PDT
I ran the following on the VM and saw no tests fail or time out:
run-tests-in-xvfb.sh media --iterations 100

I will keep trying to see if I can reproduce what you're seeing on the buildbot.
Comment 20 Csaba Osztrogonác 2011-06-23 07:45:23 PDT
(In reply to comment #19)
> I ran the following on the VM and saw no tests fail or time out:
> run-tests-in-xvfb.sh media --iterations 100
> 
> I will keep trying to see if I can reproduce what you're seeing on the buildbot.

Very strange .... I try to check it on the VM too.
Comment 21 Alexis Menard (darktears) 2011-06-23 07:54:25 PDT
(In reply to comment #20)
> (In reply to comment #19)
> > I ran the following on the VM and saw no tests fail or time out:
> > run-tests-in-xvfb.sh media --iterations 100
> > 
> > I will keep trying to see if I can reproduce what you're seeing on the buildbot.
> 
> Very strange .... I try to check it on the VM too.

That's exactly what I keep saying. The VM does not behave the same way as the bots. It also seems to behave differently whether you run with VirtualBox or VMWare, it sounds unbelievable but it is.
Comment 22 Nancy Piedra 2011-06-23 07:58:56 PDT
I ran it again and now I am seeing some failures.
Comment 23 Nancy Piedra 2011-06-23 10:42:56 PDT
Created attachment 98369 [details]
Skip only failing media tests (2nd try)

Ossy - I commented out a few more tests.  Could you try this?  If it doesn't work on the buildbot then I will look into the flakiness more.
Comment 24 Csaba Osztrogonác 2011-06-24 04:53:46 PDT
Comment on attachment 98369 [details]
Skip only failing media tests (2nd try)

r- again, because I got flakey fails and timeouts

media/video-single-valid-source.html
media/adopt-node-crash.html
media/video-source-inserted.html
media/video-src.html
media/before-load-member-access.html
media/broken-video.html
media/constructors.html
media/controls-css-overload.html
Comment 25 Zan Dobersek 2013-12-08 05:53:24 PST
The Qt port was removed from the project, so this bug is not valid anymore.