Bug 57983 – [Qt] media tests are flakey

• Component: Tools / Tests
• Status: RESOLVED INVALID

[Csaba Osztrogonác]

Nowadays regulary one or two random media tests crash on the Qt bot.

The bot must be always green to be able to catch regression, 
so flakey tests are absolutely unacceptable.

I'm going to disable all media tests until we find what cause this flakiness.

[Csaba Osztrogonác]

I disabled all media tests: http://trac.webkit.org/changeset/83100

[Csaba Osztrogonác]

Guys, first we should try to find what cause these random crashes.
If we found and fixed it, we can enable media tests.

But we shouldn't kill the reputation and the trustiness 
of the Qt bot with these crazy crashy flakey tests.

[Benjamin Poulain]

The flakey I could find in today's history: 

media/remove-from-document.html -> crashed
media/video-preload.html -> crashed
media/video-plays-past-end-of-test.html -> crashed
media/video-src.html -> crashed
media/video-volume.html -> crashed

That looks pretty random :(

[Nancy Piedra]

Created attachment 98158 [details]
Skip only failing media tests

I downloaded the Qt Webkit VM to analyze the flakey media tests.

I modified the Skipped file to skip only those tests that failed outright on the VM.

I then ran the layout tests many times and did not see any of the media tests fail.  

With this set of tests I did not see any flakiness.  I'm not sure if this bug was logged before or after the buildbot was upgraded to Qt Mobility 1.2 so it is not clear to me what version of Qt Mobility was in place when the flakiness was observerd.

In other words, I could not reproduce the flakiness in order to find a fix for it but I could find a stable set of tests.

[WebKit Review Bot]

Comment on attachment 98158 [details]
Skip only failing media tests

Rejecting attachment 98158 [details] from review queue.

[e-mail] does not have reviewer permissions according to http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/config/committers.py.

- If you do not have reviewer rights please read http://webkit.org/coding/contributing.html for instructions on how to use bugzilla flags.

- If you have reviewer rights please correct the error in Tools/Scripts/webkitpy/common/config/committers.py by adding yourself to the file (no review needed).  The commit-queue restarts itself every 2 hours.  After restart the commit-queue will correctly respect your reviewer rights.

[WebKit Review Bot]

Comment on attachment 98158 [details]
Skip only failing media tests

Rejecting attachment 98158 [details] from commit-queue.

[e-mail] does not have committer permissions according to http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/config/committers.py.

- If you do not have committer rights please read http://webkit.org/coding/contributing.html for instructions on how to use bugzilla flags.

- If you have committer rights please correct the error in Tools/Scripts/webkitpy/common/config/committers.py by adding yourself to the file (no review needed).  The commit-queue restarts itself every 2 hours.  After restart the commit-queue will correctly respect your committer rights.

[Nancy Piedra]

Sorry, meant to do review? commit-queue?  but hit + instead.

[Laszlo Gombos]

I talked to Nancy and she really did her best to filter out flaky tests. I think we should try to enable the proposed list.

[Alexis Menard (darktears)]

(In reply to comment #8)
> I talked to Nancy and she really did her best to filter out flaky tests. I think we should try to enable the proposed list.

Yep I think we should give a try.

[Alexis Menard (darktears)]

Comment on attachment 98158 [details]
Skip only failing media tests

View in context: https://bugs.webkit.org/attachment.cgi?id=98158&action=review

> LayoutTests/ChangeLog:6
> +        Skipped the Qt media tests that are failing on the Qt Webkit VM

Extra line here.

[Csaba Osztrogonác]

Comment on attachment 98158 [details]
Skip only failing media tests

$ Tools/Scripts/run-webkit-tests media 

media/video-preload.html -> failed
media/video-seek-no-src-exception.html -> timed out
media/video-seekable.html -> failed
media/video-source-inserted.html -> timed out
media/video-source-moved.html -> timed out
media/video-src-change.html -> timed out
media/video-src-remove.html -> failed
media/video-src-set.html -> timed out
media/video-src.html -> timed out
media/video-volume.html -> failed

and then run-webkit-tests waits forever ... :(

r- now, it needs more investigation

[Csaba Osztrogonác]

media/adopt-node-crash.html -> timed out

[Csaba Osztrogonác]

media/video-source-none-supported.html -> timed out
media/video-src-invalid-remove.html -> failed

[Nancy Piedra]

Ossy - Are you running the tests on the VM?  If not, what is the configuration of the system you are running the tests on.

[Csaba Osztrogonác]

media/video-source-removed.html -> crashed
media/before-load-member-access.html -> timed out
media/video-src-source.html -> timed out

Media tests are still flakey. :-S Unskipping flakey tests won't solve 
the bug, we shouldn't enable them until we find and fix the bug.

[Csaba Osztrogonác]

(In reply to comment #14)
> Ossy - Are you running the tests on the VM?  If not, what is the configuration of the system you are running the tests on.

I ran tests on the buildbot, not on the VM. But they have same configuration.

[Csaba Osztrogonác]

I tried to find flakey tests with:
$ run-tests-in-xvfb.sh media --iterations 100

[Nancy Piedra]

OK.  I will try what you have done and see if I get similar results.  If I do, I will continue to debug.

[Nancy Piedra]

I ran the following on the VM and saw no tests fail or time out:
run-tests-in-xvfb.sh media --iterations 100

I will keep trying to see if I can reproduce what you're seeing on the buildbot.

[Csaba Osztrogonác]

(In reply to comment #19)
> I ran the following on the VM and saw no tests fail or time out:
> run-tests-in-xvfb.sh media --iterations 100
> 
> I will keep trying to see if I can reproduce what you're seeing on the buildbot.

Very strange .... I try to check it on the VM too.

[Alexis Menard (darktears)]

(In reply to comment #20)
> (In reply to comment #19)
> > I ran the following on the VM and saw no tests fail or time out:
> > run-tests-in-xvfb.sh media --iterations 100
> > 
> > I will keep trying to see if I can reproduce what you're seeing on the buildbot.
> 
> Very strange .... I try to check it on the VM too.

That's exactly what I keep saying. The VM does not behave the same way as the bots. It also seems to behave differently whether you run with VirtualBox or VMWare, it sounds unbelievable but it is.

[Nancy Piedra]

I ran it again and now I am seeing some failures.

[Nancy Piedra]

Created attachment 98369 [details]
Skip only failing media tests (2nd try)

Ossy - I commented out a few more tests.  Could you try this?  If it doesn't work on the buildbot then I will look into the flakiness more.

[Csaba Osztrogonác]

Comment on attachment 98369 [details]
Skip only failing media tests (2nd try)

r- again, because I got flakey fails and timeouts

media/video-single-valid-source.html
media/adopt-node-crash.html
media/video-source-inserted.html
media/video-src.html
media/before-load-member-access.html
media/broken-video.html
media/constructors.html
media/controls-css-overload.html

[Zan Dobersek]

The Qt port was removed from the project, so this bug is not valid anymore.