Bug 76280 – media/W3C/video/networkState/networkState_during_progress.html is flaky

• Status: REOPENED
• Component: Tools / Tests

[James Robinson]

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=media%2FW3C%2Fvideo%2FnetworkState%2FnetworkState_during_progress.html&showExpectations=true

[Eric Carlson]

One problem is that the expected results contain a failure:


videoElement.networkState should be NETWORK_LOADING during progress event

On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".


FAIL "1" should be 2. Was 1.

TEST COMPLETE
spec reference

[Eric Carlson]

I *think* this test is wrong. It fails if video.networkState is not NETWORK_LOADING (2) when a progress event fires, but progress events are fired asynchronously. The "failure" (in the expected results) is that networkState is NETWORK_IDLE (1), but that just means that loading has completed.

[Eric Carlson]

Hmm, the final 'progress' event is supposed to be fired synchronously:

↪ Once the entire media resource has been fetched (but potentially before any of it has been decoded)
       Fire a simple event named progress at the media element.

 So this may well be a bug in WebKit.

[James Robinson]

(In reply to comment #1)
> One problem is that the expected results contain a failure:
> 
> 
> videoElement.networkState should be NETWORK_LOADING during progress event
> 
> On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
> 
> 
> FAIL "1" should be 2. Was 1.
> 
> TEST COMPLETE
> spec reference

Sometimes we get the PASS output, sometimes we get the (currently expected) failure message.

[Balazs Kelemen]

This is also flaky with the GStreamer backend (at least with my soon to be uploaded patch), i.e. sometimes we got the correct output. Is it ok if I mark it flakey in the general TestExpectations file?

[Balazs Kelemen]

Marked flakey in http://trac.webkit.org/changeset/150579

[Russell Epstein]

*** Bug 200354 has been marked as a duplicate of this bug. ***

[Alexey Proskuryakov]

rdar://problem/53825672

[Alexey Proskuryakov]

*** Bug 82976 has been marked as a duplicate of this bug. ***

[Charlie Turner]

Created attachment 381561 [details]
Patch

[Charlie Turner]

The reason for the flake was because we were scheduling the last progress event on an async event queue, and then setting networkState to IDLE causing a race on JS receipt of the event. This is the case on all platforms so I have adjusted the expectations accordingly.

[Eric Carlson]

Comment on attachment 381561 [details]
Patch

Thanks for the fix!

[Charlie Turner]

Created attachment 381591 [details]
Patch for landing

[Charlie Turner]

Created attachment 381592 [details]
Patch for landing

[Charlie Turner]

*** Bug 133865 has been marked as a duplicate of this bug. ***

[Charlie Turner]

Created attachment 381595 [details]
Patch for landing

[Charlie Turner]

Created attachment 381597 [details]
Patch for landing

[WebKit Commit Bot]

The commit-queue encountered the following flaky tests while processing attachment 381592 [details]:

inspector/runtime/getProperties.html bug 203271 (authors: [e-mail], [e-mail], and [e-mail])
The commit-queue is continuing to process your patch.

[WebKit Commit Bot]

Comment on attachment 381597 [details]
Patch for landing

Clearing flags on attachment: 381597

Committed r251460: <https://trac.webkit.org/changeset/251460>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Charlie Turner]

This is still flaking on Mac debug builds,

https://build.webkit.org/results/Apple%20Mojave%20Debug%20WK1%20%28Tests%29/r251541%20%286472%29/media/W3C/video/networkState/networkState_during_progress-pretty-diff.html

Sometimes I see an incorrect network state as well, but the test is clearly bugged since it's expected that it exits after one event. Perhaps we should just remove it? I noted that neither Chrome nor Firefox are spec-compliant in a similar test case either.

I can't cross check on GTK atm, because the debug bots exit early due to the many crashes there, and I don't have a powerful enough machine to build that for a few days.

[Jer Noble]

This caused a crash on macOS and iOS platforms. We need a way to de-flake this test without doing synchronous messaging when handling a message from the platform.