Bug 231084 - Regression(r282374): [ macOS ] imported/w3c/web-platform-tests/html/semantics/embedded-content/the-video-element/resize-during-playback.html is a flaky failure
Summary: Regression(r282374): [ macOS ] imported/w3c/web-platform-tests/html/semantics...
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Jean-Yves Avenard [:jya]
Keywords: InRadar
Depends on: 233406
  Show dependency treegraph
Reported: 2021-10-01 09:47 PDT by ayumi_kojima
Modified: 2021-11-20 23:43 PST (History)
4 users (show)

See Also:

Patch (3.19 KB, patch)
2021-11-09 17:20 PST, Jean-Yves Avenard [:jya]
no flags Details | Formatted Diff | Diff
[fast-cq] Patch (3.18 KB, patch)
2021-11-09 17:49 PST, Jean-Yves Avenard [:jya]
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description ayumi_kojima 2021-10-01 09:47:01 PDT

Is a flaky failure on macOS.

History: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fsemantics%2Fembedded-content%2Fthe-video-element%2Fresize-during-playback.html

Result page: https://build.webkit.org/results/Apple-BigSur-Debug-WK1-Tests/r283362%20(4465)/results.html


--- /Volumes/Data/worker/bigsur-debug-tests-wk1/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/embedded-content/the-video-element/resize-during-playback-expected.txt
+++ /Volumes/Data/worker/bigsur-debug-tests-wk1/build/layout-test-results/imported/w3c/web-platform-tests/html/semantics/embedded-content/the-video-element/resize-during-playback-actual.txt
@@ -1,4 +1,4 @@
-FAIL mp4 video assert_equals: Expected resize event, but got ended event instead expected "resize" but got "ended"
+FAIL mp4 video assert_equals: Expected resize event, but got playing event instead expected "resize" but got "playing"
 NOTRUN webm video webm supported
Comment 1 Radar WebKit Bug Importer 2021-10-01 09:47:19 PDT
Comment 2 ayumi_kojima 2021-10-01 11:06:57 PDT
Marked test expectations https://trac.webkit.org/changeset/283381/webkit
Comment 3 ayumi_kojima 2021-10-15 09:41:38 PDT
I was able to reproduce the failure at TOT locally on BigSur using run-webkit-tests --force -f -1 --iterations 50 --exit-after-n-failures 1 --exit-after-n-crashes-or-timeouts 1 --debug imported/w3c/web-platform-tests/html/semantics/embedded-content/the-video-element/resize-during-playback.html --no-timeout
Comment 4 ayumi_kojima 2021-10-29 10:26:56 PDT
The test passed with r282373 and failed with r282374.

With r282374 ~ r282379 (didn't test newer than r282379), the test failed with different diff than TOT:

--- /Volumes/Data/Builds/BigSur/debug-282374/layout-test-results/imported/w3c/web-platform-tests/html/semantics/embedded-content/the-video-element/resize-during-playback-expected.txt
+++ /Volumes/Data/Builds/BigSur/debug-282374/layout-test-results/imported/w3c/web-platform-tests/html/semantics/embedded-content/the-video-element/resize-during-playback-actual.txt
@@ -1,4 +1,7 @@
-FAIL mp4 video assert_equals: Expected resize event, but got ended event instead expected "resize" but got "ended"
-NOTRUN webm video webm supported
+CONSOLE MESSAGE: ReferenceError: Can't find variable: promise_test
+layer at (0,0) size 800x600
+  RenderView at (0,0) size 800x600
+layer at (0,0) size 800x8
+  RenderBlock {HTML} at (0,0) size 800x8
+    RenderBody {BODY} at (8,8) size 784x0
+      RenderBlock {DIV} at (0,0) size 784x0
Comment 5 Jean-Yves Avenard [:jya] 2021-11-09 00:40:51 PST
I’ve been unable to reproduce the bug on either MBP 14” (AS) or iMac Pro.
run-webkit-tests --debug --no-build --force -1 --repeat 1000 --exit-after-n-failures=1 --no-timeout LayoutTests/imported/w3c/web-platform-tests/html/semantics/embedded-content/the-video-element/resize-during-playback.html
All 1000 tests ran as expected.  

Code was modified so that webm is disabled to reproduce the testing conditions where the MediaFormatReader plugin is disabled.
I don’t believe the regression point is valid, particularly because the test will fail due to webm being enabled by default on a local or spade build which is what we are seeing in comment 4.

In comment 4, the error produced is also different.
Comment 6 Jean-Yves Avenard [:jya] 2021-11-09 17:03:55 PST
This test is failing because we do not support change of resolution mid-stream and we reach the end of the video without firing the "resize" event.

The test calls play() on the video element and wait for both `resize` and `playing` event; each event can only be fired once without causing an error.

in bug 230210 (r282374) we properly fixed stall detections.
If a stall occurs during playback (due to the system being overloaded or whatever), the readyState will move from HAVE_FUTURE_DATA to HAVE_CURRENT_DATA and the `stalled` event will be fired. [1]

When re-buffering completed and the readyState moves again from HAVE_CURRENT_DATA to HAVE_ENOUGH_DATA (or more per spec: we are to `notify about playing` [2] and fire the `playing` event.

The test doesn't handle this case, it assumes that playback will not stall and continue uninterrupted; there's no guarantee this will always be the case. It will happen more likely with WK1 as there's less parallel processing.

in bug 230210 (r282374) we properly fixed stall detections. We have a correct progression in the behaviour simply exposing an existing flaw in the test.

Seeing that the test can't pass as we do not support the feature tested, we should just skip this test.

[1] https://html.spec.whatwg.org/multipage/media.html#ready-states
[2] https://html.spec.whatwg.org/multipage/media.html#notify-about-playingv
Comment 7 Jean-Yves Avenard [:jya] 2021-11-09 17:20:06 PST
Created attachment 443759 [details]
Comment 8 Jean-Yves Avenard [:jya] 2021-11-09 17:49:34 PST
Created attachment 443762 [details]
[fast-cq] Patch
Comment 9 EWS 2021-11-09 18:48:35 PST
Committed r285554 (244067@main): <https://commits.webkit.org/244067@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 443762 [details].