Bug 221218 – [ BigSur ] imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html is flakey failing

• Component: New Bugs
• Status: RESOLVED WORKSFORME

[Robert Jenner]

imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html

Is flakey failing on MacOS Big Sur Wk2.

HISTORY URL:
https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fmedia-source%2Fmediasource-config-change-webm-v-framesize.html

DIFF URL:
https://build.webkit.org/results/Apple-BigSur-Release-WK2-Tests/r272143%20(1311)/imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize-diff.txt

DIFF TEXT:
--- /Volumes/Data/slave/bigsur-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize-expected.txt
+++ /Volumes/Data/slave/bigsur-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize-actual.txt
@@ -1,3 +1,5 @@
 
-PASS Tests webm video-only frame size changes.
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT Tests webm video-only frame size changes. Test timed out


Bug Reproducible on r272141, same failure verified.

Bug Produced using test: 
run-webkit-test imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html  --iterations 1000 -f

Regression point appears r272140, where the test did not crash. 

Bug appears to have been introduced in r272141

[Radar WebKit Bug Importer]

<rdar://problem/73837300>

[Robert Jenner]

We are unable to find a true regression point. This test appears to be very flakey. 

Tested back to r271039 just to see, and it did fail.

[Robert Jenner]

Created attachment 420026 [details]
Patch

[Ryan Haddad]

(In reply to Robert Jenner from comment #2)
> We are unable to find a true regression point. This test appears to be very
> flakey. 
> 
> Tested back to r271039 just to see, and it did fail.
I think this test has just been flaky since it was unskipped as part of https://trac.webkit.org/changeset/270720/webkit

[Ryan Haddad]

Comment on attachment 420026 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=420026&action=review

> LayoutTests/platform/mac-wk2/TestExpectations:1329
> +webkit.org/b/221218 [ BigSur ] imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html [ Pass Failure ]

Let's mark this BigSur+, since presumably it wold affect future releases too.

> LayoutTests/platform/mac-wk2/TestExpectations:1331
> +

Nit: extra newline here

[Robert Jenner]

Created attachment 420059 [details]
Patch

[Robert Jenner]

(In reply to Ryan Haddad from comment #5)
> Comment on attachment 420026 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=420026&action=review
> 
> > LayoutTests/platform/mac-wk2/TestExpectations:1329
> > +webkit.org/b/221218 [ BigSur ] imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html [ Pass Failure ]
> 
> Let's mark this BigSur+, since presumably it wold affect future releases too.
> 
> > LayoutTests/platform/mac-wk2/TestExpectations:1331
> > +
> 
> Nit: extra newline here

Changes made, how does it look now?

[Robert Jenner]

Created attachment 420062 [details]
Patch

[Robert Jenner]

It does appear to be flakey failing in wk1 as well. Updating title and patch.

[Robert Jenner]

Created attachment 420754 [details]
Update TestExpectations

[Truitt Savell]

Comment on attachment 420754 [details]
Update TestExpectations

Clearing flags on attachment: 420754

Committed r273045 (234246@main): <https://commits.webkit.org/234246@main>

[Ryan Haddad]

The test is consistently failing on Big Sur WK1 with the following diff:
-PASS Tests webm video-only frame size changes.
+FAIL Tests webm video-only frame size changes. assert_true: video/webm;codecs="vp8" is supported. expected true got false

perhaps that is expected, though.

I don't see any failures on Big Sur WK2 since 02/09/21: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fmedia-source%2Fmediasource-config-change-webm-v-framesize.html&flavor=wk2&platform=mac

We may be able to remove the expectation for WK2 and close this one out.

[Ben Schwartz]

This no longer appears to be occurring. Removing test expectations. Please file a new bug if issues persist.

Tracking test expectations removal in bug #274990 (rdar://129080660).