Bug 221218 - [ BigSur ] imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html is flakey failing
Summary: [ BigSur ] imported/w3c/web-platform-tests/media-source/mediasource-config-ch...
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2021-02-01 11:12 PST by Robert Jenner
Modified: 2021-03-22 14:48 PDT (History)
4 users (show)

See Also:


Attachments
Patch (1.52 KB, patch)
2021-02-11 12:18 PST, Robert Jenner
no flags Details | Formatted Diff | Diff
Patch (1.54 KB, patch)
2021-02-11 15:59 PST, Robert Jenner
no flags Details | Formatted Diff | Diff
Patch (1.51 KB, patch)
2021-02-11 16:26 PST, Robert Jenner
no flags Details | Formatted Diff | Diff
Update TestExpectations (1.53 KB, patch)
2021-02-17 16:20 PST, Robert Jenner
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Robert Jenner 2021-02-01 11:12:49 PST
imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html

Is flakey failing on MacOS Big Sur Wk2.

HISTORY URL:
https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fmedia-source%2Fmediasource-config-change-webm-v-framesize.html

DIFF URL:
https://build.webkit.org/results/Apple-BigSur-Release-WK2-Tests/r272143%20(1311)/imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize-diff.txt

DIFF TEXT:
--- /Volumes/Data/slave/bigsur-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize-expected.txt
+++ /Volumes/Data/slave/bigsur-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize-actual.txt
@@ -1,3 +1,5 @@
 
-PASS Tests webm video-only frame size changes.
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT Tests webm video-only frame size changes. Test timed out


Bug Reproducible on r272141, same failure verified.

Bug Produced using test: 
run-webkit-test imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html  --iterations 1000 -f

Regression point appears r272140, where the test did not crash. 

Bug appears to have been introduced in r272141
Comment 1 Radar WebKit Bug Importer 2021-02-01 11:13:15 PST
<rdar://problem/73837300>
Comment 2 Robert Jenner 2021-02-01 14:23:27 PST
We are unable to find a true regression point. This test appears to be very flakey. 

Tested back to r271039 just to see, and it did fail.
Comment 3 Robert Jenner 2021-02-11 12:18:54 PST
Created attachment 420026 [details]
Patch
Comment 4 Ryan Haddad 2021-02-11 12:53:19 PST
(In reply to Robert Jenner from comment #2)
> We are unable to find a true regression point. This test appears to be very
> flakey. 
> 
> Tested back to r271039 just to see, and it did fail.
I think this test has just been flaky since it was unskipped as part of https://trac.webkit.org/changeset/270720/webkit
Comment 5 Ryan Haddad 2021-02-11 12:56:07 PST
Comment on attachment 420026 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=420026&action=review

> LayoutTests/platform/mac-wk2/TestExpectations:1329
> +webkit.org/b/221218 [ BigSur ] imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html [ Pass Failure ]

Let's mark this BigSur+, since presumably it wold affect future releases too.

> LayoutTests/platform/mac-wk2/TestExpectations:1331
> +

Nit: extra newline here
Comment 6 Robert Jenner 2021-02-11 15:59:50 PST
Created attachment 420059 [details]
Patch
Comment 7 Robert Jenner 2021-02-11 16:04:12 PST
(In reply to Ryan Haddad from comment #5)
> Comment on attachment 420026 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=420026&action=review
> 
> > LayoutTests/platform/mac-wk2/TestExpectations:1329
> > +webkit.org/b/221218 [ BigSur ] imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html [ Pass Failure ]
> 
> Let's mark this BigSur+, since presumably it wold affect future releases too.
> 
> > LayoutTests/platform/mac-wk2/TestExpectations:1331
> > +
> 
> Nit: extra newline here

Changes made, how does it look now?
Comment 8 Robert Jenner 2021-02-11 16:26:50 PST
Created attachment 420062 [details]
Patch
Comment 9 Robert Jenner 2021-02-17 16:13:49 PST
It does appear to be flakey failing in wk1 as well. Updating title and patch.
Comment 10 Robert Jenner 2021-02-17 16:20:37 PST
Created attachment 420754 [details]
Update TestExpectations
Comment 11 Truitt Savell 2021-02-17 16:22:34 PST
Comment on attachment 420754 [details]
Update TestExpectations

Clearing flags on attachment: 420754

Committed r273045 (234246@main): <https://commits.webkit.org/234246@main>
Comment 12 Ryan Haddad 2021-03-22 14:48:21 PDT
The test is consistently failing on Big Sur WK1 with the following diff:
-PASS Tests webm video-only frame size changes.
+FAIL Tests webm video-only frame size changes. assert_true: video/webm;codecs="vp8" is supported. expected true got false

perhaps that is expected, though.

I don't see any failures on Big Sur WK2 since 02/09/21: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fmedia-source%2Fmediasource-config-change-webm-v-framesize.html&flavor=wk2&platform=mac

We may be able to remove the expectation for WK2 and close this one out.