Bug 217263 - [ iOS ] 2 mediacapture-record tests are a constant failure
Summary: [ iOS ] 2 mediacapture-record tests are a constant failure
Status: REOPENED
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: youenn fablet
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2020-10-02 18:05 PDT by Hector Lopez
Modified: 2020-10-20 09:42 PDT (History)
3 users (show)

See Also:


Attachments
Patch (3.18 KB, patch)
2020-10-12 05:15 PDT, youenn fablet
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Hector Lopez 2020-10-02 18:05:14 PDT
imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-no-sink.https.html
imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html

Tests are constant failures according to history on iOS. First occurrence of failures on iOS coincide with first testing on iOS. 

Both test were passing on macOS until r267870. From that point test imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html became flaky on macOS Catalina wk2 Debug/Release.

History:

https://results.webkit.org/?suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fmediacapture-record%2FMediaRecorder-no-sink.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fmediacapture-record%2FMediaRecorder-peerconnection.https.html

Diff iOS for imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-no-sink.https.html:

--- /Volumes/Data/slave/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-no-sink.https-expected.txt
+++ /Volumes/Data/slave/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-no-sink.https-actual.txt
@@ -1,5 +1,7 @@
+
+Harness Error (TIMEOUT), message = null
 
 PASS PeerConnection MediaRecorder records audio from PeerConnection without sinks
 PASS PeerConnection MediaRecorder records video from PeerConnection without sinks
-PASS PeerConnection MediaRecorder records audio/video from PeerConnection without sinks
+TIMEOUT PeerConnection MediaRecorder records audio/video from PeerConnection without sinks Test timed out

Diff iOS for imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html

--- /Volumes/Data/slave/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https-expected.txt
+++ /Volumes/Data/slave/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https-actual.txt
@@ -6,8 +6,8 @@
 PASS PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format [passthrough].
 PASS PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format [passthrough].
 PASS PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format [passthrough].
-PASS PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format [passthrough].
-TIMEOUT PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format [passthrough]. Test timed out
+TIMEOUT PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format [passthrough]. Test timed out
+NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format [passthrough].
 NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp8.
 NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp8.
 NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp8.

Diff macOS for imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html

--- /Volumes/Data/slave/catalina-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https-expected.txt
+++ /Volumes/Data/slave/catalina-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https-actual.txt
@@ -1,23 +1,21 @@
 
-
-Harness Error (TIMEOUT), message = null
 
 PASS PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format [passthrough].
 PASS PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format [passthrough].
 PASS PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format [passthrough].
 PASS PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format [passthrough].
 PASS PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format [passthrough].
-TIMEOUT PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format [passthrough]. Test timed out
-NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp8.
-NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp8.
-NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp8.
-NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format video/webm;codecs=vp8.
-NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format video/webm;codecs=vp8.
-NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format video/webm;codecs=vp8.
-NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp9.
-NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp9.
-NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp9.
-NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format video/webm;codecs=vp9.
-NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format video/webm;codecs=vp9.
-NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format video/webm;codecs=vp9.
+PASS PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format [passthrough].
+FAIL PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"



Reproduction for imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html on macOS Catalina wk 2 Release on r267897

run-webkit-tests imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html --iterations 1000 --exit-after-n-failures 3

Result:

Baseline search path: platform/mac-catalina -> Internal/Tests/LayoutTestsResultsForUnreleasedSoftware/mac-wk2 -> platform/mac-wk2 -> Internal/Tests/LayoutTestsResultsForUnreleasedSoftware/mac -> platform/mac -> platform/wk2 -> generic

Running 1 test

Running 1 WebKitTestRunner.

[1/1000] imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html failed unexpectedly (text diff)
[2/1000] imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html failed unexpectedly (text diff)
[3/1000] imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html failed unexpectedly (text diff)
Exiting early after 3 failures. 3 tests run.


0 tests ran as expected, 3 didn't (997 didn't run):


Regressions: Unexpected text-only failures (1)
  imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html [ Failure ]
Comment 1 Radar WebKit Bug Importer 2020-10-02 18:05:31 PDT
<rdar://problem/69904225>
Comment 2 Hector Lopez 2020-10-02 18:15:21 PDT
See webkit.org/b/217262 for test expectations for Mac

Test expectations for iOS:
Comment 3 Hector Lopez 2020-10-02 18:18:36 PDT
Test expectations for iOS:

https://trac.webkit.org/changeset/267905/webkit

See test expectation for Mac:

https://trac.webkit.org/changeset/267904/webkit
Comment 4 youenn fablet 2020-10-12 05:15:23 PDT
Created attachment 411109 [details]
Patch
Comment 5 EWS 2020-10-12 06:55:30 PDT
commit-queue failed to commit attachment 411109 [details] to WebKit repository.
Comment 6 EWS 2020-10-12 10:37:42 PDT
Committed r268361: <https://trac.webkit.org/changeset/268361>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 411109 [details].
Comment 7 youenn fablet 2020-10-12 23:40:28 PDT
Reopening for further investigation
Comment 8 Hector Lopez 2020-10-20 09:42:35 PDT
imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-no-sink.https.html

Test expectation adjustment:

https://trac.webkit.org/changeset/268737/webkit