| Summary: | iOS calling getUserMedia() again kills video display of first getUserMedia() | ||||||
|---|---|---|---|---|---|---|---|
| Product: | WebKit | Reporter: | Chad Phillips <webkit> | ||||
| Component: | WebRTC | Assignee: | Nobody <webkit-unassigned> | ||||
| Status: | RESOLVED CONFIGURATION CHANGED | ||||||
| Severity: | Normal | CC: | abarnes, bfulgham, daginge, davy.de.durpel, eric.amram, eric.carlson, gabor, jaya.allamsetty, jonlee, joseph.eugene, lee, mb, mbrice, mitch, oanguenot, ramyaddurairaj, relbeatsdev, sak126p, seba.kerckhof, selawre, shreyas, ssim, webkit, webkit-bug-importer, youennf | ||||
| Priority: | P2 | Keywords: | InRadar | ||||
| Version: | Safari 11 | ||||||
| Hardware: | iPhone / iPad | ||||||
| OS: | iOS 11 | ||||||
| See Also: | https://bugs.webkit.org/show_bug.cgi?id=238492 | ||||||
| Attachments: |
|
||||||
Created attachment 326267 [details]
Multiple getUserMedia() streams controlled by UI.
Attaching a more robust test case, allowing user triggering of multiple streams obtained by getUserMedia().
Both 'Video 1' and 'Video 2' can be started and stopped fine as long as the other is not actively streaming. But if you start one while another is streaming to screen, the first stream goes black.
I've spent some more time digging into this issue, and it turns out that the video MediaStreamTrack element of video 1 has its 'mute' property set to true upon another gUM call that requests a video stream. It's not even necessary for this gUM call to do anything with the video stream (like display it) for the muting of the previous video MediaStreamTrack element. Furthermore, I see no way via the API to unmute the muted video track -- the 'mute' property is read-only, and toggling the 'enabled' property of either video track has no effect on its state. Is this issue related to the 'Multiple Simultaneous Audio or Video Streams' as noted at https://developer.apple.com/library/content/documentation/AudioVideo/Conceptual/Using_HTML5_Audio_Video/Device-SpecificConsiderations/Device-SpecificConsiderations.html ? If so, it's going to be severely limiting for certain multiparty videoconferencing applications. For example: - It's common practice to show a user their own local video feed with one (higher resolution) stream, and publish another (lower resolution) stream to other users - To accommodate receivers with different bandwidth capabilities, a common practice is to publish both a high resolution and a low resolution stream This is still an issue with iOS 11.3, would love somebody to have a look at it. IMO, it's unnecessarily limiting to block video after a user has granted access to the camera, as mentioned in the specific use cases in previous comments. Limitations like this make it so that the in-browser WebRTC experience on iOS is the worst of any platform -- is that really what Apple wants?? Just chiming in here. This is a blocker for us in switching between front and back camera on Safari iOS. At least with minimal disruption to the user experience. > - It's common practice to show a user their own local video feed with one > (higher resolution) stream, and publish another (lower resolution) stream to > other users Understood that this is not optimal, although in most UI, the local video track usually takes a smaller part of the screen than the remote video track. Also applyConstraints should be the solution for changing the resolution of a given video track. At this point, it is not feasible to have two tracks with different resolutions. Ideally, this should be feasible using MediaStream cloning and applyConstraints. Note that general support for multiple video streams might not always be feasible, in particular if the streams are coming from multiple capture devices. > - To accommodate receivers with different bandwidth capabilities, a common > practice is to publish both a high resolution and a low resolution stream Simulcast might be a better option there. (In reply to daginge from comment #4) > Just chiming in here. This is a blocker for us in switching between front > and back camera on Safari iOS. At least with minimal disruption to the user > experience. I would be interested in what disruption you encounter. The following is expected to work without too much trouble: navigator.mediaDevices.getUserMedia({video: { facingMode: "user"} }).then((s) => { localVideo.srcObject = s; peerConnection.getSenders()[0].replaceTrack(s.getVideoTracks()[0]); }); > Understood that this is not optimal, although in most UI, > the local video track usually takes a smaller part of the > screen than the remote video track. It's not the only rational UI choice though. I have a layout where all feeds, including the local user's feed, are the same size, and many users prefer this layout. Also, it doesn't seem like a very flexible architectural mindset to make that kind of assumption about how a designer wants to lay things out? > Note that general support for multiple video streams might not > always be feasible, in particular if the streams are coming from > multiple capture devices. I want to point out here that Chrome on Android has zero restrictions/limitations in this regard. You can call gUM multiple times, grab different resolutions, clone streams, etc., and it all works flawlessly. By comparison, iOS is a nightmare to work on for anything beyond the most basic use cases. Which also makes the end user experience worse on iOS because of all the compromises necessary. It's puzzling to me Apple implemented WebRTC at all if they're going to hamstring it. > Simulcast might be a better option there. Not all clients support simulcast. For example, Chrome doesn't yet support it for h264, which is of course the required codec if you want interop with iOS devices. Adding some further clarification from more testing: 1. This issue only occurs when a subsequent gUM() request asks for an already requested media type. For example, if gUM() #1 asks for video, and gUM() #2 also asks for video, gUM() #1's video stream is affected. However, if gUM() #2 only asks for audio, then gUM() #1's video stream is NOT affected. 2. Because the mechanism is setting the track's muted property, data is still sent along a peer connection, although it's not very useful since the other side only receives muted video. This issue also occurs for audio tracks. I now believe the issue can be fully summarized as: If a getUserMedia() requests a media type requested in a previous getUserMedia(), the previously requested media track's 'muted' property is set to true, and there is no way to programmatically unmute it. This is currently an expected behavior. There seems to be two requests: - Allow multiple capture using the same capture device with different parameters (resolution, frameRate...). - Allow capture on two different capture devices at the same time. @youenn, those seem correct, and are supported by every other platform I've tried. Whether it is fixed or not. I'm also facing the same issue @Chad Phillips Please make this issue as a major one and why this is still not assigned. *** Bug 192849 has been marked as a duplicate of this bug. *** Are there any updates on this? I am working on an implementation where I need two videos in the same window. I tried MediaStream.clone() but it has its own issues. It would be great to have this feature. (In reply to Shreyas from comment #15) > Are there any updates on this? I am working on an implementation where I > need two videos in the same window. I tried MediaStream.clone() but it has > its own issues. It would be great to have this feature. What are the issues you encountered? cloning and applyConstraints should allow you to have two tracks with the same source and with different sizes. I resolved this by another approach but here's what I was facing: With clone(), When I try to apply constraints on a new video, it also applies to the original video which did not fit my solution. Can we ever expect to see this fixed? Sure, we can usually modify our code to work around it, but it doesn't mean it's not another bug in an already very buggy webRTC implementation offered by webkit. Are there any updates on this one ? Jitsi users are also facing this issue with both audio and video when switching camera because of how device selection UI is designed. We do a second gUM call for previewing the tracks. So just to be sure, it is currently not supported to "Allow capture on two different capture devices at the same time", right? I suppose it is not planned to change this behaviour as this report is on "NEW" since 2017? We would have a valid use case: One should be seen while he or she is presenting physical stuff. Think of a car show, you should see the one who is trying to sell as well as the car/interior/exterior. This is a highly asked feature, especially during covid... Had the same problem with audio only tracks. In my use case, I create several PeerConnections to connect to several audio rooms in FreeSwitch using Janus as a WebRTC Gateway. I faced the same issue: my first line was muted as soon as the second was created. Additionally to that, the complexity was that the WebRTC part is mainly managed by Janus and the SIP plugin... But hopefully, there is a way to access to the PeerConnection object from Janus. The solution I found is a workaround and fixing that issue will for sure help to handle the case correctly. 1/ I added an handler on the "onmute" event on the local track associated to each room connected. 2/ When one of my local track goes to "muted", I try to find in the other existing local tracks, a track which is not muted 3/ I clone that track 4/ I use the PeerConnection -> Sender -> replaceTrack() function "Magically", it seems to work. I'm able to speak and be heard in the room associated to that track. But, not sure on what is the consequence on that (listeners for example...) 13-08-2021: Facing the same issue with Jitsi. Hope it will be fixed soon! BUMP! Adding streams from multiple cameras (e.g. 'environment' and 'user' facingMode) would be very useful and works fine in Chrome on Android. This bug is nearly 4 years old now and still marked as NEW 8( Currently seeing this still in my company's application. Any updates on getting a fix? any news? problem still occurs on safari 15.3 This is actually: <rdar://42467754> Adding my voice to this. This is really frustrating for our iOS users in our meeting software. Please dedicate time to get this fixed. I've also found that using the SpeechRecognition API (https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition) will cause the microphone track to become muted everywhere else, including in WebRTC calls. Due to this bug, it is impossible to generate captions using this API in our meeting software without making the user go silent for everyone else. We had to disable caption generation for this reason. Seconding everything here. Unable to locally capture an audio WebRTC stream via the MediaRecorder on iOS. The originally WebRTC fails to work when the MediaRecorder instance is started. Would be very useful to work with simultaneous media streams. Bug 237359 fixes the case of muting a video track when another video track is created on iOS in https://commits.webkit.org/r291034. The new behavior in iOS is that, if a new camera device is used, we will stop the previous capture and start the new one as the current API we are using does not allow to capture both at the same time. If the new capture is using the same camera as the previous one, both will be able to continue in parallel. If your particular use case is about capturing with both cameras, please file a new bug. (In reply to Austin from comment #24) > Currently seeing this still in my company's application. Any updates on > getting a fix? The current workaround until the above fix rolls out is to clone the track instead of calling getUserMedia. This allows to get the same video feed in two tracks, and potentially apply width/height/frame rate constraints independently. (In reply to youenn fablet from comment #30) > Bug 237359 fixes the case of muting a video track when another video track > is created on iOS in https://commits.webkit.org/r291034. > > The new behavior in iOS is that, if a new camera device is used, we will > stop the previous capture and start the new one as the current API we are > using does not allow to capture both at the same time. If the new capture is > using the same camera as the previous one, both will be able to continue in > parallel. > > If your particular use case is about capturing with both cameras, please > file a new bug. In case this wasn't automatically linked back - see https://bugs.webkit.org/show_bug.cgi?id=238492 Thanks. > If the new capture is using the same camera as the previous one, both will be able to continue in parallel.
I still get a black screen (after 3 secs) on the second call to getUserMedia() on iOS in this example where the same device is returned:
<!DOCTYPE html>
<html>
<body>
<div>
<video id="video1" autoplay playsinline></video>
</div>
<script type="text/javascript">
(async function () {
var constraints = {
audio: false,
video: true,
};
const result1 = await navigator.mediaDevices.getUserMedia(constraints).then(async function(stream) {
var video1 = document.getElementById('video1');
video1.srcObject = stream;
return stream;
});
console.log(result1.getTracks()[0].getCapabilities().deviceId);
window.setTimeout(async function () {
const result2 = await navigator.mediaDevices.getUserMedia(constraints);
console.log(result2.getTracks()[0].getCapabilities().deviceId);
}, 3000);
}());
</script>
</body>
</html>
Is this expected?
|
Verified by the reference code below: On iOS, a second call to getUserMedia() kills the display of a video stream obtained by an earlier call to getUserMedia(). The original stream displays fine until the subsequent getUserMedia() call, then goes black. Note that this doesn't happen on Desktop Safari, only on iOS Safari in my tests. Reference code: <!DOCTYPE html> <html> <body> <div> <video id="video1" autoplay playsinline></video> </div> <script type="text/javascript"> var constraints1 = { audio: false, video: { height: { max: 480, }, width: { max: 640, }, }, }; navigator.mediaDevices.getUserMedia(constraints1).then(function(stream) { var video1 = document.getElementById('video1'); video1.srcObject = stream; }).catch(function(err) { console.error("Device access checks failed: ", err, constraints1); }); var constraints2 = { audio: false, video: true, }; navigator.mediaDevices.getUserMedia(constraints2).then(function(stream) { }).catch(function(err) { console.error("Device access checks failed: ", err, constraints2); }); </script> </body> </html>