Bug 176282 - autoplay hidden video element with MediaStream src that has audio track
Summary: autoplay hidden video element with MediaStream src that has audio track
Status: REOPENED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebRTC (show other bugs)
Version: Safari Technology Preview
Hardware: All All
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-09-02 13:23 PDT by Ben
Modified: 2019-10-04 06:39 PDT (History)
10 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ben 2017-09-02 13:23:00 PDT
Hidden video element with live MediaStream src from a PeerConnection don't play even when there is an audio track.
Even if the video is changed to visible it is still paused unless there is a new visible video element that starts playing.

The use case is a web conference app with multiple speakers. The dominant speaker is presented in the center (stage).
Other speakers can still speak but are not visible (hidden video element).
The app automatically promotes new dominant speaker to the stage when needed.

I understand the autoplay policy for normal video elements but for a live WebRTC conference this is really limiting.
https://webkit.org/blog/6784/new-video-policies-for-ios/
Comment 1 youenn fablet 2018-03-12 13:58:51 PDT
Hi Ben,

On iOS, there is only one video element with audio that can play at a time.
You will need to change a bit your web site to work around this, for instance by creating a single audio element containing all the audio tracks.

I am marking this bug as WontFix, let me know if I misunderstand something or if you would like further assistance.
Comment 2 Ben 2018-03-12 17:22:29 PDT
Hi Youenn,

Does it means that I can have several video elements playing at the same time but only one of them can also have audio? Or that I can only play one video element?

If I have 4 speakers in a conference I need to create 4 muted video elements and one audio element with 4 tracks?

Where is this limitation documented?
Comment 3 Chad Phillips 2018-03-12 17:43:20 PDT
@Youenn, does iOS have the same limitation for audio elements? Meaning, can you only play one audio element at a time?

@Ben, if you can play more than one audio element at a time, then you can simply split the media between audio/video elements. I've done this and have had no sync issues. However, if you're putting many people into one conference, I'd seriously recommend a media server in the middle, probably of SFU architecture. Some of them pre-mix all the audio into one stream, which would also get around the possible iOS limitation I'm asking Youenn about above.
Comment 4 youenn fablet 2018-03-12 18:28:49 PDT
(In reply to Ben from comment #2)
> Hi Youenn,
> 
> Does it means that I can have several video elements playing at the same
> time but only one of them can also have audio? Or that I can only play one
> video element?

Right, multiple video elements, but only one playing audio.


> If I have 4 speakers in a conference I need to create 4 muted video elements
> and one audio element with 4 tracks?

Right, this is the simplest approach and better than using WebAudio.

> Where is this limitation documented?

(In reply to Chad Phillips from comment #3)
> @Youenn, does iOS have the same limitation for audio elements? Meaning, can
> you only play one audio element at a time?

See https://developer.apple.com/library/content/documentation/AudioVideo/Conceptual/Using_HTML5_Audio_Video/Device-SpecificConsiderations/Device-SpecificConsiderations.html for more.
Comment 5 Ben 2018-03-13 06:14:38 PDT
Why does iOS have the limitation on multiple video elements with sound?

If I can play several muted video elements and several audio elements at the same time then I can achieve the same result only with ugly workarounds.
Comment 6 Ben 2018-03-13 06:47:48 PDT
@Youenn,any chance to remove this limitation when user gave mic permission and/or when the stream is live from a peer connection?
Comment 7 Chad Phillips 2018-03-13 11:04:17 PDT
@Youenn,

I'm pretty sure the documentation you pointed me to is incorrect. Here is the relevant section for clarity:

"Currently, all devices running iOS are limited to playback of a single audio or video stream at any time. Playing more than one video—side by side, partly overlapping, or completely overlaid—is not currently supported on iOS devices. Playing multiple simultaneous audio streams is also not supported."

This is *definitely* false for the case of multiple video streams in the WebRTC case -- I've already run scenarios with up to seven video feeds being received, each in their own video element.

It does seem to be true that I can only have one active publisher stream (see bug https://bugs.webkit.org/show_bug.cgi?id=179363 for more details on this unfortunate current limitation)

I have yet to test the case of multiple audio elements on iOS, but will report back here if I learn anything.

It does seem that the iOS media restrictions are a poor fit for WebRTC -- some of them really hamstring the capability of the standard.
Comment 8 youenn fablet 2018-03-13 11:16:48 PDT
Reopening to investigate whether to change the current behavior or not.
Comment 9 Ben 2018-04-22 01:22:55 PDT
@Youenn can you please explain how to use a single audio element with multiple tracks? I can't make the audio play.

I've muted all the video elements:
<video muted autoplay playsinline></video>
I've created a new audio element:
<audio autoplay></audio>
I've created an empty media stream for the audio tracks:
var audioStream = new MediaStream();
and set the source of the audio element:
audioEl.src = audioStream;
Every time the peer connection receive a new audio track I'm adding it to the audio stream:
if(e.track.kind == 'audio') {
  audioStream.addTrack(e.track);
}
When an audio track is removed from the peer connection I'm removing it from the audio stream:
if(e.track.kind == 'audio'){
  audioStream.removeTrack(e.track);
}
Comment 10 Ben 2018-05-10 03:06:16 PDT
@Youenn any update or workaround?
I can't make several audio tracks play at the same time.
Comment 11 Ben 2018-06-06 07:02:49 PDT
@Youenn, can you please help?
How can I play multiple webrtc audio streams in Safari?
Comment 12 youenn fablet 2018-06-06 08:00:38 PDT
(In reply to Ben from comment #11)
> @Youenn, can you please help?
> How can I play multiple webrtc audio streams in Safari?

@Ben, could you try any of the following approaches:

// srcObject
mediaElement.srcObject = new MediaStream([mediaStream1.getAudioTracks()[0], mediaStream1.getAudioTracks()[1]]);

// Web Audio
let audioContext = new webkitAudioContext();
audioContext.createMediaStreamSource(mediaStream1).connect(context.destination);
audioContext.createMediaStreamSource(mediaStream2).connect(context.destination);
Comment 13 Ben 2018-06-06 10:29:09 PDT
@Youenn, the AudioContext example is working. I'll run more tests.
Thanks.

There will probably still be an issue when playing WebRTC audio and MP4 video at the same time.

Still hoping this limitation will be removed to help conferencing web apps.
Comment 14 youenn fablet 2018-06-06 10:33:55 PDT
(In reply to Ben from comment #13)
> @Youenn, the AudioContext example is working. I'll run more tests.
> Thanks.

Let me know if the MediaStream-based approach is not working.
I agree using web audio here is more a workaround than anything else.
Comment 15 Ben 2018-06-06 11:06:08 PDT
I think I'm loosing audio/vidoe lip sync with the AudioContext hack.

I've tried to use a single MediaStream for all audio tracks but couldn't make it work yet.
I need to be able to dynamically add and remove audio and video tracks.
Not sure if Safari will let me dynamically change the audio tracks in a MediaStream.
I'll also might need to reattach the MediaStream to the audio element every time a new audio track is added.
I can maybe create a new MediaStream every time I need to add an audio track but it'll create a pause in the other audio tracks.
Comment 16 Neil Kinnish 2018-09-18 07:36:54 PDT
I've been using audiocontext with multiple audio streams in a conference scenario on iOS 11 for a while now and it's been great, since iOS 12 though no audio plays, there are no issues raised in console... can anyone throw any light on the potential issue?
Comment 17 Neil Kinnish 2018-09-18 09:20:41 PDT
To add to my last comment, it seems to work with headphones, so I'm assuming the way audiocontext pipes to destintation has changed, it seems quite glitchy tho. Removing and readding headphones, etc
Comment 18 Neil Kinnish 2018-09-18 09:48:24 PDT
Doh! my phone was muted (do not disturb button) this was previously not respected in iOS 11 so this was the reason, although it's quite glitchy switching to and from headphones and tends to lose the audio.
Comment 20 Dan 2019-02-03 08:56:31 PST
Just tested out the mediaStream approach @Youenn mentioned. I tried two different methods:
1. audioElement.srcObject = new MediaStream(allTheTracks) every time a new audio track came in.
2. Used the same mediaStream object the entire time and added tracks dynamically with mediaStrack.addTrack()

I got similar results for both methods. Sometimes, everything worked fine. Other times, only some tracks would play audio. I wasn't able to reliably reproduce the issue, it seemed to happen a random % of the time (about 50%). The times when audio wasn't playing, some of the tracks had enabled === false. Interestingly, the tracks that had enabled === false could often be heard and the tracks that had enabled === true couldn't. Very odd!

For the most part, everything works fine if I render a separate audio element for each audio track. The only weird thing I noticed here is that sometimes audio can't be heard even when all indications say it should be (audio track is enabled, audio element is playing, etc). Interestingly, this issue seemed to go away when adding the viewport meta tag to my html <meta name="viewport" content="width=device-width, initial-scale=1.0">. I'm wondering if this weird behaviour has something to do with the audio elements being rendered out of the viewport. I tried to reliably reproduce this issue but wasn't able to. It seemed to happen randomly when not including the meta tag. I'd refresh the page without changing the viewport and sometimes I could hear all the tracks, sometimes it couldn't. I don't seem to get the issue after adding that meta tag though.

Web Audio also worked for me. I didn't test it enough to know if there are lip-sync issues though as @Ben mentioned.

Just for reference, I ran these tests on an iPhone 5s and iPad Mini 2. I needed host candidates so I had the viewer grant device access with getUserMedia() then I stopped the tracks immediately before viewing remote streams from other devices.

I'm wondering if we can add a rule to allow multiple video elements to play audio if they are backed by a mediaStream. I agree with the comments above that this limitation is quite inconvenient for WebRTC. I imagine it will prevent some companies from supporting Safari in their WebRTC applications.
Comment 21 Dan 2019-02-19 15:11:38 PST
Just created a github repo to reproduce the case I mentioned where putting each mediaStream in a separate audio element has issues sometimes. https://github.com/danbriggs5/safari-multiple-audio-tracks

It's a little more complex than a simple codepen/fiddle repo because you need to form a WebRTC connection between your computer and your iOS device.

I'm very curious if others can reproduce the issue. As mentioned in my previous comment, adding the <meta name="viewport" content="width=device-width, initial-scale=1.0"> tag to my html seems to fix it.