Bug 211394 – createMediaElementSource not working

• Component: Web Audio
• Status: RESOLVED FIXED

[Luigi Pulcini]

This bug seems to be related to the previous bug #203435, which is currently reported as RESOLVED FIXED.

We asked every user we could get in touch with to test the following player on their iOS devices. Not a single one of the visitors using a device with iOS 13, including the ones who upgraded to the recent iOS 13.4.1, managed to see the player playing back the audio track. The player is stuck as soon as the user hits the play button. No warnings or errors are reported in the browser console.
https://www.staging2.waveplayer.info/

The player uses the `createMediaElementSource` WebAudio function to connect an HTML5 Audio Element as a source for an Analyzer Node so that the audio file can be analyzed and a waveform animated using the Frequency Data coming from the source can be rendered in the player canvas.

Currently, the only alternative we could find is a fallback to the regular Audio Element for all the iOS 13 visitors. A solution that excludes them from seeing the waveform animations.

Please note that this has been working just fine on any other recent versions of iOS until iOS 13 came out.

If needed, I can provide further details about the player.

Thanks.

[Radar WebKit Bug Importer]

<rdar://problem/62866132>

[Adam McAmis]

Luigi's experience with this bug matches up directly with mine. I'm also using `createMediaElementSource` to analyze audio for https://saturn.fm.

[dataexcess]

Same for me. Using audio with createMediaElementSource used in a there's positional audio environment. Not working.

[dataexcess]

three-js positional audio environment*

[David]

Have the same experience as outlined above. Please fix asap. User experience is ruined on iOS devices for my audio visualizer app.

[Dries Cleymans]

We have the same issue. Our app works fine for all users, only those with an iPhone or iPad with iOS 13 or up can't play sound. 

No error occurs. The audio element associated with the node fires the 'play' event but no playback starts. We don't receive any 'timeupdate' from the audio element. When we don't use the 'createMediaElementSource' function on the audio element, the sound plays without a problem.

[Eugene M. Joseph]

Same for me. This functionality is broken. createMediaElementSource is not working and thus analyzer nodes and visualizing audio are not working either. Everything was working fine till iOS 13 was released. It's been over half a year since this bug was first reported. Please bump this up/make this a priority.

[Abe Shakim]

Verifying what all the others have described. This is still an issue. The `createMediaElementSource `WebAudio function is broken. Please fix asap. It's really ruining our customer experience on iOS devices. Works perfectly fine on Android phones and on desktops.

[Brent Fulgham]

Created attachment 399684 [details]
Patch

[Per Arne Vollan]

Comment on attachment 399684 [details]
Patch

R=me.

[EWS]

Committed r261865: <https://trac.webkit.org/changeset/261865>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 399684 [details].

[Eugene M. Joseph]

Do we have a sense for when the fixes will land on iOS?

[Eugene M. Joseph]

Also thanks for much for the prompt handling of this!

[Adam McAmis]

+1 for the quick response, thank you Brent & Per! 🙌

[Luigi Pulcini]

I agree with Adam McAmis: this was very fast compared to the previous report of the same issue.

Any word on the possible timeframe this patch will make it to the public would be very appreciated. The list of complaining users is growing quite fast and if we can expect this to be out soon, it doesn't make sense that we think of a fallback in the meantime.

[JTerr]

(In reply to Luigi Pulcini from comment #15)
> I agree with Adam McAmis: this was very fast compared to the previous report
> of the same issue.
> 
> Any word on the possible timeframe this patch will make it to the public
> would be very appreciated. The list of complaining users is growing quite
> fast and if we can expect this to be out soon, it doesn't make sense that we
> think of a fallback in the meantime.

Yes this would be very helpful! We have had our webview audio broken for months now waiting on a fix. Thank you for looking at this! I know it is still broken on ios 13.5 but hoping for a fix on 13.5.1 or 13.5.5. Thank you guys

Is there any way to find out when this change will be GA for iOS users?

[Adel Abdelli]

Do you know when this patch will be available ? 
It seems not working on IOS 13.5.1 ...

[Adam McAmis]

I can confirm that this is now working as expected for me in the iPad 13.6 Public Beta

[Luigi Pulcini]

iOS 13.6 was released on July 15, 2020.
Can anyone please confirm this problem was permanently fixed in this release?

[Steven V.]

Can confirm it's fixed in 13.6, thanks!

[Luigi Pulcini]

That's great news! Thank you for confirming that.

[JTerr]

I just updated to ios 13.6 on iphone 11. In WkWebview audio for me is still killing 30 seconds after screen goes black. Not fixed for me

[Khushpreet]

Updated iOS 13.6 on iPad mini 4 and tested the sample, audio plays now - 'great!!'. However, audio is distorted, I mean the audio doesn't play smoothly!  Is anyone out there facing the similar issue?

[dataexcess]

Yup same, distorted.

I am using createMediaElementSource for a 3d positional audio (using three-js).
It is not smooth and distorted. Although it does play at least now.

[Luigi Pulcini]

Audio stuttering and crackling seems to be an ongoing problem in Safari, any version, any platform.

I submitted another interesting bug (#212125) about Safari itself not being able to resume playback after a ScriptProcessorNode is disconnected (when pausing) and playing back (when playing again).

It looks like the WebAudio is not stable enough in Safari. In my case I had to implement a fallback to the basic HTML5 Audio Element and give up on the WebAudio API in Safari.

[Steve]

(In reply to Khushpreet from comment #24)
> Updated iOS 13.6 on iPad mini 4 and tested the sample, audio plays now -
> 'great!!'. However, audio is distorted, I mean the audio doesn't play
> smoothly!  Is anyone out there facing the similar issue?

Yes. We have reported the issues regarding distortion when using createMediaElementSource the first time in March 2019. radar ID 48560346
No response or improvement so far. However, back then it only seemed to occur on iOS devices and not on desktop safari.

[youenn fablet]

(In reply to Steve from comment #27)
> (In reply to Khushpreet from comment #24)
> > Updated iOS 13.6 on iPad mini 4 and tested the sample, audio plays now -
> > 'great!!'. However, audio is distorted, I mean the audio doesn't play
> > smoothly!  Is anyone out there facing the similar issue?
> 
> Yes. We have reported the issues regarding distortion when using
> createMediaElementSource the first time in March 2019. radar ID 48560346
> No response or improvement so far. However, back then it only seemed to
> occur on iOS devices and not on desktop safari.

Do you still repro that on recent iOS versions.
Doing a quick check, I am not sure I repro the issue, maybe the audio is not getting data fast enough and the audio element needs to do buffering.

In any case, since this issue is closed, would you be able to file new issues, ideally with repro cases (and/or sysdiagnoses sent privately).

[Steve]

I just tested, now that 13.6 is available.

This issue, where audio would no longer be played at all, seems to be fixed.
The issue with distorted audio / crackles is still present. 

I submitted a codepen to reproduce the issue with my initial bug report. It now plays again, but it played distorted on first try (easily audible).

I'll happily open another bug report here, if that is helpful. But could you please let me know what sysdiagnoses you need and what additional information you would like to receive privately?

Do you have access to the radar issue? (48560346)

The issue likely is not related to not getting data fast enough, since the playback issues immediately appear when playback starts. If this should be the case though, this also means that the browser would try to start playback at a point where it does not have sufficient data loaded.


(In reply to youenn fablet from comment #28)
> (In reply to Steve from comment #27)
> > (In reply to Khushpreet from comment #24)
> > > Updated iOS 13.6 on iPad mini 4 and tested the sample, audio plays now -
> > > 'great!!'. However, audio is distorted, I mean the audio doesn't play
> > > smoothly!  Is anyone out there facing the similar issue?
> > 
> > Yes. We have reported the issues regarding distortion when using
> > createMediaElementSource the first time in March 2019. radar ID 48560346
> > No response or improvement so far. However, back then it only seemed to
> > occur on iOS devices and not on desktop safari.
> 
> Do you still repro that on recent iOS versions.
> Doing a quick check, I am not sure I repro the issue, maybe the audio is not
> getting data fast enough and the audio element needs to do buffering.
> 
> In any case, since this issue is closed, would you be able to file new
> issues, ideally with repro cases (and/or sysdiagnoses sent privately).

[Luigi Pulcini]

In my experience, the issue with the distorted/crackling noise comes with a mismatch between the samplerates of the audio context and the audio file being played back.

If the audio file is encoded at a samplerate different than the one assigned to the audio context (as far as I understand, the audio context is assigned the sample rate of the current audio device), then the playback will be affected by audio crackles.

I hope this helps.

[Khushpreet]

Hi, Thanks for the explanation. 



As you mentioned, if playback sample rate is different from the sample rate in which audio was recorded, audio would be glitchy. Therefore, would anyone share how browser/system decide the sample rate. 

I have a case where my media's sample rate is 48kHz, but one of my system played the playback at 44100 sample rate and audio was glitchy. 

For test, I created the audio context with default sample rate 48000 (hard-coded) in my player and audio quality got better. 



Now my question is, 
> How does system decide the sample rate? From above test, I got an understanding that this particular system could support 48000 sample rate but chose to play at 44100, why would that happen?

> Does the system/browser configuration affect the settings? Is it about the memory?

PS. I know the setting hard-coded sample rate is not a good idea due to the following reasons: 

1. User can play media at different sample rate, so it should dynamically decided by the player. 

2. I cannot set default sample rate while creating audio context in safari(please correct me if my understanding is wrong), so I'll have the same issue on safari. 

Any explanation would be very helpful for me to understand this. Thank you for your time and help in advance.

createMediaElementSource is still broken as far as I can tell (Version 14.0.3 (16610.4.3.1.4))

Running this sample still has silence:

https://mdn.github.io/webaudio-examples/media-source-buffer/

On all other browsers there is audio as expected.

Any idea of when a fix for this would be ready?

I enabled Experimental Feartures -> Modern WebAudio API and it still doesn't work....

[dataexcess]

This issue is still not fixed!!!

Please, I am running a virtual gallery space website for electronic music and sound art and right now all the iOS users cannot have a correct audio experience on my platform (it is the biggest platform according to my analytics)

You can test it with this simple audio three-js example:
(wait a few minutes until the crackles take over)
https://threejs.org/examples/webaudio_orientation.html

Or on my own website:
www.subneu.live

Please fix or give us more info!

[Brian Peiris]

Issues with crackling audio may be more specific to a regression in PannerNode as described here: https://bugs.webkit.org/show_bug.cgi?id=227199

[dataexcess]

(In reply to Brian Peiris from comment #34)
> Issues with crackling audio may be more specific to a regression in
> PannerNode as described here: https://bugs.webkit.org/show_bug.cgi?id=227199

Interesting! But the issue is still not fixed correct?

[bigflannel]

When I use createMediaElementSource and createAnalyser I get no sound. Its a local file, I create the AudioContext and analyser and make connections after a user interaction. This is some code I am working from and is a good example of what does not work:

https://codepen.io/nfj525/pen/rVBaab

Thank you.

[bigflannel]

Hm. I think my issue here is I am using audio.onplay as my user event. It doesn't qualify as a user event. I see if I use a user click event it does work. You can't get a click event off the HTML audio player (as far as I can tell) when someone clicks play, so I can't use clicking play on the audio player as the interaction event, which means layering something else for the user. Hm.

[bigflannel]

Hm. I think my issue here is I am using audio.onplay as my user event. It doesn't qualify as a user event. I see if I use a user click event it does work. You can't get a click event off the HTML audio player (as far as I can tell) when someone clicks play, so I can't use clicking play on the audio player as the interaction event, which means layering something else for the user. Hm.

[bigflannel]

Sorry about the double post.