Bug 211394 - createMediaElementSource not working
Summary: createMediaElementSource not working
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Web Audio (show other bugs)
Version: WebKit Nightly Build
Hardware: iPhone / iPad iOS 13
: P2 Major
Assignee: Brent Fulgham
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2020-05-04 13:23 PDT by Luigi Pulcini
Modified: 2021-12-15 12:43 PST (History)
24 users (show)

See Also:


Attachments
Patch (3.72 KB, patch)
2020-05-18 17:09 PDT, Brent Fulgham
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Luigi Pulcini 2020-05-04 13:23:52 PDT
This bug seems to be related to the previous bug #203435, which is currently reported as RESOLVED FIXED.

We asked every user we could get in touch with to test the following player on their iOS devices. Not a single one of the visitors using a device with iOS 13, including the ones who upgraded to the recent iOS 13.4.1, managed to see the player playing back the audio track. The player is stuck as soon as the user hits the play button. No warnings or errors are reported in the browser console.
https://www.staging2.waveplayer.info/

The player uses the `createMediaElementSource` WebAudio function to connect an HTML5 Audio Element as a source for an Analyzer Node so that the audio file can be analyzed and a waveform animated using the Frequency Data coming from the source can be rendered in the player canvas.

Currently, the only alternative we could find is a fallback to the regular Audio Element for all the iOS 13 visitors. A solution that excludes them from seeing the waveform animations.

Please note that this has been working just fine on any other recent versions of iOS until iOS 13 came out.

If needed, I can provide further details about the player.

Thanks.
Comment 1 Radar WebKit Bug Importer 2020-05-04 16:04:48 PDT
<rdar://problem/62866132>
Comment 2 Adam McAmis 2020-05-06 09:03:16 PDT
Luigi's experience with this bug matches up directly with mine. I'm also using `createMediaElementSource` to analyze audio for https://saturn.fm.
Comment 3 dataexcess 2020-05-06 10:21:24 PDT
Same for me. Using audio with createMediaElementSource used in a there's positional audio environment. Not working.
Comment 4 dataexcess 2020-05-06 10:22:08 PDT
three-js positional audio environment*
Comment 5 David 2020-05-12 14:50:21 PDT
Have the same experience as outlined above. Please fix asap. User experience is ruined on iOS devices for my audio visualizer app.
Comment 6 Dries Cleymans 2020-05-13 13:32:47 PDT
We have the same issue. Our app works fine for all users, only those with an iPhone or iPad with iOS 13 or up can't play sound. 

No error occurs. The audio element associated with the node fires the 'play' event but no playback starts. We don't receive any 'timeupdate' from the audio element. When we don't use the 'createMediaElementSource' function on the audio element, the sound plays without a problem.
Comment 7 Eugene M. Joseph 2020-05-14 17:16:07 PDT
Same for me. This functionality is broken. createMediaElementSource is not working and thus analyzer nodes and visualizing audio are not working either. Everything was working fine till iOS 13 was released. It's been over half a year since this bug was first reported. Please bump this up/make this a priority.
Comment 8 Abe Shakim 2020-05-18 08:56:04 PDT
Verifying what all the others have described. This is still an issue. The `createMediaElementSource `WebAudio function is broken. Please fix asap. It's really ruining our customer experience on iOS devices. Works perfectly fine on Android phones and on desktops.
Comment 9 Brent Fulgham 2020-05-18 17:09:42 PDT
Created attachment 399684 [details]
Patch
Comment 10 Per Arne Vollan 2020-05-18 18:24:30 PDT
Comment on attachment 399684 [details]
Patch

R=me.
Comment 11 EWS 2020-05-19 09:21:43 PDT
Committed r261865: <https://trac.webkit.org/changeset/261865>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 399684 [details].
Comment 12 Eugene M. Joseph 2020-05-19 10:19:32 PDT
Do we have a sense for when the fixes will land on iOS?
Comment 13 Eugene M. Joseph 2020-05-19 10:19:54 PDT
Also thanks for much for the prompt handling of this!
Comment 14 Adam McAmis 2020-05-19 10:37:46 PDT
+1 for the quick response, thank you Brent & Per! 🙌
Comment 15 Luigi Pulcini 2020-05-19 20:41:47 PDT
I agree with Adam McAmis: this was very fast compared to the previous report of the same issue.

Any word on the possible timeframe this patch will make it to the public would be very appreciated. The list of complaining users is growing quite fast and if we can expect this to be out soon, it doesn't make sense that we think of a fallback in the meantime.
Comment 16 JTerr 2020-06-02 14:09:15 PDT
(In reply to Luigi Pulcini from comment #15)
> I agree with Adam McAmis: this was very fast compared to the previous report
> of the same issue.
> 
> Any word on the possible timeframe this patch will make it to the public
> would be very appreciated. The list of complaining users is growing quite
> fast and if we can expect this to be out soon, it doesn't make sense that we
> think of a fallback in the meantime.

Yes this would be very helpful! We have had our webview audio broken for months now waiting on a fix. Thank you for looking at this! I know it is still broken on ios 13.5 but hoping for a fix on 13.5.1 or 13.5.5. Thank you guys
Comment 17 bruce 2020-06-21 01:11:30 PDT
Is there any way to find out when this change will be GA for iOS users?
Comment 18 Adel Abdelli 2020-06-21 14:29:17 PDT
Do you know when this patch will be available ? 
It seems not working on IOS 13.5.1 ...
Comment 19 Adam McAmis 2020-06-22 12:16:39 PDT
I can confirm that this is now working as expected for me in the iPad 13.6 Public Beta
Comment 20 Luigi Pulcini 2020-07-20 00:29:05 PDT
iOS 13.6 was released on July 15, 2020.
Can anyone please confirm this problem was permanently fixed in this release?
Comment 21 Steven V. 2020-07-20 06:00:33 PDT
Can confirm it's fixed in 13.6, thanks!
Comment 22 Luigi Pulcini 2020-07-20 06:17:40 PDT
That's great news! Thank you for confirming that.
Comment 23 JTerr 2020-07-20 06:32:57 PDT
I just updated to ios 13.6 on iphone 11. In WkWebview audio for me is still killing 30 seconds after screen goes black. Not fixed for me
Comment 24 Khushpreet 2020-07-20 06:53:11 PDT
Updated iOS 13.6 on iPad mini 4 and tested the sample, audio plays now - 'great!!'. However, audio is distorted, I mean the audio doesn't play smoothly!  Is anyone out there facing the similar issue?
Comment 25 dataexcess 2020-07-20 06:55:31 PDT
Yup same, distorted.

I am using createMediaElementSource for a 3d positional audio (using three-js).
It is not smooth and distorted. Although it does play at least now.
Comment 26 Luigi Pulcini 2020-07-20 06:59:11 PDT
Audio stuttering and crackling seems to be an ongoing problem in Safari, any version, any platform.

I submitted another interesting bug (#212125) about Safari itself not being able to resume playback after a ScriptProcessorNode is disconnected (when pausing) and playing back (when playing again).

It looks like the WebAudio is not stable enough in Safari. In my case I had to implement a fallback to the basic HTML5 Audio Element and give up on the WebAudio API in Safari.
Comment 27 Steve 2020-07-30 04:54:21 PDT
(In reply to Khushpreet from comment #24)
> Updated iOS 13.6 on iPad mini 4 and tested the sample, audio plays now -
> 'great!!'. However, audio is distorted, I mean the audio doesn't play
> smoothly!  Is anyone out there facing the similar issue?

Yes. We have reported the issues regarding distortion when using createMediaElementSource the first time in March 2019. radar ID 48560346
No response or improvement so far. However, back then it only seemed to occur on iOS devices and not on desktop safari.
Comment 28 youenn fablet 2020-07-30 06:18:04 PDT
(In reply to Steve from comment #27)
> (In reply to Khushpreet from comment #24)
> > Updated iOS 13.6 on iPad mini 4 and tested the sample, audio plays now -
> > 'great!!'. However, audio is distorted, I mean the audio doesn't play
> > smoothly!  Is anyone out there facing the similar issue?
> 
> Yes. We have reported the issues regarding distortion when using
> createMediaElementSource the first time in March 2019. radar ID 48560346
> No response or improvement so far. However, back then it only seemed to
> occur on iOS devices and not on desktop safari.

Do you still repro that on recent iOS versions.
Doing a quick check, I am not sure I repro the issue, maybe the audio is not getting data fast enough and the audio element needs to do buffering.

In any case, since this issue is closed, would you be able to file new issues, ideally with repro cases (and/or sysdiagnoses sent privately).
Comment 29 Steve 2020-08-09 05:22:37 PDT
I just tested, now that 13.6 is available.

This issue, where audio would no longer be played at all, seems to be fixed.
The issue with distorted audio / crackles is still present. 

I submitted a codepen to reproduce the issue with my initial bug report. It now plays again, but it played distorted on first try (easily audible).

I'll happily open another bug report here, if that is helpful. But could you please let me know what sysdiagnoses you need and what additional information you would like to receive privately?

Do you have access to the radar issue? (48560346)

The issue likely is not related to not getting data fast enough, since the playback issues immediately appear when playback starts. If this should be the case though, this also means that the browser would try to start playback at a point where it does not have sufficient data loaded.


(In reply to youenn fablet from comment #28)
> (In reply to Steve from comment #27)
> > (In reply to Khushpreet from comment #24)
> > > Updated iOS 13.6 on iPad mini 4 and tested the sample, audio plays now -
> > > 'great!!'. However, audio is distorted, I mean the audio doesn't play
> > > smoothly!  Is anyone out there facing the similar issue?
> > 
> > Yes. We have reported the issues regarding distortion when using
> > createMediaElementSource the first time in March 2019. radar ID 48560346
> > No response or improvement so far. However, back then it only seemed to
> > occur on iOS devices and not on desktop safari.
> 
> Do you still repro that on recent iOS versions.
> Doing a quick check, I am not sure I repro the issue, maybe the audio is not
> getting data fast enough and the audio element needs to do buffering.
> 
> In any case, since this issue is closed, would you be able to file new
> issues, ideally with repro cases (and/or sysdiagnoses sent privately).
Comment 30 Luigi Pulcini 2020-08-14 15:52:04 PDT
In my experience, the issue with the distorted/crackling noise comes with a mismatch between the samplerates of the audio context and the audio file being played back.

If the audio file is encoded at a samplerate different than the one assigned to the audio context (as far as I understand, the audio context is assigned the sample rate of the current audio device), then the playback will be affected by audio crackles.

I hope this helps.
Comment 31 Khushpreet 2020-08-17 21:44:21 PDT
Hi, Thanks for the explanation. 



As you mentioned, if playback sample rate is different from the sample rate in which audio was recorded, audio would be glitchy. Therefore, would anyone share how browser/system decide the sample rate. 

I have a case where my media's sample rate is 48kHz, but one of my system played the playback at 44100 sample rate and audio was glitchy. 

For test, I created the audio context with default sample rate 48000 (hard-coded) in my player and audio quality got better. 



Now my question is, 
> How does system decide the sample rate? From above test, I got an understanding that this particular system could support 48000 sample rate but chose to play at 44100, why would that happen?

> Does the system/browser configuration affect the settings? Is it about the memory?

PS. I know the setting hard-coded sample rate is not a good idea due to the following reasons: 

1. User can play media at different sample rate, so it should dynamically decided by the player. 

2. I cannot set default sample rate while creating audio context in safari(please correct me if my understanding is wrong), so I'll have the same issue on safari. 

Any explanation would be very helpful for me to understand this. Thank you for your time and help in advance.
Comment 32 rob.reng 2021-03-04 05:50:45 PST
createMediaElementSource is still broken as far as I can tell (Version 14.0.3 (16610.4.3.1.4))

Running this sample still has silence:

https://mdn.github.io/webaudio-examples/media-source-buffer/

On all other browsers there is audio as expected.

Any idea of when a fix for this would be ready?

I enabled Experimental Feartures -> Modern WebAudio API and it still doesn't work....
Comment 33 dataexcess 2021-09-08 06:10:41 PDT
This issue is still not fixed!!!

Please, I am running a virtual gallery space website for electronic music and sound art and right now all the iOS users cannot have a correct audio experience on my platform (it is the biggest platform according to my analytics)

You can test it with this simple audio three-js example:
(wait a few minutes until the crackles take over)
https://threejs.org/examples/webaudio_orientation.html

Or on my own website:
www.subneu.live

Please fix or give us more info!
Comment 34 Brian Peiris 2021-09-20 10:57:52 PDT
Issues with crackling audio may be more specific to a regression in PannerNode as described here: https://bugs.webkit.org/show_bug.cgi?id=227199
Comment 35 dataexcess 2021-11-11 05:29:46 PST
(In reply to Brian Peiris from comment #34)
> Issues with crackling audio may be more specific to a regression in
> PannerNode as described here: https://bugs.webkit.org/show_bug.cgi?id=227199

Interesting! But the issue is still not fixed correct?
Comment 36 bigflannel 2021-11-23 09:15:22 PST
When I use createMediaElementSource and createAnalyser I get no sound. Its a local file, I create the AudioContext and analyser and make connections after a user interaction. This is some code I am working from and is a good example of what does not work:

https://codepen.io/nfj525/pen/rVBaab

Thank you.
Comment 37 bigflannel 2021-12-15 12:39:54 PST
Hm. I think my issue here is I am using audio.onplay as my user event. It doesn't qualify as a user event. I see if I use a user click event it does work. You can't get a click event off the HTML audio player (as far as I can tell) when someone clicks play, so I can't use clicking play on the audio player as the interaction event, which means layering something else for the user. Hm.
Comment 38 bigflannel 2021-12-15 12:40:28 PST
Hm. I think my issue here is I am using audio.onplay as my user event. It doesn't qualify as a user event. I see if I use a user click event it does work. You can't get a click event off the HTML audio player (as far as I can tell) when someone clicks play, so I can't use clicking play on the audio player as the interaction event, which means layering something else for the user. Hm.
Comment 39 bigflannel 2021-12-15 12:43:56 PST
Sorry about the double post.