Bug 180696 – createMediaElementSource() not working with Hls stream

• Component: Web Audio
• Status: NEW

[Francesco Cretti]

Created attachment 329112 [details]
Source code of demo page

Hi everyone,
I'm developing an interactive web application based on WebAudio Api and Hls streams.

I know that a previous issue has been fixed (https://bugs.webkit.org/show_bug.cgi?id=143332), that prevented to import audio content from an HTML5 <audio> tag inside the WebAudio routing graph.

I found out that if the content source of the HTML5 <audio> tag it's an HLS stream (or other stream protocols, such as icecast) the bud is still present.

I prepared an online demo here: http://anaconda.polito.it:9080/~cretti/Hls_createMediaElementsource/

In this page an HLS stream is loaded with a an HTML5 <audio> tag and then is wrapped inside the WebAudio API Context with the createMediaElementSource() method. 
Later, the song is filtered with a LPF at 500 Hz using a BiquadFilter Node. Furthermore, the song volume is modulated by the mouse vertical position, such as in this example. 

The script works with Chrome, Firefox or Opera but it does not in Safari, where the stream plays, but the WebAudioAPI cannot apply any effect.

Thanks in any case,
Francesco

[Jaewon Choi]

Yeah, I also visited here for the first time to report the same problem.

Hope it gets fixed soon.

[Radar WebKit Bug Importer]

<rdar://problem/36477841>

[Francesco Cretti]

I would also add that on iOS the createMeadiaElementSource() does not work with an entire MP3 file either, while it works on Safari desktop (https://bugs.webkit.org/show_bug.cgi?id=143332).

[Tom]

We have a similar problem on desktop. It doesn't seem to work with MSE. It's as if the element is not routed into the web audio graph.

The sound is still heard if nothing is connected to the audio context destination, and any gain nodes have no effect.

Would be great to have this fixed as it means we currently cannot enable a feature on Safari.

[Tom]

This might also be related: https://openradar.appspot.com/37084774

[Guy]

This is still not working also for Safari 15 version. 

createMediaElementSource will work ok with Audio mp3 or Video mp4 as the source for the HTMLMediaElement, but for m3u8/hls stream, as the source the HTMLMediaElement's audio is not correctly routed to the AudioContext and instead the audio is not passed.

[Ollie]

Is this ever going to be fixed or even looked at? 3 years later and we still can't do audio visualisations from live HLS/MP3 streams in Safari, this is really holding our product back currently as we have to provide a CSS fallback.

Related 
https://bugs.webkit.org/show_bug.cgi?id=195043
https://bugs.webkit.org/show_bug.cgi?id=231656

Multiple reports on various JS libs as well.

https://github.com/foobar404/Wave.js/issues/17
https://audiomotion.dev/#/?id=visualization-of-live-streams-wont-work-on-safari
https://discourse.threejs.org/t/audioanalyser-not-working-with-audio-source-as-stream-type-in-safari/18950

[Sam Sneddon [:gsnedders]]

<rdar://84328473>

Our product is also severely impacted by this bug. 

While other major browsers like Chrome and Firefox have no issue with using HLS-backed media elements within their Web Audio contexts, Safari still seems to not support this. 

We have confirmed that this bug persists on Safari 15.5 and Safari Tech Preview 16.0 even in the case where all HLS assets are hosted from the same origin, so it seems that Safari is not adhering to the W3C spec (based on my understanding).

We would greatly appreciate if this bug could receive higher priority and be resolved soon. Thanks!

[Benny Lichtner]

Having the same issue here. We are unable to serve HLS media to our ios users until this bug is fixed :(

Seems to be related to or a duplicate of: https://bugs.webkit.org/show_bug.cgi?id=231656

[Sam Sneddon [:gsnedders]]

*** Bug 231656 has been marked as a duplicate of this bug. ***

[Sam Chang]

I have the same issue on latest iOS and macOS Safari.
Leave a comment to raise attention, hopefully.

[brett]

running into the same issue here. has anyone figured out any solutions yet? this seems to be quite old. 

tested on Safari 16.6 (macOS 13.5.2) + Safari Preview Release 178 + iOS 16.6.1.

[Erik Miller-Galow]

I'm running into the same thing, trying to render visuals to react to my icecast stream. It works in all browsers I've tested besides Safari. Audio does play in Safari but I've yet to find a way to retrieve the frequency data from my analyser (it's all zeroes in Safari).

[Robert]

Hello, I had the displeasure of encountering this longstanding bug today when I decided to use Safari to view the website I had been developing
Without getting into details, this is quite a kafkaesque situation considering it all works perfectly on other browsers and there are no easy solutions to this bug.
This is a StackOverflow question from 2016 about this bug: https://stackoverflow.com/questions/38936642
you can see that in October 2024, the original poster came back and said: "This still haunts me 8 years later"

I'd really appreciate if this bug could be fixed to save future devs from PTSD

[Jonas Lidén]

We're also having this issue in our products in the M&E space. We are using the Media Source Extensions API together with Web Audio API and that works great with Chrome and Firefox but not in Safari. We use the Web Audio API to extract waveform data and it's reporting all zeroes. We also use it for other purposes, like audio routing and solo/mute functionality that does not work, it's like the source nodes are not connected to the audio graph.

[dj8000]

Dear Sirs:

I would like to the add to the chorus as regards this bug. The people want stream visualizers! Let them visualize the stream! WebKit's total disregard for createMediaElementSource spec is a major blow to those who would like to explore the richness of the Web Audio API.

[Adesh Pandey]

Hey! I totally agree with @dj800, we have been using visualisers widely and this has been a pain to manage. 
What I have done rn and might be a intermediate fix for people is, play the audio using audio element and then also play it in another audio context (make sure to implement logics that ensure they are always at the same state). 
Now, play the context along with audio element but do not add the gain to the source just set the gain node as 1. 
This works like a charm but very hack.

[gogonowski]

Here is another relevant WebAudio Demo using HLS.
Specifically, this is hls.js using Managed Media Source.

https://la2.indexcom.com/player/aa/
Choose the first stream, and scroll down.

Safari fails to render the audio visualization.
This will be more than likely due to the native HLS support , and the lack of an audio pipe to support web audio when in HLS mode.
Works perfectly on Mac using Firefox, as that will more than likely be MSE.
Chrome clips the floating-point audio at +/-1.0. But it does render. Sorry Google!


Any questions:
[e-mail]
Thanks.
/greg.

[Daniel Rossi]

I had no idea there was a problem with HLS but as expected webkit don't run any tests with HLS. Their own streaming format. Its the same with WebGL they don't use HLS texture tests so there is always bugs there.

What is the work around ? It doesn't look like they care to fix it to be compatible with every other browser that works with HLS.

[Daniel Rossi]

Using mediasource is also a problem with both hls.js and shaka player. Even Dash under Shaka player there is no audio data. No idea why mp4 works but this does not. It's not just a native HLS issue. What is the work around ? Maybe Brave should become the default browser in macOS and IOS.

[Daniel Rossi]

I looked into intercepting the audio data of a chunk with hls.js to add to a buffersource. But it seems really hard to see if it even works. It can't use the chunk directly and the aac may need to be decoded into channel data via a worker creating a complex work around. What is the suggested work around ?

[Ollie]

There is no workaround for this, and seeing as though this bug was initially reported 8 years ago I don't hold out much hope they will ever fix it.

[Daniel Rossi]

Where there is a will there is a way. I believe source buffer of the audio data of the fragment in a way it accepts the aac data may work. I just cant find an example how source buffer can work with live streams and ts chunks. I tried AudioDecoder api but failed to decode. the aac data is already transmuxed from the ts via hls.js. I just need to get it working on chrome first. 

For Safari can use mediasource streaming method for hls and with hls.js or hopefully videojs can intercept the audio data about to be buffered. I can provide demo once I get it working but some suggestions how to get source buffer working with the transmuxed aac data would be good.

as annoying as it is. It's not working right yet

https://electroteque.org/dev/safari/video5.html

[Daniel Rossi]

Unfortunately I am stuck trying to make the demuxed AAC packet useable for a source buffer node. Nothing accepts partial packets including decoders. I put a call out here. If the packet can be formatted to be usable this work around may just work. There is zero resources about making partial chunk packets usable for source buffers and audio decoders. I didn't realise HLS was broken on Safari until now but not surprised as HLS is never used for any of the browser tests.

https://stackoverflow.com/questions/79794737/making-an-aac-packet-usable-with-creatsourcebuffer-audio-api-to-be-usable-with-h

[Daniel Rossi]

I can't understand why I have to keep making work arounds for webkit with anything to do with HLS. They need to add HLS sources to all their tests. It was the same with having to figure out work arounds for webgl rendering of HLS bugs years ago which I figured out. So costly and time consuming.

After many hours and weeks I figured out a solution. But it's playing back the buffers so not in sync with the video. I need to sync the buffers with the video timecode while playing back. Any ideas how to schedule the source buffers correctly ?

So my own solution needs alot of work to integrate the different setup where the element source cant be used. adding complications.

After hours researching what is required and no information out there. Basically you need to capture the start segment aac packet then append to further aac packets to be able to decode it. Decoding is required before making a buffer source. The start segment includes headers. 

So it works with hls.js but there is videojs vhs to figure out now.

https://electroteque.org/dev/safari/safarifix.html

[Daniel Rossi]

I figured out videojs didn't think it through providing an append buffer event with the data and init segment. And unfortunately mediasource doesn't allow access to the bytes appended. So it required patching the api to get to it.

https://electroteque.org/dev/safari/videojs.html

[Daniel Rossi]

The source buffer node has to be in sync with the playback. As its automatically playing back the buffered data. That is the last problem to figure out.

[Daniel Rossi]

I've figured out a synchronisation solution using texttrack cues. It will play at the exact time as the video. However another roadblock now is forcing mediasource for safari with videojs. It's broken. 

Perhaps webkit can internally do what this hack is doing externally if its broken with HLS chunks with the mediaelement source.

https://electroteque.org/dev/safari/videojs.html

[Daniel Rossi]

Here is the hack working with shaka player. They also did not think about offering an append event so requires patching their api.

video.js's built in VHS tech is broken with Safari and likely has for a while as they don't test it like webkit doesn't make tests for HLS. So requires either hls.js or shakaplayer as a tech to capture the data. I will use my shakaplayer tech then.

https://electroteque.org/dev/safari/shakaplayer.html

[Daniel Rossi]

I've released the working integrations showing audio worker nodes working with the buffer source node. To show an audio meter. the videojs version has to use shaka player which requires patching the api as they dont offer append events. The built in mediasource library is broken with safari. The other is using hls.js capturing data from the event,

I can't believe the level of effort required when dealing with webkit. There is so many missing api issues and bugs. This one has been the hardest and time consuming hack I've had to do  to date. 

https://electroteque.org/plugins/videojs/audio-visualiser/demos/hls/
https://electroteque.org/plugins/flowplayer/flowplayer7/audio-visualiser/demos/hls/