Bug 266922 – MediaElementAudioSourceNode doesn't pass audio sample data to subsequent node if based on MSE

• Component: Web Audio
• Status: NEW

[Andrey Goncharov]

MediaElementAudioSourceNode doesn't pass audio sample data to subsequent node in audio graph if video/audio element represented by the MediaElementAudioSourceNode has MediaSource object as a source. 
If however the video/audio element has Blob object with the same data as a source, then MediaElementAudioSourceNode passes audio data correctly.

There are two sample page links in the end of this description, that show buggy and correct behavior respectively. The problem is reproduced with AudioWorklet node, Analyser node and ScriptProcessor node (deprecated), so I used AudioWorklet node as the most relevant example.

Steps to reproduce:
1. Create MediaSource object:
    var mediaSource = new MediaSource;

2. Create object URL from mediaSource and assign it as media element's source:
    videoElement.src = URL.createObjectURL(mediaSource);

3. Wait for the mediaSource object to switch to 'open' state, add source buffer to it and then append AAC init segment and audio samples in ISO BMFF container.

4. On a user's gesture, instantiate AudioContext and create MediaElementAudioSourceNode interface:
    audioContext.createMediaElementSource(videoElement);

5. Assemble audio graph, connecting AudioWorklet node and destination node consequently. AudioWorklet processor has basic configuration that only transmits sample data values to the main thread in its process() method.

6. Observe sample values returned by AudioWorklet node processor.

Expected behavior: actual sample values in [-1, 1] range.
Actual behavior: all sample values are zeroes.

If the same audio samples are fed to the media element with Blob object, e.g.
    var blob = await fetchedData.blob();
    videoElement.src = URL.createObjectURL(blob);

then AudioWorklet node processor returns expected sample values.

Here are simple demo pages for both scenarios (please wait several seconds for AAC samples to download):

MediaSource related bug: https://dev.wmspanel.com/www_tmp/safari/safari_bug.html
Correct behavior with Blob source: https://dev.wmspanel.com/www_tmp/safari/safari_ok.html

Chrome and Firefox browsers show expected behavior on MacOS, Linux and Windows. The same issue is reproduced on iPadOS with all browsers.

It seems like this issue is connected to this one: https://bugs.webkit.org/show_bug.cgi?id=180696
I'm not sure that they have the same cause, but might overlap though.

[Andrey Goncharov]

Created attachment 469225 [details]
Sample page displaying the issue

[Andrey Goncharov]

Created attachment 469226 [details]
Sample page displaying the correct behavior

[Andrey Goncharov]

Created attachment 469227 [details]
Script file to include into sample pages

[Andrey Goncharov]

Created attachment 469228 [details]
Audio worklet processor file necessary for sample pages

[Andrey Goncharov]

Created attachment 469229 [details]
AAC audio samples for demo

[Radar WebKit Bug Importer]

<rdar://problem/120467987>

[gogonowski]

Here is another relevant WebAudio Demo using HLS.
Specifically, this is hls.js using Managed Media Source.

HLS Audio Analyzer
https://la2.indexcom.com/player/aa/
Choose the first stream, and scroll down.

Safari fails to render the audio visualization.
This will be more than likely due to the native HLS support , and the lack of an audio pipe to support web audio when in HLS mode.
Works perfectly on Mac using Firefox, as that will more than likely be MSE.
Chrome clips the floating-point audio at +/-1.0. But it does render. Sorry Google!


Any questions:
[e-mail]
Thanks.
/greg.