Bug 301824 – Cannot synchronise createMediaElementSource nodes

• Component: Media
• Status: NEW

[steve lee]

Created attachment 477256 [details]
Test web page (required .wavs are at the link in description)

Playing two audio elements via webAudio alows audio files to be merged as well as played / paused. This is the basis of a simple mixer app using web audio (https://mixer.vision-score.com/)

To reproduce please  see the hosted example at https://mixer.vision-score.com/test-sync/ which also show the minimal source code (also attached here).

In Chrome the two audio files play in perfect sync. In Firefox they're very slightly blurred but in Safari webkit (macOS and iOS) they are particularly out, making the mixer unusable.

This is a serious problem as users will be on iPadOS and as far as I know real chrome is STILL not available there (despite EU mumblings).

I've wracked my brains but cannot find anoteher way to do this. Using audio buffer nodes suffer from needing to be in memory and they have no pause / resume.

Is there a solution? Other than say use Chrome on Mac only!? It seems a weakness of the webAudio spec to be honest.

Thanks

[Karl Dubost]

steve,
thanks for the reports. 
https://mixer.vision-score.com/test-sync/

What am I supposed to hear?
I have different results. 

On Firefox I have consistently, be after pause or from the beginning.
1 2 3 A 1 2 3 A

On Chrome it depends on the moments.
Most of the time I have Firefox case, but sometimes I got a non aligned sound. 

On Safari, I have different paces. Such as quick pace. 
Something which sounds like pi-ti pi-ti pi-ti pa-ta
Sometimes almost aligned. 

Are the two soundwaves the same? 
https://mixer.vision-score.com/test-sync/click.wav
https://mixer.vision-score.com/test-sync/click2.wav

 ha yes. There are both the same. 


Did you try createBufferSource() instead?

[steve lee]

My appologies Karl for not making expectations clearer.

- Yes the two audio waveform files are identical (I thought safer not to share a sigle instance)
- should be A 1 2 3 ....
- With perfect sync you should hear a clear single track identical to playing one (should have add a play one option, though could download to play) The waveforms / samples will add and be mixed in the audioContext destination (or another node)
- With slightly out timing you'll hear a fuzzing or flanging effect - the clicks lose clarity
- WIth larger discrepency you hear multiple clicks - the problem I hear on iOS 

So I found the opposite to you on Windows: 
- Chrome - perfect however often I hit play / pause
- Firefox - blurred from start or after mashing play/pause
- Edge - As Chrome but louder (?)

on iPadOS 
- Safari - always multiple clicks
- "Chrome" - good first time but mutiple clicks after pause

on Mac Safari
 - as iPadOS

Looking at the spec there seems no clear requirement for Media nodes to remain in sync - oops. Also, there's little about levels when mixing multiple inputs - which may be why Edge seems to double the volume (though I understand it uses same chromium engine as Chrome?)

As I previously hinted I had concerns about using Buffer nodes because:
- need to fully load audio into memory (30 stereo channels for 5 mins is a lot of samples and could not find any browser limits) . Though for all I know media nodes do this, esp when loaded with a blob as my main mixer app does) 
- no obvious way to pause and resume

I now see I could probably simulate play after pause by using a start() offset time. But have to recreate all the nodes each time (not source buffer fortunately). Plus just spotted that AudioContext has a pause() method.

So I will now try Buffer Nodes and see how that goes.

But this might be a bug - or not - as the spec is silent on the matter. I think that Chrome on WIndows behaviour is ideal - ie keep all nodes always in sync despite pause.

Thanks for your time

[Karl Dubost]

If there is a missing feature for the spec, that could be a good opportunity to open an issue on the spec itself. 
https://github.com/WebAudio/web-audio-api/issues/

I'm not an expert in this area. Probably Jer will have better recommendations than me.

I wonder if it relates to 
https://github.com/WebAudio/web-audio-api/issues/257
which links to https://github.com/w3c/webcodecs?tab=readme-ov-file

[steve lee]

Yes, I thought I'd review the spec carefully. It'll be lime being back at the W3C!

I'll check thos bugs, thanks.

[steve lee]

Let's see - https://github.com/WebAudio/web-audio-api/issues/2652

[Jer Noble]

There's no requirement that two <audio> elements remain in sync with each other; networking issues, decoder hiccups, sample rate drift all might cause timebases to get desynchronized. 

There _used_ to be a web API that explicitly tied together the timebases of multiple media elements, but Chrome removed it, then removed the API from HTML: https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/controller

If you want sample accurate playback, you really need a BufferSource, not a HTMLMediaElement.

[steve lee]

@jer Thanks for the background.

It seems a big hole in the spec not to unify but now I've discovered I can pause the context,  using buffer nodes is less of an issue as don't have to recreate all the nodes to resume after pause. Not sure about loop yet.

As my waveforms are blobs loaded from local files there's no network issues, but without master time track we're vunerable to those other issues. Have to figure looping though.

Thanks again.

[Radar WebKit Bug Importer]

<rdar://problem/164352153>