Bug 217606 – Web Audio broken on iOS 14 after switching app

• Status: RESOLVED
• Resolution: FIXED
• Component: Web Audio

[Ashley Gullen]

Demo URL: https://downloads.scirra.com/labs/ios14audio/standard/index.html

Steps to reproduce:
1. Press button and verify a sound can be heard
2. Go to the home screen
3. Switch back to Safari so the page is showing again
4. Press the button again

Observed result:
Audio no longer plays.

Expected result:
Audio to continue playing.

It looks like all our JavaScript code is running as expected, and the AudioContext state is still "running" after switching back to the app, but no audio playback happens. So it appears to be a regression in iOS 14. This affects all web content made in Construct 3 (www.construct.net).

[Ashley Gullen]

Note it also works correctly in Safari 14 on macOS, and in Chrome on Android, so it looks like this only affects iOS.

[Radar WebKit Bug Importer]

<rdar://problem/70231769>

[Chris Dumez]

I see an exception showing in the console on homing out:
Unhandled Promise Rejection: undefined

		_SetSuspended(e)
		{
			const isSuspended = e["isSuspended"];

			console.log("[Construct 3] Setting audio suspended: ", isSuspended);
			
			// Upon resume: first resume the whole context
			if (!isSuspended && this._audioContext["resume"])
				this._audioContext["resume"]();
			
			// Suspend or resume all audio instances
			for (const ai of this._audioInstances)
				ai.SetSuspended(isSuspended);
				
			// After suspend: also suspend the whole context
			if (isSuspended && this._audioContext["suspend"])
				this._audioContext["suspend"](); // On this line.
		}

Looks like the script is not dealing with this promise rejection.

[Ashley Gullen]

Why would that promise reject? It didn't before, so this seems to be new.

[Chris Dumez]

(In reply to Ashley Gullen from comment #4)
> Why would that promise reject? It didn't before, so this seems to be new.

I will investigate. It is possible it is new behavior since we've been working on aligning our WebAudio implementation with the W3C specification.

[Chris Dumez]

Still reproduces on with WebKit trunk.

[Chris Dumez]

suspend() throws because the audio context's state is "interrupted", likely because the view is not longer visible and we interrupt audio in this case.

It seems wrong to reject the promise in this case. Seems like we should set the "suspendedByUser" flag and resolve the promise in this case.

[Ashley Gullen]

This may be off topic but the spec says the only valid AudioContext states are "running", "suspended" and "closed". Since "interrupted" isn't a specified state we haven't written any code to try to handle that. Separately, this has caused us problems in the past in WKWebView, since some things (in our case showing an AdMob ad) seemed to cause the AudioContext to end up in this non-standard state, causing audio playback to fail in that case as well.

[Chris Dumez]

(In reply to Ashley Gullen from comment #8)
> This may be off topic but the spec says the only valid AudioContext states
> are "running", "suspended" and "closed". Since "interrupted" isn't a
> specified state we haven't written any code to try to handle that.
> Separately, this has caused us problems in the past in WKWebView, since some
> things (in our case showing an AdMob ad) seemed to cause the AudioContext to
> end up in this non-standard state, causing audio playback to fail in that
> case as well.

Yes, "interrupted" is not a standard state and I agree we should stop exposing to the Web this internal implementation detail.

[Chris Dumez]

Even after I stop rejecting the suspend() promise and get rid of the "interrupted" state, clicking the button still does not play audio sometimes (it is flaky). I am not sure why (The JS is pretty complicated) and there is no longer any exception/error in the console.

When the button does not cause audio playback, I see that the AudioContext's state is "running", as expected.

[Chris Dumez]

(In reply to Chris Dumez from comment #10)
> Even after I stop rejecting the suspend() promise and get rid of the
> "interrupted" state, clicking the button still does not play audio sometimes
> (it is flaky). I am not sure why (The JS is pretty complicated) and there is
> no longer any exception/error in the console.
> 
> When the button does not cause audio playback, I see that the AudioContext's
> state is "running", as expected.

Could you point me to the code that actually plays the audio when I click the button? I tried to follow in Web Inspector but could not easily find the logic.

[Chris Dumez]

Created attachment 412355 [details]
Patch

[Ashley Gullen]

If you beautify the source and search for createBufferSource, there is one in a method named Play. That is where the actual playback is done and the call to start() made.

[Chris Dumez]

(In reply to Ashley Gullen from comment #13)
> If you beautify the source and search for createBufferSource, there is one
> in a method named Play. That is where the actual playback is done and the
> call to start() made.

Thanks, will debug further.

[Chris Dumez]

Created attachment 412364 [details]
Patch

[Chris Dumez]

I landed this first patch via Bug 218235 and am keeping this bug open because it is still very flaky. There seems to be a remaining issue where when going back to Safari, we try to resume the AudioContext and fail due to a permission error. The call to AudioOutputUnitStart() fails with an AVAudioSessionErrorCodeCannotStartPlaying error code and we still move the AudioContext's state to "running" and resolve the resume() promise. There are several issues here:
1. If the attempt to start the AudioDestination fails, we should probably reject the resume() promise and not set the state to "running".
2. Why are we getting a AVAudioSessionErrorCodeCannotStartPlaying error when trying to start the AudioDestination when going back to Safari. This is flaky so I assume we attempt to start the destination slightly before AVFoundation realizes we are allowed to play.

[Chris Dumez]

After the fixes from Bug 218235 & Bug 218251, this test page provided actually works reliably how. That said, it only works reliably because the test page keeps trying to resume the context until context.state becomes "running".

As a result, I am keeping this bug open for now. We still have a race when coming back to Safari causing us to fail to resume rendering in some cases. Without the retry logic on the JS side, it would fail which is bad. I am looking into why this race is happening and if we can resolve it. If not, I guess we could implement a retry logic on WebKit side to avoid relying on the page's JS to do so.

[Chris Dumez]

Created attachment 412556 [details]
Patch

[Chris Dumez]

Created attachment 412558 [details]
Patch

[Chris Dumez]

Created attachment 412567 [details]
Patch

[Eric Carlson]

Comment on attachment 412567 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=412567&action=review

r=me once the bots are happy

> Source/WebCore/platform/audio/cocoa/AudioDestinationCocoa.cpp:108
> +            weakThis->setIsPlaying(true);

Why not call `setIsPlaying` when success is false?

[Chris Dumez]

Comment on attachment 412567 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=412567&action=review

>> Source/WebCore/platform/audio/cocoa/AudioDestinationCocoa.cpp:108
>> +            weakThis->setIsPlaying(true);
> 
> Why not call `setIsPlaying` when success is false?

I am maintaining pre-existing behavior here.

If you try to start and starting fails, presumably there is nothing to do since you did not start playing. isPlaying should already be false?
Either way, seems out of scope of this patch no? I am not changing behavior here AFAICT.

[Chris Dumez]

Created attachment 412577 [details]
Patch

[Chris Dumez]

Created attachment 412583 [details]
Patch

[Chris Dumez]

Created attachment 412595 [details]
Patch

[Eric Carlson]

Comment on attachment 412595 [details]
Patch

r=me once the bots can deal with it

[Chris Dumez]

Created attachment 412599 [details]
Patch

[EWS]

Committed r269134: <https://trac.webkit.org/changeset/269134>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 412599 [details].