Bug 236219 – [iOS] Volume of non MediaStreamTrack-based audio reduces when capturing microphone

• Status: NEW
• Component: Media

[youenn fablet]

See discussion in https://bugs.webkit.org/show_bug.cgi?id=218012.
While we fixed the volume issue for MediaStreamTrack audio rendering, volume is still reduced for WebAudio, and potentially non MediaStream based media elements.

[youenn fablet]

See https://bugs.webkit.org/show_bug.cgi?id=218012#c31 for instance.

[Radar WebKit Bug Importer]

<rdar://problem/88670586>

[youenn fablet]

https://bugs.webkit.org/show_bug.cgi?id=248810 should help.

[John]

Has this problem been solved? 

We are seeing this exact same problem, at least on iPads running versions 15 and 16, including 16.3 

When we ask the user to accept Microphone access, if they do, the video playback has decidedly lower audio volume (even if not yet using the mic yet or not actually having yet activated mic recording).

If they do not accept the request for mic access our video clip plays at normal high volume level — but we need mic access to do what we want to do on the page. 

I do not see this problem using Safari on Mac.

[Bryce Aebi]

We are still experiencing this bug after 6 months on iOS 16.2

Our entire web app is rendered unusable on iPhone due to this bug. I am happy to assist in any way to get this bug resolved. Thank you

[youenn fablet]

Have you tried iOS 16.4 beta?
Please also note that calling getUserMedia will change the audio session category to "play-and-record" which will switch to the VC level.
To make sure "play-and-record" is consistently being used, you can set navigator.audioSession.type as per https://github.com/w3c/audio-session/blob/main/explainer.md.

[Bryce Aebi]

Youenn, thank you for the prompt reply. I will look into both of your suggestions and report back.

[Bryce Aebi]

Ok, so I tried a few things:

Describing the overall problem:
1. I play audio at the normal volume. Everything is fine
2. I then play audio after turning on the mic, usually the first bit of audio at this point plays much louder than normal.
3. I play other bits of audio while the mic is on. Now all of the audio is 1/2 as loud as normal
4. I turn off the mic and play other audio. The audio remains 1/2 as loud as normal

16.2 - web audio - result: broken
16.2 - routing web audio through MediaStreamTrack - result: broken

16.4 - web audio - broken
16.4 - routing web audio through MediaStreamTrack - result: broken


On 16.4, I tried web audio after setting navigator.audioSession.type = "play-and-record". This is the behavior
1. I play audio. It is already at 1/2 volume
2. I then play audio after turning on the mic, usually the first bit of audio at this point plays much louder than normal.
3. I play other bits of audio while the mic is on. Now all of the audio is 1/2 as loud as normal
4. I turn off the mic and play other audio. The audio remains 1/2 as loud as normal

So the behavior by setting "play-and-record" is different than without setting it, but it is still very suboptimal. 

Please let me know if there's anything else I can try out for you.

[Bryce Aebi]

My understanding is that what's happening is grabbing the mic automatically changes the audioSession.type to "play-and-record". Then after mic use, the audioSession.type remains set as "play-and-record".

So I guess what I should do is manually change audioSession.type back to "playback" after ending use of the mic. That would solve the issue of persistent low volume after mic usage.

However, this would NOT solve the issue where the audio is at first very loud after starting the mic, and then becoming 1/2 as loud during the remainder of the mic usage. Ideally I want the audio to simply be at normal volume when the mic is active.

[Bryce Aebi]

Ok, after some investigation I have figured out some more:

1. Setting audioSession.type = 'play-and-record' BEFORE mic capture and then resetting it to 'playback' AFTER mic capture ends does fix the 1/2 volume audio issue that was persisting AFTER mic capture.

2. As expected, I still have volume issues DURING mic capture, but I better understand what's going on now. The iPhone volume appears to get "saved" for each audioSession.type. For 'playback', it is saved to 50% of my iPhone's volume. For 'play-and-record' it is saved to 100% of my iPhone's volume, which I had to do because the volume gets so low during mic capture. There appears to be a secondary bug here though, because the FIRST audio I play AFTER mic capture starts is at 100% of the iPhone volume and 100% of the audio's volume, before reverting to 100% iPhone volume x 50% audio volume (that is, the iPhone's volume is still at 100%, but the audio that gets played has the low volume issue that occurs during mic capture).

The result of the above is quite jarring. Before mic capture, the user has the volume at a comfortable level. Then when they start mic capture, the user gets blasted by loud audio at first, then audio that is just below a comfortable level until mic capture ends. Then the volume reverts back to a comfortable level again.

[Bryce Aebi]

Ultimately, I think in my particular web app, I MAY be able to get around these issues by limiting the play-and-record state to ONLY when the mic is capturing.

For performance reasons (mic capture takes a while to start), I had started mic capture earlier than it was needed and left it on. Then the user would make multiple recordings and play multiple pieces of audio while the mic was capturing. 

I will see if performance is tolerable to turn on play-and-record + mic capture for each audio recording the user makes. The issue is that since mic capture is slow, it is common for the user to click the record button and then the start of their recorded audio ends up not being captured. I may be able to do something gimmicky, like setting a 3-2-1 countdown timer in the UI after they click the record button and before they actually produce noise to be recorded. Obviously this is still a subpar experience, but sounds like a separate bug/improvement.

Ideally, I think there should be an additional audioSession.type that ALLOWS for mic capture, but DOES NOT change the iPhone volume OR the audio volume playing during the mic capture.

[John]

Does anyone have a fix for this yet? 

I have been monitoring this for a while and there is no discussion since March 2, 2023. 

We have a web app wherein we want to 
a) ask for (and receive) mic acccess
b) play a short section of video (with audio)
c) Give the user the opportunity to press a "mic" button and speak a response into the mic. 

The problem (just to restate) is that if the user approves mic access, when we play the short clip of video, the audio plays low at like 1/2 volume. We want it to play a full volume. 

If the user denies mic access, then it plays at full volume, but then the app does not work because the next step is to record the user's response using the mic. 

My coder has tried implementing some of the ideas discussed here, namely:



Before mic recording (before and while we play the video clip)  set to:
navigator.audioSession.type = 'play-and-record';
navigator.mediaSession.type = 'play-and-record';

After mic recording set to:
navigator.audioSession.type = 'playback';
navigator.mediaSession.type = 'playback';

but it's still not working on iphone/iOS.
It still plays the video clip at 1/2 volume.

Is there something we are doing wrong? 

We don't need to (or want to)  play the audio WHILE the mic is recording, only before the mic is recording. But of course we do want the app to retain "permission" to access the mic because we repeat the sequential video playback and "mic record" sequence over and over again many times. We don't want the user to have to keep giving permission to access the mic over and over again. 

Any help is appreciated.

[youenn fablet]

(In reply to John from comment #12)
> 
> Before mic recording (before and while we play the video clip)  set to:
> navigator.audioSession.type = 'play-and-record';
> navigator.mediaSession.type = 'play-and-record';
> 
> After mic recording set to:
> navigator.audioSession.type = 'playback';
> navigator.mediaSession.type = 'playback';
> 
> but it's still not working on iphone/iOS.
> It still plays the video clip at 1/2 volume.
> 
> Is there something we are doing wrong? 
> 
> We don't need to (or want to)  play the audio WHILE the mic is recording,
> only before the mic is recording. But of course we do want the app to retain
> "permission" to access the mic because we repeat the sequential video
> playback and "mic record" sequence over and over again many times. We don't
> want the user to have to keep giving permission to access the mic over and
> over again.

Setting audioSession.type to playback does not help if continuing to capture the microphone.
Permission is kept even if not capturing for 1 minute on iOS by default.
We could consider updating the heuristic.

Another approach is to set it to play-and-record, user having the ability to increase the volume themselves

[John]

Thanks so much for your reply, Youenn.

Youenn Fablet wrote:
          "Setting audioSession.type to playback does not help if continuing to capture the microphone.
Permission is kept even if not capturing for 1 minute on iOS by default.
We could consider updating the heuristic."

I am not 100% sure what you mean. 

To be clear we don't have this problem in Safari on OSX, only Safari on iOS and iPadOS.

You can test our application by going here:
https://french.yabla.com/scribe_speech_speak.php?source=direct&media_id=19313

You will see a message:
"Allow “french.yabla.com” to use your microphone?"

Choose "Allow".


Click "play" for "set 1".


You will see and hear a video clip, with a subtitle, where a man says, in French:
"C'est vraiment galère, les élections..."
 

If you are using Safari on OSX, the audio will play just fine at full volume. 
If you are using Safari on iOS or iPadOS, the audio will play faintly, at something like 1/2 volume.

We have tried these settings:
> navigator.audioSession.type = 'play-and-record';
> navigator.mediaSession.type = 'play-and-record';

and we have also tried these settings:
> navigator.audioSession.type = 'playback';
> navigator.mediaSession.type = 'playback';

but either way, the audio plays back faintly, and our users (those on iPhone and iPad)  complain.

Youenn Fablet wrote:
          "Another approach is to set it to play-and-record, user having the ability to increase the volume themselves"

As I mentioned above, we have tried both 'play-and-record' and also 'playback', but either way the audio still plays faintly.

I am not sure what you mean "user having the ability to increase the volume themselves". 

How would a user "increase the volume themselves"?

It is "unexpected behavior" for Safari to suddenly reduce playback volume at this point. 
Are you suggesting that each time the user would have to using the volume control on their iPhone or iPad to push the volume back up? We would have to send them an instruction to do this, and it would be odd. It would also be tedious to have to do this each time.

I would understand reducing playback volume while the mic is actually recording, but the mic is not active or recording at this point, it is simply "allowed". So what would be the purpose? It seems like a "bug", no?


Like I said, using Safari on OSX the playback volume after "allowing" the mic is just fine, the playback volume is not reduced. 

Can the Webkit team not have it work the same way in iOS and iPadOS as it works in OSX — it works well in OSX?

Thanks so much for your attention!

[John]

Some more information: 

I have been testing on an iPad running iPadOS version 15. 


"allowing" mic resets the device volume for audio playback:

I find that after "allowing" mic access, the volume of audio playback is no longer determined by the volume set using the physical volume button on the iPad device. If the volume of the iPad device had been set to max, after hitting "allow" for mic access audio, playback will sound a softer (lower volume). However if the volume of the iPad is set to a very low volume (e.g. 1/4 of max), then after hitting "allow" for mic access, audio playback will sound comparatively much louder.

"allowing" mic seems alter the audio quality upon audio playback. 

I am testing with a video we recorded in our studio. 
The first sentence spoken in our video also contains "room tone" that is clearly audible before our female talent starts speaking (like fan or AC noise or distant traffic or that type of thing). Before "allowing" mic access when I play this video I can clearly hear this room noise. But after "allowing" mic access I can no longer hear that room tone. Also the quality of the speaker's voice is changed, it sounds more "tinny" or like an "am radio", or vaguely like she's in a tunnel. It seems like certain frequencies are being removed, it almost seems like a "high pass filter" has been applied, or something like that.

[Adesh Pandey]

There is some optimization may be necessary to add for using HTML5 audio elements. I solved it using Howler[https://howlerjs.com/] and playing it through audioContext.

[Adesh Pandey]

Not solved but avoided the hustle.

[youenn fablet]

https://bugs.webkit.org/show_bug.cgi?id=277421 may be a solution to this bug, if doing the following:
- capture microphone
- mute capture via navigator.mediaSession.setMicrophoneActive(false)
- play some audio (must be done once audio capture is muted, not before).
- stop playing audio
- unmute capture via navigator.mediaSession.setMicrophoneActive(true) if needed (requires a user gesture)

[John]

(In reply to youenn fablet from comment #18)
> https://bugs.webkit.org/show_bug.cgi?id=277421 may be a solution to this
> bug, if doing the following:
> - capture microphone
> - mute capture via navigator.mediaSession.setMicrophoneActive(false)
> - play some audio (must be done once audio capture is muted, not before).
> - stop playing audio
> - unmute capture via navigator.mediaSession.setMicrophoneActive(true) if
> needed (requires a user gesture)

[John]

(In reply to youenn fablet from comment #18)
> https://bugs.webkit.org/show_bug.cgi?id=277421 may be a solution to this
> bug, if doing the following:

Has anyone had any success with Youenn's suggestion in #18?

My coder tried but no real success — he says it "works" if he goes into the Safari settings and "mutes" by clicking "stop using microphone" and then unutes it, but of course that is not practical, something we can ask the user to do.


"if you mute/unmute manually on iphone his idea works" is what my coder says

My coder tried to use 
navigator.mediaSession.setMicrophoneActive(false)
prior to playing the audio 
and
navigator.mediaSession.setMicrophoneActive(true)
when the user initiated our "mic" button to record. 


We could not get it to work / did not solve the problem of the audio not playing at proper volume.

Did anyone get this to work?

[youenn fablet]

(In reply to John from comment #20)
> (In reply to youenn fablet from comment #18)
> > https://bugs.webkit.org/show_bug.cgi?id=277421 may be a solution to this
> > bug, if doing the following:
> 
> Has anyone had any success with Youenn's suggestion in #18?
> 
> My coder tried but no real success — he says it "works" if he goes into the
> Safari settings and "mutes" by clicking "stop using microphone" and then
> unutes it, but of course that is not practical, something we can ask the
> user to do.
> 
> 
> "if you mute/unmute manually on iphone his idea works" is what my coder says
> 
> My coder tried to use 
> navigator.mediaSession.setMicrophoneActive(false)
> prior to playing the audio 
> and
> navigator.mediaSession.setMicrophoneActive(true)
> when the user initiated our "mic" button to record. 
> 
> 
> We could not get it to work / did not solve the problem of the audio not
> playing at proper volume.
> 
> Did anyone get this to work?

setMicrophoneActive did not yet ship in Safari. It is now testable in Safari Tech Preview only (off by default for now).