Bug 244505 – REGRESSION (iOS 16 Beta 7) Outgoing video is stopped and do not recover for WebRTC call if the iPhone device is connected with a Bluetooth device (AirPods) and User uses Youtube / Spotify or other app in the middle of the call

• Component: WebRTC
• Status: RESOLVED CONFIGURATION CHANGED

[Madara Freimane]

Summary:
REGRESSION (iOS 16 Beta 7) Outgoing video is stopped and do not recover for WebRTC call if the iPhone device is connected with a Bluetooth device (AirPods) and User uses Youtube / Spotify or other app in the middle of the call

Tested devices:
Bug is reproducible on:
- iPhone 11 Pro Max | iOS 16 Beta 7 (20A5356a)

Bug is not reproducible on:
- iPhone 11 | iOS 15.6
- iPhone 11 | iOS 15.6.1

Use case:
Preconditions:
iPhone device is connected with AirPods or other Bluetooth device
User is in active WebRTC video call
Permissions for camera and microphone are set to "Ask" in browser settings

Steps:
1. iOS 16 User navigates to Youtube native app or to another native music app and listens music for ~3min
2. iOS 16 User navigates back to the Safari browser
3. iOS 16 User accepts required permissions for audio and video

Actual result:
Outgoing video is stopped and do not recover for WebRTC call if the iPhone device is connected with a Bluetooth device (AirPods) and User uses Youtube / Spotify or other app in the middle of the call
- Sometimes also audio is impacted and User does not have incoming audio anymore after the navigation back to the call

Expected result:
Incoming and outgoing audio, video are available and recovers for WebRTC call if iPhone device connected with Bluetooth device and the call is interrupted by Youtube, Siri, PSTN, Facetime call or other service

Reproducibility:
100%


Additional information:
-The bug is reproducible only in a case if we have pop-ups about the permissions requests when we return to Safari browser.
-With a low reproducibility, it is possible to reproduce a bug if we enable Siri in the middle of the call, receive PSTN / FaceTime calls (and talk more than 2mins) + if we have pop-ups which asks again approve permissions when we return to the browser.

To be able compare behaviours with iOS 15.6 and iOS 16 Beta 7, added two sysdiagnose files:
=> From iOS 16 Beta 7 (iOS_16Beta7_sysdiagnose_2022.08.29_20-41-28+0300_iPhone-OS_iPhone_20A5356a.tar.gz):
https://drive.google.com/file/d/15_tTigk6tJUFGzhW10DPK6_Vm5pnVNOA/view?usp=sharing

=> From iOS 15.6 (iOS_15.6_sysdiagnose_2022.08.29_21-32-11+0300_iPhone-OS_iPhone_19G71.tar.gz):
https://drive.google.com/file/d/11ENU4kvvSVGQbwpFHl2t-RMulW5TJjvB/view?usp=sharing


This bug can be linked with https://bugs.webkit.org/show_bug.cgi?id=243982, I created a new to get a clearer status of the existing regression.

[Dag-Inge Aas]

Tried reproducing this. I can kinda do it, but it eventually recovers. Tested on iPhone 12 Pro MGMP3QN/A running iOS 16 Beta 8 (20A5358a) using Airpods Pro A2084 running 4E71 (1.3.8).

Test procedure:
1. Go to https://webrtc.github.io/samples/src/content/peerconnection/pc1/ and initiate the call. Observe that you can see and hear yourself.
2. Go to the home screen and navigate to Spotify
3. Start playing a random song. Observe that the capture color disappears in the clock.
4. Return to Safari. Observe that both video views are black and you can no longer hear yourself.
5. Wait 3-4 seconds. Observe that the page starts capturing again.

I can also reliably reproduce the same behaviour with the YouTube app. Key seems to be starting audio playback in a native app. Expected behaviour (video mutes when Safari is not the active app) happens when I go to home screen and other apps without playing audio, and recovers as expected quickly when Safari is the active app again.

Using my trust gUM debugger (https://codepen.io/daginge/pen/ExLxZzN) I see that the video track mutes as expected when going to the home screen. When the audio starts playing, the audio track mutes. Upon returning to the page, the video and audio track unmutes after 2-3 seconds.

Logs:
Got stream with constraints: {"audio":true,"video":{"facingMode":"user","deviceId":"default","width":1280,"height":720}}
Using video device: Front Camera
The video track is currently unmuted
Using audio device: AirPods
The audio track is currently unmuted
Muted video track: true
Visibilitychange event triggered: Hidden prop is true
Muted audio track: true <-- This is when I start playing Spotify audio
Visibilitychange event triggered: Hidden prop is false
Unmuted audio track: false
Unmuted video track: false

However, if I continue playing audio in Spotify, neither the video or audio track unmutes. If I stop playback, they instantly recover.

In conclusion: I'm able to reproduce this issue on beta 8 if I continuously play audio. It seems to be an OS mute issue, but the capture is luckily never ended, and will recover.

[youenn fablet]

(In reply to daginge from comment #1)
> Tried reproducing this. I can kinda do it, but it eventually recovers.
> Tested on iPhone 12 Pro MGMP3QN/A running iOS 16 Beta 8 (20A5358a) using
> Airpods Pro A2084 running 4E71 (1.3.8).
> 
> Test procedure:
> 1. Go to https://webrtc.github.io/samples/src/content/peerconnection/pc1/
> and initiate the call. Observe that you can see and hear yourself.
> 2. Go to the home screen and navigate to Spotify
> 3. Start playing a random song. Observe that the capture color disappears in
> the clock.
> 4. Return to Safari. Observe that both video views are black and you can no
> longer hear yourself.
> 5. Wait 3-4 seconds. Observe that the page starts capturing again.

@daginge, this is somehow expected behaviour.
Is your request that restarting capture should happen more quickly?

> 
> I can also reliably reproduce the same behaviour with the YouTube app. Key
> seems to be starting audio playback in a native app. Expected behaviour
> (video mutes when Safari is not the active app) happens when I go to home
> screen and other apps without playing audio, and recovers as expected
> quickly when Safari is the active app again.
> 
> Using my trust gUM debugger (https://codepen.io/daginge/pen/ExLxZzN) I see
> that the video track mutes as expected when going to the home screen. When
> the audio starts playing, the audio track mutes. Upon returning to the page,
> the video and audio track unmutes after 2-3 seconds.
> 
> Logs:
> Got stream with constraints:
> {"audio":true,"video":{"facingMode":"user","deviceId":"default","width":1280,
> "height":720}}
> Using video device: Front Camera
> The video track is currently unmuted
> Using audio device: AirPods
> The audio track is currently unmuted
> Muted video track: true
> Visibilitychange event triggered: Hidden prop is true
> Muted audio track: true <-- This is when I start playing Spotify audio
> Visibilitychange event triggered: Hidden prop is false
> Unmuted audio track: false
> Unmuted video track: false
> 
> However, if I continue playing audio in Spotify, neither the video or audio
> track unmutes. If I stop playback, they instantly recover.

Can you clarify this?
I would have thought that, when going back to Safari, Safari would restart capture and would interrupt Spotify.
Isn't this happening?

[Dag-Inge Aas]

(In reply to youenn fablet from comment #2)
> (In reply to daginge from comment #1)
> > Tried reproducing this. I can kinda do it, but it eventually recovers.
> > Tested on iPhone 12 Pro MGMP3QN/A running iOS 16 Beta 8 (20A5358a) using
> > Airpods Pro A2084 running 4E71 (1.3.8).
> > 
> > Test procedure:
> > 1. Go to https://webrtc.github.io/samples/src/content/peerconnection/pc1/
> > and initiate the call. Observe that you can see and hear yourself.
> > 2. Go to the home screen and navigate to Spotify
> > 3. Start playing a random song. Observe that the capture color disappears in
> > the clock.
> > 4. Return to Safari. Observe that both video views are black and you can no
> > longer hear yourself.
> > 5. Wait 3-4 seconds. Observe that the page starts capturing again.
> 
> @daginge, this is somehow expected behaviour.
> Is your request that restarting capture should happen more quickly?

Honestly I don't really mind, as long as it reliably restarts. It's riiight at the limit of users being confused I think, but let's see if we get any reports. I think this is such a corner case that we won't hear much. It is faster without the AirPods in though, can't explain that.


> > However, if I continue playing audio in Spotify, neither the video or audio
> > track unmutes. If I stop playback, they instantly recover.
> 
> Can you clarify this?
> I would have thought that, when going back to Safari, Safari would restart
> capture and would interrupt Spotify.
> Isn't this happening?

I'll do you one better and share a video: https://www.icloud.com/photos/#05bDb-5-djR1WZFp3U53sf0Gw (a bit loud due to the music getting picked up).

Strangely, with screen recording the behaviour kinda changed, it never recovered. Or rather, the capture seems to recover, but the video went white (which in this context I think means transparent). But if you ignore this fact for now, the behaviour I described is there. The video and audio track doesn't recover until I pause playback.

Hope this helps!

[youenn fablet]

> Strangely, with screen recording the behaviour kinda changed, it never
> recovered. Or rather, the capture seems to recover, but the video went white
> (which in this context I think means transparent).

We might still have some bad interaction between autoplay and interruptions here.
It might be that calling video.play() will fix the white issue.

[Radar WebKit Bug Importer]

<rdar://problem/99574139>

[youenn fablet]

> => From iOS 16 Beta 7
> (iOS_16Beta7_sysdiagnose_2022.08.29_20-41-28+0300_iPhone-OS_iPhone_20A5356a.
> tar.gz):
> https://drive.google.com/file/d/15_tTigk6tJUFGzhW10DPK6_Vm5pnVNOA/
> view?usp=sharing

On this sysdiagnose, I do not find anything wrong.
The encoded frame counter is increasing from the beginning of the call 15:20 to the end 15:29. Ditto for audio.

I do not see also where the audio get interrupted. Or maybe I did not look at the right timestamp?

[youenn fablet]

> I'll do you one better and share a video:
> https://www.icloud.com/photos/#05bDb-5-djR1WZFp3U53sf0Gw (a bit loud due to
> the music getting picked up).

It does not seem bad that Spotify is not interrupted by moving Safari to foreground.
It seems good that user does an action so that it happens. In that case it would be unmuting capture using the red camera icon in the menu bar.
I'll have a try with Spotify to verify this is working as expected.

[Madara Freimane]

Hi Youenn!

I added a new sysdiagnose file:
https://drive.google.com/file/d/1Bsq3gIAJOJ7NEK5aBSYBhh8Z9-v5OhXE/view?usp=sharing

Bug reproduced on 2022.09.07_10-48-03

[Madara Freimane]

Hi, Youenn!

After the investigation we see more than one issue:

1. Bluetooth device issue:

Use case:
Preconditions:
iPhone device is connected with AirPods or other Bluetooth device
Permissions for camera and microphone are set to "Ask" in browser settings
User is in active WebRTC video call

Steps:
1. iOS 16 User navigates to Youtube native app or to another native music app and listens music for ~3min
2. iOS 16 User navigates back to the Safari browser
3. iOS 16 User accepts required permissions for audio and video

=> AirPods option disappears and Microphone device in settings is switching from iPhone mic to "No mic device found".
Added sysdiagnose file:
https://drive.google.com/file/d/14i4QbOQvr8K3ItZ79S3jaMcP3QUt5X34/view?usp=sharing



2. H264 encoder/decoder issue:
Use case:
Preconditions:
iPhone device is connected with AirPods or other Bluetooth device
Permissions for camera and microphone are set to "Ask" in browser settings
The WebRTC call is an active, video enabled for other Users in the call

Steps:
1. iOS 16 User joins a call
1. iOS 16 User navigates to Youtube native app or to another native music app and listens music for ~3min
2. iOS 16 User navigates back to the Safari browser
3. iOS 16 User accepts required permissions for audio and video
=> video becomes broken and sometimes also incoming audio disappears + observed that call becomes broken immediately sometimes if iOS User joins an active WebRTC call while cameras enabled for other Users => we don't have incoming video for iOS User in that case.
Added sysdiagnose file with this use case:
https://drive.google.com/file/d/1oYuYD_mwaIATrpTBcTZjsWWPuFxEwOHo/view?usp=sharing
In my opinion there is a higher repro if iPhone device is connected with AirPods, but I was able reproduce defect also with other Bluetooth devices (for example, with Huawei FreeBuds 4i), but with a lower reproducibility.

All sysdiagnostics files were generated immediately the defects were reproduced.


Tested device:
* iPhone 11 Pro Max | iOS 16, build: 20A362

[Madara Freimane]

Hi, Youenn!



Still can reproduce defects.
Added new sysdiagnostics files, recorded logs using iOS 16.1 Beta 2 (Build: 20B5050f)

Here using 2nd use case I reproduced this issue: H264 encoder/decoder issue
When came back to Safari browser => AirPods was still connected with a device, no audio issue, only outgoing video was broken (Time: 16:46, EET):

https://drive.google.com/file/d/1M48-cn5JbkLW57eHzsj_K1ymfZh9gdxW/view?usp=sharing

Here using 1st use case use case I reproduced this issue: Bluetooth device issue
When came back to Safari browser => AirPods option was not available anymore, didn’t have incoming audio + outgoing video was broken (Time: 17:43, EET):


https://drive.google.com/file/d/1rgXXZ6BCZEsLOC7VqVidNkqeb5YabWnf/view?usp=sharing

Tested device:
* iPhone 11 Pro Max | iOS 16.1 Beta 2, Build: 20B5050f

[Madara Freimane]

Hi, Youenn!



Bug is still reproducible.
Tested device:
iPhone 11 Pro Max, iOS 16.2 Beta, Build: 20C5032e.

Added new sysdiagnostics file (bug reproduced at 11:05, 07.11.2022 (EET time zone):
https://drive.google.com/file/d/1Iek-6-IjsruI4UXepEDRgjfWJWyPK9oP/view?usp=share_link

[Madara Freimane]

I’m not able to reproduce described issue in ticket with iOS 16.2 Beta 2, build: 20C5043e, iPhone 11 Pro Max.

@Youenn, should we see the fix with this version?

Bug linked with https://bugs.webkit.org/show_bug.cgi?id=243982

[Madara Freimane]

Bug is not reproducible anymore.

Tested OK on:
* iPhone 11 Pro Max | iOS 16.2, build: 20C65
* iPhone 12 Pro Max | iOS 16.2, build: 20C65
* iPhone 14 | iOS 16.2, build: 20C65

Thanks!