Frequent situations where iOS device locks up, requiring a hard reboot when consuming webrtc streams on IOS

• Component: WebRTC
• Status: RESOLVED CONFIGURATION CHANGED

I'm encountering fairly frequent hard locks/crashes when viewing webrtc streams using Safari on the current iOS 11 beta 3.

How can I go about providing crash dumps to you for this issue? Thank you.

Correction, iOS 11 beta 4.

Verified on two different devices so far both running beta 4.

[Alexey Proskuryakov]

After syncing the iPhone or iPad to the Mac, the crash logs should be visible in Console.app, ~/Library/Logs -> CrashReporter -> MobileDevice.

Could you please also add steps to reproduce?

[youenn fablet]

Could you also share on which webrtc sites you are experiencing hard locks and crashes?

I can provide a link and logs, but it involves an internal site, so I provided the details to both of you in a private email.

[Alexey Proskuryakov]

I don't see the e-mail. Youenn, do you?

Please confirm that the information is OK to share internally within Apple.

Yes, you can share within Apple.

Alexey, It's unfortunate that you cannot see the information :( Is your actual email different than the one listed here? I'm getting replies from Youenn, so I know he can see it. If you send an email to my address, perhaps any spam blocking mechanisms on that side will be opened up. Youenn asked me some questions, so I'm sending the below to the private email thread. I'm posting it here so that Alexey can see it as well.

There seems to be some confusion. By "crash/hard lock", I mean that the entire iPad freezes and has to be hard rebooted (home button + sleep button). That is the only issue that I'm encountering -- "Crash" and "hard lock" are not two separate issues. I've changed the title to reflect this.

The application on the server also uses libwebrtc and is basically a screensharing app of sorts. More participants are not necessary to cause the locking. If the character is not quite smooth, that is not a big deal... that is not the issue that I'm describing. You can try interacting with the view ("use single finger drag, use two finger drag, use pinch, etc") and this may help with reproducing it.

> If you can try with an iPad with more memory, maybe the crashes will no longer happen which is why I do not see the crashes.
Regarding using an iPad with more memory: As I mentioned in my report that I'm using an iPad 12" which has 4GB. As far as I know, the most memory of any iPad. The hard lockup has still occurred many times, the logs that I provided were from that iPad. I suppose it could be a memory thing on the client side, but should a webpage really be able to cause an out of memory condition that locks up the iPad, requiring a hard reboot like this?

Some questions for Youenn and anyone else try to reproduce this:

Is there something more I can do on my end? More dignostic information I can provide?

If you weren't able to reproduce the issue, how long did you try to to reproduce the issue for? Unfortunately, I have had long periods (10+ minutes) where it works fine without hard locking, then immediately crashes on the next refresh. So it can be difficult to reproduce sometimes, and requires some patience, and many refreshes.

What version of iOS did you try to reproduce it on? I am using iOS beta 4. If you're using something newer (i.e. only available within Apple), I have not tested with that.

Some other points:

If a lockup doesn't occur within a 30-60 seconds of loading the page, just reload and try again. If you try 5-6 times with no lockup. Just wait and try again later (30 minutes later, 60 minutes later, etc).

Most of the lockups I've seen occur within 30 seconds of pageload, so if you haven't seen one by then, there's a good chance you won't see one during that pageload.

I mention that because I notice someone on a pretty long session right now. Long session length is not the thing that seems to bring about the hard lock, so when trying to reproduce this, you're probably better off refreshing 4-5 times, and  letting it run for about 30 seconds each time after you refresh, then if the issue doesn't happen, try another 4-5 reloads a bit later on.

[Alexey Proskuryakov]

> Alexey, It's unfortunate that you cannot see the information :(

Spam blocker failure on an intermediate server. Youenn shared the data with me, thanks!

Was someone able to repro it, by chance?

This still happens in beta 5. I've emailed ap and youenn with further details.

[Radar WebKit Bug Importer]

<rdar://problem/33781374>

Any updates on this? Really hoping a fix goes in before iOS 11 is released...
Thanks.

[Mike Block]

One other consideration on this, we've noticed the freezing sometimes occurs when there's considerable audio echo / feedback when doing local testing.

Is it possible that echo cancellation is involved in the lock up?

I'd originally thought it was cropping on the video, but if there's no audio involved, it seems to be okay. Maybe I'm just lucky on these tests as it seems inconsistent.

[youenn fablet]

Latest iOS 11.2 beta might solve the issue.
Let me known if this is still not working, otherwise I will close this bug in a few weeks.

[Mike Block]

When is 11.2 due for release?

[Jay Charles]

I see no change in this behavior with 11.2 beta.

In my case I'm consuming streams from Wowza Streaming Engine. I've tested against many encoder sources (chrome, firefox, ffmpeg, wirecast), all using the same video encoding parameters (the wirecast test case involves transcoding audio as the input is AAC, but that's not particularly relevant here)

In nearly all cases, playback of video freezes after as little as 1 second, but as long as several minutes. In my case the video freeze takes the safari tab with it (tab goes unresponsive and has to be closed), but does not take down safari or the OS.

I've also tested with streams encoded with flash. I have to modify the profile level id to make this work (what comes out of flash isn't constrained, so I have to lie a bit to prevent outright rejection of the sdp). Playback works perfectly everywhere except webkit. In webkit, it can take several minutes for playback to begin, but once it starts it never freezes. Go figure.

[zzxx53]

Testing with the latest 11.2 beta on iPhone 6; with Google Chrome on the other side. The situation has definitely improved; the system no longer freezes up. However I am getting issues where the webpage will freeze up, and JS code will stop running. The page will stop transmitting data to the other client, and video will freeze on the current page even though you can still scroll the page. I get this 1/3 of the time and it usually happens a few seconds into the call. 
Seems like this is very similar to what Jay Charles has reported. Let me know if you can reproduce on your end.

[Jay Charles]

One thing I failed to mention about my tests... in all cases audio continues to play while the video is frozen and the browser tab is unresponsive.

[Mike Block]

I too have experienced the audio continuing, and the video stream being delivered to the remote peer after the lockup. That said, I've been testing with iOS 11.1.1 and the few tests I have done have played okay. I was concerned it was visual clipping or audio feedback loops, but have tested both. Echo cancellation seems better now too (was getting wild feedback loops before). That said, there still seems to be instability.

[Jay Charles]

FWIW, I'm still having issues on 11.3 beta 6.

P2P works fine, but when receiving streams from Wowza Streaming Engine video in safari/ios freezes frequently. When this happens the browser tab remains responsive and audio continues to play... until I attempt to close the peerconnection at which point the tab goes unresponsive (no response from remote debugger either) and the only way out is to close the browser tab.

(In reply to Jay Charles from comment #21)
> FWIW, I'm still having issues on 11.3 beta 6.
> 
> P2P works fine, but when receiving streams from Wowza Streaming Engine video
> in safari/ios freezes frequently. When this happens the browser tab remains
> responsive and audio continues to play... until I attempt to close the
> peerconnection at which point the tab goes unresponsive (no response from
> remote debugger either) and the only way out is to close the browser tab.

We have the same issue. I'm receiving videos from a webrtc streaming server. After using the app for a while (circling through different video streams), the app is either freezing or crashing after calling the stop function of a peerconnection. Removing the close call, the app runs more stable. We are using iOS 11.2.5.

[Ben]

In iOS 11.3 stable Safari still freezes.

[youenn fablet]

Is there a way I can reproduce the freezing issue?
Would it be possible to get a crash log from one of you guys?

[Jay Charles]

(In reply to youenn fablet from comment #24)
> Is there a way I can reproduce the freezing issue?
> Would it be possible to get a crash log from one of you guys?

In my case no crash logs are generated (either that or I'm looking in the wrong place.

If you'd like to coordinate time, I can set up a test case and leave a stream running for you to test against for as long as you need. Please contact me offlist and I'll be happy to help

[Peter Vertenten]

I'm also seeing this freezing behavior using the twilio-video javascript libraries. It seems to happen when removing/adding video streams, from changing the camera from back/front etc.

It works better if we just kill the call entirely and reinitiate the call while changing the video streams, which is less than ideal but it seems to prevent the browser from locking up.

[youenn fablet]

(In reply to Peter Vertenten from comment #26)
> I'm also seeing this freezing behavior using the twilio-video javascript
> libraries. It seems to happen when removing/adding video streams, from
> changing the camera from back/front etc.
> 
> It works better if we just kill the call entirely and reinitiate the call
> while changing the video streams, which is less than ideal but it seems to
> prevent the browser from locking up.

Hi Peter,
We are investigating this issue and it would be good if we could validate it with as many WebRTC middleware as we can.
Let me know if you can give access to a repro case with this particular middleware.

[Peter Vertenten]

(In reply to youenn fablet from comment #27)
> (In reply to Peter Vertenten from comment #26)
> > I'm also seeing this freezing behavior using the twilio-video javascript
> > libraries. It seems to happen when removing/adding video streams, from
> > changing the camera from back/front etc.
> > 
> > It works better if we just kill the call entirely and reinitiate the call
> > while changing the video streams, which is less than ideal but it seems to
> > prevent the browser from locking up.
> 
> Hi Peter,
> We are investigating this issue and it would be good if we could validate it
> with as many WebRTC middleware as we can.
> Let me know if you can give access to a repro case with this particular
> middleware.

Private message me and we can schedule a time for me to setup a link for you as our video invites are ephemeral, otherwise I can spend some time making a more permanent test. Also of note, I can reproduce the issue on Safari desktop, if that makes things easier for testing on your end.

[youenn fablet]

(In reply to Peter Vertenten from comment #28)
> (In reply to youenn fablet from comment #27)
> > (In reply to Peter Vertenten from comment #26)
> > > I'm also seeing this freezing behavior using the twilio-video javascript
> > > libraries. It seems to happen when removing/adding video streams, from
> > > changing the camera from back/front etc.
> > > 
> > > It works better if we just kill the call entirely and reinitiate the call
> > > while changing the video streams, which is less than ideal but it seems to
> > > prevent the browser from locking up.
> > 
> > Hi Peter,
> > We are investigating this issue and it would be good if we could validate it
> > with as many WebRTC middleware as we can.
> > Let me know if you can give access to a repro case with this particular
> > middleware.
> 
> Private message me and we can schedule a time for me to setup a link for you
> as our video invites are ephemeral, otherwise I can spend some time making a
> more permanent test. Also of note, I can reproduce the issue on Safari
> desktop, if that makes things easier for testing on your end.

If you reproduce it on Safari desktop, that is probably another issue.
Can you reproduce it in STP52?

[Peter Vertenten]

I'm not sure what you mean by STP52. I can reproduce this on IOS 11.3 and Desktop Safari 11.1 (13605.1.25.1).


There does appear to be 2 separate issues.

Our setup may be not the most common, but here is what we do.

We programmatically ensure there is only ever 1 active video stream. We do allow multiple audio streams at once. So when the user on 1 side enables video they see their local camera and they publish their video stream, other users on the same call would stop their own video streams and unpublish it. 

So our calls start our like this.

Safari Peer Video ----->  Chrome Peer (one way)
Safari Peer Audio <---->  Chrome Peer Audio

The first issue we have happens when we try to swap who is the video holder. This is the issue that brought us here.

So moving to.

Chrome Peer Video ----->  Safari Peer (one way)
Safari Peer Audio <---->  Chrome Peer Audio

At this point one of a few things will happen.

1) It works! Yeah!
2) It works for a few seconds, then the video stream freezes in safari. Both on IOS and desktop safari.
3) The stream freezes and a notification that there is a problem occurs. Both on IOS and desktop safari.
4) The page reloads. Both on IOS and desktop.


The second issue.

The call starts out the same.

Safari Peer Video ----->  Chrome Peer
Safari Peer Audio <---->  Chrome Peer Audio

Then the safari peer disables video, and the call is audio only.

Safari Peer Audio <---->  Chrome Peer Audio (no change)

Then the safari peer re-adds video.
Safari Peer Video ----->  Chrome Peer
Safari Peer Audio <---->  Chrome Peer Audio (no change)

At this point again a few things happen.

1) It works!
2) The local video stream is frozen and the peer is getting no video. (IOS and desktop safari.)

If I manually call play on the video tag nothing happens, both autoplay and playsinline are already on the tag.

If I disable video again and reenable it, I don't even see a local video frame anymore.

[[e-mail]]

I encounter the same freezing issues.
When going peer to peer with the appr.tc example everything seams to work well. However, my current setup is Chrome <-> Wowza <-> iOS Safari. The video freezes in the safari webbrowser after some time. On that iOS device I can still hear the audio, but the video is frozen. On the Chrome side I can both see and hear the other side. So in Wowza, all streams are ok.

If I run the same setup with Chrome <-> Wowza <-> Desktop Safari or Chrome <-> Wowza <-> Chrome (Desktop or Android) I don't see this issue. So it looks like a iOS Safari issue.

It also appears to trigger this freeze bug more easily when there is a lot of movement in the video, eg when the resolution changes due to limited bandwidth or maybe other h264 parameters. 

I tried this with both UDP and TCP and give the same results.

I also tried different css styles in the hope to have it rendered differently as described here: https://bugs.webkit.org/show_bug.cgi?id=176439

I am using h264 with profile id 42e01f, to be the same as appr.tc and Tried to run the chrome side on Windows, Ubuntu and Mac, all with the same results.

[youenn fablet]

*** Bug 184648 has been marked as a duplicate of this bug. ***

I have another case that causes remote video from https://appr.tc to freeze.

After a call is established, if you pull down the notification center, wait a few seconds, and then swipe it back up, the remote video freezes (however the audio can still be heard). I use Safari on iOS 11.2, and am fairly confident that this applies to all released versions of iOS 11.x.

One workaround I found is that after the remote video freezes, if you call pause() and then play() on the video element, the video would resume. Except, there's no way to detect if the notification center has been pulled down. This StackOverflow question has received no answers: https://stackoverflow.com/questions/49962055/ios-safari-know-when-notification-center-is-displayed-and-dismissed

Another interesting thing to note, is that the video stream is still being sent to the browser during this time. So if you capture the video frame using canvas context's drawImage(), you'll get the up to date image. But also means you can't detect this problem by checking if video frames from two points in time are identical...

Any word on when this can be fixed or suggestions on other workarounds would be appreciated.

[elbecita]

Hi, I am seeing similar behavior. My scenario is:
- Screensharing from Chrome (the presenter is using Chrome).
- The stream goes to Kurento server.
- A viewer starts viewing from Safari (Desktop usually, but I see same behavior testing from iPhone).
So it's: Chrome > Kurento server > Safari (not P2P)


The viewer gets to see the stream, webRTC connection seems ok, but after a while, the browser reloads and I can see a message saying an error happened and that is why the browser reloaded.

I've been trying to debug this, and I see that every time the browser reloads there is a memory issue (I saw this in the Mac Console, in the system.log):
malloc mach_vm_map(size=blabla_some_size) failed (error code=3)
error: can't allocate region

I see this error every single time that a reloads in the browser happens. Sometimes the viewer gets to view, the ICE state of the connection goes to complete and everything goes normal, but most of the time the reload happens.

Is this something others are experiencing? I am a bit blocked now, I don't know where to look at. I was trying to get webrtc stats, check the nackCount or the pliCount, but I haven't detected a pattern. Ideally I would love to at least detect when this is going to happen and stop the viewer and retry, before the reload happens.

[Mike Block]

I'm on iOS 11.4 and I too am experiencing an increase in reloads of the tab due to failures. I've tried debugging through the Safari desktop debugger, but as soon as the error occurs and the tab is reloaded, the console is shut down. If there is a way to report error dumps for a tab that has been removed, I can very easily repeat the errors.

I'd thought we had some stability, but it seems to be even worse.

I'm using Twilio Programmable Video. Even basic tests are causing tabs to die.

My feeling is that the libwebrtc framework consumes more resources (CPU, memory) than permitted in iOS for a tab in Safari mobile. I think when the full framework loads (not sure which point) it simply shuts it down.

[Mike Block]

Created attachment 339533 [details]
Crash Report from Console

[elbecita]

Hey! How did you manage to get that crash report?
I was right now trying to catch a memory leak following the steps WebKit describes in here: https://webkit.org/leak-hunting/

Was that what you did?

[Mike Block]

This was from an iPad mini (ME277C/A) running iOS 11.4. I am using the Twilio Programmable Video SDK barebones (no other resources loaded). I launch the same page in Google Chrome (66.0.3359.139) on a separate machine (windows 10) and then repeatedly reload the page on the Chrome tab. This eventually crashes the Safari Mobile tab on the iPad mini.

You can retrieve crash reports this way:

1. Connect the device to your MacBook
2. Launch Console on your MacBook and under devices, find your iOS device
3. Add a filter for ReportCrash, hit enter to accept the term, and then change filter to be against the Process value
4. Monitor during crash event

[elbecita]

Alright, I run my screenshare sessions using a webKit debug build.
- Presenting a stream that goes from Chrome browser to a Kurento server (webRTC connection)
- Viewing the stream from Safari Desktop on macOS High Sierra 10.13.3 (webRTC connection between the Kurento server and the Safari browser).

The connection is established and the viewer views the stream, but eventually the browser reloads. I attached the crash report. Also when the crash happens, I can see this in the MAC Console app:

ASSERTION FAILED: clipRectsContext.rootLayer == m_clipRectsCache->m_clipRectsRoot[clipRectsType]
./rendering/RenderLayer.cpp(5367) : Ref<WebCore::ClipRects> WebCore::RenderLayer::updateClipRects(const WebCore::RenderLayer::ClipRectsContext &)

And a WTFCrash

At the same exact moment I see this in system.log:
(com.apple.WebKit.WebContent.80542A86-22FD-451C-B3F4-22CCA469D2B9[38299]): Service exited due to signal: Segmentation fault: 11 sent by exc handler[0]

^ This is in the crash report.

[elbecita]

Created attachment 339552 [details]
Safari desktop crash report

Explained in prev comment.

[Ben]

In iOS 11.4 Safari still freezes.

[youenn fablet]

(In reply to elbecita from comment #40)
> Created attachment 339552 [details]
> Safari desktop crash report
> 
> Explained in prev comment.

Elbecita, thanks for the crash report.
It seems to be a different issue happening in the rendering engine.
I believe you should file another bug and upload the crash report there.

[youenn fablet]

(In reply to Ben from comment #41)
> In iOS 11.4 Safari still freezes.

What device are you testing on?

[Jay Charles]

Are there any details available about the cause of this issue that we might be able to provide to our server vendor? My tests suggest that this issue does not affect some media server implementations (Kurento, MediaSoup), and I'm hoping that armed with some additional details our server vendor may be able to find workarounds for us.

[Ben]

> What device are you testing on?

@youenn, iPhone 6 running iOS 11.4

[Simon Fraser (smfr)]

T(In reply to elbecita from comment #39)
> And a WTFCrash
> 
> At the same exact moment I see this in system.log:
> (com.apple.WebKit.WebContent.80542A86-22FD-451C-B3F4-22CCA469D2B9[38299]):
> Service exited due to signal: Segmentation fault: 11 sent by exc handler[0]
> 
> ^ This is in the crash report.

That crash is a debug assertion that is unrelated to WebRTC (and harmless for users).

[youenn fablet]

It would be nice if some of you that hit that issue could test against latest iOS12 beta.