Bug 313426 – Leak due to retain cycle between LibWebRTCRtp{Sender,Receiver}TransformBackend and libwebrtc sender/receiver after RTCPeerConnection closes

• Component: WebRTC
• Status: RESOLVED FIXED

[David Kilzer (:ddkilzer)]

`WebCore::LibWebRTCRtpSenderTransformBackend` and `WebCore::LibWebRTCRtpReceiverTransformBackend` leak due to retain cycles whenever a page uses an `RTCRtpScriptTransform` or `RTCRtpSFrameTransform` on an `RTCRtpSender` or `RTCRtpReceiver`. The cycle persists after `RTCPeerConnection.close()` and after JavaScript has dropped all references to the sender/receiver, so the leak is permanent for the remaining lifetime of the WebContent process.

Each cycle has this structure (shown by `leaks --groupByType` on a memgraph captured from a WebContent process that ran `LayoutTests/http/wpt/webrtc/` tests):

```
541 (5.61M) ROOT CYCLE: webrtc::RefCountedObject<webrtc::RtpSenderProxyWithInternal<webrtc::RtpSenderInternal>>
  528 (5.61M) ROOT CYCLE: webrtc::RefCountedObject<webrtc::AudioRtpSender>
    26 (1.83K) ROOT CYCLE: WebCore::LibWebRTCRtpSenderTransformBackend
      CYCLE BACK TO webrtc::RefCountedObject<webrtc::RtpSenderProxyWithInternal<webrtc::RtpSenderInternal>>
    13 (1.22K) ROOT CYCLE: webrtc::RefCountedObject<webrtc::DtmfSender>
    13 (624 bytes) ROOT CYCLE: webrtc::RefCountedObject<webrtc::DtmfSenderProxyWithInternal<webrtc::DtmfSenderInterface>>
```

The receiver side has an analogous cycle involving `LibWebRTCRtpReceiverTransformBackend`, `webrtc::AudioRtpReceiver`/`webrtc::VideoRtpReceiver`, and `webrtc::RtpReceiverProxyWithInternal`.

Root cause:

1. `LibWebRTCRtpSenderTransformBackend` stores the libwebrtc sender as `const Ref<webrtc::RtpSenderInterface> m_rtcSender;` (strong ref, from backend -> sender).

2. On the first call to `LibWebRTCRtpSenderTransformBackend::setTransformableFrameCallback()`, the backend registers itself with the sender via `m_rtcSender->SetFrameTransformer(scoped_refptr<FrameTransformerInterface>(this))`. libwebrtc stores this in `RtpSenderBase::frame_transformer_`, a `scoped_refptr<FrameTransformerInterface>` (strong ref, from sender -> backend).

3. Both sides hold strong refs, so neither drops first. The cycle keeps both objects alive indefinitely.

`LibWebRTCRtpReceiverTransformBackend` has the same two-way relationship with its `webrtc::RtpReceiverInterface` via `SetFrameTransformer()` in `setTransformableFrameCallback()`.

Reproduction:

Run any WebRTC layout test that attaches a transform to a sender or receiver (for example `http/wpt/webrtc/unset-transform.html`, `http/wpt/webrtc/audiovideo-script-transform.html`, `webrtc/sframe-keys.html`, or `webrtc/video-sframe.html`) with `run-webkit-tests --debug --leaks`. `leaks --groupByType` on the resulting WebContent memgraph reports multiple `ROOT CYCLE` entries naming `WebCore::LibWebRTCRtpSenderTransformBackend` or `WebCore::LibWebRTCRtpReceiverTransformBackend`.

The cycle has existed since WebRTC transform infrastructure was introduced in Bug 218750, 231538@main (2020-11-13).

[Radar WebKit Bug Importer]

<rdar://problem/175674797>

[David Kilzer (:ddkilzer)]

Pull request: https://github.com/WebKit/WebKit/pull/63695

[EWS]

Committed 312177@main (0f476dfaa434): <https://commits.webkit.org/312177@main>

Reviewed commits have been landed. Closing PR #63695 and removing active labels.