Bug 257803 – Terrible video quality when using TransformStream with Simulcast

• Component: WebRTC
• Status: RESOLVED FIXED

Created attachment 466619 [details]
The crazy jitter value coming from the Peer Connection Stats

1. Have a WebRTC app which publishes/receives simulcast streams (through a SFU media server).

2. Have a TransformStream in a worker which just passes unmodified the stream (RTCRtpScriptTransform):

    const transformStream = new TransformStream({
      transform: async (chunk, controller) => {
        controller.enqueue(chunk); //Unmodified chunk
      }
    })

3. Observe the received video quality is pixelated and terrible.

4. Observe only the L layer 320x180 remains with a very high bitrate and suddenly jitter value in PeerConnection stats becomes HUGE:
   7737 seconds - 2 hours which is practically an impossible value.

   Attaching screenshots.

If simulcast is not used, then video quality is fine, but the moment simulcast and a TransformStream is used, the quality goes bad and jitter of 2 hours appears.

Jitsi (a videoconference provider) also experiences this issue on Safari -> https://community.jitsi.org/t/e2ee-on-ios-safari-with-enableencodedtransformsupport/114517/9

So it seems a widespread issue under Safari.

Under Chrome on MacOS everything is fine (with the same SFU media server, simulcast and Insertable Streams). So that seems to confirm it is a Safari/Webkit issue.

[Radar WebKit Bug Importer]

<rdar://problem/110395571>

[youenn fablet]

@pminchev, do you have a test site where the issue can be reproduced?
What WebRTC logs do you get on the sending side?

Created attachment 466832 [details]
WebRTC logs from Safari

(In reply to youenn fablet from comment #2)
> @pminchev, do you have a test site where the issue can be reproduced?
> What WebRTC logs do you get on the sending side?

Yes, https://demos.digitalsamba.com/safari_simulcast.html
Just enter some random name and click join.

You need at least 2 people from Safari in the room, since the local video preview doesn't go through the media servers, only the subscribed videos. Of course you can just open 2 browser tabs if you don't want to ask a second person to join. A heavy pixelation is soon observed for the consumed streams. It fixes itself for a few seconds, then it appears again - kind of random. Also as I mentioned in the original description - the M simulcast layer often disappears and L layer with a > 7000 seconds jitter appears.

Note this demo uses end-to-end encryption, but even without it, we experience the same issues - the moment TransformStream is used with simulcast.

I am also attaching some WebRTC logs, hope they are helpful.

@youenn fablet - Did you manage to reproduce? Do you need something more as a report? It has been 3 months, and we still cannot support E2EE under Safari because of the issue.

Thanks in advance!

[youenn fablet]

Thanks for the ping.
I had a try at https://demos.digitalsamba.com/safari_simulcast.html and was able to reproduce the bad quality.

Via web inspector, I set the transformers to null and it did not change the quality overall.
I saw repeated packet loss though on sending and receiving side, I need to retry on a different network with ethernet probably.

(In reply to youenn fablet from comment #6)
> Thanks for the ping.
> I had a try at https://demos.digitalsamba.com/safari_simulcast.html and was
> able to reproduce the bad quality.
> 
> Via web inspector, I set the transformers to null and it did not change the
> quality overall.
> I saw repeated packet loss though on sending and receiving side, I need to
> retry on a different network with ethernet probably.

Thanks for checking - you can compare to Firefox (quality there is good). For Firefox & Safari we use the same RTCRtpScriptTransform API, but under Safari receiving quality is bad.

[youenn fablet]

OK, I was able to reproduce and there is indeed an issue as the transform triggers mixing the simulcast streams in one ssrc. I'll fix this shortly

[youenn fablet]

Pull request: https://github.com/WebKit/WebKit/pull/18681

[EWS]

Committed 268912@main (1fc26a73bcc7): <https://commits.webkit.org/268912@main>

Reviewed commits have been landed. Closing PR #18681 and removing active labels.

(In reply to youenn fablet from comment #9)
> Pull request: https://github.com/WebKit/WebKit/pull/18681

Thank you very much for the PR! I see it is merged to webkit master.

Do you know more about when it would become part of Safari?

Maybe you know more how does Safari patches and release cycle work when there are webkit fixes being done.