Bug 195124 – Call was negotiated with H264 Base Profile 42e01f but encoded in High Profile

• Status: REOPENED
• Component: Media

[Long Cheng]

Problem:

  In a WebRTC call, the local SDP from Safari claims supporting video codec H264 with profile-level-id 42e01f which is the Base Profile. The remote SDP also supports video codec H264 with profile-level-id 42c01f which is also the Base Profile. In this case, both sides should agree on H264 Base Profile. However, we observed the Safari browser is sending High Profile stream instead.

Expected:

  H264 Base Profile stream is important for many video systems. Please fix this issue by sending Base Profile stream according to the negotiated profile-level-id. Or please let us know how to configure Safari browser to send Base Profile stream only.

[Radar WebKit Bug Importer]

<rdar://problem/48453085>

[Long Cheng]

Hello, 

Do you need any further information? Did you successfully reproduce this issue? When should I anticipate a fix on this issue? Please let me know if there is any progress. Thank you.

Thanks,
Long Cheng

[Long Cheng]

Hello, 

Do you need any further information? 

Thanks,
Long Cheng

[youenn fablet]

Hi Long Chen,
Thanks for the report and sorry for the delay.
I think we have sufficient information now.
Please feel free to share additional information here or privately like repro cases or services/devices that are affected.

[Long Cheng]

Thanks for your reply. 

May I ask for the status of this bug? Is it in progress of a fix? 

Thanks,
Long Cheng

[youenn fablet]

This is under investigation.
Any information that may help us prioritizing this would be useful.

[Long Cheng]

Hello Youenn,

Thanks for your reply. Please investigate the issue and keep us updated. Appreciate your help. 

We are providing video conference services here in Compunetix. One major job we do is to mix all media streams from a variety of devices (including legacy video room terminals). We are excited to see a great progress of WebRTC implementation in WebKit in the recent two years. There are many customers prefer to use Safari on iOS and Mac. However, due to this issue, these customers have to use Chrome instead.

Here is a test conference you may use to reproduce the issue as well as verify the fix.
https://evergreen.compunetix.com/cx-full/conference/join?autoJoinPasscode=35678554

Thanks,
Long Cheng

[youenn fablet]

Safari now supports VP8, H264 base profile and H264 high profile.
The default codec is H264 high profile, which might be your issue here.

Selection of the base profile can be forced by munging SDP to remove VP8 and H264 high profile.
Or it should be selected if the other party only supports H264 base profile.

I put up https://youennf.github.io/webrtc-tests/src/content/peerconnection/pc-h264-baseline/ which shows that baseline can be forced by munging SDP.

Can you verify on your side whether munging SDP would fix your issue?

I am marking this issue as Invalid for now.
Please reopen the bug if needed.

[Long Cheng]

Created attachment 367321 [details]
SDP Offer from Safari 12.1

[Long Cheng]

Created attachment 367322 [details]
SDP Answer from our product

[Long Cheng]

Comment from our team member:

Hello Youenn,

We are currently already doing what you suggest. If you examine the offer and answer images we have attached you will see that each side supports H.264 base profile and neither supports H.264 high profile. What we have observed is that even with this SDP exchange we are receiving an H.264 RTP stream that contains a profile-level-id set to high profile in the H.264 header and features only available in high profile are present in the video stream. This should not be allowed based on the SDP exchange. Can you please examine this issue from this perspective and let us know if you need anything from us to expedite this? This was tested on a brand new and fully updated Mac Book Pro running Mojave and the latest version of Safari.

[Long Cheng]

Hello Youenn,

Have you successfully verified the H.264 stream headers and features? We believe with the SDP forcing base profile, the H.264 RTP stream still contains a profile-level-id set to high profile in the H.264 header and features only available in high profile are present in the video stream. 

Thanks,
Long Cheng

[youenn fablet]

When using VTB, it seems to work fine with the hardware H264 encoder on my MacBookPro.
This does not work when not using the hardware H264 encoder.
When using VCP, this does not work properly, no matter whether hardware or not.
kVTCompressionPropertyKey_ProfileLevel does not seem to be respected.

[youenn fablet]

Created attachment 371293 [details]
Patch

[EWS Watchlist]

Comment on attachment 371293 [details]
Patch

Attachment 371293 [details] did not pass win-ews (win):
Output: https://webkit-queues.webkit.org/results/12374570

New failing tests:
fast/block/float/float-with-anonymous-previous-sibling.html

[EWS Watchlist]

Created attachment 371302 [details]
Archive of layout-test-results from ews215 for win-future

The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews215  Port: win-future  Platform: CYGWIN_NT-10.0-17763-3.0.5-338.x86_64-x86_64-64bit

[youenn fablet]

Created attachment 371305 [details]
Patch

[youenn fablet]

Created attachment 371306 [details]
Patch

[youenn fablet]

Created attachment 371307 [details]
Patch

[youenn fablet]

Created attachment 371345 [details]
Patch

[EWS Watchlist]

Comment on attachment 371345 [details]
Patch

Attachment 371345 [details] did not pass mac-wk2-ews (mac-wk2):
Output: https://webkit-queues.webkit.org/results/12378704

Number of test failures exceeded the failure limit.

[EWS Watchlist]

Created attachment 371358 [details]
Archive of layout-test-results from ews107 for mac-highsierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews107  Port: mac-highsierra-wk2  Platform: Mac OS X 10.13.6

[youenn fablet]

Created attachment 371365 [details]
Patch

[youenn fablet]

Created attachment 371408 [details]
Patch

[Eric Carlson]

Comment on attachment 371408 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=371408&action=review

> Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/VideoProcessingSoftLink.h:37
>  #define ENABLE_VCP_ENCODER 0

Nit: not new, but there is an extra space before the "&&" in the line above this

> Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/VideoProcessingSoftLink.h:39
>  #elif (defined(TARGET_OS_IPHONE)  && TARGET_OS_IPHONE)

Nit: Ditto.

> Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/VideoProcessingSoftLink.h:44
> +//#define ENABLE_VCP_ENCODER __MAC_OS_X_VERSION_MIN_REQUIRED >= 101300
> +//#define ENABLE_VCP_VTB_ENCODER __MAC_OS_X_VERSION_MIN_REQUIRED >= 101500

Oops!

> Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/RTCVideoEncoderH264.mm:321
> +  VTCompressionSessionRef _compressionSession;

Nit: something like "_vtCompressionSession" would be clearer.

> Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/RTCVideoEncoderH264.mm:507
> +    status = 1;

Is it worth logging an error here to make it clear what failed?

> Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/RTCVideoEncoderH264.mm:876
> +      RTC_LOG(LS_ERROR) << "VTSessionSetProperty failed to set: " << kVTCompressionPropertyKey_DataRateLimits

Nit: kVTCompressionPropertyKey_DataRateLimits is a constant, why not hard code it in the string?

> Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/helpers.cc:38
> +void SetVTSessionProperty(VTCompressionSessionRef session, VCPCompressionSessionRef vcpSession,

Nit: something like "vtSession" would be clearer.

> Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/helpers.cc:48
> +  if (session)
> +    status = VTSessionSetProperty(session, key, cfNum);
> +#if ENABLE_VCP_ENCODER
> +  else
> +    status = webrtc::VCPCompressionSessionSetProperty(vcpSession, key, cfNum);

If we are going to assume vcpSession is valid when session is NULL, maybe ASSERT(session || vcpSession)?

> Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/helpers.cc:70
> +  if (session)
> +    status = VTSessionSetProperty(session, key, cfNum);
> +#if ENABLE_VCP_ENCODER
> +  else
> +    status = webrtc::VCPCompressionSessionSetProperty(vcpSession, key, cfNum);

Ditto.

> Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/helpers.cc:88
> +  if (session)
> +    status = VTSessionSetProperty(session, key, cf_bool);
> +#if ENABLE_VCP_ENCODER
> +  else
> +    status = webrtc::VCPCompressionSessionSetProperty(vcpSession, key, cf_bool);

Ditto.

> Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/helpers.cc:106
> +  if (session)
> +    status = VTSessionSetProperty(session, key, value);
> +#if ENABLE_VCP_ENCODER
> +  else
> +    status = webrtc::VCPCompressionSessionSetProperty(vcpSession, key, value);

Ditto

[youenn fablet]

Thanks for the review.

(In reply to Eric Carlson from comment #25)
> Comment on attachment 371408 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=371408&action=review
> 
> > Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/VideoProcessingSoftLink.h:37
> >  #define ENABLE_VCP_ENCODER 0
> 
> Nit: not new, but there is an extra space before the "&&" in the line above
> this

OK

> > Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/VideoProcessingSoftLink.h:39
> >  #elif (defined(TARGET_OS_IPHONE)  && TARGET_OS_IPHONE)
> 
> Nit: Ditto.

OK

> > Source/ThirdParty/libwebrtc/Source/webrtc/sdk/WebKit/VideoProcessingSoftLink.h:44
> > +//#define ENABLE_VCP_ENCODER __MAC_OS_X_VERSION_MIN_REQUIRED >= 101300
> > +//#define ENABLE_VCP_VTB_ENCODER __MAC_OS_X_VERSION_MIN_REQUIRED >= 101500
> 
> Oops!
> 

Thanks!
 Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/RTCVideoEncoderH264.mm:321
> > +  VTCompressionSessionRef _compressionSession;
> 
> Nit: something like "_vtCompressionSession" would be clearer.

OK

> > Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/RTCVideoEncoderH264.mm:507
> > +    status = 1;
> 
> Is it worth logging an error here to make it clear what failed?

It is logged below.
 
> > Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/RTCVideoEncoderH264.mm:876
> > +      RTC_LOG(LS_ERROR) << "VTSessionSetProperty failed to set: " << kVTCompressionPropertyKey_DataRateLimits
> 
> Nit: kVTCompressionPropertyKey_DataRateLimits is a constant, why not hard
> code it in the string?

I prefer keeping it this way to limit the changes compared to upstream.
Admittedly, we have already branched quite a bit...

> > Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/helpers.cc:38
> > +void SetVTSessionProperty(VTCompressionSessionRef session, VCPCompressionSessionRef vcpSession,
> 
> Nit: something like "vtSession" would be clearer.

OK

[youenn fablet]

Created attachment 371533 [details]
Patch for landing

[WebKit Commit Bot]

Comment on attachment 371533 [details]
Patch for landing

Clearing flags on attachment: 371533

Committed r246263: <https://trac.webkit.org/changeset/246263>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Daniele]

Created attachment 375695 [details]
global end-to-end sip trace

How to read the trace:

(WSC-SE-01) is the webrtc gateway. The SDP contained in the first INVITE on the left comes directly from the Iphone 6s.

10.132.4.50 is the Cisco Jabber SIP client

SBC-WSC-MILANO is the Oracle SIP SBC

Cucm-CC is the Cisco Call Manager, SIP PBX. Jabber is registered here.

WSC-ME is the webrtc gateway Media Engine...media flow through here.

Calling: [e-mail] // webrtc caller
Called: 10090 // Jabber extension number

[Daniele]

Hello,

even though this bug is FIXED, I am experiencing the exact same problem whenever I try to set up a webrtc call from latest ios/Safari on iPhone 6s.

Scenario:

Iphone 6s --- Webrtc gateway --- Oracle SBC --- Cisco Call Manager --- Cisco Jabber

Problem: Cisco Jabber does not show remote video (audio is fine).

Same scenario with Android/Chrome works fine. It works also on Iphone/Safari with 12.1.4 release.


In a nutshell, safari ios is sending to Jabber H264 high profile even though the offer/answer negotiation agrees on h264 base profile on both parts.

Reasoning:

in the attached message-flow.html:

SDP INVITE from ios/Safari contains #2 h264 m-lines (high and base profile):

a=rtpmap:96 H264/90000
a=fmtp:96 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640c1f <--- high-profile

a=rtpmap:98 H264/90000
a=fmtp:98 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f <--- base profile

final 200 OK sent back to ios/Safari contains just one h264 m line:

a=rtpmap:126 H264/90000
a=fmtp:126 profile-level-id=42E01F;packetization-mode=1;max-fs=3601;max-rcmd-nalu-size=32000;level-asymmetry-allowed=1

This is ACKed by Safari so they both agree on using h264 base profile.

But if you open the attached ios-call-ok-old-iphone.pcapng, isolate the h264 track with this filter:

(ip.src==10.133.8.53 && udp.srcport==20530 && ip.dst==10.132.4.59 && udp.dstport==32220 && rtp.ssrc==0x29b6f70a) and look at the very first h264 packet, you'll see: 

0110 0100 = Profile_idc: High profile (100)

[Daniele]

Created attachment 375696 [details]
complete wireshark trace taken on the Jabber laptop

it shows the rtp and sip dialog between my sip client and cucm/sbc

[Daniele]

Forgot to add iphone user agent string:

Mozilla/5.0 (iPhone; CPU iPhone OS 12_4 like Mac OS X)
AppleWebKit/605.1.15 /KHTML, like Gecko) Version/12.1.2 Mobile/15E148
Safari/604.1

[Daniele]

Hi,

any comment on this? Do you need additional information?

Thank you

[youenn fablet]

> any comment on this? Do you need additional information?

Which OS are you testing on?
Have you tried on the latest iOS beta?

[Daniele]

> Which OS are you testing on?

Mozilla/5.0 (iPhone; CPU iPhone OS 12_4 like Mac OS X)
AppleWebKit/605.1.15 /KHTML, like Gecko) Version/12.1.2 Mobile/15E148
Safari/604.1

> Have you tried on the latest iOS beta?

Not yet. Doing that asap.

[Daniele]

Hi Youenn,

with ios 13 beta 7 it is finally working fine. I can see h264 base profile sent by safari!

User Agent:

Mozilla/5.0 (iPhone; CPU iPhone OS 13_0 like Mac OS X)
AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0 Mobile/15E148
Safari/604.1

Thanks

[youenn fablet]

*** Bug 211862 has been marked as a duplicate of this bug. ***

[Long Cheng]

Hello Youenn,

We are seeing exact same problem on iPad Pro 11-inch 2nd generation, OS 14.7, Can you help me check on it, please?

user agent says: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15

Thanks,
Long Cheng

[Gaowei Nian]

Hello, I am experiencing the exact same problem on macos10.15.7 safari. I think this problem is related to the system version, not the browser version, I reproduced it on two Safari devices with different versions. their system versions are 10.15.7.

[Gaowei Nian]

Created attachment 459155 [details]
Specified profile-level-id, but no working.

this is safari webrtc log.

[Gaowei Nian]

Call was negotiated with H264 Constrained Baseline Profile 42e01f but encoded in Baseline Profile

[Gaowei Nian]

And Baseline cannot decoded by Constrained Baseline on M1 mac's safari. chrome is work fine.

Has any more work been done on this?