Bug 218637 - Textures Fail to Render in WebGL from HLS Stream on iPhone 12 [iOS 14.2]
Summary: Textures Fail to Render in WebGL from HLS Stream on iPhone 12 [iOS 14.2]
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebGL (show other bugs)
Version: Other
Hardware: iPhone / iPad Other
: P2 Critical
Assignee: Kimmo Kinnunen
URL:
Keywords: InRadar
Depends on: 215908
Blocks: 216915 218970 218971 219026
  Show dependency treegraph
 
Reported: 2020-11-05 15:29 PST by Tony Tomarchio
Modified: 2022-02-17 22:52 PST (History)
22 users (show)

See Also:


Attachments
Image comparing HLS WebHL playback on iPhone12 vs iPhone11 in iOS 14.2 (2.07 MB, image/png)
2020-11-05 15:29 PST, Tony Tomarchio
no flags Details
Patch (22.73 KB, patch)
2020-11-12 05:44 PST, Kimmo Kinnunen
no flags Details | Formatted Diff | Diff
Patch (8.76 KB, patch)
2020-11-16 00:21 PST, Kimmo Kinnunen
ews-feeder: commit-queue-
Details | Formatted Diff | Diff
Patch (8.76 KB, patch)
2020-11-16 00:39 PST, Kimmo Kinnunen
no flags Details | Formatted Diff | Diff
Patch (3.89 KB, patch)
2021-02-24 04:59 PST, Kimmo Kinnunen
no flags Details | Formatted Diff | Diff
Patch (3.88 KB, patch)
2021-02-25 07:39 PST, Kimmo Kinnunen
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Tony Tomarchio 2020-11-05 15:29:35 PST
Created attachment 413357 [details]
Image comparing HLS WebHL playback on iPhone12 vs iPhone11 in iOS 14.2

As a follow-up to https://bugs.webkit.org/show_bug.cgi?id=215908 HLS WebGL playback is still broken on iPhone 12 devices in today's iOS 14.2 release (18B92)

*Examples to Reproduce Issue*

Device: iPhone 12

three.js example: Visit https://8w.8thwall.app/hls-threejs on an iPhone 12 running iOS 14.2 (18B92). The rotating cube has a video texture applied from an HLS stream. The video in the top left corner is using the same HLS stream but rendering on top of the page outside WebGL.

babylon.js example: Visit https://www.babylonjs-playground.com/#9FSDC7#49 on iPhone 12 running iOS 14.2. The plane has a video texture applied from an HLS stream.

Both examples work correctly on:
iPhone 11 (Model MWLC2LL/A) running iOS 14.2 (18B92)
iPhone SE (Model MX9Q2LL/A) running iOS 14.2 (18B92)
Comment 1 Alexey Proskuryakov 2020-11-05 17:45:00 PST
I can reproduce this in iPhone 12 Pro.
Comment 2 Radar WebKit Bug Importer 2020-11-05 17:45:19 PST
<rdar://problem/71102126>
Comment 3 Kimmo Kinnunen 2020-11-11 08:36:27 PST
On iPhone 12 fails with: 
Asked to copy an unsupported pixel format ('&8v0').
when interpreting the result of CVPixelBufferGetPixelFormatType.
The supported ones are '420v' and '420f'. Unsure if the above pixel format is real fourcc format or just misinterpretation of OSType enumeration.
Comment 4 Kimmo Kinnunen 2020-11-11 11:00:11 PST
And to clarify:
Not all WebGL video copies fail. It appears that at least the bunny one fails, but at least one other test case in bug 215908 succeeds
Comment 5 Kenneth Russell 2020-11-11 12:18:38 PST
Kimmo, since you've triaged this already could you officially take it? Unfortunately James and I can't feasibly debug anything that happens only on real iOS hardware.
Comment 6 Kimmo Kinnunen 2020-11-12 05:44:52 PST
Created attachment 413929 [details]
Patch
Comment 7 Kenneth Russell 2020-11-12 11:55:47 PST
Comment on attachment 413929 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=413929&action=review

Very nice and well thought through fix. It looks good to me, especially given you've verified the fix on hardware - two small comments above. However I think it would be best if jer.noble reviewed this too and approved it.

> Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2216
> +        (__bridge NSString *)kCVPixelBufferIOSurfacePropertiesKey : @{ /*empty dictionary*/ },

Just checking - it's okay to specify this property for all video playback?

> Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2363
> +    CVPixelBufferRef pixelBuffer = updateLastPixelBufferForWebGL();

One note - the software upload path into GraphicsContext goes through the call chain paintCurrentFrameInContext -> paintWithVideoOutput -> updateLastImage, which calls updateLastPixelBuffer. However, this doesn't have the same requirements on the pixel buffer format that copyVideoTextureToPlatformTexture does, so that call to updateLastPixelBuffer doesn't need to be updated to updateLastPixelBufferForWebGL.
Comment 8 Kenneth Russell 2020-11-13 10:42:48 PST
Comment on attachment 413929 [details]
Patch

Since other reviewers seem busy, let me set r+ since you've tested this locally and it fixes a major regression. Please set cq? if you'd like me to cq+ it too.
Comment 9 Kimmo Kinnunen 2020-11-16 00:21:53 PST
Created attachment 414198 [details]
Patch
Comment 10 Kimmo Kinnunen 2020-11-16 00:39:53 PST
Created attachment 414199 [details]
Patch
Comment 11 Kimmo Kinnunen 2020-11-16 00:50:58 PST
(In reply to Kenneth Russell from comment #8)
> Comment on attachment 413929 [details]
> Patch
> 
> Since other reviewers seem busy, let me set r+ since you've tested this
> locally and it fixes a major regression. Please set cq? if you'd like me to
> cq+ it too.

Thanks for the review.
Jer explained to me that using PixelBufferTransferSession is a software fallback. I moved the implementation to bug 218971 where we should improve the software fallback, since currently it's not really working (contributing to two regressions in iOS releases).

I uploaded Jer's patch for treating the currently frequent-but-not-working format similar to a format we already support.
Comment 12 EWS 2020-11-16 21:23:46 PST
Committed r269893: <https://trac.webkit.org/changeset/269893>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 414199 [details].
Comment 13 Simon Taylor 2021-02-17 03:48:50 PST
I'm still seeing this behaviour on the latest iOS 14.4 (18D52) on iPhone 12 Mini.

I know Apple doesn't comment on release plans but just wanted to mention again as this is a pretty major regression that has persisted for multiple public releases since the webkit patch landed and this bug was closed.

Kimmo - a couple of practical questions if I may:

(In reply to Kimmo Kinnunen from comment #4)
> And to clarify:
> Not all WebGL video copies fail. It appears that at least the bunny one
> fails, but at least one other test case in bug 215908 succeeds

This is interesting, as it suggests a change in the encoding may allow HLS videos to work in current iOS releases. From looking at the patches it seems the video decoder on the iPhone 12 produces frames with a new internal format, so I'd be surprised if the behaviour was different with different videos, but perhaps there's a resolution threshold where it switches output format? Any further hints appreciated...

Related to Bug 216250 (the performance regression) - I assume even videos that do work on iPhone 12 series are using the software fallback? Performance numbers strongly indicate that's the case, but as this hardware has always shipped with at least iOS 14 it's hard to know for sure.
Comment 14 Simon Taylor 2021-02-18 00:10:56 PST
The bug still present in the iOS 14.5 beta 2 (18E5154f), released a couple of days ago (Feb 16).
Comment 15 Kimmo Kinnunen 2021-02-18 06:22:18 PST
Thanks, I'll inspect this the coming week.
Comment 16 Kimmo Kinnunen 2021-02-24 04:59:25 PST
Reopening to attach new patch.
Comment 17 Kimmo Kinnunen 2021-02-24 04:59:29 PST
Created attachment 421397 [details]
Patch
Comment 18 EWS Watchlist 2021-02-24 05:00:38 PST
Note that there are important steps to take when updating ANGLE. See https://trac.webkit.org/wiki/UpdatingANGLE
Comment 19 Kimmo Kinnunen 2021-02-24 05:07:20 PST
Thanks for the report, sorry it took so long.

Repened this as per the effect of the problem is the same.
Comment 20 Brent Fulgham 2021-02-24 10:38:34 PST
Tracking under
<rdar://problem/74702474>
Comment 21 Kenneth Russell 2021-02-24 12:18:11 PST
Comment on attachment 421397 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=421397&action=review

Looks good. r+

> Source/ThirdParty/ANGLE/src/libANGLE/renderer/gl/eagl/IOSurfaceSurfaceEAGL.mm:321
> +    // However, there caller might supply us non-public pixel format, which makes exhaustive checks

Should this be:
  there caller -> the caller
?

> Source/ThirdParty/ANGLE/src/libANGLE/renderer/gl/eagl/IOSurfaceSurfaceEAGL.mm:322
> +    // problematic.

Can any validation be done at all? What are the consequences of failing this validation? It's important to not expose uninitialized GPU memory, etc.
Comment 22 Kimmo Kinnunen 2021-02-25 07:39:33 PST
Created attachment 421524 [details]
Patch
Comment 23 Kimmo Kinnunen 2021-02-25 07:41:31 PST
(In reply to Kenneth Russell from comment #21)
> Can any validation be done at all? What are the consequences of failing this
> validation? It's important to not expose uninitialized GPU memory, etc.

I'll try to ask what kind of checks we should do, e.g. what is the impact of passing internalFormat, format, type to texImageIOSurface
Comment 24 EWS 2021-02-26 12:44:10 PST
Committed r273582: <https://commits.webkit.org/r273582>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 421524 [details].
Comment 25 Taeheon Kim 2021-02-26 13:18:34 PST
Thanks for the fix.
Since this issue has been pending for a few months, is there any possibility of ensuring/expediting this fix to be included in the next release? We have multiple clients blocked and waiting for this to be resolved so any visibility would be appreciated.
Comment 26 Mateusz Śpiewak 2021-03-02 15:20:12 PST
I'm unable to confirm this issue is resolved. I can reproduce the issue on iPhone 12 iOS 14.5 Beta.

@Kimmo Kinnunen I'm not really sure why this issue has status Resolved Fixed. It is not resolved.
Comment 27 Kimmo Kinnunen 2021-03-02 22:46:06 PST
(In reply to Mateusz Śpiewak from comment #26)
> I'm unable to confirm this issue is resolved. I can reproduce the issue on
> iPhone 12 iOS 14.5 Beta.
> 
> @Kimmo Kinnunen I'm not really sure why this issue has status Resolved
> Fixed. It is not resolved.

The issue is fixed in the WebKit development source repository. This will be released in iOS Safari in a future iOS release. Unfortunately I cannot give a timeline when or in which public beta this can be tested.
Comment 28 Taeheon Kim 2021-04-02 18:02:15 PDT
(In reply to Kimmo Kinnunen from comment #27)
> (In reply to Mateusz Śpiewak from comment #26)
> > I'm unable to confirm this issue is resolved. I can reproduce the issue on
> > iPhone 12 iOS 14.5 Beta.
> > 
> > @Kimmo Kinnunen I'm not really sure why this issue has status Resolved
> > Fixed. It is not resolved.
> 
> The issue is fixed in the WebKit development source repository. This will be
> released in iOS Safari in a future iOS release. Unfortunately I cannot give
> a timeline when or in which public beta this can be tested.

@Kimmo Kinnunen Sorry to bug you again but it's been another month and was wondering if there's any information you could share regarding timelines. The issue itself has been pending for months and we have multiple clients blocked and waiting for this to be resolved so any visibility would be appreciated.
Comment 29 Brent Fulgham 2021-04-06 12:04:11 PDT
(In reply to Taeheon Kim from comment #28)
> (In reply to Kimmo Kinnunen from comment #27)
> > (In reply to Mateusz Śpiewak from comment #26)
> > > I'm unable to confirm this issue is resolved. I can reproduce the issue on
> > > iPhone 12 iOS 14.5 Beta.
> > > 
> > > @Kimmo Kinnunen I'm not really sure why this issue has status Resolved
> > > Fixed. It is not resolved.
> > 
> > The issue is fixed in the WebKit development source repository. This will be
> > released in iOS Safari in a future iOS release. Unfortunately I cannot give
> > a timeline when or in which public beta this can be tested.
> 
> @Kimmo Kinnunen Sorry to bug you again but it's been another month and was
> wondering if there's any information you could share regarding timelines.
> The issue itself has been pending for months and we have multiple clients
> blocked and waiting for this to be resolved so any visibility would be
> appreciated.

Apple does not discuss release schedules or plans for future updates. We will notify you when a public build is available with such a fix.
Comment 30 Hannah Bianchini 2021-04-30 17:32:46 PDT
Hi Kimmo - 

Hannah Bianchini, Executive Producer of Microsoft Mixed Reality Capture Studios.  Do you know if there's a manager on the Safari team that I can escalate this issue to? I'd like to find a way to unblock a number of clients.

Thanks
Hannah@microsoft.com
Comment 31 Brent Fulgham 2021-05-01 12:57:01 PDT
(In reply to Hannah Bianchini from comment #30)
> Hi Kimmo - 
> 
> Hannah Bianchini, Executive Producer of Microsoft Mixed Reality Capture
> Studios.  Do you know if there's a manager on the Safari team that I can
> escalate this issue to? I'd like to find a way to unblock a number of
> clients.
> 
> Thanks
> Hannah@microsoft.com

Hannah, I would recommend you reach out to the Microsoft business relationship contact at Apple. The WebKit Open Source project is not really the right venue for these discussions as it has no ability to decide what bug fixes go into which release, nor the ability to share details about upcoming releases.

Microsoft's existing communications channels with Apple are designed for these kinds of discussions, and will likely get you better information than posting to this bug.
Comment 32 Brent Fulgham 2021-05-01 13:43:45 PDT
I just checked, and I believe the fix is part of iOS 14.6 Beta 2.
Comment 33 Daniel Rossi 2021-05-20 22:09:53 PDT
Ok, I'll try and get an Iphone 12 to confirm but won't be able to install beta on it to test. I think the original regression was affecting Iphone 12. I'll have to let people know. So within another 6 months to get 14.6 seeing 14.5 was just released ?
Comment 34 Brent Fulgham 2021-08-13 11:52:12 PDT
(In reply to Daniel Rossi from comment #33)
> Ok, I'll try and get an Iphone 12 to confirm but won't be able to install
> beta on it to test. I think the original regression was affecting Iphone 12.
> I'll have to let people know. So within another 6 months to get 14.6 seeing
> 14.5 was just released ?

Hopefully you noticed, but iOS 14.6 was released on 5/24, so less than a week after your message.
Comment 35 Daniel Rossi 2022-02-17 22:52:12 PST
This was ages ago. I am seeing first frame HLS texture render issues on IOS 15 still and its random. Page reload will show it. Resizing the canvas on a resize event doesn't help anymore. 

I've been dealing with this bug and finding work arounds since IOS 10 so going to just leave it at that. 

It might be good if webkit had a HLS texture as part of their webgl tests. I found when reporting they don't.