Bug 230617 - gl.texImage2D upload of getUserMedia streams via <video> element fails
Summary: gl.texImage2D upload of getUserMedia streams via <video> element fails
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebGL (show other bugs)
Version: Other
Hardware: iPhone / iPad Other
: P2 Critical
Assignee: Kimmo Kinnunen
URL:
Keywords: InRadar
: 230589 230879 (view as bug list)
Depends on:
Blocks: 232727
  Show dependency treegraph
 
Reported: 2021-09-22 05:59 PDT by connell
Modified: 2022-01-29 21:10 PST (History)
32 users (show)

See Also:


Attachments
Patch (52.27 KB, patch)
2021-09-24 08:39 PDT, Kimmo Kinnunen
ews-feeder: commit-queue-
Details | Formatted Diff | Diff
Patch (14.27 KB, patch)
2022-01-14 04:22 PST, Kimmo Kinnunen
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description connell 2021-09-22 05:59:04 PDT
Uploading a camera stream to a WebGL texture using getUserMedia(...), a <video> element, and gl.texImage2D(...) is failing in Safari on the iOS 15.1 beta 1 release.

It works correctly on the release version of iOS 15.0 and earlier.

I've prepared a minimal test case here:
https://clv.zappar.io/3571850569055456732/1.0.0/

The page should show a live camera feed in the canvas after allowing camera permissions. On iOS 15.1 beta 1 no camera feed appears. The source for this test case is entirely within the page itself should you wish to take a look at how it's implemented.

This is a common flow for web-based augmented reality and image processing applications. We have many sites and customers who rely on this behaviour to work correctly, so this is a major regression and concern for us.

The same issue is also reported in the following bug but I've filed this one with a minimal test case and a more specific title in the hopes that it helps surface this significant regression to the right team :-)
https://bugs.webkit.org/show_bug.cgi?id=230589

Let me know if I can help with any questions!
Comment 1 Radar WebKit Bug Importer 2021-09-22 10:45:26 PDT
<rdar://problem/83407577>
Comment 2 Dean Jackson 2021-09-22 11:37:39 PDT
*** Bug 230589 has been marked as a duplicate of this bug. ***
Comment 3 Brent Fulgham 2021-09-22 11:43:48 PDT
It looks like this regressed in Bug 228821.
Comment 4 Simon Taylor 2021-09-23 02:51:38 PDT
Thanks Brett. I see the commit in that bug references it affecting the CPU codepath only - @kkinnunen is this perhaps another case like in 216259 and 215908 where some change has forced these uploads down the CPU path rather than the GPU one?
Comment 5 Kimmo Kinnunen 2021-09-24 08:39:52 PDT
Created attachment 439150 [details]
Patch
Comment 6 Eric Carlson 2021-09-24 09:03:37 PDT
Comment on attachment 439150 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=439150&action=review

> Source/WebCore/html/HTMLVideoElement.cpp:-318
> -RefPtr<NativeImage> HTMLVideoElement::nativeImageForCurrentTime()
> -{
> -    if (!player())
> -        return nullptr;
> -
> -    return player()->nativeImageForCurrentTime();
> -}
> -

I would rather not remove this as I think MediaPlayer should be an implementation detail, and I'd like to make it private.

> Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2559
> +    if (m_lastImage)
> +        return std::nullopt;

You return null if there *is* an image?
Comment 7 Kenneth Russell 2021-09-24 11:00:19 PDT
Comment on attachment 439150 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=439150&action=review

>> Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2559
>> +        return std::nullopt;
> 
> You return null if there *is* an image?

Looks like a typo compared to MediaPlayerPrivateMediaStreamAVFObjC - maybe the cause of the widespread test failures.
Comment 8 Kenneth Russell 2021-09-24 11:00:28 PDT
Comment on attachment 439150 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=439150&action=review

>> Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2559
>> +        return std::nullopt;
> 
> You return null if there *is* an image?

Looks like a typo compared to MediaPlayerPrivateMediaStreamAVFObjC - maybe the cause of the widespread test failures.
Comment 9 Kenneth Russell 2021-09-24 11:01:32 PDT
The new accelerated code paths added in this patch look very cool though!
Comment 10 connell 2021-09-29 07:39:51 PDT
Just a note to say that we still see this issue on iOS 15.1 Beta 2.
Comment 11 Brent Fulgham 2021-09-29 08:55:59 PDT
(In reply to connell from comment #10)
> Just a note to say that we still see this issue on iOS 15.1 Beta 2.

I believe this will be part of Beta 3.
Comment 12 Kimmo Kinnunen 2021-09-29 22:36:25 PDT
*** Bug 230879 has been marked as a duplicate of this bug. ***
Comment 13 Kimmo Kinnunen 2022-01-13 05:11:59 PST
Regression was fixed by reverting r280963 (Bug 228821) for iOS 15.2 (rdar://83587220)

This bug still persists in trunk. Repurposing this bug to fix in trunk.
Comment 14 Kimmo Kinnunen 2022-01-14 04:22:31 PST
Created attachment 449160 [details]
Patch
Comment 15 youenn fablet 2022-01-14 06:15:19 PST
Comment on attachment 449160 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=449160&action=review

> Source/WebCore/ChangeLog:9
> +        Fix MSE camera to WebGL texture uploads.

s/MSE/MediaStreamTrack/

> Source/WebCore/ChangeLog:12
> +        Turns out MSE does not have these implemented, so currently fall back

It should be easy to add such methods.
We have m_currentVideoSample we can grab for that using a lock.
Comment 16 EWS 2022-01-14 12:42:29 PST
Committed r288025 (246051@main): <https://commits.webkit.org/246051@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 449160 [details].
Comment 17 rigel 2022-01-29 21:10:11 PST
I'm currently seeing this behavior (black screen, no camera feed) on iOS 15.4b1 (iPhone 11) on all 8th Wall and Zappar websites, including the test case listed at the top: https://clv.zappar.io/3571850569055456732/1.0.0/