Bug 254094 – REGRESSION: Error on camera startup on iOS 16.4 Beta 4

• Status: NEW
• Component: WebGL

[Joel Udwin]

8th Wall WebAR experiences are failing on camera startup in iOS 16.4 Beta 4. This previously worked on iOS 16.4 Beta 3. This impacts major brands including: Marvel, LA Dodgers, Universal Pictures, MGM, Xbox, & More

The behavior we observe on iOS 16.4 Beta 4 is that there is no request for camera or device orientation permissions before camera startup. We also observe that there appears to be some code called with a null GL context. 

The following experiences can be accessed on iOS 16.4 Beta 4 to reproduce the issue: 
- https://lensthat.8thwall.app/m-and-m-s-find-the-crew/ (M&Ms / Mars Company)
- https://haribohunt.com/ (Haribo)

Sample code with a similar structure shared below does not raise the issue so our team is still trying to find the minimum reproduction case. 
Sample code: https://8w.8thwall.app/webgl-camera/

[Radar WebKit Bug Importer]

<rdar://problem/106883366>

[[e-mail]]

Created attachment 465488 [details]
OffscreenCanvas 2d only

Looks like only 2d contexts are supported at this time.

[[e-mail]]

We believe the issue is caused by the addition of OffscreenCanvas to 16.4. 8th Wall uses OffscreenCanvas when available to run webgl operations.  However, it seems that only 2d contexts are available at this time on iOS for OffscreenCanvas.  See the "OffscreenCanvas 2d only" screenshot to see how we tested this.  We now have a workaround so that we continue not using OffscreenCanvas on iOS and our WebAR experiences are working again.

[[e-mail]]

In the near-term, we've implemented a workaround, but it would be great if OffscreenCanvas worked for webgl and webgl2.

[Dean Jackson]

Yeah - it's likely that you're detecting OffscreenCanvas and assuming WebGL is supported. That's not quite ready yet.

It's good you can work around it, but I assume that won't automatically update every page that uses your library. We had a similar bug where we added a quirk that disabled OffscreenCanvas.

Is there something identifiable your JS library does that we could detect?

[[e-mail]]

Thanks for the context Dean.

It would be great if you all added a warning for when an OffscreenCanvas was trying to create a webgl or webgl2 context when it isn’t supported. That would help explain the issue to our customers.

[Karl Dubost]

Trying with Haribo.

```
[Warning] No device motion events will be fired, reason: Permission to use the API was not yet requested. (8frame-1.2.0.min.js, line 1688)
[Warning] No DPDB device match. (8frame-1.2.0.min.js, line 1688)
[Error] Failed to recalculate device parameters.
	(anonymous function) (8frame-1.2.0.min.js:1688:22871)
	o (8frame-1.2.0.min.js:1687:5325)
	w (8frame-1.2.0.min.js:1687:10361)
	(anonymous function) (8frame-1.2.0.min.js:1690:19579)
	(anonymous function) (8frame-1.2.0.min.js:1690:21054)
	e (8frame-1.2.0.min.js:1687:552)
	(anonymous function) (8frame-1.2.0.min.js:3622:653)
	o (8frame-1.2.0.min.js:1:636)
	r (8frame-1.2.0.min.js:1:799)
	(anonymous function) (8frame-1.2.0.min.js:1:827)
	(anonymous function) (8frame-1.2.0.min.js:1:318)
	Global Code (8frame-1.2.0.min.js:1:323)
[Warning] Using fallback iOS device measurements. (8frame-1.2.0.min.js, line 1688)
[Log] core:schema:warn Default value `0` does not match type `color` in component `undefined`  (8frame-1.2.0.min.js, line 19)
[Log] 8-Frame Version: 1.2.0 (Date 2021-10-29, Commit #e28557b1) (8frame-1.2.0.min.js, line 3622)
[Log] three Version (https://github.com/supermedium/three.js): – "^0.125.1" (8frame-1.2.0.min.js, line 3622)
[Log] WebVR Polyfill Version: – "^0.10.12" (8frame-1.2.0.min.js, line 3622)
[Warning] No device motion events will be fired, reason: Permission to use the API was not yet requested. (xrextras.js, line 1)
[Log] core:schema:warn Default value `false` does not match type `boolean` in component `sharing-screenshot`  (8frame-1.2.0.min.js, line 19, x2)
[Log] core:a-assets:warn Cross-origin element (e.g., <img>) was requested without `crossorigin` set. A-Frame will re-request the asset with `crossorigin` attribute set. Please set `crossorigin` on the element (e.g., <img crossorigin="anonymous">)  – "https://haribohunt.com/3f52a3cc851c86ca4f6a.png" (8frame-1.2.0.min.js, line 19)
[Error] TypeError: null is not an object (evaluating 'document.getElementById("qrcode").insertAdjacentElement')
	(anonymous function) (bundle.js:2:357372)
[Warning] No device orientation events will be fired, reason: Permission to use the API was not yet requested. (xr-simd-21.3.8.997.js, line 1)
[Warning] No device motion events will be fired, reason: Permission to use the API was not yet requested. (xr-simd-21.3.8.997.js, line 1)
[Log] 8th Wall XR Version: 21.3.8.997s (xr-simd-21.3.8.997.js, line 1)
[Warning] No device orientation events will be fired, reason: Permission to use the API was not yet requested. (8frame-1.2.0.min.js, line 3722)
[Log] ["friedegg", "eggsgalore2", "bunnywhite", "bunnypurple", "eggsgalore3", "eggsgalore1"] (6) (bundle.js, line 2)
[Log] 323.23801157521916 (bundle.js, line 2)
[Log] tapped (bundle.js, line 2)
[Log] User is over 14 (bundle.js, line 2)
[Error] XR threw an exception – TypeError: null is not an object (evaluating 'A.TEXTURE_2D') — xr-simd-21.3.8.997.js:1:9270138
TypeError: null is not an object (evaluating 'A.TEXTURE_2D') — xr-simd-21.3.8.997.js:1:9270138
	onException (xr-simd-21.3.8.997.js:1:9443081)
	forEach
	f (xr-simd-21.3.8.997.js:1:9483636)
	run (xr-simd-21.3.8.997.js:1:9496174)
	G (xr-simd-21.3.8.997.js:1:9435783)
	init (xr-simd-21.3.8.997.js:1:9451797)
	initComponent (8frame-1.2.0.min.js:2582:4891)
	updateProperties (8frame-1.2.0.min.js:2582:4644)
	(anonymous function) (8frame-1.2.0.min.js:2582:2828)
	i (8frame-1.2.0.min.js:2582:9575)
	initComponent (8frame-1.2.0.min.js:2574:4294)
	updateComponent (8frame-1.2.0.min.js:2574:5984)
	updateComponent (8frame-1.2.0.min.js:3330)
	setAttribute (8frame-1.2.0.min.js:2574:8111)
	setAttribute (8frame-1.2.0.min.js:3145)
	(anonymous function) (bundle.js:2:338853)
[Error]  (1)
UC — xr-simd-21.3.8.997.js:1:9270138(anonymous function) — xr-simd-21.3.8.997.js:1:9492131run — xr-simd-21.3.8.997.js:1:9492372run — xr-simd-21.3.8.997.js:1:9496148G — xr-simd-21.3.8.997.js:1:9435783init — xr-simd-21.3.8.997.js:1:9451797initComponent — component.js:320updateProperties — component.js:302(anonymous function) — component.js:78i — component.js:667initComponent — a-entity.js:332updateComponent — a-entity.js:495updateComponent — a-scene.js:743setAttribute — a-entity.js:727setAttribute — a-scene.js:558(anonymous function) — click-event-manager.js:64
	onException (xr-simd-21.3.8.997.js:1:9443134)
	forEach
	f (xr-simd-21.3.8.997.js:1:9483636)
	run (xr-simd-21.3.8.997.js:1:9496174)
	G (xr-simd-21.3.8.997.js:1:9435783)
	init (xr-simd-21.3.8.997.js:1:9451797)
	initComponent (8frame-1.2.0.min.js:2582:4891)
	updateProperties (8frame-1.2.0.min.js:2582:4644)
	(anonymous function) (8frame-1.2.0.min.js:2582:2828)
	i (8frame-1.2.0.min.js:2582:9575)
	initComponent (8frame-1.2.0.min.js:2574:4294)
	updateComponent (8frame-1.2.0.min.js:2574:5984)
	updateComponent (8frame-1.2.0.min.js:3330)
	setAttribute (8frame-1.2.0.min.js:2574:8111)
	setAttribute (8frame-1.2.0.min.js:3145)
	(anonymous function) (bundle.js:2:338853)
[Error] TypeError: null is not an object (evaluating 'g.readyState')
	(anonymous function) (analytics.js:33:404)
```
WindowObject);
      if (typeof DeviceOrientationEvent !== 'undefined' && DeviceOrientationEvent.requestPermission) {
        magicWindowControls.enabled = false;
        // we removed 'device-orientation-permission-ui' as a default enabled component.  This is
        // a VR specific component.
        this.el.sceneEl.addEventListener('deviceorientationpermissiongranted', function () {
          magicWindowControls.enabled = data.magicWindowTrackingEnabled;
        });
      }
    }
  }
```


The detection of OffscreenCanvas in three.js is 
https://cdn.8thwall.com/web/aframe/node_modules/super-three/build/three.js

```
var useOffscreenCanvas = false;
try {
  useOffscreenCanvas = typeof OffscreenCanvas !== 'undefined' && new OffscreenCanvas(1, 1).getContext('2d') !== null;
  } catch (err) {
  // Ignore any errors
}
```

it will be set to true.




Another detection in 
https://cdn.8thwall.com/xr-simd-21.3.8.997.js

```
                oA = function() {
                    return window.OffscreenCanvas ? new OffscreenCanvas(0, 0) : i("compute")
                },
```

so far so good but this is used in

```
var I = !1,
g = aQ(aQ({}, iA), D.glContextConfig || {});
D.webgl2 && (I = (Q = C.getContext("webgl2", g)) && n(Q)),
Q || (Q = C.getContext("webgl", g)),
Q || (I = (Q = C.getContext("webgl2", g)) && n(Q)),

// Here B is the OffscreenCanvas

B || (B = oA()),
(E = B.getContext(I ? "webgl2" : "webgl", g)) && !E.isContextLost() || (B = oA(), E = B.getContext(I ? "webgl2" :
"webgl", g)),


t("[XR] Using WebGL", I ? "2" : "1"),
UC(Q, D.verbose, !0, I),


//calling UC with E null
UC(E, D.verbose, !0, I);

var i = E.getContextAttributes();
```

B -> OffscreenCanvas
I -> false

So we got B.getContext("webgl") or B.getContext("webgl2") returning null
so E is null.

Then E is passed to UC

```
        function UC(A) {
            var I,
                g,
                C,
                Q,
                B = arguments.length > 1 && void 0 !== arguments[1] && arguments[1],
                E = arguments.length > 3 && void 0 !== arguments[3] && arguments[3],
                D = (I = {}, c()(I, A.TEXTURE_2D, A.TEXTURE_BINDING_2D), c()(I, A.TEXTURE_CUBE_MAP, A.TEXTURE_BINDING_CUBE_MAP), I);
            E && Object.assign(D, (Q = {}, c()(Q, A.TEXTURE_3D, A.TEXTURE_BINDING_3D), c()(Q, A.TEXTURE_2D_ARRAY, A.TEXTURE_BINDING_2D_ARRAY), Q));
```


So it will throw here as soon it is requesting things like  
A.TEXTURE_2D which would be basically null.TEXTURE_2D.




Currently in Safari 16.4

> offscreen = new OffscreenCanvas(1,1)
< OffscreenCanvas {width: 1, height: 1, getContext: function, transferToImageBitmap: function, convertToBlob: function, …}
> offscreen.getContext("webgl")
< null
> offscreen.getContext("webgl2")
null

[Karl Dubost]

Probably the oA() function in https://cdn.8thwall.com/xr-simd-21.3.8.997.js

oA = function() {
     return window.OffscreenCanvas ? new OffscreenCanvas(0, 0) : i("compute")
}

could right away test for webgl or webgl2 with getContext("webgl")
or later in the code if E is null to not go the current path.

Avoid user agent sniffing so that it is resilient once webgl and webgl2 are activated.

[Karl Dubost]

Nathan,

is everyone using the CDN for this library? Or are people using their own hosted version?

If people are hosting their own version, is there a window object defined early enough that would help detect the library and make it possible for Safari to deactivate offscreenCanvas early enough. 

Maybe window._XR8 would be a good candidate?

[[e-mail]]

Hello Karl,

Thanks for the suggestions! We went with a similar approach to the one you suggested and have made the release with that fix.  People are using the CDN for our library so they all should receive the update.  I don’t think there is a need to add additional logic to Safari to deactivate offscreenCanvas early, but I appreciate the offer.

I’ll look into the Three.js issue you discovered and will followup with what I’ve found.  Thank you!  We really appreciate the effort you put into helping us with this issue and look forward to full WebGl support for offscreen canvas in Safari when it is available.