Bug 246049 – [Wasm-GC] Fix handling of recursive types using shorthand encoding

• Status: RESOLVED
• Resolution: FIXED
• Component: WebAssembly

[Asumu Takikawa]

Recent Wasm GC patches added support for recursive types. There are different ways to encode a recursion group, and currently some shorthands are not handled correctly.

Examples (these test cases would go in `JSTests/wasm/gc/rec.js`):

```
  /*
   * This test needs to be in binary format, as it tests the specific encoding
   * of the recursion group. This one omits the explicit `rec`.
   *
   *  (module
   *    (rec (type (array (ref 0))))
   *    (func (result (ref null 0)) (ref.null 0)))
   */
  new WebAssembly.Instance(module("\x00\x61\x73\x6d\x01\x00\x00\x00\x01\x8a\x80\x80\x80\x00\x02\x5e\x6b\x00\x00\x60\x00\x01\x6c\x00\x03\x82\x80\x80\x80\x00\x01\x01\x0a\x8a\x80\x80\x80\x00\x
01\x84\x80\x80\x80\x00\x00\xd0\x00\x0b"))

  // Same test as above but using a non-shorthand encoding.
  new WebAssembly.Instance(module("\x00\x61\x73\x6d\x01\x00\x00\x00\x01\x8c\x80\x80\x80\x00\x02\x4f\x01\x5e\x6b\x00\x00\x60\x00\x01\x6c\x00\x03\x82\x80\x80\x80\x00\x01\x01\x0a\x8a\x80\x80\x80\x00\x01\x84\x80\x80\x80\x00\x00\xd0\x00\x0b"))
```

Currently the second example works as expected with no errors. It uses an explicit `rec` operator in the binary format ("\x4f"). The first example fails with a parse error, because the `rec` operator is left implicit in the binary format.

Error for first example:

```
Exception: CompileError: WebAssembly.Module doesn't parse at byte 18: can't get array's element Type (evaluating 'new WebAssembly.Module(buffer)')
```

This can be fixed by either uniformly representing all types as recursive groups when GC is turned on and parsing them as such, or by parsing structural types assuming the possibility that they will be recursive and adding a recursion group around them if recursion variable shows up in the type.

[Asumu Takikawa]

Pull request: https://github.com/WebKit/WebKit/pull/5124

[Radar WebKit Bug Importer]

<rdar://problem/101055691>

[EWS]

Committed 255460@main (cc59ec0dd950): <https://commits.webkit.org/255460@main>

Reviewed commits have been landed. Closing PR #5124 and removing active labels.