Bug 247536 – [Wasm-GC] Fix some interactions between subtyping and recursion

• Status: RESOLVED
• Resolution: FIXED
• Component: WebAssembly

[Asumu Takikawa]

Recently both subtyping and recursive types were added for Wasm GC support. There are a few cases of interactions between these two features that don't quite work as intended. In particular, when the parent type of a `sub` declaration is a recursive type, the display-based subtyping check and also the structural subtyping check between type definitions can crash due to assertion failures.

Example test cases (can be added to `JSTests/wasm/gc/sub.js`):

```
  // This type-checks in the reference interpreter.
  instantiate(`
    (module
      (rec (type (func (result (ref 0)))))
      (rec (type (sub 0 (func (result (ref 1))))))
      (type (sub 1 (func (result (ref 1))))) ;; parent is a recursive subtype, whose parent is also a recursive type

      (func (result (ref null 0))
        (ref.null 2))
    )
  `);
```

Another example is from the Wasm GC spec tests:

```
  // This fails because during the structural type-check between a `sub` clause and its
  // parent, the parent is not a projection (because the recursion group has not been
  // created at that point), but then the recursive references cannot be resolved correctly.
  // This requires the structural type-check to bring in the entire recursion group and
  // expanding parent references if needed, or some variation on that strategy.
  instantiate(`
    (module
      (rec
        (type $t1 (func (param i32 (ref $t3))))
        (type $t2 (sub $t1 (func (param i32 (ref $t2)))))
        (type $t3 (sub $t2 (func (param i32 (ref $t1)))))
      )

      (func $f1 (param $r (ref $t1))
        (call $f1 (local.get $r)))
    )
  `);
```

The problems all seem to stem from `sub` clause parent references needing to point to the projection to the whole recursion group (and this also needs to be factored in for creating the displays for fast subtype checks).

[Radar WebKit Bug Importer]

<rdar://problem/102280174>

[Asumu Takikawa]

Pull request: https://github.com/WebKit/WebKit/pull/6713

[EWS]

Committed 257945@main (0d872184b6e1): <https://commits.webkit.org/257945@main>

Reviewed commits have been landed. Closing PR #6713 and removing active labels.