REGRESSION (r286982): ASSERTION FAILED: clipRectsContext.rootLayer == m_clipRectsCache->m_clipRectsRoot[clipRectsType]

[Ryan Haddad]

fast/layers/top-layer-ancestor-opacity-and-transform-crash.html is asserting on iOS and macOS debug bots since https://trac.webkit.org/changeset/286982/webkit, which was a revert of https://trac.webkit.org/changeset/283354/webkit 

ASSERTION FAILED: clipRectsContext.rootLayer == m_clipRectsCache->m_clipRectsRoot[clipRectsType]
./rendering/RenderLayer.cpp(4410) : Ref<WebCore::ClipRects> WebCore::RenderLayer::updateClipRects(const WebCore::RenderLayer::ClipRectsContext &)
1   0x14fa1c950 WTFCrash
2   0x105812230 WebCore::Converter<WebCore::IDLDOMString>::ReturnType WebCore::convert<WebCore::IDLDOMString>(JSC::JSGlobalObject&, JSC::JSValue)
3   0x109b7f31c WebCore::RenderLayer::updateClipRects(WebCore::RenderLayer::ClipRectsContext const&)
4   0x109b7f840 WebCore::RenderLayer::parentClipRects(WebCore::RenderLayer::ClipRectsContext const&) const
5   0x109b75ef4 WebCore::RenderLayer::backgroundClipRect(WebCore::RenderLayer::ClipRectsContext const&) const
6   0x109b75604 WebCore::RenderLayer::paintLayerWithEffects(WebCore::GraphicsContext&, WebCore::RenderLayer::LayerPaintingInfo const&, WTF::OptionSet<WebCore::RenderLayer::PaintLayerFlag>)
7   0x109b74be8 WebCore::RenderLayer::paintLayer(WebCore::GraphicsContext&, WebCore::RenderLayer::LayerPaintingInfo const&, WTF::OptionSet<WebCore::RenderLayer::PaintLayerFlag>)
8   0x109b7a64c WebCore::RenderLayer::paintList(WebCore::RenderLayer::LayerList, WebCore::GraphicsContext&, WebCore::RenderLayer::LayerPaintingInfo const&, WTF::OptionSet<WebCore::RenderLayer::PaintLayerFlag>)
9   0x109b770b4 WebCore::RenderLayer::paintLayerContents(WebCore::GraphicsContext&, WebCore::RenderLayer::LayerPaintingInfo const&, WTF::OptionSet<WebCore::RenderLayer::PaintLayerFlag>)
10  0x109b76620 WebCore::RenderLayer::paintLayerContentsAndReflection(WebCore::GraphicsContext&, WebCore::RenderLayer::LayerPaintingInfo const&, WTF::OptionSet<WebCore::RenderLayer::PaintLayerFlag>)
11  0x109b76400 WebCore::RenderLayer::paintLayerByApplyingTransform(WebCore::GraphicsContext&, WebCore::RenderLayer::LayerPaintingInfo const&, WTF::OptionSet<WebCore::RenderLayer::PaintLayerFlag>, WebCore::LayoutSize const&)
12  0x109b756dc WebCore::RenderLayer::paintLayerWithEffects(WebCore::GraphicsContext&, WebCore::RenderLayer::LayerPaintingInfo const&, WTF::OptionSet<WebCore::RenderLayer::PaintLayerFlag>)
13  0x109b74be8 WebCore::RenderLayer::paintLayer(WebCore::GraphicsContext&, WebCore::RenderLayer::LayerPaintingInfo const&, WTF::OptionSet<WebCore::RenderLayer::PaintLayerFlag>)
14  0x109b7a64c WebCore::RenderLayer::paintList(WebCore::RenderLayer::LayerList, WebCore::GraphicsContext&, WebCore::RenderLayer::LayerPaintingInfo const&, WTF::OptionSet<WebCore::RenderLayer::PaintLayerFlag>)
15  0x109b770b4 WebCore::RenderLayer::paintLayerContents(WebCore::GraphicsContext&, WebCore::RenderLayer::LayerPaintingInfo const&, WTF::OptionSet<WebCore::RenderLayer::PaintLayerFlag>)
16  0x109b95d84 WebCore::RenderLayerBacking::paintIntoLayer(WebCore::GraphicsLayer const*, WebCore::GraphicsContext&, WebCore::IntRect const&, WTF::OptionSet<WebCore::PaintBehavior>, WebCore::EventRegionContext*)::$_25::operator()(WebCore::RenderLayer&, WTF::OptionSet<WebCore::RenderLayer::PaintLayerFlag>) const
17  0x109b95804 WebCore::RenderLayerBacking::paintIntoLayer(WebCore::GraphicsLayer const*, WebCore::GraphicsContext&, WebCore::IntRect const&, WTF::OptionSet<WebCore::PaintBehavior>, WebCore::EventRegionContext*)
18  0x109b969cc WebCore::RenderLayerBacking::paintContents(WebCore::GraphicsLayer const*, WebCore::GraphicsContext&, WebCore::FloatRect const&, unsigned int)
19  0x1094abe38 WebCore::GraphicsLayer::paintGraphicsLayerContents(WebCore::GraphicsContext&, WebCore::FloatRect const&, unsigned int)
20  0x10958bbdc WebCore::GraphicsLayerCA::platformCALayerPaintContents(WebCore::PlatformCALayer*, WebCore::GraphicsContext&, WebCore::FloatRect const&, unsigned int)
21  0x106e8a3d4 WebCore::PlatformCALayer::drawLayerContents(WebCore::GraphicsContext&, WebCore::PlatformCALayer*, WTF::Vector<WebCore::FloatRect, 5ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&, unsigned int)
22  0x1095cb29c WebCore::TileGrid::platformCALayerPaintContents(WebCore::PlatformCALayer*, WebCore::GraphicsContext&, WebCore::FloatRect const&, unsigned int)
23  0x106f483ec -[WebSimpleLayer drawInContext:]
24  0x1b8ab6660 CABackingStoreUpdate_
25  0x1b8b15464 invocation function for block in CA::Layer::display_()
26  0x1b8ab58d0 -[CALayer _display]
27  0x106f481d8 -[WebSimpleLayer display]
28  0x1b8ab49d8 CA::Layer::display_if_needed(CA::Transaction*)
29  0x1b8bf6aac CA::Context::commit_transaction(CA::Transaction*, double, double*)
30  0x1b8a96d78 CA::Transaction::commit()
31  0x12c3b8fcc WebKit::TiledCoreAnimationDrawingArea::forceRepaint()


https://results.webkit.org/?suite=layout-tests&test=fast%2Flayers%2Ftop-layer-ancestor-opacity-and-transform-crash.html&limit=31116

[Radar WebKit Bug Importer]

<rdar://problem/86488039>

[Ryan Haddad]

Skipped the test for debug in r287055.

[cathiechen]

Created attachment 447933 [details]
WIP-patch

Looks like it hits this assert.
ASSERT(clipRectsContext.rootLayer == m_clipRectsCache->m_clipRectsRoot[clipRectsType]);

IIUC, `m_clipRectsRoot` of ClipRectsCache is set while `updateLayerPositions()` which happens after layout.
And  `clipRectsContext.rootLayer` comes from `paintInfo.rootLayer`.
Looks like the rootLayer is changed in RenderLayer::paintLayerByApplyingTransform(see [1]).
Also happens in RenderLayer::hitTestLayerByApplyingTransform(see [2]).

Then the assert doesn't pass.

Regarding the approach, I tried to update `m_clipRectsRoot` of ClipRectsCache, it seems not work, for there are other layers don't has transform.
So maybe we shouldn't let the transform layers use the clipRects in `m_clipRectsRoot`, i.e, use TemporaryClipRects instead. The WIP patch is attached, not sure if this makes sense or not?

[1] https://webkit-search.igalia.com/webkit/source/Source/WebCore/rendering/RenderLayer.cpp#3495
[2] https://webkit-search.igalia.com/webkit/source/Source/WebCore/rendering/RenderLayer.cpp#4333

[cathiechen]

Created attachment 448494 [details]
Patch

[cathiechen]

Created attachment 448497 [details]
Patch

[Simon Fraser (smfr)]

Comment on attachment 448497 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=448497&action=review

> Source/WebCore/rendering/RenderLayer.cpp:4457
> +    if (establishesTopLayer() && !parent()->establishesTopLayer())

Even if both this and parent establish a top layer, they are in separate top layers so I think you should just check establishesTopLayer().

[cathiechen]

Comment on attachment 448497 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=448497&action=review

>> Source/WebCore/rendering/RenderLayer.cpp:4457
>> +    if (establishesTopLayer() && !parent()->establishesTopLayer())
> 
> Even if both this and parent establish a top layer, they are in separate top layers so I think you should just check establishesTopLayer().

Done! Thanks!

[cathiechen]

Created attachment 448563 [details]
Patch

[cathiechen]

Still crash in "mac-AS-debug-wk2" of EWS, it seems the crash exists before the patch. See, https://bugs.webkit.org/show_bug.cgi?id=230079

[Rob Buis]

(In reply to cathiechen from comment #9)
> Still crash in "mac-AS-debug-wk2" of EWS, it seems the crash exists before
> the patch. See, https://bugs.webkit.org/show_bug.cgi?id=230079

It may make sense to change the patch to skip top-layer-ancestor-opacity-and-transform-crash.html on Mac Debug with bug 230079 as reference.

[cathiechen]

(In reply to Rob Buis from comment #10)
> (In reply to cathiechen from comment #9)
> > Still crash in "mac-AS-debug-wk2" of EWS, it seems the crash exists before
> > the patch. See, https://bugs.webkit.org/show_bug.cgi?id=230079
> 
> It may make sense to change the patch to skip
> top-layer-ancestor-opacity-and-transform-crash.html on Mac Debug with bug
> 230079 as reference.

Done! Thanks, Rob!

[cathiechen]

Created attachment 448725 [details]
[fast-cq] Patch

[Simon Fraser (smfr)]

Comment on attachment 448725 [details]
[fast-cq] Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=448725&action=review

> Source/WebCore/ChangeLog:11
> +        treated as a root layer. When the tranformed layer updates its clip rects, the root layer won't

tranformed

[Tim Nguyen (:ntim)]

Comment on attachment 448725 [details]
[fast-cq] Patch

Win failure is false alarm.

[Tim Nguyen (:ntim)]

Comment on attachment 448725 [details]
[fast-cq] Patch

let me address simon's comment first

[Tim Nguyen (:ntim)]

Created attachment 448776 [details]
[fast-cq] Patch for landing

[EWS]

Committed r287847 (245896@main): <https://commits.webkit.org/245896@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 448776 [details].

[cathiechen]

Thanks, Tim and Simon!