Defend against stack overflow in GraphicsLayerCA::recursiveCommitChanges

• Component: Compositing
• Status: RESOLVED FIXED

[Cameron McCormack (:heycam)]

It looks like GraphicsLayerCA::recursiveCommitChanges can cause a stack overflow in some configurations.  We already have CommitState::treeDepth, which we use to avoid creating deep CALayer trees, so we should be able to stop the recursion at the same level.

[Cameron McCormack (:heycam)]

<rdar://42584284>

[Cameron McCormack (:heycam)]

All CommitState::treeDepth does currently is avoid hooking up the too-deep PlatformCALayers to their intended parents.  I'm slightly concerned with leaving other aspects of the PlatformCALayers out of date, though most of the LayerChange reasons are just about rendering, geometry, etc., and shouldn't be a big deal, especially when not hooked up.  ScrollingNodeChanged is a bit different, but we tend to check for scrolling node ID validity when we use them.

Another concern is that the treeDepth counts structural layers, and so a given GraphicsLayerCA may be too deep in one call to recursiveCommitChanges, and not be too deep the next.  In such a case we should then continue to process the old m_uncommittedChanges on the children, so that should be OK?

[Cameron McCormack (:heycam)]

Created attachment 433107 [details]
Patch

[Cameron McCormack (:heycam)]

Created attachment 433120 [details]
Patch

[EWS]

Committed r279756 (239529@main): <https://commits.webkit.org/239529@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 433120 [details].