Executing "insertunorderedlist" while selecting a contenteditable element inside a shadow dom hangs the browser

Created attachment 336598 [details]
Reproduction case

Steps to reproduce the problem:
1. Create an element with a shadow dom
2. In the shadow dom, create an element with contenteditable
3. Select all text in the contenteditable element
4. Execute "insertunorderedlist" (a couple of times)

In the supplied test case the first text is without shadow dom, while the second text is within shadow dom.

What is the expected behavior?
The behavior is the same as in the non-shadow dom version, it create 3 bullet points for each line or removes all bullet points from all lines.

What went wrong?
The browser tab hangs and is completely unresponsive. Indeterminate spinner is running forever

[Radar WebKit Bug Importer]

<rdar://problem/38931033>

[Ryosuke Niwa]

Hm... I can't reproduce this problem with STP70. Can you still reproduce the issue?

Please feel free to re-open the bug if you can.

I am not yet on Mojave, so I am unable to run on STP70. It was still broken on STP69 on High Sierra. Hopefully I can verify this soon after upgrading.

[Ryosuke Niwa]

Hm... I don't think there was any change between STP69 and STP70 in this area. In fact, it works just fine in STP68 for me.

Oh, I see, you'd have to select the entire list. Now I can reproduce it!

[Ryosuke Niwa]

Looks like TextIterator is getting stuck:

1213 WebCore::JSExecState::profiledEvaluate(JSC::ExecState*, JSC::ProfilingReason, JSC::SourceCode const&, JSC::JSValue, WTF::NakedPtr<JSC::Exception>&)  (in WebCore) + 84  [0x10be059a4]
  1213 JSC::evaluate(JSC::ExecState*, JSC::SourceCode const&, JSC::JSValue, WTF::NakedPtr<JSC::Exception>&)  (in JavaScriptCore) + 287  [0x578a193df]
    1213 JSC::Interpreter::executeProgram(JSC::SourceCode const&, JSC::ExecState*, JSC::JSObject*)  (in JavaScriptCore) + 11170  [0x57887a1e2]
      1213 vmEntryToJavaScript  (in JavaScriptCore) + 200  [0x57835d029]
        1213 llint_entry  (in JavaScriptCore) + 26835  [0x578363ab6]
          1213 ???  (in <unknown binary>)  [0x48e61f28177]
            1213 WebCore::jsDocumentPrototypeFunctionExecCommand(JSC::ExecState*)  (in WebCore) + 533  [0x10b4807d5]
              1213 WebCore::Document::execCommand(WTF::String const&, bool, WTF::String const&)  (in WebCore) + 73  [0x10b480999]
                1213 WebCore::executeInsertOrderedList(WebCore::Frame&, WebCore::Event*, WebCore::EditorCommandSource, WTF::String const&)  (in WebCore) + 48  [0x10c0ba740]
                  1213 WebCore::CompositeEditCommand::apply()  (in WebCore) + 268  [0x10b439c5c]
                    959 WebCore::InsertListCommand::doApply()  (in WebCore) + 879  [0x10c0cab2f]
                    ! 702 WebCore::indexForVisiblePosition(WebCore::VisiblePosition const&, WTF::RefPtr<WebCore::ContainerNode, WTF::DumbPtrTraits<WebCore::ContainerNode> >&)  (in WebCore) + 386  [0x10c09d1f2]
                    ! : 455 WebCore::TextIterator::rangeLength(WebCore::Range const*, bool)  (in WebCore) + 36  [0x10b436e84]
                    ! : | 191 WebCore::TextIterator::init()  (in WebCore) + 386  [0x10c0e84e2]
                    ! : | + 46 WebCore::TextIterator::advance()  (in WebCore) + 436  [0x10b3aa644]

[Ryosuke Niwa]

Created attachment 354684 [details]
Reduction

[Ryosuke Niwa]

Created attachment 355964 [details]
Fixes the hang

[Ryosuke Niwa]

Created attachment 355965 [details]
Added the forgotten tests

[Ryosuke Niwa]

Created attachment 355966 [details]
Added the forgotten tests

[Ryosuke Niwa]

Committed r238693: <https://trac.webkit.org/changeset/238693>