Bug 218607 – [GTK] [REGRESSSION(r269390) Several editing/ tests are flaky

• Component: New Bugs
• Status: RESOLVED FIXED

[Diego Pino]

Several editing/ related tests became flaky in [r268384-r268400] https://build.webkit.org/builders/GTK-Linux-64-bit-Release-Tests/builds/16812

I bisected the range and I found I cannot reproduce the issue in r268384, but I can in r268390 (next revision in commit history). I can only reproduce the flaky failures if I run all the tests. If I run the offending tests only, they're not flaky.

So this might be a regression caused by r268390 or some other commits in the range r268385-r268389.

I append below the test reports for revisions r268384 and r268390 for comparison.

r268384:

=> Results: 53649/57126 tests passed (93.9%)

=> Tests to be fixed (13403):
     20 crashes                  ( 0.1%)
    119 timeouts                 ( 0.9%)
     20 missing results          ( 0.1%)
   2129 image-only failures      (15.9%)

=> Tests that will only be fixed if they crash (WONTFIX) (308):
      1 timeouts                 ( 0.3%)


Expected to fail, but passed: (6)
  imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/block-aspect-ratio-022.html
  imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/flex-aspect-ratio-010.html
  imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/parsing/contain-intrinsic-size-invalid.html
  imported/w3c/web-platform-tests/css/css-sizing/whitespace-and-break.html
  storage/domstorage/sessionstorage/blocked-file-access.html
  webrtc/peerconnection-page-cache-long.html


Unexpected flakiness: text-only failures (1)
  imported/w3c/web-platform-tests/webaudio/the-audio-api/the-channelmergernode-interface/active-processing.https.html [ Failure Pass ]

Unexpected flakiness: timeouts (1)
  media/video-as-img-output-pts.html [ Timeout Pass ]


Regressions: Unexpected text-only failures (1)
  inspector/unit-tests/heap-snapshot-collection-event.html [ Failure ]

Regressions: Unexpected crashes (1)
  inspector/page/frameScheduledNavigation.html [ Crash ]

Regressions: Unexpected timeouts (1)
  workers/wasm-hashset-many.html [ Timeout ]

------------------------------------------------------------------------------

r268390:

=> Results: 53629/57126 tests passed (93.9%)

=> Tests to be fixed (13423):
     20 crashes                  ( 0.1%)
    118 timeouts                 ( 0.9%)
     20 missing results          ( 0.1%)
   2128 image-only failures      (15.9%)

=> Tests that will only be fixed if they crash (WONTFIX) (308):
      1 timeouts                 ( 0.3%)


Expected to crash, but passed: (1)
  workers/wasm-hashset-many.html

Expected to fail, but passed: (5)
  imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/block-aspect-ratio-022.html
  imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/flex-aspect-ratio-010.html
  imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/parsing/contain-intrinsic-size-invalid.html
  imported/w3c/web-platform-tests/css/css-sizing/whitespace-and-break.html
  storage/domstorage/sessionstorage/blocked-file-access.html


Unexpected flakiness: text-only failures (21)
  editing/selection/extend-after-mouse-selection.html [ Failure Pass ]
  editing/selection/extend-selection-backward-at-beginning-of-line-by-sentence-granularity.html [ Failure Pass ]
  editing/selection/extend-selection-backward-at-beginning-of-line-by-word-granularity.html [ Failure Pass ]
  editing/selection/focus-contenteditable-iframe.html [ Failure Pass ]
  editing/selection/getRangeAt.html [ Failure Pass ]
  editing/selection/move-by-word-visually-across-object-element-1.html [ Failure Pass ]
  editing/selection/move-by-word-visually-across-object-element-2.html [ Failure Pass ]
  editing/selection/move-by-word-visually-across-object-element-3.html [ Failure Pass ]
  editing/selection/select-all-user-select-none.html [ Failure Pass ]
  editing/selection/select-out-of-floated-non-editable-02.html [ Failure Pass ]
  editing/selection/select-out-of-floated-non-editable-07.html [ Failure Pass ]
  editing/selection/select-out-of-floated-non-editable-10.html [ Failure Pass ]
  editing/selection/shift-click-includes-existing-selection.html [ Failure Pass ]
  editing/selection/shift-click.html [ Failure Pass ]
  editing/selection/toString-1.html [ Failure Pass ]
  editing/selection/user-select-all-selection.html [ Failure Pass ]
  editing/selection/user-select-all-with-shift.html [ Failure Pass ]
  editing/selection/user-select-all-with-single-click.html [ Failure Pass ]
  imported/w3c/web-platform-tests/html/semantics/forms/form-submission-0/url-encoded.html [ Failure Pass ]
  imported/w3c/web-platform-tests/html/semantics/forms/the-label-element/labelable-elements.html [ Failure Pass ]
  imported/w3c/web-platform-tests/html/semantics/interfaces.html [ Failure Pass ]


Regressions: Unexpected text-only failures (1)
  inspector/unit-tests/heap-snapshot-collection-event.html [ Failure ]

Regressions: Unexpected crashes (1)
  inspector/dom/setInspectedNode.html [ Crash ]

------------------------------------------------------------------------------

[Diego Pino]

There's a mistake in the references to the revisions. Everything that is r268XXX should be r269XXX.

[Diego Pino]

I run layoutest on top of r269389 and the offending flaky tests are not failing in that revision. Here are the results:

=> Results: 53641/57126 tests passed (93.9%)

=> Tests to be fixed (13411):
     21 crashes                  ( 0.2%)
    124 timeouts                 ( 0.9%)
     20 missing results          ( 0.1%)
   2127 image-only failures      (15.9%)

=> Tests that will only be fixed if they crash (WONTFIX) (308):
      1 timeouts                 ( 0.3%)


Expected to crash, but passed: (1)
  workers/wasm-hashset-many.html

Expected to fail, but passed: (6)
  http/wpt/beacon/cors/crossorigin-arraybufferview-no-preflight.html
  imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/block-aspect-ratio-022.html
  imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/flex-aspect-ratio-010.html
  imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/parsing/contain-intrinsic-size-invalid.html
  imported/w3c/web-platform-tests/css/css-sizing/whitespace-and-break.html
  storage/domstorage/sessionstorage/blocked-file-access.html


Unexpected flakiness: timeouts (1)
  imported/w3c/web-platform-tests/html/browsers/offline/manifest_url_check.https.https.html [ Timeout Pass ]


Regressions: Unexpected text-only failures (1)
  inspector/unit-tests/heap-snapshot-collection-event.html [ Failure ]

Regressions: Unexpected crashes (2)
  inspector/model/remote-object/iterators-mutated.html [ Crash ]
  inspector/page/setBootstrapScript-sub-frame.html [ Crash ]

------------------------------------------------------------------------------

[Diego Pino]

An easier way to reproduce this issue is by running only the 'editing/selection/' folder.

[Carlos Alberto Lopez Perez]

I confirm that r268390 caused this regression. Tested locally and I can reproduce the issue with r268390 but not before it

To reproduce the issue simply run the tests inside editing/selection and you will see this:

$ Tools/Scripts/run-webkit-tests --debug-rwt-logging --release --gtk editing/selection

Unexpected flakiness: text-only failures (18)
  editing/selection/extend-after-mouse-selection.html [ Failure Pass ]
  editing/selection/extend-selection-backward-at-beginning-of-line-by-sentence-granularity.html [ Failure Pass ]
  editing/selection/extend-selection-backward-at-beginning-of-line-by-word-granularity.html [ Failure Pass ]
  editing/selection/focus-contenteditable-iframe.html [ Failure Pass ]
  editing/selection/getRangeAt.html [ Failure Pass ]
  editing/selection/move-by-word-visually-across-object-element-1.html [ Failure Pass ]
  editing/selection/move-by-word-visually-across-object-element-2.html [ Failure Pass ]
  editing/selection/move-by-word-visually-across-object-element-3.html [ Failure Pass ]
  editing/selection/select-all-user-select-none.html [ Failure Pass ]
  editing/selection/select-out-of-floated-non-editable-02.html [ Failure Pass ]
  editing/selection/select-out-of-floated-non-editable-07.html [ Failure Pass ]
  editing/selection/select-out-of-floated-non-editable-10.html [ Failure Pass ]
  editing/selection/shift-click-includes-existing-selection.html [ Failure Pass ]
  editing/selection/shift-click.html [ Failure Pass ]
  editing/selection/toString-1.html [ Failure Pass ]
  editing/selection/user-select-all-selection.html [ Failure Pass ]
  editing/selection/user-select-all-with-shift.html [ Failure Pass ]
  editing/selection/user-select-all-with-single-click.html [ Failure Pass ]


You can check the text diffs they give here: https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/r269612%20(16890)/results.html


Some observations:

 * Those failed tests failed on the first run, but when they were retried by WTR on the second run they all passed.
 * And if you run the tests one by one, they always pass.
 * Also if you run those tests with several workers (by passing the flags "--child-process=32 -f") then the failures don't happen or happen less (depending on the number of workers you spawn)
 * And finally you can see that if you run those tests by restarting WTR after each test (by passing the flag --run-singly) then the failures also don't happen.

So an hypothesis is that after r268390 one or more tests inside the folder editing/selection is leaving WTR in a inconsistent state that then causes failures on the rest of tests later executed in that WTR process.

[Carlos Alberto Lopez Perez]

I confirm that r269390 caused this regression. Tested locally and I can reproduce the issue with r269390 but not before it

To reproduce the issue simply run the tests inside editing/selection and you will see this:

$ Tools/Scripts/run-webkit-tests --debug-rwt-logging --release --gtk editing/selection

Unexpected flakiness: text-only failures (18)
  editing/selection/extend-after-mouse-selection.html [ Failure Pass ]
  editing/selection/extend-selection-backward-at-beginning-of-line-by-sentence-granularity.html [ Failure Pass ]
  editing/selection/extend-selection-backward-at-beginning-of-line-by-word-granularity.html [ Failure Pass ]
  editing/selection/focus-contenteditable-iframe.html [ Failure Pass ]
  editing/selection/getRangeAt.html [ Failure Pass ]
  editing/selection/move-by-word-visually-across-object-element-1.html [ Failure Pass ]
  editing/selection/move-by-word-visually-across-object-element-2.html [ Failure Pass ]
  editing/selection/move-by-word-visually-across-object-element-3.html [ Failure Pass ]
  editing/selection/select-all-user-select-none.html [ Failure Pass ]
  editing/selection/select-out-of-floated-non-editable-02.html [ Failure Pass ]
  editing/selection/select-out-of-floated-non-editable-07.html [ Failure Pass ]
  editing/selection/select-out-of-floated-non-editable-10.html [ Failure Pass ]
  editing/selection/shift-click-includes-existing-selection.html [ Failure Pass ]
  editing/selection/shift-click.html [ Failure Pass ]
  editing/selection/toString-1.html [ Failure Pass ]
  editing/selection/user-select-all-selection.html [ Failure Pass ]
  editing/selection/user-select-all-with-shift.html [ Failure Pass ]
  editing/selection/user-select-all-with-single-click.html [ Failure Pass ]


You can check the text diffs they give here: https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/r269612%20(16890)/results.html


Some observations:

 * Those failed tests failed on the first run, but when they were retried by WTR on the second run they all passed.
 * And if you run the tests one by one, they always pass.
 * Also if you run those tests with several workers (by passing the flags "--child-process=32 -f") then the failures don't happen or happen less (depending on the number of workers you spawn)
 * And finally you can see that if you run those tests by restarting WTR after each test (by passing the flag --run-singly) then the failures also don't happen.

So an hypothesis is that after r269390 one or more tests inside the folder editing/selection is leaving WTR in a inconsistent state that then causes failures on the rest of tests later executed in that WTR process.

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #5)
> 
> So an hypothesis is that after r269390 one or more tests inside the folder
> editing/selection is leaving WTR in a inconsistent state that then causes
> failures on the rest of tests later executed in that WTR process.

It seems this is the case.

I have identified the test that causes this and is: editing/selection/expando.html

This test has on on the top of the file the following declaration

-<!DOCTYPE html><!-- webkit-test-runner [ LiveRangeSelectionEnabled=true ] -->


Which I understands it enables LiveRangeSelectionEnabled on WTR, and that seems
to be affecting the tests that run after it on that WTR process.

I checked to remove that "LiveRangeSelectionEnabled=true" part from this test and then I can not longer reproduce the issue (the other tests on this folder work as expected).


So the issue seems to be that WTR is not resetting anymore the internal web-preferences settings to the default values after finishing running a test or something like that

[Carlos Alberto Lopez Perez]

Adding back the call to WKPreferencesResetAllInternalDebugFeatures(preferences) that was removed on r269390 from TestController::resetPreferencesToConsistentValues() fixes the issue. I will upload a patch for review

[Carlos Alberto Lopez Perez]

Created attachment 413661 [details]
Patch

[EWS]

Committed r269625: <https://trac.webkit.org/changeset/269625>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 413661 [details].

[Radar WebKit Bug Importer]

<rdar://problem/71235438>