| Summary: | [GTK] [REGRESSSION(r269390) Several editing/ tests are flaky | ||||||
|---|---|---|---|---|---|---|---|
| Product: | WebKit | Reporter: | Diego Pino <dpino> | ||||
| Component: | New Bugs | Assignee: | Carlos Alberto Lopez Perez <clopez> | ||||
| Status: | RESOLVED FIXED | ||||||
| Severity: | Normal | CC: | aperez, bugs-noreply, clopez, Hironori.Fujii, sam, simon.fraser, thorton, webkit-bug-importer | ||||
| Priority: | P2 | Keywords: | InRadar | ||||
| Version: | WebKit Nightly Build | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| See Also: | https://bugs.webkit.org/show_bug.cgi?id=218569 | ||||||
| Attachments: |
|
||||||
|
Description
Diego Pino
2020-11-05 02:10:04 PST
There's a mistake in the references to the revisions. Everything that is r268XXX should be r269XXX. I run layoutest on top of r269389 and the offending flaky tests are not failing in that revision. Here are the results: => Results: 53641/57126 tests passed (93.9%) => Tests to be fixed (13411): 21 crashes ( 0.2%) 124 timeouts ( 0.9%) 20 missing results ( 0.1%) 2127 image-only failures (15.9%) => Tests that will only be fixed if they crash (WONTFIX) (308): 1 timeouts ( 0.3%) Expected to crash, but passed: (1) workers/wasm-hashset-many.html Expected to fail, but passed: (6) http/wpt/beacon/cors/crossorigin-arraybufferview-no-preflight.html imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/block-aspect-ratio-022.html imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/flex-aspect-ratio-010.html imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/parsing/contain-intrinsic-size-invalid.html imported/w3c/web-platform-tests/css/css-sizing/whitespace-and-break.html storage/domstorage/sessionstorage/blocked-file-access.html Unexpected flakiness: timeouts (1) imported/w3c/web-platform-tests/html/browsers/offline/manifest_url_check.https.https.html [ Timeout Pass ] Regressions: Unexpected text-only failures (1) inspector/unit-tests/heap-snapshot-collection-event.html [ Failure ] Regressions: Unexpected crashes (2) inspector/model/remote-object/iterators-mutated.html [ Crash ] inspector/page/setBootstrapScript-sub-frame.html [ Crash ] ------------------------------------------------------------------------------ An easier way to reproduce this issue is by running only the 'editing/selection/' folder. I confirm that r268390 caused this regression. Tested locally and I can reproduce the issue with r268390 but not before it To reproduce the issue simply run the tests inside editing/selection and you will see this: $ Tools/Scripts/run-webkit-tests --debug-rwt-logging --release --gtk editing/selection Unexpected flakiness: text-only failures (18) editing/selection/extend-after-mouse-selection.html [ Failure Pass ] editing/selection/extend-selection-backward-at-beginning-of-line-by-sentence-granularity.html [ Failure Pass ] editing/selection/extend-selection-backward-at-beginning-of-line-by-word-granularity.html [ Failure Pass ] editing/selection/focus-contenteditable-iframe.html [ Failure Pass ] editing/selection/getRangeAt.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-1.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-2.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-3.html [ Failure Pass ] editing/selection/select-all-user-select-none.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-02.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-07.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-10.html [ Failure Pass ] editing/selection/shift-click-includes-existing-selection.html [ Failure Pass ] editing/selection/shift-click.html [ Failure Pass ] editing/selection/toString-1.html [ Failure Pass ] editing/selection/user-select-all-selection.html [ Failure Pass ] editing/selection/user-select-all-with-shift.html [ Failure Pass ] editing/selection/user-select-all-with-single-click.html [ Failure Pass ] You can check the text diffs they give here: https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/r269612%20(16890)/results.html Some observations: * Those failed tests failed on the first run, but when they were retried by WTR on the second run they all passed. * And if you run the tests one by one, they always pass. * Also if you run those tests with several workers (by passing the flags "--child-process=32 -f") then the failures don't happen or happen less (depending on the number of workers you spawn) * And finally you can see that if you run those tests by restarting WTR after each test (by passing the flag --run-singly) then the failures also don't happen. So an hypothesis is that after r268390 one or more tests inside the folder editing/selection is leaving WTR in a inconsistent state that then causes failures on the rest of tests later executed in that WTR process. I confirm that r269390 caused this regression. Tested locally and I can reproduce the issue with r269390 but not before it To reproduce the issue simply run the tests inside editing/selection and you will see this: $ Tools/Scripts/run-webkit-tests --debug-rwt-logging --release --gtk editing/selection Unexpected flakiness: text-only failures (18) editing/selection/extend-after-mouse-selection.html [ Failure Pass ] editing/selection/extend-selection-backward-at-beginning-of-line-by-sentence-granularity.html [ Failure Pass ] editing/selection/extend-selection-backward-at-beginning-of-line-by-word-granularity.html [ Failure Pass ] editing/selection/focus-contenteditable-iframe.html [ Failure Pass ] editing/selection/getRangeAt.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-1.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-2.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-3.html [ Failure Pass ] editing/selection/select-all-user-select-none.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-02.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-07.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-10.html [ Failure Pass ] editing/selection/shift-click-includes-existing-selection.html [ Failure Pass ] editing/selection/shift-click.html [ Failure Pass ] editing/selection/toString-1.html [ Failure Pass ] editing/selection/user-select-all-selection.html [ Failure Pass ] editing/selection/user-select-all-with-shift.html [ Failure Pass ] editing/selection/user-select-all-with-single-click.html [ Failure Pass ] You can check the text diffs they give here: https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/r269612%20(16890)/results.html Some observations: * Those failed tests failed on the first run, but when they were retried by WTR on the second run they all passed. * And if you run the tests one by one, they always pass. * Also if you run those tests with several workers (by passing the flags "--child-process=32 -f") then the failures don't happen or happen less (depending on the number of workers you spawn) * And finally you can see that if you run those tests by restarting WTR after each test (by passing the flag --run-singly) then the failures also don't happen. So an hypothesis is that after r269390 one or more tests inside the folder editing/selection is leaving WTR in a inconsistent state that then causes failures on the rest of tests later executed in that WTR process. (In reply to Carlos Alberto Lopez Perez from comment #5) > > So an hypothesis is that after r269390 one or more tests inside the folder > editing/selection is leaving WTR in a inconsistent state that then causes > failures on the rest of tests later executed in that WTR process. It seems this is the case. I have identified the test that causes this and is: editing/selection/expando.html This test has on on the top of the file the following declaration -<!DOCTYPE html><!-- webkit-test-runner [ LiveRangeSelectionEnabled=true ] --> Which I understands it enables LiveRangeSelectionEnabled on WTR, and that seems to be affecting the tests that run after it on that WTR process. I checked to remove that "LiveRangeSelectionEnabled=true" part from this test and then I can not longer reproduce the issue (the other tests on this folder work as expected). So the issue seems to be that WTR is not resetting anymore the internal web-preferences settings to the default values after finishing running a test or something like that Adding back the call to WKPreferencesResetAllInternalDebugFeatures(preferences) that was removed on r269390 from TestController::resetPreferencesToConsistentValues() fixes the issue. I will upload a patch for review Created attachment 413661 [details]
Patch
Committed r269625: <https://trac.webkit.org/changeset/269625> All reviewed patches have been landed. Closing bug and clearing flags on attachment 413661 [details]. |