218607 2020-11-05 02:10:04 -0800 [GTK] [REGRESSSION(r269390) Several editing/ tests are flaky 2020-11-10 05:32:22 -0800 1 1 1 Unclassified WebKit New Bugs WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=218569 InRadar P2 Normal --- 1 dpino clopez aperez bugs-noreply clopez fujii sam simon.fraser thorton webkit-bug-importer oldest_to_newest 1704792 0 dpino 2020-11-05 02:10:04 -0800 Several editing/ related tests became flaky in [r268384-r268400] https://build.webkit.org/builders/GTK-Linux-64-bit-Release-Tests/builds/16812 I bisected the range and I found I cannot reproduce the issue in r268384, but I can in r268390 (next revision in commit history). I can only reproduce the flaky failures if I run all the tests. If I run the offending tests only, they're not flaky. So this might be a regression caused by r268390 or some other commits in the range r268385-r268389. I append below the test reports for revisions r268384 and r268390 for comparison. r268384: => Results: 53649/57126 tests passed (93.9%) => Tests to be fixed (13403): 20 crashes ( 0.1%) 119 timeouts ( 0.9%) 20 missing results ( 0.1%) 2129 image-only failures (15.9%) => Tests that will only be fixed if they crash (WONTFIX) (308): 1 timeouts ( 0.3%) Expected to fail, but passed: (6) imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/block-aspect-ratio-022.html imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/flex-aspect-ratio-010.html imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/parsing/contain-intrinsic-size-invalid.html imported/w3c/web-platform-tests/css/css-sizing/whitespace-and-break.html storage/domstorage/sessionstorage/blocked-file-access.html webrtc/peerconnection-page-cache-long.html Unexpected flakiness: text-only failures (1) imported/w3c/web-platform-tests/webaudio/the-audio-api/the-channelmergernode-interface/active-processing.https.html [ Failure Pass ] Unexpected flakiness: timeouts (1) media/video-as-img-output-pts.html [ Timeout Pass ] Regressions: Unexpected text-only failures (1) inspector/unit-tests/heap-snapshot-collection-event.html [ Failure ] Regressions: Unexpected crashes (1) inspector/page/frameScheduledNavigation.html [ Crash ] Regressions: Unexpected timeouts (1) workers/wasm-hashset-many.html [ Timeout ] ------------------------------------------------------------------------------ r268390: => Results: 53629/57126 tests passed (93.9%) => Tests to be fixed (13423): 20 crashes ( 0.1%) 118 timeouts ( 0.9%) 20 missing results ( 0.1%) 2128 image-only failures (15.9%) => Tests that will only be fixed if they crash (WONTFIX) (308): 1 timeouts ( 0.3%) Expected to crash, but passed: (1) workers/wasm-hashset-many.html Expected to fail, but passed: (5) imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/block-aspect-ratio-022.html imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/flex-aspect-ratio-010.html imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/parsing/contain-intrinsic-size-invalid.html imported/w3c/web-platform-tests/css/css-sizing/whitespace-and-break.html storage/domstorage/sessionstorage/blocked-file-access.html Unexpected flakiness: text-only failures (21) editing/selection/extend-after-mouse-selection.html [ Failure Pass ] editing/selection/extend-selection-backward-at-beginning-of-line-by-sentence-granularity.html [ Failure Pass ] editing/selection/extend-selection-backward-at-beginning-of-line-by-word-granularity.html [ Failure Pass ] editing/selection/focus-contenteditable-iframe.html [ Failure Pass ] editing/selection/getRangeAt.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-1.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-2.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-3.html [ Failure Pass ] editing/selection/select-all-user-select-none.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-02.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-07.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-10.html [ Failure Pass ] editing/selection/shift-click-includes-existing-selection.html [ Failure Pass ] editing/selection/shift-click.html [ Failure Pass ] editing/selection/toString-1.html [ Failure Pass ] editing/selection/user-select-all-selection.html [ Failure Pass ] editing/selection/user-select-all-with-shift.html [ Failure Pass ] editing/selection/user-select-all-with-single-click.html [ Failure Pass ] imported/w3c/web-platform-tests/html/semantics/forms/form-submission-0/url-encoded.html [ Failure Pass ] imported/w3c/web-platform-tests/html/semantics/forms/the-label-element/labelable-elements.html [ Failure Pass ] imported/w3c/web-platform-tests/html/semantics/interfaces.html [ Failure Pass ] Regressions: Unexpected text-only failures (1) inspector/unit-tests/heap-snapshot-collection-event.html [ Failure ] Regressions: Unexpected crashes (1) inspector/dom/setInspectedNode.html [ Crash ] ------------------------------------------------------------------------------ 1704854 1 dpino 2020-11-05 07:54:17 -0800 There's a mistake in the references to the revisions. Everything that is r268XXX should be r269XXX. 1705574 2 dpino 2020-11-07 19:41:58 -0800 I run layoutest on top of r269389 and the offending flaky tests are not failing in that revision. Here are the results: => Results: 53641/57126 tests passed (93.9%) => Tests to be fixed (13411): 21 crashes ( 0.2%) 124 timeouts ( 0.9%) 20 missing results ( 0.1%) 2127 image-only failures (15.9%) => Tests that will only be fixed if they crash (WONTFIX) (308): 1 timeouts ( 0.3%) Expected to crash, but passed: (1) workers/wasm-hashset-many.html Expected to fail, but passed: (6) http/wpt/beacon/cors/crossorigin-arraybufferview-no-preflight.html imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/block-aspect-ratio-022.html imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/flex-aspect-ratio-010.html imported/w3c/web-platform-tests/css/css-sizing/aspect-ratio/parsing/contain-intrinsic-size-invalid.html imported/w3c/web-platform-tests/css/css-sizing/whitespace-and-break.html storage/domstorage/sessionstorage/blocked-file-access.html Unexpected flakiness: timeouts (1) imported/w3c/web-platform-tests/html/browsers/offline/manifest_url_check.https.https.html [ Timeout Pass ] Regressions: Unexpected text-only failures (1) inspector/unit-tests/heap-snapshot-collection-event.html [ Failure ] Regressions: Unexpected crashes (2) inspector/model/remote-object/iterators-mutated.html [ Crash ] inspector/page/setBootstrapScript-sub-frame.html [ Crash ] ------------------------------------------------------------------------------ 1705581 3 dpino 2020-11-07 21:35:23 -0800 An easier way to reproduce this issue is by running only the 'editing/selection/' folder. 1705975 4 clopez 2020-11-09 20:32:19 -0800 I confirm that r268390 caused this regression. Tested locally and I can reproduce the issue with r268390 but not before it To reproduce the issue simply run the tests inside editing/selection and you will see this: $ Tools/Scripts/run-webkit-tests --debug-rwt-logging --release --gtk editing/selection Unexpected flakiness: text-only failures (18) editing/selection/extend-after-mouse-selection.html [ Failure Pass ] editing/selection/extend-selection-backward-at-beginning-of-line-by-sentence-granularity.html [ Failure Pass ] editing/selection/extend-selection-backward-at-beginning-of-line-by-word-granularity.html [ Failure Pass ] editing/selection/focus-contenteditable-iframe.html [ Failure Pass ] editing/selection/getRangeAt.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-1.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-2.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-3.html [ Failure Pass ] editing/selection/select-all-user-select-none.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-02.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-07.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-10.html [ Failure Pass ] editing/selection/shift-click-includes-existing-selection.html [ Failure Pass ] editing/selection/shift-click.html [ Failure Pass ] editing/selection/toString-1.html [ Failure Pass ] editing/selection/user-select-all-selection.html [ Failure Pass ] editing/selection/user-select-all-with-shift.html [ Failure Pass ] editing/selection/user-select-all-with-single-click.html [ Failure Pass ] You can check the text diffs they give here: https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/r269612%20(16890)/results.html Some observations: * Those failed tests failed on the first run, but when they were retried by WTR on the second run they all passed. * And if you run the tests one by one, they always pass. * Also if you run those tests with several workers (by passing the flags "--child-process=32 -f") then the failures don't happen or happen less (depending on the number of workers you spawn) * And finally you can see that if you run those tests by restarting WTR after each test (by passing the flag --run-singly) then the failures also don't happen. So an hypothesis is that after r268390 one or more tests inside the folder editing/selection is leaving WTR in a inconsistent state that then causes failures on the rest of tests later executed in that WTR process. 1705977 5 clopez 2020-11-09 20:34:25 -0800 I confirm that r269390 caused this regression. Tested locally and I can reproduce the issue with r269390 but not before it To reproduce the issue simply run the tests inside editing/selection and you will see this: $ Tools/Scripts/run-webkit-tests --debug-rwt-logging --release --gtk editing/selection Unexpected flakiness: text-only failures (18) editing/selection/extend-after-mouse-selection.html [ Failure Pass ] editing/selection/extend-selection-backward-at-beginning-of-line-by-sentence-granularity.html [ Failure Pass ] editing/selection/extend-selection-backward-at-beginning-of-line-by-word-granularity.html [ Failure Pass ] editing/selection/focus-contenteditable-iframe.html [ Failure Pass ] editing/selection/getRangeAt.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-1.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-2.html [ Failure Pass ] editing/selection/move-by-word-visually-across-object-element-3.html [ Failure Pass ] editing/selection/select-all-user-select-none.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-02.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-07.html [ Failure Pass ] editing/selection/select-out-of-floated-non-editable-10.html [ Failure Pass ] editing/selection/shift-click-includes-existing-selection.html [ Failure Pass ] editing/selection/shift-click.html [ Failure Pass ] editing/selection/toString-1.html [ Failure Pass ] editing/selection/user-select-all-selection.html [ Failure Pass ] editing/selection/user-select-all-with-shift.html [ Failure Pass ] editing/selection/user-select-all-with-single-click.html [ Failure Pass ] You can check the text diffs they give here: https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/r269612%20(16890)/results.html Some observations: * Those failed tests failed on the first run, but when they were retried by WTR on the second run they all passed. * And if you run the tests one by one, they always pass. * Also if you run those tests with several workers (by passing the flags "--child-process=32 -f") then the failures don't happen or happen less (depending on the number of workers you spawn) * And finally you can see that if you run those tests by restarting WTR after each test (by passing the flag --run-singly) then the failures also don't happen. So an hypothesis is that after r269390 one or more tests inside the folder editing/selection is leaving WTR in a inconsistent state that then causes failures on the rest of tests later executed in that WTR process. 1705980 6 clopez 2020-11-09 21:12:30 -0800 (In reply to Carlos Alberto Lopez Perez from comment #5) > > So an hypothesis is that after r269390 one or more tests inside the folder > editing/selection is leaving WTR in a inconsistent state that then causes > failures on the rest of tests later executed in that WTR process. It seems this is the case. I have identified the test that causes this and is: editing/selection/expando.html This test has on on the top of the file the following declaration -<!DOCTYPE html><!-- webkit-test-runner [ LiveRangeSelectionEnabled=true ] --> Which I understands it enables LiveRangeSelectionEnabled on WTR, and that seems to be affecting the tests that run after it on that WTR process. I checked to remove that "LiveRangeSelectionEnabled=true" part from this test and then I can not longer reproduce the issue (the other tests on this folder work as expected). So the issue seems to be that WTR is not resetting anymore the internal web-preferences settings to the default values after finishing running a test or something like that 1705981 7 clopez 2020-11-09 21:29:05 -0800 Adding back the call to WKPreferencesResetAllInternalDebugFeatures(preferences) that was removed on r269390 from TestController::resetPreferencesToConsistentValues() fixes the issue. I will upload a patch for review 1705982 8 413661 clopez 2020-11-09 21:40:01 -0800 Created attachment 413661 Patch 1706055 9 ews-feeder 2020-11-10 05:30:57 -0800 Committed r269625: <https://trac.webkit.org/changeset/269625> All reviewed patches have been landed. Closing bug and clearing flags on attachment 413661. 1706056 10 webkit-bug-importer 2020-11-10 05:32:22 -0800 <rdar://problem/71235438> 413661 2020-11-09 21:40:01 -0800 2020-11-10 05:31:06 -0800 Patch bug-218607-20201110063959.patch text/plain 1874 clopez U3VidmVyc2lvbiBSZXZpc2lvbjogMjY5NjE1CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggM2M0MTIyMWFlZTIzOTU2NTZkNTBjNjI1OWMwOTJiYTA4 NjFkOTg0OS4uODBiZDY4NzQ5YTZlZDAyMTU3OGU1MmZiNDdlMDg3NjE5Yzk2Y2Q2NSAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE4 IEBACisyMDIwLTExLTA5ICBDYXJsb3MgQWxiZXJ0byBMb3BleiBQZXJleiAgPGNsb3BlekBpZ2Fs aWEuY29tPgorCisgICAgICAgIFtHVEtdIFtSRUdSRVNTU0lPTihyMjY5MzkwKSBTZXZlcmFsIGVk aXRpbmcvIHRlc3RzIGFyZSBmbGFreQorICAgICAgICBodHRwczovL2J1Z3Mud2Via2l0Lm9yZy9z aG93X2J1Zy5jZ2k/aWQ9MjE4NjA3CisKKyAgICAgICAgUmV2aWV3ZWQgYnkgTk9CT0RZIChPT1BT ISkuCisKKyAgICAgICAgQWZ0ZXIgcjI2OTM5MCBzb21lIG9mIHRoZSBpbnRlcm5hbCBwcmVmZXJl bmNlcyBsaWtlIExpdmVSYW5nZVNlbGVjdGlvbkVuYWJsZWQKKyAgICAgICAgYXJlIG5vdCByZXNl dHRlZCB0byB0aGVpciBkZWZhdWx0IHZhbHVlIG9uIFdUUiBiZXR3ZWVuIHRlc3QgcnVucy4KKyAg ICAgICAgU28gd2hlbiBhIHRlc3QgY2hhbmdlcyBzb21lIG9mIHRoaXMgcHJvcGVydGllcyBpdCBt YXkgY2F1c2UgZmFpbHVyZXMgb24KKyAgICAgICAgdGhlIG5leHQgdGVzdHMgdGhhdCB3aWxsIHJ1 biBvbiB0aGUgc2FtZSBXVFIgcHJvY2Vzcy4KKworICAgICAgICAqIFdlYktpdFRlc3RSdW5uZXIv VGVzdENvbnRyb2xsZXIuY3BwOgorICAgICAgICAoV1RSOjpUZXN0Q29udHJvbGxlcjo6cmVzZXRQ cmVmZXJlbmNlc1RvQ29uc2lzdGVudFZhbHVlcyk6IEFkZCBiYWNrIHRoZSBjYWxsIHRvIFdLUHJl ZmVyZW5jZXNSZXNldEFsbEludGVybmFsRGVidWdGZWF0dXJlcygpIHRoYXQgd2FzIHJlbW92ZWQg b24gcjI2OTM5MC4KKwogMjAyMC0xMS0wOSAgQWFrYXNoIEphaW4gIDxhYWthc2hfamFpbkBhcHBs ZS5jb20+CiAKICAgICAgICAgW2J1aWxkLndlYmtpdC5vcmddIERvIG5vdCBwcmludCBtYXN0ZXIg ZW52aXJvbm1lbnQgdmFyaWFibGVzIGluIGJ1aWxkIHN0ZXBzCmRpZmYgLS1naXQgYS9Ub29scy9X ZWJLaXRUZXN0UnVubmVyL1Rlc3RDb250cm9sbGVyLmNwcCBiL1Rvb2xzL1dlYktpdFRlc3RSdW5u ZXIvVGVzdENvbnRyb2xsZXIuY3BwCmluZGV4IDZkYmM2ODRlODg1MGFmZTA0NTkxMzlmYmUxOWQ1 ZTVjY2RiNzI2ZTcuLmU0YWFmMzhiNThiYzJkZjA1NjBjNTI5MGYxYWQ4ZjRkNGRmYjM1NWEgMTAw NjQ0Ci0tLSBhL1Rvb2xzL1dlYktpdFRlc3RSdW5uZXIvVGVzdENvbnRyb2xsZXIuY3BwCisrKyBi L1Rvb2xzL1dlYktpdFRlc3RSdW5uZXIvVGVzdENvbnRyb2xsZXIuY3BwCkBAIC04MjksNiArODI5 LDcgQEAgdm9pZCBUZXN0Q29udHJvbGxlcjo6cmVzZXRQcmVmZXJlbmNlc1RvQ29uc2lzdGVudFZh bHVlcyhjb25zdCBUZXN0T3B0aW9ucyYgb3B0aW8KICAgICBXS1ByZWZlcmVuY2VzUmVzZXRUZXN0 UnVubmVyT3ZlcnJpZGVzKHByZWZlcmVuY2VzKTsKIAogICAgIFdLUHJlZmVyZW5jZXNFbmFibGVB bGxFeHBlcmltZW50YWxGZWF0dXJlcyhwcmVmZXJlbmNlcyk7CisgICAgV0tQcmVmZXJlbmNlc1Jl c2V0QWxsSW50ZXJuYWxEZWJ1Z0ZlYXR1cmVzKHByZWZlcmVuY2VzKTsKIAogICAgIC8vIEZJWE1F OiBDb252ZXJ0IHRoZXNlIHRvIGRlZmF1bHQgdmFsdWVzIGZvciBUZXN0T3B0aW9ucy4KICAgICBX S1ByZWZlcmVuY2VzU2V0UHJvY2Vzc1N3YXBPbk5hdmlnYXRpb25FbmFibGVkKHByZWZlcmVuY2Vz LCBvcHRpb25zLnNob3VsZEVuYWJsZVByb2Nlc3NTd2FwT25OYXZpZ2F0aW9uKCkpOwo=