191002 2018-10-27 17:43:47 -0700 At least two dozen IntersectionObserver tests are flaky 2019-05-14 13:40:22 -0700 1 1 1 Unclassified WebKit WebKitGTK Other PC Linux NEW P2 Normal --- 1 mcatanzaro ajuma ajuma ap bugs-noreply fred.wang jlewis3 magomez mcatanzaro ryanhaddad simon.fraser thorton tsavell zan oldest_to_newest 1472766 0 mcatanzaro 2018-10-27 17:43:47 -0700 https://build.webkit.org/results/GTK%20Linux%2064-bit%20Release%20(Tests)/r237491%20(8613)/results.html shows two dozen flaky IntersectionObserver tests. It looks like these tests all usually pass, but sometimes fail all together? The couple I cheked have been flaky since they were added in r234723 "Import WPTs for IntersectionObserver" and unfortunately not fixed by r237218 "Flaky IntersectionObserver web platform tests involving style updates". I'm marking the entire imported/w3c/web-platform-tests/intersection-observer/ directory as flaky. 1472767 1 mcatanzaro 2018-10-27 17:45:43 -0700 Flakiness is observed also on Apple bots, so I'll do this in the global expectations file. 1472768 2 mcatanzaro 2018-10-27 17:47:22 -0700 (In reply to Michael Catanzaro from comment #1) > Flakiness is observed also on Apple bots, so I'll do this in the global > expectations file. (That said, the GTK bots are *way* more flaky than the Apple bots.) 1472848 3 mcatanzaro 2018-10-28 18:20:08 -0700 Here's an example failure that occurred after r237218: https://build.webkit.org/results/Apple%20High%20Sierra%20Debug%20WK2%20(Tests)/r237322%20(5225)/results.html --- /Volumes/Data/slave/highsierra-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/intersection-observer/containing-block-expected.txt +++ /Volumes/Data/slave/highsierra-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/intersection-observer/containing-block-actual.txt @@ -1,7 +1,7 @@ PASS IntersectionObserver should only report intersections if root is a containing block ancestor of target. -PASS In containing block and intersecting. -PASS In containing block and not intersecting. -PASS Not in containing block and intersecting. -PASS Not in containing block and not intersecting. +FAIL In containing block and intersecting. assert_equals: entries.length expected 1 but got 0 +FAIL In containing block and not intersecting. assert_equals: entries.length expected 2 but got 1 +FAIL Not in containing block and intersecting. assert_equals: entries.length expected 2 but got 1 +FAIL Not in containing block and not intersecting. assert_equals: entries.length expected 2 but got 1 1472899 4 ajuma 2018-10-29 06:47:01 -0700 The root cause of this flakiness is that the test suite assumes HTMLEventLoop timing, and WebKit doesn't implement the HTMLEventLoop spec (and, when I brought this up at the Contributors' Meeting, there wasn't consensus that we should implement that spec). More specifically, the tests assume that if changes to style/layout are made in a rAF callback, then intersection observations will be updated before the next rAF callback, and events will fire within one setTimeout of that second callback. In HTMLEventLoop-based engines, this is straightforward to implement -- rAF is immediately followed by style/layout, and immediately followed by an update of intersection observations. After spending a while considering different approaches for WebKit, the most predictable (and closest to the spec) approach was to update intersection observations at the same time as we flush layers. On iOS, flushing and rAF are both driven by didUpdate messages received from the UIProcess. None of the tests are flaky on iOS. Mac is slightly different, but rAF still gets throttled when we fall behind on flushes. There's still a bit of flakiness, but still, the vast majority of tests are not flaky. And those that are flaky, are only flaky on Debug bots (where we're more likely to fall behind on flushes), not on Release bots. So I suspect that on GTK, there's perhaps weaker coupling between rAF and flushes, and so on some runs we get into a state where flushing/drawing is slower for some reason, so the tests fail because the observations they're expecting haven't been computed yet. In any case, given the above, I think that marking the entire suite as flaky in the global expectations file is too broad. So I'm going to move that over to the GTK expectations file. 1472908 5 mcatanzaro 2018-10-29 07:42:42 -0700 Thanks for the detailed explanation! Sounds like some investigation of the GTK graphics stack would be needed here. (In reply to Ali Juma from comment #4) > In any case, given the above, I think that marking the entire suite as flaky > in the global expectations file is too broad. So I'm going to move that over > to the GTK expectations file. FWIW I don't agree with this because the flakes are still occurring on several Apple bots so it's going to result in spurious complaints from the EWS. 1472910 6 ajuma 2018-10-29 07:47:05 -0700 (In reply to Michael Catanzaro from comment #5) > Thanks for the detailed explanation! Sounds like some investigation of the > GTK graphics stack would be needed here. > > (In reply to Ali Juma from comment #4) > > In any case, given the above, I think that marking the entire suite as flaky > > in the global expectations file is too broad. So I'm going to move that over > > to the GTK expectations file. > > FWIW I don't agree with this because the flakes are still occurring on > several Apple bots so it's going to result in spurious complaints from the > EWS. I'm also going to add expectations for the specific tests that are flaky on Mac Debug (rather than having a global expectation for the whole test suite). 1472913 7 mcatanzaro 2018-10-29 07:57:22 -0700 OK, sounds good! 1472936 8 353294 ajuma 2018-10-29 10:07:45 -0700 Created attachment 353294 Update expectations 1472937 9 353294 ajuma 2018-10-29 10:09:16 -0700 Comment on attachment 353294 Update expectations Clearing flags on attachment: 353294 Committed r237556: <https://trac.webkit.org/changeset/237556> 1472950 10 mcatanzaro 2018-10-29 10:57:45 -0700 There were definitely some failures on WK1 bots too, which should be accounted for. 1472952 11 ajuma 2018-10-29 11:04:14 -0700 (In reply to Michael Catanzaro from comment #10) > There were definitely some failures on WK1 bots too, which should be > accounted for. These tests were skipped entirely on mac-wk1 in r237218. And on the Win bots, all web platform tests are skipped. Looking at https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#tests=intersection-observer, the only flakiness I see is on Mac WK2 Debug and GTK. Are there others that I'm missing? 1473241 12 mcatanzaro 2018-10-30 06:43:02 -0700 Hmm, I don't know. I'm sure I saw them, but maybe they were too far back in the history? Not sure. Clearly it looks good now! 1536227 13 ajuma 2019-05-14 13:40:22 -0700 After recent changes to rAF timing to align with the spec, these tests are no longer flaky on Mac Debug, so I've updated the expectations in https://trac.webkit.org/changeset/245304. 353294 2018-10-29 10:07:45 -0700 2018-10-29 10:07:45 -0700 Update expectations bug-191002-20181029130744.patch text/plain 4494 ajuma U3VidmVyc2lvbiBSZXZpc2lvbjogMjM3NTQ3CmRpZmYgLS1naXQgYS9MYXlvdXRUZXN0cy9DaGFu Z2VMb2cgYi9MYXlvdXRUZXN0cy9DaGFuZ2VMb2cKaW5kZXggNjM2ZWQwOGM1ZTE3MjQ3MGYyYTM1 MTIxODRkNDRiOTM4NzUwM2QwNy4uYjI2NTU5OGRkOTVmNTA0Njc2Mjk0NTEyNjE5YzZmMzEzNWRl MzgyYiAxMDA2NDQKLS0tIGEvTGF5b3V0VGVzdHMvQ2hhbmdlTG9nCisrKyBiL0xheW91dFRlc3Rz L0NoYW5nZUxvZwpAQCAtMSwzICsxLDE0IEBACisyMDE4LTEwLTI5ICBBbGkgSnVtYSAgPGFqdW1h QGNocm9taXVtLm9yZz4KKworICAgICAgICBVbnJldmlld2VkIHRlc3QgZ2FyZGVuaW5nCisgICAg ICAgIGh0dHBzOi8vYnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0xOTEwMDIKKworICAg ICAgICBNYWtlIGV4cGVjdGF0aW9ucyBmb3IgSW50ZXJzZWN0aW9uT2JzZXJ2ZXIgV1BUcyBtb3Jl IHNwZWNpZmljLgorCisgICAgICAgICogVGVzdEV4cGVjdGF0aW9uczoKKyAgICAgICAgKiBwbGF0 Zm9ybS9ndGsvVGVzdEV4cGVjdGF0aW9uczoKKyAgICAgICAgKiBwbGF0Zm9ybS9tYWMtd2syL1Rl c3RFeHBlY3RhdGlvbnM6CisKIDIwMTgtMTAtMjkgIFRhZGV1IFphZ2FsbG8gIDx0emFnYWxsb0Bh cHBsZS5jb20+CiAKICAgICAgICAgTmV3IGJ5dGVjb2RlIGZvcm1hdCBmb3IgSlNDCmRpZmYgLS1n aXQgYS9MYXlvdXRUZXN0cy9UZXN0RXhwZWN0YXRpb25zIGIvTGF5b3V0VGVzdHMvVGVzdEV4cGVj dGF0aW9ucwppbmRleCA5MjFlOGE3ZWVhOGRmZGU5MjY0YWI4NWU0NjQwOWMwMzFhNzM0MDk5Li4z NDc0NTc4YWQyNWRhZmVjY2Q4OTMyNWFhM2U1NDNhNzcxMzZlOTIyIDEwMDY0NAotLS0gYS9MYXlv dXRUZXN0cy9UZXN0RXhwZWN0YXRpb25zCisrKyBiL0xheW91dFRlc3RzL1Rlc3RFeHBlY3RhdGlv bnMKQEAgLTI4NDYsOCArMjg0Niw2IEBAIHdlYmtpdC5vcmcvYi8xNzU2MDkgaW1wb3J0ZWQvdzNj L3dlYi1wbGF0Zm9ybS10ZXN0cy9JbmRleGVkREIvaWRiY3Vyc29yLWNvbnRpbnVlCiB3ZWJraXQu b3JnL2IvMTc1NjA5IGltcG9ydGVkL3czYy93ZWItcGxhdGZvcm0tdGVzdHMvSW5kZXhlZERCL2lk YmRhdGFiYXNlLWRlbGV0ZU9iamVjdFN0b3JlLWV4Y2VwdGlvbi1vcmRlci5odG0gWyBQYXNzIEZh aWx1cmUgXQogd2Via2l0Lm9yZy9iLzE3NTYwOSBpbXBvcnRlZC93M2Mvd2ViLXBsYXRmb3JtLXRl c3RzL0luZGV4ZWREQi9pZGJvYmplY3RzdG9yZV9nZXRBbGxLZXlzLmh0bWwgWyBQYXNzIEZhaWx1 cmUgXQogCi13ZWJraXQub3JnL2IvMTkwODA4IFsgRGVidWcgXSBpbXBvcnRlZC93M2Mvd2ViLXBs YXRmb3JtLXRlc3RzL2ludGVyc2VjdGlvbi1vYnNlcnZlci9jb250YWluaW5nLWJsb2NrLmh0bWwg WyBQYXNzIEZhaWx1cmUgXQotCiB3ZWJraXQub3JnL2IvMTg2ODQ4IGltcG9ydGVkL3czYy93ZWIt cGxhdGZvcm0tdGVzdHMvRmlsZUFQSS9ibG9iL0Jsb2Itc2xpY2UuaHRtbCBbIFBhc3MgRmFpbHVy ZSBdCiB3ZWJraXQub3JnL2IvMTc5MTc2IHN2Zy93aWNkL3Rlc3QtcmlnaHRzaXppbmctYS54aHRt bCBbIFBhc3MgRmFpbHVyZSBdCiAKQEAgLTI5MTIsNSArMjkxMCwzIEBAIHdlYmtpdC5vcmcvYi8x OTAwMzIgaW1wb3J0ZWQvdzNjL3dlYi1wbGF0Zm9ybS10ZXN0cy93ZWItYW5pbWF0aW9ucy90aW1p bmctbW9kZWwvCiBmYXN0L21lZGlhY2FwdHVyZWZyb21lbGVtZW50L0NhbnZhc0NhcHR1cmVNZWRp YVN0cmVhbS1pbWFnZWJpdG1hcHJlbmRlcmluZ2NvbnRleHQuaHRtbCBbIFNraXAgXQogZmFzdC9t ZWRpYWNhcHR1cmVmcm9tZWxlbWVudC9DYW52YXNDYXB0dXJlTWVkaWFTdHJlYW0tZnJhbWVyYXRl LTAuaHRtbCBbIFNraXAgXQogZmFzdC9tZWRpYWNhcHR1cmVmcm9tZWxlbWVudC9DYW52YXNDYXB0 dXJlTWVkaWFTdHJlYW0tY2FwdHVyZS1vdXQtb2YtRE9NLWVsZW1lbnQuaHRtbCBbIFNraXAgXQot Ci13ZWJraXQub3JnL2IvMTkxMDAyIGltcG9ydGVkL3czYy93ZWItcGxhdGZvcm0tdGVzdHMvaW50 ZXJzZWN0aW9uLW9ic2VydmVyLyBbIFBhc3MgRmFpbHVyZSBdCmRpZmYgLS1naXQgYS9MYXlvdXRU ZXN0cy9wbGF0Zm9ybS9ndGsvVGVzdEV4cGVjdGF0aW9ucyBiL0xheW91dFRlc3RzL3BsYXRmb3Jt L2d0ay9UZXN0RXhwZWN0YXRpb25zCmluZGV4IDAxZDA3YjgzMTAzODM0ZjI0ZGQwNDNjNWRiMzFm OWE4OTJkMDE4ZmEuLjM5YjRjMTNiNTVhNjgxOGM3M2ZmMzg5NTU3YzlkYmU5NWVjMjI3OTIgMTAw NjQ0Ci0tLSBhL0xheW91dFRlc3RzL3BsYXRmb3JtL2d0ay9UZXN0RXhwZWN0YXRpb25zCisrKyBi L0xheW91dFRlc3RzL3BsYXRmb3JtL2d0ay9UZXN0RXhwZWN0YXRpb25zCkBAIC0zNTgyLDYgKzM1 ODIsOCBAQCB3ZWJraXQub3JnL2IvMTkxMDAwIGltcG9ydGVkL3czYy93ZWItcGxhdGZvcm0tdGVz dHMvc2hhZG93LWRvbS9pbnB1dC1lbGVtZW50LWxpcwogCiB3ZWJraXQub3JnL2IvMTkxMDAxIGlt cG9ydGVkL3czYy93ZWItcGxhdGZvcm0tdGVzdHMvaW50ZXJzZWN0aW9uLW9ic2VydmVyL3NhbWUt ZG9jdW1lbnQtcm9vdC5odG1sIFsgRmFpbHVyZSBdCiAKK3dlYmtpdC5vcmcvYi8xOTEwMDIgaW1w b3J0ZWQvdzNjL3dlYi1wbGF0Zm9ybS10ZXN0cy9pbnRlcnNlY3Rpb24tb2JzZXJ2ZXIvIFsgUGFz cyBGYWlsdXJlIF0KKwogd2Via2l0Lm9yZy9iLzE5MTAwMyBpbXBvcnRlZC93M2Mvd2ViLXBsYXRm b3JtLXRlc3RzL21pbWVzbmlmZi9taW1lLXR5cGVzL2NoYXJzZXQtcGFyYW1ldGVyLndpbmRvdy5o dG1sIFsgRmFpbHVyZSBdCiAKIHdlYmtpdC5vcmcvYi8xOTEwMDQgbWVkaWEvdHJhY2svdHJhY2st Y3VlLW1pc3NpbmcuaHRtbCBbIEZhaWx1cmUgXQpkaWZmIC0tZ2l0IGEvTGF5b3V0VGVzdHMvcGxh dGZvcm0vbWFjLXdrMi9UZXN0RXhwZWN0YXRpb25zIGIvTGF5b3V0VGVzdHMvcGxhdGZvcm0vbWFj LXdrMi9UZXN0RXhwZWN0YXRpb25zCmluZGV4IGQwMmRmYmMyMThmOTRiMzY0ZjViY2QyYzhkOWM2 MmMyMTg2ZWUwOWMuLmRjMjBkNDY3ZjU2ZjQ4YTUzNjliMzQxMDQ2YWRiOWZjMWFlNWUwMzcgMTAw NjQ0Ci0tLSBhL0xheW91dFRlc3RzL3BsYXRmb3JtL21hYy13azIvVGVzdEV4cGVjdGF0aW9ucwor KysgYi9MYXlvdXRUZXN0cy9wbGF0Zm9ybS9tYWMtd2syL1Rlc3RFeHBlY3RhdGlvbnMKQEAgLTg0 NCw2ICs4NDQsMTQgQEAgd2Via2l0Lm9yZy9iLzE4NjM2MiBbIFJlbGVhc2UgXSBodHRwL3Rlc3Rz L3Jlc291cmNlTG9hZFN0YXRpc3RpY3MvcHJldmFsZW50LXJlc28KIAogd2Via2l0Lm9yZy9iLzE4 NjQyNSBbIERlYnVnIF0gaW5zcGVjdG9yL2NvbnNvbGUvd2ViY29yZS1sb2dnaW5nLmh0bWwgWyBQ YXNzIEZhaWx1cmUgXQogCit3ZWJraXQub3JnL2IvMTkwODA4IFsgRGVidWcgXSBpbXBvcnRlZC93 M2Mvd2ViLXBsYXRmb3JtLXRlc3RzL2ludGVyc2VjdGlvbi1vYnNlcnZlci9jb250YWluaW5nLWJs b2NrLmh0bWwgWyBQYXNzIEZhaWx1cmUgXQord2Via2l0Lm9yZy9iLzE5MTAwMiBbIERlYnVnIF0g aW1wb3J0ZWQvdzNjL3dlYi1wbGF0Zm9ybS10ZXN0cy9pbnRlcnNlY3Rpb24tb2JzZXJ2ZXIvYm91 bmRpbmctYm94Lmh0bWwgWyBQYXNzIEZhaWx1cmUgXQord2Via2l0Lm9yZy9iLzE5MTAwMiBbIERl YnVnIF0gaW1wb3J0ZWQvdzNjL3dlYi1wbGF0Zm9ybS10ZXN0cy9pbnRlcnNlY3Rpb24tb2JzZXJ2 ZXIvZGlzcGxheS1ub25lLmh0bWwgWyBQYXNzIEZhaWx1cmUgXQord2Via2l0Lm9yZy9iLzE5MTAw MiBbIERlYnVnIF0gaW1wb3J0ZWQvdzNjL3dlYi1wbGF0Zm9ybS10ZXN0cy9pbnRlcnNlY3Rpb24t b2JzZXJ2ZXIvZWRnZS1pbmNsdXNpdmUtaW50ZXJzZWN0aW9uLmh0bWwgWyBQYXNzIEZhaWx1cmUg XQord2Via2l0Lm9yZy9iLzE5MTAwMiBbIERlYnVnIF0gaW1wb3J0ZWQvdzNjL3dlYi1wbGF0Zm9y bS10ZXN0cy9pbnRlcnNlY3Rpb24tb2JzZXJ2ZXIvcmVtb3ZlLWVsZW1lbnQuaHRtbCBbIFBhc3Mg RmFpbHVyZSBdCit3ZWJraXQub3JnL2IvMTkxMDAyIFsgRGVidWcgXSBpbXBvcnRlZC93M2Mvd2Vi LXBsYXRmb3JtLXRlc3RzL2ludGVyc2VjdGlvbi1vYnNlcnZlci9zaGFkb3ctY29udGVudC5odG1s IFsgUGFzcyBGYWlsdXJlIF0KK3dlYmtpdC5vcmcvYi8xOTEwMDIgWyBEZWJ1ZyBdIGltcG9ydGVk L3czYy93ZWItcGxhdGZvcm0tdGVzdHMvaW50ZXJzZWN0aW9uLW9ic2VydmVyL3VuY2xpcHBlZC1y b290Lmh0bWwgWyBQYXNzIEZhaWx1cmUgXQorCiAjIHNraXAgbWFudWFsIHBheW1lbnQtcmVxdWVz dCB0ZXN0cwogaW1wb3J0ZWQvdzNjL3dlYi1wbGF0Zm9ybS10ZXN0cy9wYXltZW50LXJlcXVlc3Qv UGF5bWVudFJlcXVlc3RVcGRhdGVFdmVudC91cGRhdGVXaXRoLWNhbGwtaW1tZWRpYXRlLW1hbnVh bC5odHRwcy5odG1sIFsgU2tpcCBdCiBpbXBvcnRlZC93M2Mvd2ViLXBsYXRmb3JtLXRlc3RzL3Bh eW1lbnQtcmVxdWVzdC9QYXltZW50UmVxdWVzdFVwZGF0ZUV2ZW50L3VwZGF0ZVdpdGgtZHVwbGlj YXRlLXNoaXBwaW5nLW9wdGlvbnMtbWFudWFsLmh0dHBzLmh0bWwgWyBTa2lwIF0K