Bug 279747 – REGRESSION (October 2023): [ macOS wk2 ] editing/input/scroll-viewport-page-up-down.html is a flaky text failure

• Status: NEW
• Component: HTML Editing

[Fady Farag]

editing/input/scroll-viewport-page-up-down.html

This test is flakily failing on macOS wk2.

HISTORY:

https://results.webkit.org/?suite=layout-tests&test=editing%2Finput%2Fscroll-viewport-page-up-down.html&platform=mac

TEXT DIFF:

--- /Volumes/Data/worker/Apple-Ventura-Release-WK2-Tests/build/layout-test-results/editing/input/scroll-viewport-page-up-down-expected.txt
+++ /Volumes/Data/worker/Apple-Ventura-Release-WK2-Tests/build/layout-test-results/editing/input/scroll-viewport-page-up-down-actual.txt
@@ -1,3 +1,4 @@
+CONSOLE MESSAGE: Frame viewport should be around 75px , not at 0
 Page up/down (option+page up/down on Mac) should move the move cursor and scroll the content in contenteditable elements. This sample covers scroll position test of a) iframe element containing contenteditable body and b) content editable div element. Even though the cursor will move exactly to the same location on all platforms (covered by test option-page-up-down.html), please note that Mac will scroll the visible area by placing the cursor position in the middle. All other platforms will scroll by keeping the cursor aligned with the top edge of the visible area.
 
 line 0
@@ -21,5 +22,4 @@
 line 9
 
 iframe PASS
-div PASS

DIFF URL:

https://build.webkit.org/results/Apple-Ventura-Release-WK2-Tests/283610@main%20(8094)/editing/input/scroll-viewport-page-up-down-pretty-diff.html

REGRESSION:

The regression point seems to be after 269887@main landed.

[Radar WebKit Bug Importer]

<rdar://problem/136051507>

[Fady Farag]

Pull request: https://github.com/WebKit/WebKit/pull/33684

[Alexey Proskuryakov]

I don't think that this regression point is likely. How did you determine it?

[Fady Farag]

(In reply to Alexey Proskuryakov from comment #3)
> I don't think that this regression point is likely. How did you determine it?

It was based on the fact that there was not a single failure before 269887@main landed. The test started to exhibit flaky behavior after that commit landed according to the results database.

[EWS]

Test gardening commit 283784@main (2f2ae76988ba): <https://commits.webkit.org/283784@main>

Reviewed commits have been landed. Closing PR #33684 and removing active labels.

[Alexey Proskuryakov]

Thank you for the clarification. When we blame a specific revision, this means that to the best of our knowledge, this is the revision that caused the problem. For flaky tests, blaming the revision of first failure is typically inaccurate.

[Fady Farag]

(In reply to Alexey Proskuryakov from comment #6)
> Thank you for the clarification. When we blame a specific revision, this
> means that to the best of our knowledge, this is the revision that caused
> the problem. For flaky tests, blaming the revision of first failure is
> typically inaccurate.

Interesting. Thank you for making that clear. Any feedback on how to accurately identify regression points for flaky tests? Would it be even acceptable to provide a range? In case specific releases were not built or there was a build failure in between them.

[Alexey Proskuryakov]

Sometimes people reproduce locally to bisect. Other times, we just post the rough estimate that we have (as I did when changing to the title "October 2023" here). What's important is not to post information that can be misleading.

Let's take further process discussions elsewhere, as they are not relevant to people who will be investigating this scrolling problem.