Bug 167263 – LayoutTest media/modern-media-controls/media-controller/media-controller-auto-hide.html is flaky

• Component: New Bugs
• Status: RESOLVED FIXED

[Ryan Haddad]

LayoutTest media/modern-media-controls/media-controller/media-controller-auto-hide.html is flaky

https://build.webkit.org/results/Apple%20El%20Capitan%20Debug%20WK1%20(Tests)/r210986%20(10966)/results.html

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=media%2Fmodern-media-controls%2Fmedia-controller%2Fmedia-controller-auto-hide.html

--- /Volumes/Data/slave/elcapitan-debug-tests-wk1/build/layout-test-results/media/modern-media-controls/media-controller/media-controller-auto-hide-expected.txt
+++ /Volumes/Data/slave/elcapitan-debug-tests-wk1/build/layout-test-results/media/modern-media-controls/media-controller/media-controller-auto-hide-actual.txt
@@ -7,7 +7,7 @@
 PASS controlsBar.classList.contains('faded') is false
 
 Auto-hide timer has been going for 200ms, controls bar should still be visible.
-PASS controlsBar.classList.contains('faded') is false
+FAIL controlsBar.classList.contains('faded') should be false. Was true.
 
 Auto-hide timer has elapsed, controls bar should be faded.
 PASS controlsBar.classList.contains('faded') is true

[Ryan Haddad]

Marked as flaky in https://trac.webkit.org/changeset/210994

[Antoine Quint]

This is tricky because this test, and a host of others, is timing-sensitive by design since it's testing the auto-hide timer. The test checks as close as possible to the auto-hide delay that the controls are still visible, and then again as soon as possible after the auto-hide delay has elapsed to check that the controls are no longer visible. So in certain cases, probably where the system is under pressure and timers run late, we get in cases where the check prior to the timer firing fails because we've actually run past auto-hide delay. We're going to have to find a different way to test this.

[Antoine Quint]

OK, I think I understand what makes this test flaky.

First, the test makes the assumption that the auto-hide timer can only be started as a result of the "play" event firing… but it turns out that when "loadedmetadata" fires, media.paused can be set to "false", and ControlsVisibilitySupport which governs when the controls bar is set to have fadesWhileIdle = true, uses both "loadedmetadata" and "play" as triggers provided media.paused = "false". So that's the first issue, since we could potentially wait enough time until we'd assume the auto-hide timer started to make our 200ms and 300ms timers off-base enough.

Second, while the system is under stress (in my case I have a full WK build and running the test with GuardMalloc), there might be a significant delta between the moment the same event is handled within the media controls code and when it's handled in the test. Locally, I can see something around 85ms as the delay. This would cause another significant delay to make the 200ms and 300ms timers off-base.

So this test is definitely flaky by nature, and all of the other auto-hide tests, many of which are showing flakiness, suffer from the same design flaw. I'll need to think of a better, more robust way to write those tests.

[Radar WebKit Bug Importer]

<rdar://problem/30821398>

[Antoine Quint]

Created attachment 303270 [details]
Patch

[Antoine Quint]

Created attachment 303275 [details]
Patch for landing

[WebKit Commit Bot]

Comment on attachment 303275 [details]
Patch for landing

Clearing flags on attachment: 303275

Committed r213337: <http://trac.webkit.org/changeset/213337>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.