Bug 173978 – Multiple Layout Tests from media/ tests have become flaky.

• Component: New Bugs
• Status: NEW

[Matt Lewis]

Multiple test from LayoutTest/media/ have become flaky after this revision and are failing together: https://trac.webkit.org/changeset/218903/webkit

  media/video-click-dblckick-standalone.html [ Failure ]
  media/video-controls-audiotracks-trackmenu.html [ Failure ]
  media/video-controls-captions-trackmenu-localized.html [ Failure ]
  media/video-controls-captions-trackmenu-only-captions-descriptions-and-subtitles.html [ Failure ]
  media/video-controls-captions-trackmenu-sorted.html [ Failure ]
  media/video-controls-captions-trackmenu.html [ Failure ]
  media/video-controls-drop-and-restore-timeline.html [ Failure ]
  media/video-controls-in-media-document.html [ Failure ]
  media/video-controls-no-display-with-text-track.html [ Failure ]
  media/video-controls-rendering.html [ Failure ]


https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r218928%20(2615)/results.html
https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20%28Tests%29/builds/2615

[youenn fablet]

https://trac.webkit.org/changeset/218903/webkit is only modifying werbrtc which is not tested in the media/xx tests. Maybe antoine will know more here.

imported/w3c/web-platform-tests/webrtc/simplecall.html is also failing there.
Maybe this is a regression there.

[Alexey Proskuryakov]

What's the next step on this? Should we roll back to confirm whether r218903 was the culprit?

[youenn fablet]

r218903 was rolled out and re landed successfully.

[Alexey Proskuryakov]

The first failure on internal bots was with r218901. 

A couple preceding changes that may be relevant:

https://trac.webkit.org/r218892
https://trac.webkit.org/r218891

As there are multiple tests failing in different ways, it seems like it should be possible to figure out what's going wrong from the failures. Antoine, what do these failures imply?

https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r218928%20(2615)/results.html

[Alexey Proskuryakov]

<rdar://problem/33103102>

[Antoine Quint]

None of the tests listed above use modern media controls, they all have <!-- webkit-test-runner [ enableModernMediaControls=false ] -->. Maybe something else broke that makes DRT/WKTR ignore these flags?

[Antoine Quint]

(In reply to Alexey Proskuryakov from comment #4)
> As there are multiple tests failing in different ways, it seems like it
> should be possible to figure out what's going wrong from the failures.
> Antoine, what do these failures imply?

The failures are consistent with modern media controls being enabled. Is there a revision that points towards a change in behaviour in dealing with the webkit-test-runner flags?

[Antoine Quint]

Furthermore, it seems that when this happens, all the affected tests are running in the same shard.

[Antoine Quint]

I haven't been able to reproduce the failures locally over 25 runs with 6 WKTR instances.

[Alexey Proskuryakov]

Did you try running the tests in the same order as the failing bot?

[Antoine Quint]

(In reply to Alexey Proskuryakov from comment #10)
> Did you try running the tests in the same order as the failing bot?

Yes, I ran them in the same order as the initial issue reported, using all the tests ran for worker/0 in https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20%28Tests%29/builds/2615/steps/layout-test/logs/stdio, and I didn't see the failures. I'll try again across multiple runs.

[Ryan Haddad]

Examples from today:
https://build.webkit.org/results/Apple%20El%20Capitan%20Debug%20WK2%20(Tests)/r221186%20(2720)/results.html
https://build.webkit.org/results/Apple%20El%20Capitan%20Debug%20WK2%20(Tests)/r221185%20(2719)/results.html
https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r221190%20(3851)/results.html