Bug 173978

Summary: Multiple Layout Tests from media/ tests have become flaky.
Product: WebKit Reporter: Matt Lewis <jlewis3>
Component: New BugsAssignee: Nobody <webkit-unassigned>
Status: NEW ---    
Severity: Normal CC: ap, graouts, ryanhaddad, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=175700
https://bugs.webkit.org/show_bug.cgi?id=176054
https://bugs.webkit.org/show_bug.cgi?id=176122

Description Matt Lewis 2017-06-29 11:42:11 PDT
Multiple test from LayoutTest/media/ have become flaky after this revision and are failing together: https://trac.webkit.org/changeset/218903/webkit

  media/video-click-dblckick-standalone.html [ Failure ]
  media/video-controls-audiotracks-trackmenu.html [ Failure ]
  media/video-controls-captions-trackmenu-localized.html [ Failure ]
  media/video-controls-captions-trackmenu-only-captions-descriptions-and-subtitles.html [ Failure ]
  media/video-controls-captions-trackmenu-sorted.html [ Failure ]
  media/video-controls-captions-trackmenu.html [ Failure ]
  media/video-controls-drop-and-restore-timeline.html [ Failure ]
  media/video-controls-in-media-document.html [ Failure ]
  media/video-controls-no-display-with-text-track.html [ Failure ]
  media/video-controls-rendering.html [ Failure ]


https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r218928%20(2615)/results.html
https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20%28Tests%29/builds/2615
Comment 1 youenn fablet 2017-06-29 11:45:32 PDT
https://trac.webkit.org/changeset/218903/webkit is only modifying werbrtc which is not tested in the media/xx tests. Maybe antoine will know more here.

imported/w3c/web-platform-tests/webrtc/simplecall.html is also failing there.
Maybe this is a regression there.
Comment 2 Alexey Proskuryakov 2017-07-02 02:11:25 PDT
What's the next step on this? Should we roll back to confirm whether r218903 was the culprit?
Comment 3 youenn fablet 2017-07-02 10:19:22 PDT
r218903 was rolled out and re landed successfully.
Comment 4 Alexey Proskuryakov 2017-07-03 01:13:49 PDT
The first failure on internal bots was with r218901. 

A couple preceding changes that may be relevant:

https://trac.webkit.org/r218892
https://trac.webkit.org/r218891

As there are multiple tests failing in different ways, it seems like it should be possible to figure out what's going wrong from the failures. Antoine, what do these failures imply?

https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r218928%20(2615)/results.html
Comment 5 Alexey Proskuryakov 2017-07-03 01:14:49 PDT
<rdar://problem/33103102>
Comment 6 Antoine Quint 2017-07-05 10:03:55 PDT
None of the tests listed above use modern media controls, they all have <!-- webkit-test-runner [ enableModernMediaControls=false ] -->. Maybe something else broke that makes DRT/WKTR ignore these flags?
Comment 7 Antoine Quint 2017-07-05 10:13:23 PDT
(In reply to Alexey Proskuryakov from comment #4)
> As there are multiple tests failing in different ways, it seems like it
> should be possible to figure out what's going wrong from the failures.
> Antoine, what do these failures imply?

The failures are consistent with modern media controls being enabled. Is there a revision that points towards a change in behaviour in dealing with the webkit-test-runner flags?
Comment 8 Antoine Quint 2017-07-05 10:23:30 PDT
Furthermore, it seems that when this happens, all the affected tests are running in the same shard.
Comment 9 Antoine Quint 2017-07-06 02:47:24 PDT
I haven't been able to reproduce the failures locally over 25 runs with 6 WKTR instances.
Comment 10 Alexey Proskuryakov 2017-07-06 11:51:45 PDT
Did you try running the tests in the same order as the failing bot?
Comment 11 Antoine Quint 2017-07-07 01:01:56 PDT
(In reply to Alexey Proskuryakov from comment #10)
> Did you try running the tests in the same order as the failing bot?

Yes, I ran them in the same order as the initial issue reported, using all the tests ran for worker/0 in https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20%28Tests%29/builds/2615/steps/layout-test/logs/stdio, and I didn't see the failures. I'll try again across multiple runs.