NEW 173978
Multiple Layout Tests from media/ tests have become flaky. 
https://bugs.webkit.org/show_bug.cgi?id=173978
Summary Multiple Layout Tests from media/ tests have become flaky.
Matt Lewis
Reported 2017-06-29 11:42:11 PDT
Multiple test from LayoutTest/media/ have become flaky after this revision and are failing together: https://trac.webkit.org/changeset/218903/webkit media/video-click-dblckick-standalone.html [ Failure ] media/video-controls-audiotracks-trackmenu.html [ Failure ] media/video-controls-captions-trackmenu-localized.html [ Failure ] media/video-controls-captions-trackmenu-only-captions-descriptions-and-subtitles.html [ Failure ] media/video-controls-captions-trackmenu-sorted.html [ Failure ] media/video-controls-captions-trackmenu.html [ Failure ] media/video-controls-drop-and-restore-timeline.html [ Failure ] media/video-controls-in-media-document.html [ Failure ] media/video-controls-no-display-with-text-track.html [ Failure ] media/video-controls-rendering.html [ Failure ] https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r218928%20(2615)/results.html https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20%28Tests%29/builds/2615
Attachments
Add attachment proposed patch, testcase, etc.
youenn fablet
Comment 1 2017-06-29 11:45:32 PDT
https://trac.webkit.org/changeset/218903/webkit is only modifying werbrtc which is not tested in the media/xx tests. Maybe antoine will know more here. imported/w3c/web-platform-tests/webrtc/simplecall.html is also failing there. Maybe this is a regression there.
Alexey Proskuryakov
Comment 2 2017-07-02 02:11:25 PDT
What's the next step on this? Should we roll back to confirm whether r218903 was the culprit?
youenn fablet
Comment 3 2017-07-02 10:19:22 PDT
r218903 was rolled out and re landed successfully.
Alexey Proskuryakov
Comment 4 2017-07-03 01:13:49 PDT
The first failure on internal bots was with r218901. A couple preceding changes that may be relevant: https://trac.webkit.org/r218892 https://trac.webkit.org/r218891 As there are multiple tests failing in different ways, it seems like it should be possible to figure out what's going wrong from the failures. Antoine, what do these failures imply? https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r218928%20(2615)/results.html
Alexey Proskuryakov
Comment 5 2017-07-03 01:14:49 PDT
<rdar://problem/33103102>
Antoine Quint
Comment 6 2017-07-05 10:03:55 PDT
None of the tests listed above use modern media controls, they all have <!-- webkit-test-runner [ enableModernMediaControls=false ] -->. Maybe something else broke that makes DRT/WKTR ignore these flags?
Antoine Quint
Comment 7 2017-07-05 10:13:23 PDT
(In reply to Alexey Proskuryakov from comment #4) > As there are multiple tests failing in different ways, it seems like it > should be possible to figure out what's going wrong from the failures. > Antoine, what do these failures imply? The failures are consistent with modern media controls being enabled. Is there a revision that points towards a change in behaviour in dealing with the webkit-test-runner flags?
Antoine Quint
Comment 8 2017-07-05 10:23:30 PDT
Furthermore, it seems that when this happens, all the affected tests are running in the same shard.
Antoine Quint
Comment 9 2017-07-06 02:47:24 PDT
I haven't been able to reproduce the failures locally over 25 runs with 6 WKTR instances.
Alexey Proskuryakov
Comment 10 2017-07-06 11:51:45 PDT
Did you try running the tests in the same order as the failing bot?
Antoine Quint
Comment 11 2017-07-07 01:01:56 PDT
(In reply to Alexey Proskuryakov from comment #10) > Did you try running the tests in the same order as the failing bot? Yes, I ran them in the same order as the initial issue reported, using all the tests ran for worker/0 in https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20%28Tests%29/builds/2615/steps/layout-test/logs/stdio, and I didn't see the failures. I'll try again across multiple runs.
Note You need to log in before you can comment on or make changes to this bug.