Bug 172034 – [macOS Sierra Release] LayoutTest webrtc/closing-peerconnection.html is a flaky timeout.

• Component: New Bugs
• Status: NEW

[Matt Lewis]

The test webrtc/closing-peerconnection.html is a flaky timeout on Sierra Release Wk2.

This doesn't appear to be a recent regression. The test was last edited in https://trac.webkit.org/changeset/215558/webkit

History:
https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=webrtc%2Fclosing-peerconnection.html

Build:
https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20(Tests)/builds/1418
https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r216765%20(1418)/results.html

Diff:
--- /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/webrtc/closing-peerconnection-expected.txt
+++ /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/webrtc/closing-peerconnection-actual.txt
@@ -1,3 +1,5 @@
+#PID UNRESPONSIVE - com.apple.WebKit.WebContent.Development (pid 24347)
+FAIL: Timed out waiting for notifyDone to be called
 
-PASS closing and stopping peer connection in the middle of gathering candidates 
-
+#EOF
+#EOF

[Matt Lewis]

Marked test as flaky.
https://trac.webkit.org/changeset/216767/webkit

[youenn fablet]

(In reply to Matt Lewis from comment #1)
> Marked test as flaky.
> https://trac.webkit.org/changeset/216767/webkit

I think it is fine to create a bugzilla for it and CC me.
Adding the TestExpectation too quickly hinder the possibility to catch regressions if the test starts to always time out.
We need to find the right balance between raising too many false alarms and missing true alarms.

In that particular case, I would think we should add the test expectation only if is timing out a few times.

Not sure if there are some kinds of rule for that.

[Matt Lewis]

(In reply to youenn fablet from comment #2)
> (In reply to Matt Lewis from comment #1)
> > Marked test as flaky.
> > https://trac.webkit.org/changeset/216767/webkit
> 
> I think it is fine to create a bugzilla for it and CC me.
> Adding the TestExpectation too quickly hinder the possibility to catch
> regressions if the test starts to always time out.
> We need to find the right balance between raising too many false alarms and
> missing true alarms.
> 
> In that particular case, I would think we should add the test expectation
> only if is timing out a few times.
> 
> Not sure if there are some kinds of rule for that.

Sure thing! I'll keep that in mind!