[macOS WK2] LayoutTest webaudio/silent-audio-interrupted-in-background.html is a flaky timeout on older OSes (El Capitan / Sierra)

[Matt Lewis]

The test webaudio/silent-audio-interrupted-in-background.html is a flaky timeout on macOS wk2.

The test was last edited in r214721
https://trac.webkit.org/changeset/214721/webkit

https://build.webkit.org/builders/Apple%20Sierra%20Release%20WK2%20(Tests)/builds/1525
https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r216951%20(1525)/results.html

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=webaudio%2Fsilent-audio-interrupted-in-background.html

diff:
--- /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/webaudio/silent-audio-interrupted-in-background-expected.txt
+++ /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/webaudio/silent-audio-interrupted-in-background-actual.txt
@@ -1,14 +1,5 @@
-Tests that silent WebAudio rendering gets interrupted in hidden pages.
+#PID UNRESPONSIVE - com.apple.WebKit.WebContent.Development (pid 95499)
+FAIL: Timed out waiting for notifyDone to be called
 
-On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
-
-
-PASS context.state became 'running'
-* Setting page visibility to hidden
-PASS context.state became 'interrupted'
-* Setting page visibility to visible
-PASS context.state became 'running'
-PASS successfullyParsed is true
-
-TEST COMPLETE
-
+#EOF
+#EOF

[Matt Lewis]

I was able to reproduce the timeout on Sierra Release with:

run-webkit-tests webaudio/silent-audio-interrupted-in-background.html -f --iterations=100 --no-retry-failure -g

It is flaky so it did occasionally pass all test.

[Alexey Proskuryakov]

+#PID UNRESPONSIVE - com.apple.WebKit.WebContent.Development (pid 95499)

I believe that this error means that the issue is caused by the preceding test. Perhaps both this and preceding test can cause this?

Definitely seems worth a deeper look.

[Alexey Proskuryakov]

Sorry, I misread. "Timed out waiting for notifyDone to be called" is normally about the current test.

[Alexey Proskuryakov]

This test is normally very fast, but times out fairly frequently.

[Chris Dumez]

I will take a look soon.

[Radar WebKit Bug Importer]

<rdar://problem/32239545>

[Chris Dumez]

(In reply to Matt Lewis from comment #1)
> I was able to reproduce the timeout on Sierra Release with:
> 
> run-webkit-tests webaudio/silent-audio-interrupted-in-background.html -f
> --iterations=100 --no-retry-failure -g
> 
> It is flaky so it did occasionally pass all test.

I have tried this repeatedly:
Tools/Scripts/run-webkit-tests webaudio/silent-audio-interrupted-in-background.html -f --iterations=1000 --no-retry-failure -g

So far, no failures though :/

[Chris Dumez]

(In reply to Chris Dumez from comment #7)
> (In reply to Matt Lewis from comment #1)
> > I was able to reproduce the timeout on Sierra Release with:
> > 
> > run-webkit-tests webaudio/silent-audio-interrupted-in-background.html -f
> > --iterations=100 --no-retry-failure -g
> > 
> > It is flaky so it did occasionally pass all test.
> 
> I have tried this repeatedly:
> Tools/Scripts/run-webkit-tests
> webaudio/silent-audio-interrupted-in-background.html -f --iterations=1000
> --no-retry-failure -g
> 
> So far, no failures though :/

Looks like I have managed to reproduce with:
Tools/Scripts/run-webkit-tests webaudio/silent-audio-interrupted-in-background.html -f --iterations=1000 --no-retry-failure -g --child-processes=50

[Chris Dumez]

This is somehow related to page visibility. The onvisibilitychange event handler on document sometimes does not get called after calling testRunner.setPageVisibility("hidden").

[Chris Dumez]

(In reply to Chris Dumez from comment #9)
> This is somehow related to page visibility. The onvisibilitychange event
> handler on document sometimes does not get called after calling
> testRunner.setPageVisibility("hidden").

When setting the page visibility to hidden, we end up calling:
void TestController::setHidden(bool hidden)
{
    NSWindow *window = [mainWebView()->platformView() window];
    if (!window)
        return;

    if (hidden)
        [window orderOut:nil];
    else
        [window makeKeyAndOrderFront:nil];
}

[Ryan Haddad]

Marked test as flaky in http://trac.webkit.org/projects/webkit/changeset/218706

[Alexey Proskuryakov]

It continues to be a flaky timeout on new OSes.

[youenn fablet]

It is no longer flaky on bots

[Alexey Proskuryakov]

Seems like flaky expectations still need to be removed?