Bug 225665 – [ BigSur Release wk2 arm64 & iOS] http/tests/appcache/fail-on-update-2.html (layout-test) is a flaky timeout

• Status: NEW
• Component: New Bugs

[Robert Jenner]

http/tests/appcache/fail-on-update-2.html

is a flaky timeout on BigSur Release wk2 on Apple Silicon Macs only. 

HISTORY:
https://results.webkit.org/?suite=layout-tests&test=http%2Ftests%2Fappcache%2Ffail-on-update-2.html

TIMEOUT TEXT:
-SUCCESS: No crash.
+#PID UNRESPONSIVE - WebKitTestRunner (pid 21811)
+FAIL: Timed out waiting for notifyDone to be called
+
+#EOF
+#EOF


STDERR TEXT:
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: Application Cache manifest could not be fetched, because the manifest had a 404 response.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: Application Cache manifest could not be fetched, because the manifest had a 404 response.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: Application Cache manifest could not be fetched, because the manifest had a 404 response.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: Application Cache manifest could not be fetched, because the manifest had a 404 response.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: Application Cache manifest could not be fetched, because the manifest had a 404 response.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: Application Cache manifest could not be fetched, because the manifest had a 404 response.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: Application Cache manifest could not be fetched, because the manifest had a 404 response.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: Application Cache manifest could not be fetched, because the manifest had a 404 response.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
CONSOLE MESSAGE: Application Cache manifest could not be fetched, because the manifest had a 404 response.
CONSOLE MESSAGE: ApplicationCache is deprecated. Please use ServiceWorkers instead.
Wait on notifyDone timed out, process WebKitTestRunner pid = 21811#EOF

[Robert Jenner]

There were some consistent timeouts on Intel Macs. But they stopped, and haven't occurred for over a month. I did attempt to reproduce it at BigSur Release ToT wk1, but was unable to. It doesn't appear this test is having a problem on Intel anymore, only Apple Silicon. 

Since this appears to only occur on Apple Silicon, I can't reproduce the timeout, as I don't have access to said system type. 

I have gone ahead and updated the test expectations to Pass Timeout here:
https://trac.webkit.org/changeset/277339/webkit

[Radar WebKit Bug Importer]

<rdar://problem/77861895>

[Robert Jenner]

As it turns out, there is an expectation set for this test in: sources/OpenSource/LayoutTests/TestExpectations

The expectation is set as:
[ DumpJSConsoleLogInStdErr ]

That expectation is set to hide the deprecation error from the stdout, and puts it as a stderr instead. When I updated my expectation it over-rode that and caused constant text failures. 

Updated test expectations to DumpJSConsoleLogInStdErr Slow here:
https://trac.webkit.org/changeset/277410/webkit

This should stop the constant text failures, and help confirm if these timeouts are actual timeouts, or just a test being slow.

[Robert Jenner]

After marking the test is slow, it appears to have actually gotten significantly faster. 

Before marking as slow, the test would take anywhere between 3 to 19 seconds to pass, and would timeout at roughly one minute. Since marking as slow, the test has been passing only taking between .86 seconds and 3.26 seconds for the test to run and pass. I know we also were looking at the ram usage on this particular Mac mini, and found at the time that it was using approx. 10 gigs of its ram. We have addressed this issue, and the test is no longer timing out.

The same flaky timeout is seen on iOS 14 E Simulator wk2. 

I was not able to reproduce the timeout on simulator (iOS 15) on my local machine using run-webkit-tests --ios-simulator --iterations 500  --exit-after-n-crashes-or-timeouts 1 --exit-after-n-failures 1 http/tests/appcache/fail-on-update-2.html -f --release --no-build --clobber-old-results

The flaky timeout on iOS is seen since 5/6/21 (No test results are available on iOS before 5/6/21)

Updated test expectations for ios-wk2 https://trac.webkit.org/changeset/280148/webkit

[Eric Hutchison]

Test is still flaking on iOS15; marking as timeout. https://trac.webkit.org/changeset/285266/webkit