Bug 203732

Summary: Flaky API Test TestWebKitAPI.WebKit.ApplicationCacheDirectories
Product: WebKit Reporter: Aakash Jain <aakash_jain>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: RESOLVED DUPLICATE    
Severity: Normal CC: aakash_jain, achristensen, ap, jbedard, webkit-bot-watchers-bugzilla, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   

Description Aakash Jain 2019-11-01 06:02:06 PDT
TestWebKitAPI.WebKit.ApplicationCacheDirectories seems flaky on iOS. In https://ews-build.webkit.org/#/builders/9/builds/11269, the test failed in run-api-tests step. However, in the immediately next retry step (re-run-api-tests), it passed.

Also, this test failed on clean tree in https://ews-build.webkit.org/#/builders/9/builds/11291/steps/13/logs/json
Comment 1 Aakash Jain 2019-11-01 06:03:52 PDT
This test was added 3 months back in https://trac.webkit.org/changeset/247686/webkit
Comment 2 Radar WebKit Bug Importer 2019-11-01 06:20:40 PDT
<rdar://problem/56814641>
Comment 3 Aakash Jain 2019-11-01 06:27:41 PDT
This test flaked in https://ews-build.webkit.org/#/builders/9/builds/11281 as well.
Comment 5 Aakash Jain 2019-11-01 15:13:19 PDT
This test has become too much flaky recently and slowing down EWS.

This failure/flakiness started yesterday, and was first seen in: https://ews-build.webkit.org/#/builders/9/builds/11260

More examples of flakiness/failures:
https://ews-build.webkit.org/#/builders/9/builds/11307
https://ews-build.webkit.org/#/builders/9/builds/11314
https://ews-build.webkit.org/#/builders/9/builds/11318
https://ews-build.webkit.org/#/builders/9/builds/11320
Comment 6 Alexey Proskuryakov 2019-11-03 10:36:01 PST
There isn't much in this test to tie out on, unless TCPServer is broken...
Comment 7 Alexey Proskuryakov 2019-11-03 10:37:05 PST
Looks like another one that isn't failing in post-commit testing.
Comment 8 Aakash Jain 2019-11-03 15:31:59 PST
(In reply to Alexey Proskuryakov from comment #7)
> Looks like another one that isn't failing in post-commit testing.
Yes, also this test isn't failing on EWS bots which are on Catalina (ews156 and ews157), it is only failing/flaky on bots which are on Mojave (ews158 and ews159).
Comment 9 Alex Christensen 2019-11-04 03:28:32 PST
Do we know when this flakiness started?
I don't have a Mojave machine with me, so this won't be investigated this week.  Could we disable this test on Mojave for now?
Comment 10 Aakash Jain 2019-11-13 05:46:00 PST
Committed r252404: <https://trac.webkit.org/changeset/252404>
Comment 11 Aakash Jain 2019-11-13 05:46:53 PST
(In reply to Alex Christensen from comment #9)
> Do we know when this flakiness started?
First failure was noticed in https://ews-build.webkit.org/#/builders/9/builds/11260 on Oct 31st.

> I don't have a Mojave machine with me, so this won't be investigated this
> week.  Could we disable this test on Mojave for now?
I don't think we have a way to disable API tests on specific platforms. I have disabled this test for now.

This is still flaking frequently and causing false-positives on EWS:
https://ews-build.webkit.org/#/builders/9/builds/11936
https://ews-build.webkit.org/#/builders/9/builds/11928
https://ews-build.webkit.org/#/builders/9/builds/11884
https://ews-build.webkit.org/#/builders/9/builds/11876
Comment 12 Alex Christensen 2019-11-13 15:51:55 PST

*** This bug has been marked as a duplicate of bug 203734 ***