Bug 200086 – REGRESSION (r247486?): Flaky API Test TestWebKitAPI.WKWebView.LocalStorageProcessSuspends

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Aakash Jain]

TestWebKitAPI.WKWebView.LocalStorageProcessSuspends seems flaky. In https://ews-build.webkit.org/#/builders/9/builds/4903, the test Failed in run-api-tests step. However, in the immediately next retry step (re-run-api-tests), it passed.

Same thing happened in https://ews-build.webkit.org/#/builders/9/builds/4901

In https://ews-build.webkit.org/#/builders/9/builds/4784, it just failed consistently.

[Ryan Haddad]

It can also be seen failing on a trunk bot here:
https://build.webkit.org/builders/Apple%20Mojave%20Debug%20WK2%20%28Tests%29/builds/3730/steps/run-api-tests/logs/stdio

[Radar WebKit Bug Importer]

<rdar://problem/53501721>

[Ryan Haddad]

This is indeed quite flaky. The bots are often red because of this failure.

    TestWebKitAPI.WKWebView.LocalStorageProcessSuspends
        
        /Volumes/Data/slave/ios-simulator-12-release/build/Tools/TestWebKitAPI/Tests/WebKitCocoa/LocalStoragePersistence.mm:156
        Expected equality of these values:
          @"value"
            Which is: "value"
          result
            Which is: "newValue"

[Aakash Jain]

This test is creating issue in EWS infrastrucutre. It is extremely flaky and various times flakes in a manner that misleads EWS into blaming the patch being tested (this test failed consistently twice with the patch, and pass without the patch).

e.g.:
https://ews-build.webkit.org/#/builders/9/builds/5177
https://ews-build.webkit.org/#/builders/9/builds/5145
https://ews-build.webkit.org/#/builders/9/builds/5140
https://ews-build.webkit.org/#/builders/9/builds/5109

[Sihui Liu]

(In reply to Aakash Jain from comment #4)
> This test is creating issue in EWS infrastrucutre. It is extremely flaky and
> various times flakes in a manner that misleads EWS into blaming the patch
> being tested (this test failed consistently twice with the patch, and pass
> without the patch).
> 
> e.g.:
> https://ews-build.webkit.org/#/builders/9/builds/5177
> https://ews-build.webkit.org/#/builders/9/builds/5145
> https://ews-build.webkit.org/#/builders/9/builds/5140
> https://ews-build.webkit.org/#/builders/9/builds/5109

Do you mean this API test always passes without those patches being tested but fails with them? It sounds like some states are kept between test runs..

Does this happen since last week? If this starts to happen recently, it may help narrow down the blamed changes.

[Ryan Haddad]

FWIW this isn't specific to iOS.

Based on history, this may have started after:

Speed up StorageManager::getValues()
https://trac.webkit.org/changeset/247486/webkit

[Chris Dumez]

(In reply to Ryan Haddad from comment #6)
> FWIW this isn't specific to iOS.
> 
> Based on history, this may have started after:
> 
> Speed up StorageManager::getValues()
> https://trac.webkit.org/changeset/247486/webkit

Ok, so probably making it a WorkQueueMessageReceiver again.

[Aakash Jain]

> Do you mean this API test always passes without those patches being tested
> but fails with them?
Not always. It flakes in various different manners. This particular manner is particularly bad for EWS (as it incorrectly blames a patch).

In following examples, the test is clearly flaky; it failed in first run (run-api-tests step), but in immediate retry step (in re-run-api-tests) this test passed:

https://ews-build.webkit.org/#/builders/9/builds/5199
https://ews-build.webkit.org/#/builders/9/builds/5183
https://ews-build.webkit.org/#/builders/9/builds/5179
https://ews-build.webkit.org/#/builders/9/builds/5176
https://ews-build.webkit.org/#/builders/9/builds/5170
https://ews-build.webkit.org/#/builders/9/builds/5163
https://ews-build.webkit.org/#/builders/9/builds/5157
https://ews-build.webkit.org/#/builders/9/builds/5151
https://ews-build.webkit.org/#/builders/9/builds/5123
https://ews-build.webkit.org/#/builders/9/builds/5120
https://ews-build.webkit.org/#/builders/9/builds/5119
https://ews-build.webkit.org/#/builders/9/builds/5114
https://ews-build.webkit.org/#/builders/9/builds/5103
https://ews-build.webkit.org/#/builders/9/builds/5102

> 
> Does this happen since last week? If this starts to happen recently, it may
> help narrow down the blamed changes.

Seems to be happening atleast since July 16:
https://ews-build.webkit.org/#/builders/9/builds/4550
https://ews-build.webkit.org/#/builders/9/builds/4547
https://ews-build.webkit.org/#/builders/9/builds/4545

[Chris Dumez]

Created attachment 375227 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 375227 [details]
Patch

Clearing flags on attachment: 375227

Committed r248047: <https://trac.webkit.org/changeset/248047>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.