Bug 202850 – Regression(r250786) API Test TestWebKitAPI.WebKit.DefaultQuota is flaky

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Aakash Jain]

API Test TestWebKitAPI.WebKit.DefaultQuota is failing consistently on macOS Release build.

e.g.:
https://build.webkit.org/builders/Apple%20High%20Sierra%20Release%20WK2%20%28Tests%29/builds/13637/steps/run-api-tests/logs/stdio

https://ews-build.webkit.org/#/builders/3/builds/10036

[Aakash Jain]

This test was modified recently in https://trac.webkit.org/changeset/250728/webkit

[Alex Christensen]

That was a week ago.  Has it been failing consistently since then?  A sampling of the latest API test failures shows it is not failing consistently.

[Jonathan Bedard]

Pretty much always fails on OpenSource High Sierra: <https://results.webkit.org/?suite=api-tests&test=TestWebKitAPI.WebKit.DefaultQuota> (unfortunately, we just brought up the database with the Catalina testers, so we don't have more results....)

[Aakash Jain]

Actually it's not consistently failing, it's very flaky. For e.g.: in https://ews-build.webkit.org/#/builders/3/builds/10056, the test failed in  run-api-tests step, but in immediate retry (re-run-api-tests) it passed.

Similar thing happened in:
https://ews-build.webkit.org/#/builders/3/builds/10054
https://ews-build.webkit.org/#/builders/3/builds/10048
https://ews-build.webkit.org/#/builders/3/builds/10018
https://ews-build.webkit.org/#/builders/3/builds/10005


Also, in following patches, it was flaky in a manner, which caused EWS to wrongly blame the patch being tested:
https://ews-build.webkit.org/#/builders/3/builds/10037
https://ews-build.webkit.org/#/builders/3/builds/10019
https://ews-build.webkit.org/#/builders/3/builds/10014
https://ews-build.webkit.org/#/builders/3/builds/10006

[Aakash Jain]

The first failure seems to be in https://ews-build.webkit.org/#/builders/3/builds/9629
followed by https://ews-build.webkit.org/#/builders/3/builds/9643 and https://ews-build.webkit.org/#/builders/3/builds/9729. All these patches belongs to Bug 202590.

Bug 202590 has red EWS bubble for api-mac, indicating this failure, and that patch was committed. Seems like that commit broke this test.

[Chris Dumez]

(In reply to Aakash Jain from comment #5)
> The first failure seems to be in
> https://ews-build.webkit.org/#/builders/3/builds/9629
> followed by https://ews-build.webkit.org/#/builders/3/builds/9643 and
> https://ews-build.webkit.org/#/builders/3/builds/9729. All these patches
> belongs to Bug 202590.
> 
> Bug 202590 has red EWS bubble for api-mac, indicating this failure, and that
> patch was committed. Seems like that commit broke this test.

I will take a look but Bug 202590 is a PageCache patch and should have nothing to do with quotas.

[Chris Dumez]

The test passes for me locally.

[Chris Dumez]

+Youenn who wrote this test because I doubt this can be related to my PageCache change. Looking at this, I have no idea how it could be related.

[Chris Dumez]

I was able to reproduce the timeout one time. I'll try running the test in a loop for make it more reproducible.

[Chris Dumez]

(In reply to Chris Dumez from comment #9)
> I was able to reproduce the timeout one time. I'll try running the test in a
> loop for make it more reproducible.

So, then it reproduces, with some extra logging I see:
    doTest(10) BEGIN (0)
    doTest(10) END (0)
    doTest(10) BEGIN (1)
    doTest(10) END (1)
    doTest(10) BEGIN (2)
    doTest(10) END (2)
    doTest(10) BEGIN (3)
    doTest(10) END (3)
    doTest(10) BEGIN (4)
    doTest(10) END (4)
    doTest(10) BEGIN (5)
    doTest(10) END (5)
    doTest(10) BEGIN (6)
    doTest(10) END (6)
    doTest(10) BEGIN (7)


So the 7 first calls to doTest() caused us to receive a message back from the page, but the 8th one did not.

[Chris Dumez]

(In reply to Chris Dumez from comment #10)
> (In reply to Chris Dumez from comment #9)
> > I was able to reproduce the timeout one time. I'll try running the test in a
> > loop for make it more reproducible.
> 
> So, then it reproduces, with some extra logging I see:
>     doTest(10) BEGIN (0)
>     doTest(10) END (0)
>     doTest(10) BEGIN (1)
>     doTest(10) END (1)
>     doTest(10) BEGIN (2)
>     doTest(10) END (2)
>     doTest(10) BEGIN (3)
>     doTest(10) END (3)
>     doTest(10) BEGIN (4)
>     doTest(10) END (4)
>     doTest(10) BEGIN (5)
>     doTest(10) END (5)
>     doTest(10) BEGIN (6)
>     doTest(10) END (6)
>     doTest(10) BEGIN (7)
> 
> 
> So the 7 first calls to doTest() caused us to receive a message back from
> the page, but the 8th one did not.

I am making good progress debugging this. I should be able to fix this shortly.

[Chris Dumez]

Created attachment 380798 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 380798 [details]
Patch

Clearing flags on attachment: 380798

Committed r251031: <https://trac.webkit.org/changeset/251031>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/56313729>

[Aakash Jain]

This test flaked again 2 days ago. In https://ews-build.webkit.org/#/builders/3/builds/11094, it timed out in run-api-tests step. However, in the immediately next retry step (re-run-api-tests), it passed.