Bug 160780 - IndexedDB.IndexedDBMultiProcess is a flaky timeout
Summary: IndexedDB.IndexedDBMultiProcess is a flaky timeout
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebCore Misc. (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Alexey Proskuryakov
URL:
Keywords: InRadar
Depends on:
Blocks: 154968
  Show dependency treegraph
 
Reported: 2016-08-11 13:26 PDT by Alexey Proskuryakov
Modified: 2017-03-01 09:25 PST (History)
4 users (show)

See Also:


Attachments
Patch (9.70 KB, patch)
2016-12-02 10:41 PST, Brady Eidson
no flags Details | Formatted Diff | Diff
more fixing (1.79 KB, patch)
2017-02-28 23:19 PST, Alexey Proskuryakov
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey Proskuryakov 2016-08-11 13:26:59 PDT
IndexedDB.IndexedDBMultiProcess still times out: https://build.webkit.org/builders/Apple%20El%20Capitan%20Release%20WK2%20%28Tests%29/builds/8429/steps/run-api-tests/logs/stdio

I don't see any crash log on the server in this case.
Comment 1 Alexey Proskuryakov 2016-09-23 16:37:44 PDT
This reproduces for me pretty easily when running the test in a loop, about 20% of attempts time out:

while true; do run-api-tests --no-build IndexedDB.IndexedDBMultiProcess; done

or with GuardMalloc,

while true; do run-api-tests --no-build --guard-malloc IndexedDB.IndexedDBMultiProcess; done

The database process is not crashing.
Comment 2 Radar WebKit Bug Importer 2016-09-26 09:23:53 PDT
<rdar://problem/28475282>
Comment 3 Alexey Proskuryakov 2016-12-01 13:27:05 PST
This is still quite frequent, I see this failure on two bots right now.

Especially given that this is reproducible, a fix would be really appreciated.
Comment 4 Brady Eidson 2016-12-01 13:29:40 PST
I missed your comment from Sept claiming this is reproducible.

I have still *never* seen it locally, but I haven't sat down to forcefully try.

I will do so in the next couple of days.
Comment 5 Brady Eidson 2016-12-02 09:17:32 PST
I've gotten it to timeout once in 45 runs.

So yes, I can witness it happen. Exploring will still be a bit of a pain.
Comment 6 Brady Eidson 2016-12-02 09:28:29 PST
Taking a different approach.
I need to see logging from the test, so I'm running TestWebKitAPI directly:
DYLD_FRAMEWORK_PATH=~/build/Debug/ ~/build/Debug/TestWebKitAPI --gtest_filter=IndexedDB.IndexedDBMultiProcess --gtest_repeat=-1
Comment 7 Brady Eidson 2016-12-02 10:09:20 PST
Figured it out.

Definitely a bug in the test.
Comment 8 Brady Eidson 2016-12-02 10:41:48 PST
Created attachment 295962 [details]
Patch
Comment 9 Brady Eidson 2016-12-02 10:52:43 PST
This patch will also fix https://bugs.webkit.org/show_bug.cgi?id=161001
Comment 10 Alexey Proskuryakov 2016-12-02 10:55:46 PST
Comment on attachment 295962 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=295962&action=review

> Tools/ChangeLog:11
> +        If two messages came in from the WebProcesses one after another, the runloop in the UI process could
> +        sometimes spin them one after another, eating the first one.

I don't quite understand this explanation, perhaps it can be reworded for clarity. Notably, what does it mean for he runloop to "spin" a message?

The code change looks reasonable.
Comment 11 Brady Eidson 2016-12-02 11:55:52 PST
https://trac.webkit.org/changeset/209251
Comment 12 Ryan Haddad 2016-12-02 20:51:09 PST
Just saw a IndexedDB.IndexedDBMultiProcess timeout as of r209293:

https://build.webkit.org/builders/Apple%20Yosemite%20Release%20WK2%20%28Tests%29/builds/19746/steps/run-api-tests/logs/stdio
Comment 13 Brady Eidson 2016-12-02 21:04:38 PST
(In reply to comment #12)
> Just saw a IndexedDB.IndexedDBMultiProcess timeout as of r209293:
> 
> https://build.webkit.org/builders/
> Apple%20Yosemite%20Release%20WK2%20%28Tests%29/builds/19746/steps/run-api-
> tests/logs/stdio

Mother hell.

I can no longer reproduce locally, so we're back in a holding pattern.
Comment 14 Alexey Proskuryakov 2016-12-05 09:18:55 PST
I've been running this test in a loop over the weekend, and on a Mac Pro, it times out 1% of the time (127 timeouts out of 13527 runs).

This was a release build, no GuardMalloc. Revision 209309.
Comment 15 Brady Eidson 2016-12-05 09:40:49 PST
(In reply to comment #14)
> I've been running this test in a loop over the weekend, and on a Mac Pro, it
> times out 1% of the time (127 timeouts out of 13527 runs).
> 
> This was a release build, no GuardMalloc. Revision 209309.

How were you running it?

I'm willing to give it a shot, but don't want to do something subtly different and therefore unproductive.
Comment 16 Alexey Proskuryakov 2016-12-05 09:49:11 PST
set-webkit-configuration --release
make
while true; do run-api-tests --no-build IndexedDB.IndexedDBMultiProcess; done
Comment 17 Alexey Proskuryakov 2017-02-27 22:31:06 PST
This reproduces nearly every time on some of the bots. Deleting /Users/buildbot/Library/WebKit and /Users/buildbot/Library/Caches/TestWebKitAPI doesn't change it, so I don't see any persistent state for this.
Comment 18 Brady Eidson 2017-02-28 17:18:38 PST
I will not have time to dig deep on this soon.

We should disable it if it's getting in the way of things.
Comment 19 Alexey Proskuryakov 2017-02-28 22:58:34 PST
Easy fix.
Comment 20 Alexey Proskuryakov 2017-02-28 23:19:21 PST
Created attachment 303044 [details]
more fixing

I don't know if this will necessarily fix all the issues, but it doesn't time out in my testing so far.
Comment 21 WebKit Commit Bot 2017-03-01 09:25:30 PST
Comment on attachment 303044 [details]
more fixing

Clearing flags on attachment: 303044

Committed r213225: <http://trac.webkit.org/changeset/213225>
Comment 22 WebKit Commit Bot 2017-03-01 09:25:34 PST
All reviewed patches have been landed.  Closing bug.