Bug 160780 – IndexedDB.IndexedDBMultiProcess is a flaky timeout

• Component: WebCore Misc.
• Status: RESOLVED FIXED

[Alexey Proskuryakov]

IndexedDB.IndexedDBMultiProcess still times out: https://build.webkit.org/builders/Apple%20El%20Capitan%20Release%20WK2%20%28Tests%29/builds/8429/steps/run-api-tests/logs/stdio

I don't see any crash log on the server in this case.

[Alexey Proskuryakov]

This reproduces for me pretty easily when running the test in a loop, about 20% of attempts time out:

while true; do run-api-tests --no-build IndexedDB.IndexedDBMultiProcess; done

or with GuardMalloc,

while true; do run-api-tests --no-build --guard-malloc IndexedDB.IndexedDBMultiProcess; done

The database process is not crashing.

[Radar WebKit Bug Importer]

<rdar://problem/28475282>

[Alexey Proskuryakov]

This is still quite frequent, I see this failure on two bots right now.

Especially given that this is reproducible, a fix would be really appreciated.

[Brady Eidson]

I missed your comment from Sept claiming this is reproducible.

I have still *never* seen it locally, but I haven't sat down to forcefully try.

I will do so in the next couple of days.

[Brady Eidson]

I've gotten it to timeout once in 45 runs.

So yes, I can witness it happen. Exploring will still be a bit of a pain.

[Brady Eidson]

Taking a different approach.
I need to see logging from the test, so I'm running TestWebKitAPI directly:
DYLD_FRAMEWORK_PATH=~/build/Debug/ ~/build/Debug/TestWebKitAPI --gtest_filter=IndexedDB.IndexedDBMultiProcess --gtest_repeat=-1

[Brady Eidson]

Figured it out.

Definitely a bug in the test.

[Brady Eidson]

Created attachment 295962 [details]
Patch

[Brady Eidson]

This patch will also fix https://bugs.webkit.org/show_bug.cgi?id=161001

[Alexey Proskuryakov]

Comment on attachment 295962 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=295962&action=review

> Tools/ChangeLog:11
> +        If two messages came in from the WebProcesses one after another, the runloop in the UI process could
> +        sometimes spin them one after another, eating the first one.

I don't quite understand this explanation, perhaps it can be reworded for clarity. Notably, what does it mean for he runloop to "spin" a message?

The code change looks reasonable.

[Brady Eidson]

https://trac.webkit.org/changeset/209251

[Ryan Haddad]

Just saw a IndexedDB.IndexedDBMultiProcess timeout as of r209293:

https://build.webkit.org/builders/Apple%20Yosemite%20Release%20WK2%20%28Tests%29/builds/19746/steps/run-api-tests/logs/stdio

[Brady Eidson]

(In reply to comment #12)
> Just saw a IndexedDB.IndexedDBMultiProcess timeout as of r209293:
> 
> https://build.webkit.org/builders/
> Apple%20Yosemite%20Release%20WK2%20%28Tests%29/builds/19746/steps/run-api-
> tests/logs/stdio

Mother hell.

I can no longer reproduce locally, so we're back in a holding pattern.

[Alexey Proskuryakov]

I've been running this test in a loop over the weekend, and on a Mac Pro, it times out 1% of the time (127 timeouts out of 13527 runs).

This was a release build, no GuardMalloc. Revision 209309.

[Brady Eidson]

(In reply to comment #14)
> I've been running this test in a loop over the weekend, and on a Mac Pro, it
> times out 1% of the time (127 timeouts out of 13527 runs).
> 
> This was a release build, no GuardMalloc. Revision 209309.

How were you running it?

I'm willing to give it a shot, but don't want to do something subtly different and therefore unproductive.

[Alexey Proskuryakov]

set-webkit-configuration --release
make
while true; do run-api-tests --no-build IndexedDB.IndexedDBMultiProcess; done

[Alexey Proskuryakov]

This reproduces nearly every time on some of the bots. Deleting /Users/buildbot/Library/WebKit and /Users/buildbot/Library/Caches/TestWebKitAPI doesn't change it, so I don't see any persistent state for this.

[Brady Eidson]

I will not have time to dig deep on this soon.

We should disable it if it's getting in the way of things.

[Alexey Proskuryakov]

Easy fix.

[Alexey Proskuryakov]

Created attachment 303044 [details]
more fixing

I don't know if this will necessarily fix all the issues, but it doesn't time out in my testing so far.

[WebKit Commit Bot]

Comment on attachment 303044 [details]
more fixing

Clearing flags on attachment: 303044

Committed r213225: <http://trac.webkit.org/changeset/213225>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.