Bug 195952 - REGRESSION (r242911?): High Sierra Release WK2 Perf bot timing out while running IndexedDB/large-number-of-inserts.html
Summary: REGRESSION (r242911?): High Sierra Release WK2 Perf bot timing out while runn...
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: youenn fablet
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2019-03-19 10:02 PDT by Ryan Haddad
Modified: 2019-06-16 20:52 PDT (History)
8 users (show)

See Also:


Attachments
Patch (1.30 KB, patch)
2019-03-20 11:56 PDT, youenn fablet
youennf: review?
Details | Formatted Diff | Diff
Archive of layout-test-results from ews105 for mac-highsierra-wk2 (2.97 MB, application/zip)
2019-03-20 15:42 PDT, Build Bot
no flags Details
Archive of layout-test-results from ews124 for ios-simulator-wk2 (2.60 MB, application/zip)
2019-03-20 16:34 PDT, Build Bot
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ryan Haddad 2019-03-19 10:02:28 PDT
The High Sierra Release WK2 Perf bot hitting a Buildbot step timeout of 20m without output while running IndexedDB/large-number-of-inserts.html

https://build.webkit.org/builders/Apple%20High%20Sierra%20Release%20WK2%20%28Perf%29/builds/612/steps/perf-test/logs/stdio

Based on the history of the queue, it looks like the last successful run was with r242891 and the first failing was r 242912
Comment 1 Ryan Haddad 2019-03-19 10:03:16 PDT
"Check IDB quota usage through QuotaManager" seems like a potential culprit:
https://trac.webkit.org/changeset/242911/webkit
Comment 2 youenn fablet 2019-03-19 10:07:03 PDT
Will look at it tonight
Comment 3 youenn fablet 2019-03-20 10:48:02 PDT
So, a test that is doing about the same thing (PerformanceTests/IndexedDB/large-number-of-inserts-responsiveness.html) is taking twice the time between run 610 and 612.
The median test run remains of the same order though.

It might be that cache quota is hit. This triggers some IPC to the UIProcess which triggers some additional overhead.
We might be able to fix that by increasing the default quota for these tests.
Comment 4 youenn fablet 2019-03-20 10:48:35 PDT
Additionally, we could also increase the build bot timeout above 20 minutes for perf tests and/or decrease the test iteration from 20 down to 10.
Comment 5 youenn fablet 2019-03-20 11:56:37 PDT
Created attachment 365376 [details]
Patch
Comment 6 Build Bot 2019-03-20 15:42:35 PDT
Comment on attachment 365376 [details]
Patch

Attachment 365376 [details] did not pass mac-wk2-ews (mac-wk2):
Output: https://webkit-queues.webkit.org/results/11585671

New failing tests:
http/wpt/mediarecorder/MediaRecorder-AV-audio-video-dataavailable.html
Comment 7 Build Bot 2019-03-20 15:42:37 PDT
Created attachment 365417 [details]
Archive of layout-test-results from ews105 for mac-highsierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews105  Port: mac-highsierra-wk2  Platform: Mac OS X 10.13.6
Comment 8 Build Bot 2019-03-20 16:34:43 PDT
Comment on attachment 365376 [details]
Patch

Attachment 365376 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: https://webkit-queues.webkit.org/results/11586009

New failing tests:
fast/visual-viewport/ios/min-scale-greater-than-one.html
Comment 9 Build Bot 2019-03-20 16:34:44 PDT
Created attachment 365432 [details]
Archive of layout-test-results from ews124 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews124  Port: ios-simulator-wk2  Platform: Mac OS X 10.13.6
Comment 10 youenn fablet 2019-03-20 16:36:02 PDT
Comment on attachment 365376 [details]
Patch

Both errors unrelated
Comment 11 Brady Eidson 2019-03-20 16:46:34 PDT
Comment on attachment 365376 [details]
Patch

242911 slowed down IDB enough to make this test timeout, and our solution is to reduce the magnitude of the test?
Comment 12 youenn fablet 2019-03-20 16:51:37 PDT
(In reply to Brady Eidson from comment #11)
> Comment on attachment 365376 [details]
> Patch
> 
> 242911 slowed down IDB enough to make this test timeout, and our solution is
> to reduce the magnitude of the test?

The first step is to make the test run again instead of timing out, hence reducing the magnitude of the test. This patch is only doing this.

The second step is to add an API to increase the default quota to a bigger value so that we do not hit the quota-check IPC cost from NetworkProcess to UIProcess.
My guess is that this eats time for the first test.
Comment 13 Ryosuke Niwa 2019-03-20 19:19:55 PDT
I feel like the latest time Sihui fixed this test, the issue was that our IDB limit was too small? Do you recall that, Sihui?
Comment 14 youenn fablet 2019-03-20 19:53:32 PDT
(In reply to Ryosuke Niwa from comment #13)
> I feel like the latest time Sihui fixed this test, the issue was that our
> IDB limit was too small? Do you recall that, Sihui?

I think that something similar to setIDBPerOriginQuota with a big enough value should do the trick. Step 2 is probably just renaming setIDBPerOriginQuota to setPerOriginQuota, piping the provided value to StorageQuotaManager(s) and updating that perf test to set the quota accordingly.
Comment 15 Radar WebKit Bug Importer 2019-03-21 20:39:36 PDT
<rdar://problem/49137688>
Comment 16 Sihui Liu 2019-03-22 10:20:09 PDT
(In reply to youenn fablet from comment #14)
> (In reply to Ryosuke Niwa from comment #13)
> > I feel like the latest time Sihui fixed this test, the issue was that our
> > IDB limit was too small? Do you recall that, Sihui?
> 
> I think that something similar to setIDBPerOriginQuota with a big enough
> value should do the trick. Step 2 is probably just renaming
> setIDBPerOriginQuota to setPerOriginQuota, piping the provided value to
> StorageQuotaManager(s) and updating that perf test to set the quota
> accordingly.

At that time we had a lower default IDB quota for layout tests. Default quota for single layout test is 50MB and for non-test is 500MB. So increasing the quota for some tests still showed us what to expect when opening test page in browser.

If we were about to change the default quota for some test now with presence of StorageQuotaManager, it seems we are ignoring the performance issue? Could we still wait for UI process to respond and return a high quota in the first response?
Comment 17 Ryan Haddad 2019-03-29 08:49:55 PDT
I skipped the test in https://trac.webkit.org/changeset/243604/webkit so that the bot would finish its test runs.