Bug 165222 - REGRESSION (r209144): imported/w3c/web-platform-tests/IndexedDB/idbindex-* tests are flaky in debug builds
Summary: REGRESSION (r209144): imported/w3c/web-platform-tests/IndexedDB/idbindex-* te...
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: Other
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on:
Blocks: 154968
  Show dependency treegraph
 
Reported: 2016-11-30 14:24 PST by Ryan Haddad
Modified: 2017-10-31 09:56 PDT (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ryan Haddad 2016-11-30 14:24:41 PST
The following tests have been flaky since https://trac.webkit.org/changeset/209144 landed.

imported/w3c/web-platform-tests/IndexedDB/idbindex-multientry.htm
imported/w3c/web-platform-tests/IndexedDB/idbindex-rename-abort.html
imported/w3c/web-platform-tests/IndexedDB/idbindex-rename-errors.html

https://build.webkit.org/results/Apple%20El%20Capitan%20Debug%20WK2%20(Tests)/r209145%20(9666)/results.html

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2FIndexedDB%2Fidbindex-multientry.htm

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2FIndexedDB%2Fidbindex-rename-abort.html

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2FIndexedDB%2Fidbindex-rename-errors.html

--- /Volumes/Data/slave/elcapitan-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/IndexedDB/idbindex-multientry-expected.txt
+++ /Volumes/Data/slave/elcapitan-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/IndexedDB/idbindex-multientry-actual.txt
@@ -1,3 +1,5 @@
 
-PASS IDBIndex.multiEntry - adding keys 
+Harness Error (TIMEOUT), message = null
 
+NOTRUN IDBIndex.multiEntry - adding keys 
+
Comment 1 Ryan Haddad 2016-11-30 14:25:55 PST
This affects macOS Debug WK2
Comment 2 Brady Eidson 2016-11-30 17:03:55 PST
Hammering these in WK2 Debug locally, all sorts of configurations of run-webkit-tests.

Cannot reproduce.
Comment 3 Brady Eidson 2016-12-01 13:09:31 PST
Wonder if there's any Database process crashes and they're just not coming up in the results?
Comment 4 Ryan Haddad 2016-12-01 13:10:31 PST
(In reply to comment #3)
> Wonder if there's any Database process crashes and they're just not coming
> up in the results?

I'll see if I can find anything.
Comment 5 Alexey Proskuryakov 2016-12-01 13:18:41 PST
Brady, how long do these tests take when you run them? If they are slow locally, timing out on the bots wouldn't be surprising.
Comment 6 Ryan Haddad 2016-12-01 13:22:14 PST
(In reply to comment #4)
> (In reply to comment #3)
> > Wonder if there's any Database process crashes and they're just not coming
> > up in the results?
> 
> I'll see if I can find anything.

I don't see any database process crashlogs on the bot that most recently had these failures.
Comment 7 Brady Eidson 2016-12-01 14:07:46 PST
(In reply to comment #5)
> Brady, how long do these tests take when you run them? If they are slow
> locally, timing out on the bots wouldn't be surprising.

Extremely quickly.

With webkit-httpd running, the command:
`DYLD_FRAMEWORK_PATH=~/build/Debug/ ~/build/Debug/WebKitTestRunner http://127.0.0.1:8800/IndexedDB/idbindex-multientry.htm http://127.0.0.1:8800/IndexedDB/idbindex-rename-abort.html http://127.0.0.1:8800/IndexedDB/idbindex-rename-errors.html`

...takes well under a second to successfully complete.
Comment 8 Brady Eidson 2016-12-02 14:00:47 PST
According to the flakiness dashboard, these literally never timeout on Release builds.
But on Debug builds they take *a long* time.

As mentioned in comment 7, I can't reproduce these locally even with a debug build. They are very fast.
Comment 9 Brady Eidson 2016-12-02 14:02:17 PST
I'm also trying to reason through why they could possibly be related to this patch, and I got nothin'
Comment 10 Brady Eidson 2016-12-02 14:03:56 PST
Even completes quickly under gmalloc (locally)
Comment 11 Brady Eidson 2016-12-02 14:06:49 PST
We should mark them flakey for debug only.
Comment 12 Ryan Haddad 2016-12-02 14:43:41 PST
(In reply to comment #11)
> We should mark them flakey for debug only.

Done in http://trac.webkit.org/projects/webkit/changeset/209272