Bug 155050

Summary: storage/indexeddb/intversion-open-in-upgradeneeded.html is flaky on mac
Product: WebKit Reporter: Ryan Haddad <ryanhaddad>
Component: WebCore Misc.Assignee: Brady Eidson <beidson>
Status: RESOLVED FIXED    
Severity: Normal CC: alecflett, beidson, commit-queue, jsbell
Priority: P2    
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=154706
https://bugs.webkit.org/show_bug.cgi?id=155131
Bug Depends on:    
Bug Blocks: 154968    
Attachments:
Description Flags
Patch achristensen: review+

Description Ryan Haddad 2016-03-04 14:58:16 PST
storage/indexeddb/intversion-open-in-upgradeneeded.html is flaky on mac

This test was previously skipped due to flakiness (https://bugs.webkit.org/show_bug.cgi?id=154706) and a fix was made. However, it still appears to be flaky.

Recent failing run:
<https://build.webkit.org/results/Apple%20El%20Capitan%20Release%20WK2%20(Tests)/r197589%20(3949)/results.html>

Flakiness dashboard:
<http://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=storage%2Findexeddb%2Fintversion-open-in-upgradeneeded.html>

--- /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/storage/indexeddb/intversion-open-in-upgradeneeded-expected.txt
+++ /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/storage/indexeddb/intversion-open-in-upgradeneeded-actual.txt
@@ -34,15 +34,10 @@
 onVersionChange():
 db.close()
 
-onBlocked():
-
 upgradeNeeded2():
 db = event.target.result
 PASS event.newVersion is 3
-
-openSuccess2():
-db = event.target.result
-PASS db.version is 3
+FAIL Error function called unexpectedly: (AbortError) undefined
 PASS successfullyParsed is true
 
 TEST COMPLETE
Comment 1 Brady Eidson 2016-03-04 16:23:26 PST
The flakiness is different now.

It used to be a text failure that was well understood.

Now it's timing out.
Comment 2 Brady Eidson 2016-03-04 16:23:45 PST
(In reply to comment #1)
> The flakiness is different now.
> 
> It used to be a text failure that was well understood.
> 
> Now it's timing out.

(Of course, perhaps the text failure was *not* well understood)
Comment 3 Ryan Haddad 2016-03-07 13:44:11 PST
Skipped test on mac-wk2 with <http://trac.webkit.org/projects/webkit/changeset/197703>

The test also seems to be crashing intermittently. Filed https://bugs.webkit.org/show_bug.cgi?id=155131
Comment 4 Brady Eidson 2016-03-25 15:29:41 PDT
Regardless, it is the same problem. With the entire test suite cruising along, and with the unpredictability of GC, and with some leaks still outstanding, we're clearly keeping some file handles open that we don't need.

Fixing leaks is going to be one way out of this mess.

Another is:
https://bugs.webkit.org/show_bug.cgi?id=154923
Comment 5 Brady Eidson 2016-03-25 15:30:11 PDT
(In reply to comment #4)
> Regardless, it is the same problem. With the entire test suite cruising
> along, and with the unpredictability of GC, and with some leaks still
> outstanding, we're clearly keeping some file handles open that we don't need.
> 
> Fixing leaks is going to be one way out of this mess.
> 
> Another is:
> https://bugs.webkit.org/show_bug.cgi?id=154923

Whoops - That comment was meant for 155131
Comment 6 Ryan Haddad 2016-05-03 16:02:46 PDT
Un-skipped test but left marked as flaky in <https://trac.webkit.org/r200392> to allow us to see test results on the flakiness dashboard.
Comment 7 Brady Eidson 2016-05-03 22:16:28 PDT
Strangely, hasn't started running on any WK1 bots yet (which is where my enhanced logging is in place)

Gripping around the TestExpectations files now...
Comment 8 Brady Eidson 2016-05-03 22:19:18 PDT
Odd!

LayoutTests$ grep -r intversion-open-in-upgradeneeded.html .  | grep -v ChangeLog
./platform/efl/TestExpectations:storage/indexeddb/intversion-open-in-upgradeneeded.html [ Skip ]
./platform/mac-wk2/TestExpectations:webkit.org/b/155050 storage/indexeddb/intversion-open-in-upgradeneeded.html [ Pass Crash Timeout ]

So why isn't it running on WK1 bots? :(
Comment 9 Ryan Haddad 2016-05-04 08:54:45 PDT
(In reply to comment #8)
> Odd!
> 
> LayoutTests$ grep -r intversion-open-in-upgradeneeded.html .  | grep -v
> ChangeLog
> ./platform/efl/TestExpectations:storage/indexeddb/intversion-open-in-
> upgradeneeded.html [ Skip ]
> ./platform/mac-wk2/TestExpectations:webkit.org/b/155050
> storage/indexeddb/intversion-open-in-upgradeneeded.html [ Pass Crash Timeout
> ]
> 
> So why isn't it running on WK1 bots? :(

It is running on WK1, but it is passing. You can see the test listed in this WK1 LayoutTest log:
<https://build.webkit.org/builders/Apple%20El%20Capitan%20Release%20WK1%20(Tests)/builds/5907/steps/layout-test/logs/stdio>
Comment 10 Brady Eidson 2016-05-04 11:02:44 PDT
(In reply to comment #9)
> (In reply to comment #8)
> > Odd!
> > 
> > LayoutTests$ grep -r intversion-open-in-upgradeneeded.html .  | grep -v
> > ChangeLog
> > ./platform/efl/TestExpectations:storage/indexeddb/intversion-open-in-
> > upgradeneeded.html [ Skip ]
> > ./platform/mac-wk2/TestExpectations:webkit.org/b/155050
> > storage/indexeddb/intversion-open-in-upgradeneeded.html [ Pass Crash Timeout
> > ]
> > 
> > So why isn't it running on WK1 bots? :(
> 
> It is running on WK1, but it is passing. You can see the test listed in this
> WK1 LayoutTest log:
> <https://build.webkit.org/builders/
> Apple%20El%20Capitan%20Release%20WK1%20(Tests)/builds/5907/steps/layout-test/
> logs/stdio>

It has no WK1 entries on the flakiness dashboard which - I assumed - shows all results for all bots...?
Comment 11 Ryan Haddad 2016-05-04 11:04:24 PDT
> It has no WK1 entries on the flakiness dashboard which - I assumed - shows
> all results for all bots...?

It will only start showing data once the test fails / times out / crashes at least once.
Comment 12 Brady Eidson 2016-05-04 11:58:43 PDT
A few things I can see:

WK1-mac is, apparently, working great.

Release builds sometimes fail with text differences, but never crash
Debug builds "crash" - They're all asserts.

Example assert:

No crash log found for com.apple.WebKit.Databases.Development:81065.

stdout:

stderr:
ASSERTION FAILED: !m_db.m_transactionInProgress
/Volumes/Data/slave/yosemite-debug/build/Source/WebCore/platform/sql/SQLiteTransaction.cpp(53) : void WebCore::SQLiteTransaction::begin()
1   0x105e99b60 WTFCrash
2   0x10a2a01f8 WebCore::SQLiteTransaction::begin()
3   0x10a29656e WebCore::IDBServer::SQLiteIDBTransaction::begin(WebCore::SQLiteDatabase&)
4   0x10a27d5ac WebCore::IDBServer::SQLiteIDBBackingStore::beginTransaction(WebCore::IDBTransactionInfo const&)
5   0x10a5cfb15 WebCore::IDBServer::UniqueIDBDatabase::beginTransactionInBackingStore(WebCore::IDBTransactionInfo const&)
6   0x10a61ea5f WebCore::CrossThreadTaskImpl<WebCore::IDBServer::UniqueIDBDatabase, WebCore::IDBTransactionInfo const&>::CrossThreadTaskImpl(WebCore::IDBServer::UniqueIDBDatabase*, void (WebCore::IDBServer::UniqueIDBDatabase::*)(WebCore::IDBTransactionInfo const&), WebCore::IDBTransactionInfo const&)::'lambda'()::operator()() const
7   0x10a61e9cc std::__1::__function::__func<WebCore::CrossThreadTaskImpl<WebCore::IDBServer::UniqueIDBDatabase, WebCore::IDBTransactionInfo const&>::CrossThreadTaskImpl(WebCore::IDBServer::UniqueIDBDatabase*, void (WebCore::IDBServer::UniqueIDBDatabase::*)(WebCore::IDBTransactionInfo const&), WebCore::IDBTransactionInfo const&)::'lambda'(), std::__1::allocator<WebCore::CrossThreadTaskImpl<WebCore::IDBServer::UniqueIDBDatabase, WebCore::IDBTransactionInfo const&>::CrossThreadTaskImpl(WebCore::IDBServer::UniqueIDBDatabase*, void (WebCore::IDBServer::UniqueIDBDatabase::*)(WebCore::IDBTransactionInfo const&), WebCore::IDBTransactionInfo const&)::'lambda'()>, void ()>::operator()()
8   0x10818103a std::__1::function<void ()>::operator()() const
9   0x108f2d125 WebCore::CrossThreadTask::performTask()
10  0x108f2a7a3 WebCore::IDBServer::IDBServer::databaseRunLoop()
11  0x108f27e14 WebCore::IDBServer::IDBServer::databaseThreadEntry(void*)
12  0x105f03379 WTF::createThread(void (*)(void*), void*, char const*)::$_0::operator()() const
13  0x105f0334c std::__1::__function::__func<WTF::createThread(void (*)(void*), void*, char const*)::$_0, std::__1::allocator<WTF::createThread(void (*)(void*), void*, char const*)::$_0>, void ()>::operator()()
14  0x10578184a std::__1::function<void ()>::operator()() const
15  0x105f0209e WTF::threadEntryPoint(void*)
16  0x105f03901 WTF::wtfThreadEntryPoint(void*)
17  0x7fff9441b05a _pthread_body
18  0x7fff9441afd7 _pthread_body
19  0x7fff944183ed thread_start

-Windows WK1 debug is seeing the crashes too, but I can't get windows crash logs.
Comment 13 Brady Eidson 2016-05-04 12:12:27 PDT
Locally, running:
run-webkit-tests storage/indexeddb/intversion-open-in-upgradeneeded.html --child-processes=1 --repeat-each=1000

Causes about a 0.5% crash rate of the DatabaseProcess with this same ASSERT.

I might be able to explore from here.
Comment 14 Brady Eidson 2016-06-03 11:12:18 PDT
This was timing out in all bot configs for a large span of time, but then around 5/26-5/27 it stopped timing out.

There's still *very* frequent text failures like the following:

--- /Volumes/Data/slave/elcapitan-debug-tests-wk2/build/layout-test-results/storage/indexeddb/intversion-open-in-upgradeneeded-expected.txt
+++ /Volumes/Data/slave/elcapitan-debug-tests-wk2/build/layout-test-results/storage/indexeddb/intversion-open-in-upgradeneeded-actual.txt
@@ -34,8 +34,6 @@
 onVersionChange():
 db.close()
 
-onBlocked():
-
 upgradeNeeded2():
 db = event.target.result
 PASS event.newVersion is 3

That might be a test bug. I'll look in to that.
Comment 15 Brady Eidson 2016-06-03 11:15:13 PDT
Getting blocked is:
1 - A timing race.
2 - Not what the test is testing.

So I'm removing the onblocked handler.
Comment 16 Brady Eidson 2016-06-03 12:08:30 PDT
(In reply to comment #14)
> This was timing out in all bot configs for a large span of time, but then
> around 5/26-5/27 it stopped timing out.
> 

Note: There's actually an expected reason for this. http://trac.webkit.org/changeset/201461 was a *really* important change that potentially affected *so many tests*...
Comment 17 Brady Eidson 2016-06-03 12:11:44 PDT
Created attachment 280450 [details]
Patch
Comment 18 Brady Eidson 2016-06-03 12:27:45 PDT
http://trac.webkit.org/changeset/201652
Comment 19 Brady Eidson 2016-06-03 16:49:12 PDT
*** Bug 155131 has been marked as a duplicate of this bug. ***