Bug 155050 – storage/indexeddb/intversion-open-in-upgradeneeded.html is flaky on mac

• Status: RESOLVED
• Resolution: FIXED
• Component: WebCore Misc.

[Ryan Haddad]

storage/indexeddb/intversion-open-in-upgradeneeded.html is flaky on mac

This test was previously skipped due to flakiness (https://bugs.webkit.org/show_bug.cgi?id=154706) and a fix was made. However, it still appears to be flaky.

Recent failing run:
<https://build.webkit.org/results/Apple%20El%20Capitan%20Release%20WK2%20(Tests)/r197589%20(3949)/results.html>

Flakiness dashboard:
<http://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=storage%2Findexeddb%2Fintversion-open-in-upgradeneeded.html>

--- /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/storage/indexeddb/intversion-open-in-upgradeneeded-expected.txt
+++ /Volumes/Data/slave/elcapitan-release-tests-wk2/build/layout-test-results/storage/indexeddb/intversion-open-in-upgradeneeded-actual.txt
@@ -34,15 +34,10 @@
 onVersionChange():
 db.close()
 
-onBlocked():
-
 upgradeNeeded2():
 db = event.target.result
 PASS event.newVersion is 3
-
-openSuccess2():
-db = event.target.result
-PASS db.version is 3
+FAIL Error function called unexpectedly: (AbortError) undefined
 PASS successfullyParsed is true
 
 TEST COMPLETE

[Brady Eidson]

The flakiness is different now.

It used to be a text failure that was well understood.

Now it's timing out.

[Brady Eidson]

(In reply to comment #1)
> The flakiness is different now.
> 
> It used to be a text failure that was well understood.
> 
> Now it's timing out.

(Of course, perhaps the text failure was *not* well understood)

[Ryan Haddad]

Skipped test on mac-wk2 with <http://trac.webkit.org/projects/webkit/changeset/197703>

The test also seems to be crashing intermittently. Filed https://bugs.webkit.org/show_bug.cgi?id=155131

[Brady Eidson]

Regardless, it is the same problem. With the entire test suite cruising along, and with the unpredictability of GC, and with some leaks still outstanding, we're clearly keeping some file handles open that we don't need.

Fixing leaks is going to be one way out of this mess.

Another is:
https://bugs.webkit.org/show_bug.cgi?id=154923

[Brady Eidson]

(In reply to comment #4)
> Regardless, it is the same problem. With the entire test suite cruising
> along, and with the unpredictability of GC, and with some leaks still
> outstanding, we're clearly keeping some file handles open that we don't need.
> 
> Fixing leaks is going to be one way out of this mess.
> 
> Another is:
> https://bugs.webkit.org/show_bug.cgi?id=154923

Whoops - That comment was meant for 155131

[Ryan Haddad]

Un-skipped test but left marked as flaky in <https://trac.webkit.org/r200392> to allow us to see test results on the flakiness dashboard.

[Brady Eidson]

Strangely, hasn't started running on any WK1 bots yet (which is where my enhanced logging is in place)

Gripping around the TestExpectations files now...

[Brady Eidson]

Odd!

LayoutTests$ grep -r intversion-open-in-upgradeneeded.html .  | grep -v ChangeLog
./platform/efl/TestExpectations:storage/indexeddb/intversion-open-in-upgradeneeded.html [ Skip ]
./platform/mac-wk2/TestExpectations:webkit.org/b/155050 storage/indexeddb/intversion-open-in-upgradeneeded.html [ Pass Crash Timeout ]

So why isn't it running on WK1 bots? :(

[Ryan Haddad]

(In reply to comment #8)
> Odd!
> 
> LayoutTests$ grep -r intversion-open-in-upgradeneeded.html .  | grep -v
> ChangeLog
> ./platform/efl/TestExpectations:storage/indexeddb/intversion-open-in-
> upgradeneeded.html [ Skip ]
> ./platform/mac-wk2/TestExpectations:webkit.org/b/155050
> storage/indexeddb/intversion-open-in-upgradeneeded.html [ Pass Crash Timeout
> ]
> 
> So why isn't it running on WK1 bots? :(

It is running on WK1, but it is passing. You can see the test listed in this WK1 LayoutTest log:
<https://build.webkit.org/builders/Apple%20El%20Capitan%20Release%20WK1%20(Tests)/builds/5907/steps/layout-test/logs/stdio>

[Brady Eidson]

(In reply to comment #9)
> (In reply to comment #8)
> > Odd!
> > 
> > LayoutTests$ grep -r intversion-open-in-upgradeneeded.html .  | grep -v
> > ChangeLog
> > ./platform/efl/TestExpectations:storage/indexeddb/intversion-open-in-
> > upgradeneeded.html [ Skip ]
> > ./platform/mac-wk2/TestExpectations:webkit.org/b/155050
> > storage/indexeddb/intversion-open-in-upgradeneeded.html [ Pass Crash Timeout
> > ]
> > 
> > So why isn't it running on WK1 bots? :(
> 
> It is running on WK1, but it is passing. You can see the test listed in this
> WK1 LayoutTest log:
> <https://build.webkit.org/builders/
> Apple%20El%20Capitan%20Release%20WK1%20(Tests)/builds/5907/steps/layout-test/
> logs/stdio>

It has no WK1 entries on the flakiness dashboard which - I assumed - shows all results for all bots...?

[Ryan Haddad]

> It has no WK1 entries on the flakiness dashboard which - I assumed - shows
> all results for all bots...?

It will only start showing data once the test fails / times out / crashes at least once.

[Brady Eidson]

A few things I can see:

WK1-mac is, apparently, working great.

Release builds sometimes fail with text differences, but never crash
Debug builds "crash" - They're all asserts.

Example assert:

No crash log found for com.apple.WebKit.Databases.Development:81065.

stdout:

stderr:
ASSERTION FAILED: !m_db.m_transactionInProgress
/Volumes/Data/slave/yosemite-debug/build/Source/WebCore/platform/sql/SQLiteTransaction.cpp(53) : void WebCore::SQLiteTransaction::begin()
1   0x105e99b60 WTFCrash
2   0x10a2a01f8 WebCore::SQLiteTransaction::begin()
3   0x10a29656e WebCore::IDBServer::SQLiteIDBTransaction::begin(WebCore::SQLiteDatabase&)
4   0x10a27d5ac WebCore::IDBServer::SQLiteIDBBackingStore::beginTransaction(WebCore::IDBTransactionInfo const&)
5   0x10a5cfb15 WebCore::IDBServer::UniqueIDBDatabase::beginTransactionInBackingStore(WebCore::IDBTransactionInfo const&)
6   0x10a61ea5f WebCore::CrossThreadTaskImpl<WebCore::IDBServer::UniqueIDBDatabase, WebCore::IDBTransactionInfo const&>::CrossThreadTaskImpl(WebCore::IDBServer::UniqueIDBDatabase*, void (WebCore::IDBServer::UniqueIDBDatabase::*)(WebCore::IDBTransactionInfo const&), WebCore::IDBTransactionInfo const&)::'lambda'()::operator()() const
7   0x10a61e9cc std::__1::__function::__func<WebCore::CrossThreadTaskImpl<WebCore::IDBServer::UniqueIDBDatabase, WebCore::IDBTransactionInfo const&>::CrossThreadTaskImpl(WebCore::IDBServer::UniqueIDBDatabase*, void (WebCore::IDBServer::UniqueIDBDatabase::*)(WebCore::IDBTransactionInfo const&), WebCore::IDBTransactionInfo const&)::'lambda'(), std::__1::allocator<WebCore::CrossThreadTaskImpl<WebCore::IDBServer::UniqueIDBDatabase, WebCore::IDBTransactionInfo const&>::CrossThreadTaskImpl(WebCore::IDBServer::UniqueIDBDatabase*, void (WebCore::IDBServer::UniqueIDBDatabase::*)(WebCore::IDBTransactionInfo const&), WebCore::IDBTransactionInfo const&)::'lambda'()>, void ()>::operator()()
8   0x10818103a std::__1::function<void ()>::operator()() const
9   0x108f2d125 WebCore::CrossThreadTask::performTask()
10  0x108f2a7a3 WebCore::IDBServer::IDBServer::databaseRunLoop()
11  0x108f27e14 WebCore::IDBServer::IDBServer::databaseThreadEntry(void*)
12  0x105f03379 WTF::createThread(void (*)(void*), void*, char const*)::$_0::operator()() const
13  0x105f0334c std::__1::__function::__func<WTF::createThread(void (*)(void*), void*, char const*)::$_0, std::__1::allocator<WTF::createThread(void (*)(void*), void*, char const*)::$_0>, void ()>::operator()()
14  0x10578184a std::__1::function<void ()>::operator()() const
15  0x105f0209e WTF::threadEntryPoint(void*)
16  0x105f03901 WTF::wtfThreadEntryPoint(void*)
17  0x7fff9441b05a _pthread_body
18  0x7fff9441afd7 _pthread_body
19  0x7fff944183ed thread_start

-Windows WK1 debug is seeing the crashes too, but I can't get windows crash logs.

[Brady Eidson]

Locally, running:
run-webkit-tests storage/indexeddb/intversion-open-in-upgradeneeded.html --child-processes=1 --repeat-each=1000

Causes about a 0.5% crash rate of the DatabaseProcess with this same ASSERT.

I might be able to explore from here.

[Brady Eidson]

This was timing out in all bot configs for a large span of time, but then around 5/26-5/27 it stopped timing out.

There's still *very* frequent text failures like the following:

--- /Volumes/Data/slave/elcapitan-debug-tests-wk2/build/layout-test-results/storage/indexeddb/intversion-open-in-upgradeneeded-expected.txt
+++ /Volumes/Data/slave/elcapitan-debug-tests-wk2/build/layout-test-results/storage/indexeddb/intversion-open-in-upgradeneeded-actual.txt
@@ -34,8 +34,6 @@
 onVersionChange():
 db.close()
 
-onBlocked():
-
 upgradeNeeded2():
 db = event.target.result
 PASS event.newVersion is 3

That might be a test bug. I'll look in to that.

[Brady Eidson]

Getting blocked is:
1 - A timing race.
2 - Not what the test is testing.

So I'm removing the onblocked handler.

[Brady Eidson]

(In reply to comment #14)
> This was timing out in all bot configs for a large span of time, but then
> around 5/26-5/27 it stopped timing out.
> 

Note: There's actually an expected reason for this. http://trac.webkit.org/changeset/201461 was a *really* important change that potentially affected *so many tests*...

[Brady Eidson]

Created attachment 280450 [details]
Patch

[Brady Eidson]

http://trac.webkit.org/changeset/201652

[Brady Eidson]

*** Bug 155131 has been marked as a duplicate of this bug. ***

[Robert Junior]

Created attachment 280450 [details]
Patched on https://emp3juice.blog/ https://tubidy.diy/ https://y2mate.diy/