Bug 195961 – [ Mojave Debug ] REGRESSION (r242975) Layout Test imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex7-event_order.html is a flaky failure

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Shawn Roberts]

The following layout test is flaky on Mojave Debug WK2

imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex7-event_order.html
imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex6-event_order.html

Probable cause:

Test recently showed up on the dashboard ~242943. Recently started to be more flaky. 

Through testing found that as of r242975 the failure rate jumped to about 25/600 failures. Exited early with 25 failures around 600 iterations. 
Testing prior to 242975 show that the test was a highly flaky failure, only 1 or 2 failures in 1500 iterations. I tested with various builds from 242974 to 242400 for these results.
Dashboard also shows flaky failures with WK1, iOS Simulator Debug, and Mac WK2. Those have similar highly flaky results, but did not get worse with 242943 like it did with Mojave Debug WK2.

Reproduced with: 

run-webkit-tests imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex6-event_order.html --iterations 1500 -f --debug --exit-after-n-failures=25 --no-retry

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2FIndexedDB-private-browsing%2Fidbobjectstore_createIndex7-event_order.html

Index 7 Diff: 

--- /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex7-event_order-expected.txt
+++ /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex7-event_order-actual.txt
@@ -1,3 +1,3 @@
 
-PASS IDBObjectStore.createIndex() - Event ordering for ConstraintError on request 
+FAIL IDBObjectStore.createIndex() - Event ordering for ConstraintError on request assert_array_equals: events lengths differ, expected 10 got 8

Index 6 Diff:

--- /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex6-event_order-expected.txt
+++ /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex6-event_order-actual.txt
@@ -1,3 +1,3 @@
 
-PASS IDBObjectStore.createIndex() - event order when unique constraint is triggered 
+FAIL IDBObjectStore.createIndex() - event order when unique constraint is triggered assert_array_equals: events lengths differ, expected 8 got 6

[Radar WebKit Bug Importer]

<rdar://problem/49031397>

[Shawn Roberts]

Marked flaky in https://trac.webkit.org/changeset/244026/webkit while waiting for a fix.

[Sihui Liu]

Created attachment 369600 [details]
Patch

[EWS Watchlist]

Comment on attachment 369600 [details]
Patch

Attachment 369600 [details] did not pass win-ews (win):
Output: https://webkit-queues.webkit.org/results/12157353

New failing tests:
http/tests/security/local-video-src-from-remote.html

[EWS Watchlist]

Created attachment 369621 [details]
Archive of layout-test-results from ews212 for win-future

The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews212  Port: win-future  Platform: CYGWIN_NT-10.0-17763-3.0.5-338.x86_64-x86_64-64bit

[Sihui Liu]

(In reply to Build Bot from comment #5)
> Created attachment 369621 [details]
> Archive of layout-test-results from ews212 for win-future
> 
> The attached test failures were seen while running run-webkit-tests on the
> win-ews.
> Bot: ews212  Port: win-future  Platform:
> CYGWIN_NT-10.0-17763-3.0.5-338.x86_64-x86_64-64bit

I only modified test expectation. This should not be caused by this patch.

[youenn fablet]

Comment on attachment 369600 [details]
Patch

For patches that modify only TestExpectations and/or -expected.txt, I do not think there is a need for an explicit review.

[WebKit Commit Bot]

Comment on attachment 369600 [details]
Patch

Clearing flags on attachment: 369600

Committed r245235: <https://trac.webkit.org/changeset/245235>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.