Bug 195961 - [ Mojave Debug ] REGRESSION (r242975) Layout Test imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex7-event_order.html is a flaky failure
Summary: [ Mojave Debug ] REGRESSION (r242975) Layout Test imported/w3c/IndexedDB-priv...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Sihui Liu
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2019-03-19 13:05 PDT by Shawn Roberts
Modified: 2019-05-13 08:39 PDT (History)
8 users (show)

See Also:


Attachments
Patch (1.83 KB, patch)
2019-05-10 14:51 PDT, Sihui Liu
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews212 for win-future (13.54 MB, application/zip)
2019-05-10 16:37 PDT, Build Bot
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Shawn Roberts 2019-03-19 13:05:25 PDT
The following layout test is flaky on Mojave Debug WK2

imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex7-event_order.html
imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex6-event_order.html

Probable cause:

Test recently showed up on the dashboard ~242943. Recently started to be more flaky. 

Through testing found that as of r242975 the failure rate jumped to about 25/600 failures. Exited early with 25 failures around 600 iterations. 
Testing prior to 242975 show that the test was a highly flaky failure, only 1 or 2 failures in 1500 iterations. I tested with various builds from 242974 to 242400 for these results.
Dashboard also shows flaky failures with WK1, iOS Simulator Debug, and Mac WK2. Those have similar highly flaky results, but did not get worse with 242943 like it did with Mojave Debug WK2.

Reproduced with: 

run-webkit-tests imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex6-event_order.html --iterations 1500 -f --debug --exit-after-n-failures=25 --no-retry

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2FIndexedDB-private-browsing%2Fidbobjectstore_createIndex7-event_order.html

Index 7 Diff: 

--- /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex7-event_order-expected.txt
+++ /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex7-event_order-actual.txt
@@ -1,3 +1,3 @@
 
-PASS IDBObjectStore.createIndex() - Event ordering for ConstraintError on request 
+FAIL IDBObjectStore.createIndex() - Event ordering for ConstraintError on request assert_array_equals: events lengths differ, expected 10 got 8

Index 6 Diff:

--- /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex6-event_order-expected.txt
+++ /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/imported/w3c/IndexedDB-private-browsing/idbobjectstore_createIndex6-event_order-actual.txt
@@ -1,3 +1,3 @@
 
-PASS IDBObjectStore.createIndex() - event order when unique constraint is triggered 
+FAIL IDBObjectStore.createIndex() - event order when unique constraint is triggered assert_array_equals: events lengths differ, expected 8 got 6
Comment 1 Radar WebKit Bug Importer 2019-03-19 13:06:54 PDT
<rdar://problem/49031397>
Comment 2 Shawn Roberts 2019-04-08 10:35:18 PDT
Marked flaky in https://trac.webkit.org/changeset/244026/webkit while waiting for a fix.
Comment 3 Sihui Liu 2019-05-10 14:51:42 PDT
Created attachment 369600 [details]
Patch
Comment 4 Build Bot 2019-05-10 16:37:00 PDT
Comment on attachment 369600 [details]
Patch

Attachment 369600 [details] did not pass win-ews (win):
Output: https://webkit-queues.webkit.org/results/12157353

New failing tests:
http/tests/security/local-video-src-from-remote.html
Comment 5 Build Bot 2019-05-10 16:37:03 PDT
Created attachment 369621 [details]
Archive of layout-test-results from ews212 for win-future

The attached test failures were seen while running run-webkit-tests on the win-ews.
Bot: ews212  Port: win-future  Platform: CYGWIN_NT-10.0-17763-3.0.5-338.x86_64-x86_64-64bit
Comment 6 Sihui Liu 2019-05-13 07:42:19 PDT
(In reply to Build Bot from comment #5)
> Created attachment 369621 [details]
> Archive of layout-test-results from ews212 for win-future
> 
> The attached test failures were seen while running run-webkit-tests on the
> win-ews.
> Bot: ews212  Port: win-future  Platform:
> CYGWIN_NT-10.0-17763-3.0.5-338.x86_64-x86_64-64bit

I only modified test expectation. This should not be caused by this patch.
Comment 7 youenn fablet 2019-05-13 08:12:26 PDT
Comment on attachment 369600 [details]
Patch

For patches that modify only TestExpectations and/or -expected.txt, I do not think there is a need for an explicit review.
Comment 8 WebKit Commit Bot 2019-05-13 08:39:48 PDT
Comment on attachment 369600 [details]
Patch

Clearing flags on attachment: 369600

Committed r245235: <https://trac.webkit.org/changeset/245235>
Comment 9 WebKit Commit Bot 2019-05-13 08:39:49 PDT
All reviewed patches have been landed.  Closing bug.