Bug 217193 – REGRESSION (r267814): [ Mojave+ release wk2 ] inspector/console/queryHolders.html is a flaky failure

• Status: NEW
• Component: Web Inspector

[Karl Rackler]

Description:
inspector/console/queryHolders.html

The first failure that I see this on the dashboard was 09/30/20 at r267815, but does not appear to be relevant to what would have caused this.

I can reproduce this on r267814, but unable to reproduce on 267813 or earlier.
Commit r267814 has to do with implementing an {Array, %TypedArray%, String}.prototype.item, which could have introduced the flaky failure.  The change was introduced here https://trac.webkit.org/changeset/267814/webkit.

REPRODUCTION STEPS
Command: 
run-webkit-tests --exit-after-n-failures 1 --iterations 50 inspector/console/queryHolders.html

Result: 
Regressions: Unexpected text-only failures (1)
  inspector/console/queryHolders.html [ Failure ]

History:
https://results.webkit.org/?suite=layout-tests&test=inspector%2Fconsole%2FqueryHolders.html&platform=mac

Diff Log:
--- /Volumes/Data/slave/catalina-release-tests-wk2/build/layout-test-results/inspector/formatting/formatting-css-expected.txt
+++ /Volumes/Data/slave/catalina-release-tests-wk2/build/layout-test-results/inspector/formatting/formatting-css-actual.txt
@@ -1,14 +1,5 @@
-Test CSS formatting.
+#PID UNRESPONSIVE - WebKitTestRunner (pid 84131)
+FAIL: Timed out waiting for notifyDone to be called
 
-
-== Running test suite: CSSFormatter
--- Running test case: CSSFormatter
-PASS: basic.css
-PASS: comment.css
-PASS: gradient.css
-PASS: keyframes.css
-PASS: media-query.css
-PASS: selectors.css
-PASS: url.css
-PASS: wrapping.css
-
+#EOF
+#EOF

[Radar WebKit Bug Importer]

<rdar://problem/69858825>

[Aakash Jain]

Interesting that in https://bugs.webkit.org/show_bug.cgi?id=217115#c5 EWS blamed the patch for r267814 for this test failure.

[Karl Rackler]

I have marked this test as failing while this issue is investigated.
https://trac.webkit.org/changeset/267856/webkit

[Ross Kirsling]

The test has no overt connection to r267814, but there must be some sort of odd dependence, since it evidently wasn't flaky before.

I'll try to see what possible interaction there could be...

[Ross Kirsling]

A few more details:
- Repro rate seems to be around 15%.
- When it fails, the queryHolders result (which is expected to be empty) contains two WI.RemoteObjects, which appear to differ only in `objectId.id`. In particular, they have the same `target` and a `description` of "Window".

This very much feels like an unreliable test that just happened to be stable until something shifted underneath it...?

[Ross Kirsling]

Playing around more, it appears that adding *any* new built-in function to Array.prototype (regardless of name, order, or behavior) will cause this test to turn flaky.

And yet the procedure for, say, r228266 was the same as my procedure for r267814, and this had no associated regression.

I'm happy to update the test to simply expect the array to be of length 2, but I'm not sure this preserves the purpose of the test, so I'd feel more comfortable leaving it to the WI folks to adjust more suitably.

[Diego Pino]

*** Bug 217966 has been marked as a duplicate of this bug. ***

[Diego Pino]

This regression could also be reproduced in WebKitGTK. In the case of WebKitGTK, after r268760 this test is no longer flaky, although it times out sometimes as it used to happen before r267814.

https://results.webkit.org/?suite=layout-tests&test=inspector%2Fconsole%2FqueryHolders.html&platform=GTK&platform=WPE