Bug 188394 - REGRESSION (r234330): [mac-wk1] Layout Test inspector/layers/layer-tree-manager.html and inspector/layers/layers-anonymous.html are flaky
Summary: REGRESSION (r234330): [mac-wk1] Layout Test inspector/layers/layer-tree-manag...
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2018-08-07 16:42 PDT by Truitt Savell
Modified: 2018-09-04 16:44 PDT (History)
7 users (show)

See Also:


Attachments
Test List (54.11 KB, text/plain)
2018-08-08 14:14 PDT, Truitt Savell
no flags Details
Test List (3.01 KB, text/plain)
2018-08-09 10:52 PDT, Truitt Savell
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Truitt Savell 2018-08-07 16:42:02 PDT
The following layout tests are flaky on High Sierra Leaks

inspector/layers/layer-tree-manager.html
inspector/layers/layers-anonymous.html

Probable cause:

These two tests fail in tandem. A state is being set somewhere in the test runner. I have not reproduced yet. This started somewhere closely around r234344

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=inspector%2Flayers%2Flayers-anonymous.html%20inspector%2Flayers%2Flayer-tree-manager.html
Comment 1 Ryan Haddad 2018-08-07 16:56:46 PDT
inspector/layers/layer-tree-manager.html is a text failure:

--- /Volumes/Data/slave/sierra-release-tests-wk1/build/layout-test-results/inspector/layers/layer-tree-manager-expected.txt
+++ /Volumes/Data/slave/sierra-release-tests-wk1/build/layout-test-results/inspector/layers/layer-tree-manager-actual.txt
@@ -3,7 +3,9 @@
 
 == Running test suite: LayerTreeManager
 -- Running test case: LayerTreeManager.layersForNode.root
-PASS: Returned array should include all layers.
+FAIL: Returned array should include all layers.
+    Expected: 5
+    Actual: 0
 PASS: Array elements should be Layer instances.
 
 -- Running test case: LayerTreeManager.layerTreeMutations.overlappingNodeIds

--- /Volumes/Data/slave/sierra-release-tests-wk1/build/layout-test-results/inspector/layers/layers-anonymous-expected.txt
+++ /Volumes/Data/slave/sierra-release-tests-wk1/build/layout-test-results/inspector/layers/layers-anonymous-actual.txt
@@ -1,3 +1,5 @@
+CONSOLE MESSAGE: line 58: TypeError: undefined is not an object (evaluating 'anonymousLayers[0].nodeId')
+FAIL: Timed out waiting for notifyDone to be called
 This is a test
 
inspector/layers/layers-anonymous.html times out with output:

@@ -15,14 +17,5 @@
 
 === Check layers ===
 
-PASS: Expected number of anonymous layers.
-PASS: The sole anonymous layer has a non-zero node id.
-PASS: The sole anonymous layer has a :first-letter pseudo-element.
+FAIL: Expected number of anonymous layers. Expected 1 but got 0
 
-=== Check node ===
-
-PASS: Node was found.
-PASS: Node has expected localName.
-PASS: Node has id.
-PASS: Node has expected id.
-

Also, this isn't limited to leaks, but it definitely happens far more often there.
Comment 2 Ryan Haddad 2018-08-07 16:57:02 PDT
Possibly related to https://trac.webkit.org/changeset/234330/webkit?
Comment 3 Truitt Savell 2018-08-08 14:14:30 PDT
Created attachment 346793 [details]
Test List

I was able to reproduce the two failures by running the tests from the imported/ directory that ran directly before the inspector/ tests from the same test runner. I am going to work on bisecting down the blamed tests. 

Command Used:
run-webkit-tests --test-list Test_list_directory --child-processes 1 -1 --leaks
Comment 4 Truitt Savell 2018-08-09 10:52:51 PDT
Created attachment 346846 [details]
Test List

New list of tests. Noting that the imported/ directory is no longer ran. The results are flakey at around a 2/3 chance of having the two tests fail.
Comment 5 Truitt Savell 2018-08-09 11:05:31 PDT
(In reply to Truitt Savell from comment #4)
> Created attachment 346846 [details]
> Test List
> 
> New list of tests. Noting that the imported/ directory is no longer ran. The
> results are flakey at around a 2/3 chance of having the two tests fail.

For now I think this is the best list I can provide. removing any more tests from it either stops the failure condition or makes the test failure far less consistent.
Comment 6 Ryan Haddad 2018-08-23 17:17:09 PDT
Updating title to reflect that this isn't limited to the leaks bot, but it does happen far more frequently there.
Comment 7 Truitt Savell 2018-09-04 16:15:57 PDT
Found that these three tests run in succession makes the failure occur around 90% of the time:

imported/w3c/web-platform-tests/html/dom/interfaces.html
inspector/layers/layer-tree-manager.html
inspector/layers/layers-anonymous.html

I used the command:
run-webkit-tests --root testbuild-235621 imported/w3c/web-platform-tests/html/dom/interfaces.html inspector/layers/layer-tree-manager.html inspector/layers/layers-anonymous.html --child-processes 1 -1 --iterations 10

Using this I was able to begin bisecting the webkit revisions to find the start of the regression. I found the revision that caused these failures to occur to be https://trac.webkit.org/changeset/234330/webkit. 

If you run the above command on r234329 no failures will occur, if it is ran on r234330 than the failure reproduces.
Comment 8 Radar WebKit Bug Importer 2018-09-04 16:44:09 PDT
<rdar://problem/44118147>