Bug 111004 – Layout Test accessibility/svg-remote-element.html is flaky

• Status: NEW
• Component: Tools / Tests

[Eric Seidel (no email)]

The following layout test is flaky on linux/win

accessibility/svg-remote-element.html

--- /src/WebKit/Source/WebKit/chromium/webkit/Debug/layout-test-results/accessibility/svg-remote-element-expected.txt
+++ /src/WebKit/Source/WebKit/chromium/webkit/Debug/layout-test-results/accessibility/svg-remote-element-actual.txt
@@ -4,29 +4,29 @@
 On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
 
 
-container location: (7, 7)
+container location: (7, 7.5)
 Face role: AXRole: AXButton
 Face label: AXDescription: face
 FaceX: 0
-FaceY: 0
+FaceY: 0.5
 
 
 Eye role: AXRole: AXButton
 Eye label: AXDescription: left-eye
 EyeX: 103
-EyeY: 148
+EyeY: 147.5
 
 
 Nose role: AXRole: AXButton
 Nose label: AXDescription: nose
 NoseX: 193
-NoseY: 206
+NoseY: 205.5
 
 
 Mouth role: AXRole: AXButton
 Mouth label: AXDescription: smile
 MouthX: 115
-MouthY: 275
+MouthY: 274
 
 
 PASS successfullyParsed is true

[Eric Seidel (no email)]

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=accessibility%2Fsvg-remote-element.html

[Eric Seidel (no email)]

http://trac.webkit.org/browser/trunk/LayoutTests/accessibility/svg-remote-element.html

[chris fleizach]

maybe making the the positions be within some sort of tolerance, like

shouldBeTrue(face.X > 200 && face.X < 205) 

would reduce this flakiness.

[Dominic Mazzoni]

To clarify, do you mean "flaky" as in sometimes succeeding and sometimes failing, or is it giving slightly different expectations on different platforms?

If it really is returning slightly different results on the same platform sometimes, what's the underlying cause?

[Eric Seidel (no email)]

The bots are seeing it sometimes return one result and sometimes another on both linux and win.  See the test-results link in comment 1.

[Eric Seidel (no email)]

The diff in comment 0 is from running the test locally on my linux machine with run-webkit-tests --chromium --iterations=100 accessibility/svg-remote-element.html

[chris fleizach]

(In reply to comment #6)
> The diff in comment 0 is from running the test locally on my linux machine with run-webkit-tests --chromium --iterations=100 accessibility/svg-remote-element.html

Following up to Dominics comment, I agree this is probably exposing an more fundamental problem with the layout of SVG remote items or with the values it reports. It's unlikely that the actual cause is accessibility. However, I'm not opposed to making the test more tolerant, given that the differences are so small

[Dominic Mazzoni]

pdr@ graciously volunteered to take a look and see if the bug is in SVG.

In the meantime, let's not mask the problem by adding tolerance...at least until we understand it better.

I'd rather mark the test as flaky in TestExpectations.

[Philip Rogers]

(In reply to comment #8)
> pdr@ graciously volunteered to take a look and see if the bug is in SVG.
> 
> In the meantime, let's not mask the problem by adding tolerance...at least until we understand it better.
> 
> I'd rather mark the test as flaky in TestExpectations.

Small update: running with --run-singly does not expose the flakiness, so I think this is likely a state issue between tests.

The test fails due to differences in the following values: container.clickPointY, container.clickPointX, container.height and container.width.

Good: clickPointX=208, clickPointY=208, width=400, height=400
Bad: clickPointX=56, clickPointY=18, width=96, height=19

I don't see a logical source for these values other than accessibilityController containing stale state.

[Philip Rogers]

*** Bug 111062 has been marked as a duplicate of this bug. ***