Bug 63186 – svg/zoom/page/zoom-svg-through-object-with-*.xhtml are flaky

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Adam Roben (:aroben)]

The svg/zoom/page/zoom-svg-through-object-with-*.xhtml tests are flaky. See dupes for examples.

[Adam Roben (:aroben)]

*** Bug 61664 has been marked as a duplicate of this bug. ***

[Adam Roben (:aroben)]

*** Bug 61757 has been marked as a duplicate of this bug. ***

[Adam Roben (:aroben)]

*** Bug 61918 has been marked as a duplicate of this bug. ***

[Adam Roben (:aroben)]

*** Bug 62023 has been marked as a duplicate of this bug. ***

[Adam Roben (:aroben)]

*** Bug 62555 has been marked as a duplicate of this bug. ***

[Adam Roben (:aroben)]

*** Bug 62558 has been marked as a duplicate of this bug. ***

[Adam Roben (:aroben)]

*** Bug 62560 has been marked as a duplicate of this bug. ***

[Adam Roben (:aroben)]

*** Bug 62563 has been marked as a duplicate of this bug. ***

[Adam Roben (:aroben)]

Disabled the tests in r89484.

[Nikolas Zimmermann]

What a pity, these tests are pretty important :(
I haven't been able to reproduce this on my SL machines so far, I'll try --random -p runs soon, maybe I can finally see the problem on my own...

[Nikolas Zimmermann]

(In reply to comment #10)
> What a pity, these tests are pretty important :(
> I haven't been able to reproduce this on my SL machines so far, I'll try --random -p runs soon, maybe I can finally see the problem on my own...

Some results:

run-webkit-tests --tolerance 0 -p --random svg/zoom/page svg/zoom/page....
No errors.

run-webkit-tests --tolerance 0 -p --iterations 20 svg/zoom/page svg/...
No errors.

Also running without pixel tests makes no difference for me :(

Note: I moved the foo-disabled files to foo before running the tests, so I am sure they are enabled.
I've also tried -1 (singly) runs, w/o luck in reproducing.

[Adam Roben (:aroben)]

Perhaps some earlier test affects these ones? Then the tests would pass or fail depending on exactly when DRT is relaunched (which is dependent on which tests crash, how many tests are run in total, etc.).

[Nikolas Zimmermann]

(In reply to comment #12)
> Perhaps some earlier test affects these ones? Then the tests would pass or fail depending on exactly when DRT is relaunched (which is dependent on which tests crash, how many tests are run in total, etc.).

I'll try to debug more tomorrow.

[Nikolas Zimmermann]

The root of all evil is probably the bug described in 64974. I'm working on a proper fix.

[Nikolas Zimmermann]

(In reply to comment #14)
> The root of all evil is probably the bug described in 64974. I'm working on a proper fix.

The fix landed in r92545. It's better now, but there's still a problem with the page zoom tests in general, seems easy to trigger on non-mac eg gtk/win.

I'm testing following, as it seems to be a problem that we call notifyDone() right after the zooming.
-            layoutTestController.notifyDone();
+            setTimeout(function() { layoutTestController.notifyDone(); }, 0);

Let's see.

[Andrei Popescu]

(In reply to comment #15)
> (In reply to comment #14)
> > The root of all evil is probably the bug described in 64974. I'm working on a proper fix.
> 
> The fix landed in r92545. It's better now, but there's still a problem with the page zoom tests in general, seems easy to trigger on non-mac eg gtk/win.
> 
> I'm testing following, as it seems to be a problem that we call notifyDone() right after the zooming.
> -            layoutTestController.notifyDone();
> +            setTimeout(function() { layoutTestController.notifyDone(); }, 0);
> 
> Let's see.

FYI, these tests currently fail on Chromium too.

[Nikolas Zimmermann]

(In reply to comment #16)
> (In reply to comment #15)
> > (In reply to comment #14)
> > > The root of all evil is probably the bug described in 64974. I'm working on a proper fix.
> > 
> > The fix landed in r92545. It's better now, but there's still a problem with the page zoom tests in general, seems easy to trigger on non-mac eg gtk/win.
> > 
> > I'm testing following, as it seems to be a problem that we call notifyDone() right after the zooming.
> > -            layoutTestController.notifyDone();
> > +            setTimeout(function() { layoutTestController.notifyDone(); }, 0);
> > 
> > Let's see.
> 
> FYI, these tests currently fail on Chromium too.
You mean they're flakey or fail every time? Or do they just need a re-baseline?

[Andrei Popescu]

(In reply to comment #17)
> (In reply to comment #16)
> > (In reply to comment #15)
> > > (In reply to comment #14)
> > > > The root of all evil is probably the bug described in 64974. I'm working on a proper fix.
> > > 
> > > The fix landed in r92545. It's better now, but there's still a problem with the page zoom tests in general, seems easy to trigger on non-mac eg gtk/win.
> > > 
> > > I'm testing following, as it seems to be a problem that we call notifyDone() right after the zooming.
> > > -            layoutTestController.notifyDone();
> > > +            setTimeout(function() { layoutTestController.notifyDone(); }, 0);
> > > 
> > > Let's see.
> > 
> > FYI, these tests currently fail on Chromium too.
> You mean they're flakey or fail every time? Or do they just need a re-baseline?

They fail every time. I am not sure whether they need a re-baseline or not, the actual results look slightly suspicious:

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#showExpectations=true&tests=svg%2Fzoom%2Fpage%2Fzoom-svg-through-object-with-absolute-size-2.xhtml

[Nikolas Zimmermann]

(In reply to comment #18)
> They fail every time. I am not sure whether they need a re-baseline or not, the actual results look slightly suspicious:
> 
> http://test-results.appspot.com/dashboards/flakiness_dashboard.html#showExpectations=true&tests=svg%2Fzoom%2Fpage%2Fzoom-svg-through-object-with-absolute-size-2.xhtml

This is a known problem on Chromium (the extra scrollbar).
When these svg/zoom/page/zoom-svg-through-object* tests were landed the first time, they were flakey on at least Win/Chromium/Gtk, producing extra-scrollbars on Chromium all the time, but never on other platforms.

The tests should now no longer be flakey - the extra scrollbar problem on chromium remains though.
We need someone with Chromium to debug this, I fear.

Could you rebaseline them for now, to see whether they are stable at least now on Chromium?

[Andrei Popescu]

(In reply to comment #19)
> (In reply to comment #18)
> > They fail every time. I am not sure whether they need a re-baseline or not, the actual results look slightly suspicious:
> > 
> > http://test-results.appspot.com/dashboards/flakiness_dashboard.html#showExpectations=true&tests=svg%2Fzoom%2Fpage%2Fzoom-svg-through-object-with-absolute-size-2.xhtml
> 
> This is a known problem on Chromium (the extra scrollbar).
> When these svg/zoom/page/zoom-svg-through-object* tests were landed the first time, they were flakey on at least Win/Chromium/Gtk, producing extra-scrollbars on Chromium all the time, but never on other platforms.
> 
> The tests should now no longer be flakey - the extra scrollbar problem on chromium remains though.
> We need someone with Chromium to debug this, I fear.
> 
> Could you rebaseline them for now, to see whether they are stable at least now on Chromium?

Thanks for the background! I have filed:

http://code.google.com/p/chromium/issues/detail?id=92037

and Mike Reed was kind enough to volunteer looking into the issue. Mike, would you be ok with me rebaselining the tests for now?

[Adam Roben (:aroben)]

*** Bug 62143 has been marked as a duplicate of this bug. ***

[James Robinson]

They are not stable, the tests are still flaky on all chromium platforms.  The text output differs from run to run.

[James Robinson]

Example test failure (from a mac):

--- /b/build/slave/Webkit_Mac10_6__CG__dbg_/build/layout-test-results/svg/zoom/page/zoom-svg-through-object-with-percentage-size-expected.txt 
+++ /b/build/slave/Webkit_Mac10_6__CG__dbg_/build/layout-test-results/svg/zoom/page/zoom-svg-through-object-with-percentage-size-actual.txt 
@@ -25,7 +25,7 @@
                     RenderTableCell {td} at (1,90) size 343x263 [r=2 c=0 rs=1 cs=1]
                       RenderEmbeddedObject {object} at (5,5) size 333x250
                         layer at (0,0) size 333x250
-                          RenderView at (0,0) size 333x250
+                          RenderView at (0,0) size 318x235
                         layer at (0,0) size 333x250
                           RenderSVGRoot {svg} at (0,0) size 333x250
                             RenderSVGContainer {g} at (18,31) size 260x164

[Nikolas Zimmermann]

(In reply to comment #22)
> They are not stable, the tests are still flaky on all chromium platforms.  The text output differs from run to run.

That's correct, I found the root of the problem. Bug 71673 has a patch, I expect all problems to be gone then.

[Nikolas Zimmermann]

(In reply to comment #24)
> That's correct, I found the root of the problem. Bug 71673 has a patch, I expect all problems to be gone then.

r99937 has the fix. We probably need some rebaselines and expectations update for chromium. Crossing fingers and watching the flakiness dashboard, I hope extra scrollbars are gone everywhere.

[Adam Klein]

Committed r101028: <http://trac.webkit.org/changeset/101028>