Bug 71849 – [GTK] Move towards Xvfb-less testing

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Zan Dobersek]

Gtk should start running tests in a real desktop environment and abandon current usage of Xvfb.

Here's a short and incomplete pros & cons list:
Pros:
- WebGL tests could be run
- The future Clutter port is currently doing the same, so testing facilities (as in bots) could be shared between the ports
- If possible, the standardized theme (look at first con) could apply mac-style-like appearance to some elements (e.g. list menu items), making tests that are tailored for mac port style passable

Cons:
- A theme must be standardized and applied when testing to ensure same results across different systems
- There would be windows popping around when testing is performed, but this annoyance could be reduced by minimizing them upon showing
- Lots of rebaselining due to theme change

If you see any other pros/cons, do share.

This shall server as a metabug for these changes.

[Martin Robinson]

(In reply to comment #0)

> This shall server as a metabug for these changes.

I think it would be good to make this an option or even to intelligently figure out whether to run some tests in Xvfb or not.

[Dan Winship]

(In reply to comment #0)
> - WebGL tests could be run

It should be possible to run those under Xvfb soon too (as a result of the work being done for http://fedoraproject.org/wiki/Features/Gnome_shell_software_rendering)

> - If possible, the standardized theme (look at first con) could apply mac-style-like appearance to some elements (e.g. list menu items), making tests that are tailored for mac port style passable

How is that a pro for *not* using Xvfb? If we change the default theme under the real X server then we're interfering with the user's other windows, whereas if we run Xvfb we can just run a separate xsettings manager on that server to change the default theme only there.

[Martin Robinson]

(In reply to comment #2)
> How is that a pro for *not* using Xvfb? If we change the default theme under the real X server then we're interfering with the user's other windows, whereas if we run Xvfb we can just run a separate xsettings manager on that server to change the default theme only there.

If run outside Xvfb, DumpRenderTree will need to override the GTK+ theme at runtime. This  should not affect any other applications.

[Martin Robinson]

(In reply to comment #2)
> (In reply to comment #0)
> > - WebGL tests could be run
> 
> It should be possible to run those under Xvfb soon too (as a result of the work being done for http://fedoraproject.org/wiki/Features/Gnome_shell_software_rendering)

It's actually possible to run the WebGL tests in Xvfb now with some graphics hardware. I do not think Nvidia cards have this feature though.

[Zan Dobersek]

I'm linking test expectations for the following entries to this bug:
compositing/
fast/canvas/webgl/
fast/transforms/selection-bounds-in-transformed-view.html
http/tests/canvas/webgl/
http/tests/security/webgl-remote-read-remote-image-allowed.html
http/tests/security/webgl-remote-read-remote-image-allowed-with-credentials.html
http/tests/security/webgl-remote-read-remote-image-blocked-no-crossorigin.html
transforms/3d/

All these tests require drivers with OpenGL support under Xvfb to properly run.

[Carlos Alberto Lopez Perez]

Is this still an issue?

I believe that now with llvmpipe we can run GL tests just fine inside Xvfb.

And we are skipping all the webgl layout tests.

I have locally tested to unskip it and everything passed.

So maybe we can unskip those tests, mark the ones that fail/crash/timeout on a new bug and close this one? WDYT?

I have tested to run all the 1641 tests skipped due to this bug and this was the result:

=> Results: 1301/1660 tests passed (78.4%)

=> Tests to be fixed (378):
      4 crashes                  ( 1.1%)
     51 timeouts                 (13.5%)
    142 missing results          (37.6%)
     35 image-only failures      ( 9.3%)

[Martin Robinson]

(In reply to comment #6)

> So maybe we can unskip those tests, mark the ones that fail/crash/timeout on
> a new bug and close this one? WDYT?

I like this plan, personally.

[Carlos Alberto Lopez Perez]

I have opened the following bugs to track the WebGL, 3D and compositing related failures:

bug 169916 (css3/blending failures)
bug 169918 (compositing failures)
bug 169917 (webgl failures)
bug 166536 (webgl2 not enabled)

I also rebaselined a bunch of tests that looked to me like they were passing fine
(either the produced image was very similar to the Mac ports or the text diff was only a few pixels of difference)

I have tested to run all this tests inside Xvfb vs native X11/Xorg and the differences
where minimal and all related to WebGL. I summarized those differences here: https://bugs.webkit.org/show_bug.cgi?id=169917#c2

I expect the differences when running on wayland to be bigger.
I tried to run the tests with the virtualized weston (by passing --display-server=weston to run-webkit-tests script)
and indeed I got a big diff:

Regressions: Unexpected text-only failures (156)
Regressions: Unexpected image-only failures (4)
Regressions: Unexpected timeouts (8)


My bet is that this is caused because the nested compositor is unable to run in that environment. I will watch our Wayland bot (that runs on real wayland with an Intel GPU) to check what differences are compared to X.

[Carlos Alberto Lopez Perez]

Committed r214223: <http://trac.webkit.org/changeset/214223>

[Carlos Alberto Lopez Perez]

Committed r214229: <http://trac.webkit.org/changeset/214229>