Gtk should start running tests in a real desktop environment and abandon current usage of Xvfb. Here's a short and incomplete pros & cons list: Pros: - WebGL tests could be run - The future Clutter port is currently doing the same, so testing facilities (as in bots) could be shared between the ports - If possible, the standardized theme (look at first con) could apply mac-style-like appearance to some elements (e.g. list menu items), making tests that are tailored for mac port style passable Cons: - A theme must be standardized and applied when testing to ensure same results across different systems - There would be windows popping around when testing is performed, but this annoyance could be reduced by minimizing them upon showing - Lots of rebaselining due to theme change If you see any other pros/cons, do share. This shall server as a metabug for these changes.
(In reply to comment #0) > This shall server as a metabug for these changes. I think it would be good to make this an option or even to intelligently figure out whether to run some tests in Xvfb or not.
(In reply to comment #0) > - WebGL tests could be run It should be possible to run those under Xvfb soon too (as a result of the work being done for http://fedoraproject.org/wiki/Features/Gnome_shell_software_rendering) > - If possible, the standardized theme (look at first con) could apply mac-style-like appearance to some elements (e.g. list menu items), making tests that are tailored for mac port style passable How is that a pro for *not* using Xvfb? If we change the default theme under the real X server then we're interfering with the user's other windows, whereas if we run Xvfb we can just run a separate xsettings manager on that server to change the default theme only there.
(In reply to comment #2) > How is that a pro for *not* using Xvfb? If we change the default theme under the real X server then we're interfering with the user's other windows, whereas if we run Xvfb we can just run a separate xsettings manager on that server to change the default theme only there. If run outside Xvfb, DumpRenderTree will need to override the GTK+ theme at runtime. This should not affect any other applications.
(In reply to comment #2) > (In reply to comment #0) > > - WebGL tests could be run > > It should be possible to run those under Xvfb soon too (as a result of the work being done for http://fedoraproject.org/wiki/Features/Gnome_shell_software_rendering) It's actually possible to run the WebGL tests in Xvfb now with some graphics hardware. I do not think Nvidia cards have this feature though.
I'm linking test expectations for the following entries to this bug: compositing/ fast/canvas/webgl/ fast/transforms/selection-bounds-in-transformed-view.html http/tests/canvas/webgl/ http/tests/security/webgl-remote-read-remote-image-allowed.html http/tests/security/webgl-remote-read-remote-image-allowed-with-credentials.html http/tests/security/webgl-remote-read-remote-image-blocked-no-crossorigin.html transforms/3d/ All these tests require drivers with OpenGL support under Xvfb to properly run.
Is this still an issue? I believe that now with llvmpipe we can run GL tests just fine inside Xvfb. And we are skipping all the webgl layout tests. I have locally tested to unskip it and everything passed. So maybe we can unskip those tests, mark the ones that fail/crash/timeout on a new bug and close this one? WDYT? I have tested to run all the 1641 tests skipped due to this bug and this was the result: => Results: 1301/1660 tests passed (78.4%) => Tests to be fixed (378): 4 crashes ( 1.1%) 51 timeouts (13.5%) 142 missing results (37.6%) 35 image-only failures ( 9.3%)
(In reply to comment #6) > So maybe we can unskip those tests, mark the ones that fail/crash/timeout on > a new bug and close this one? WDYT? I like this plan, personally.
I have opened the following bugs to track the WebGL, 3D and compositing related failures: bug 169916 (css3/blending failures) bug 169918 (compositing failures) bug 169917 (webgl failures) bug 166536 (webgl2 not enabled) I also rebaselined a bunch of tests that looked to me like they were passing fine (either the produced image was very similar to the Mac ports or the text diff was only a few pixels of difference) I have tested to run all this tests inside Xvfb vs native X11/Xorg and the differences where minimal and all related to WebGL. I summarized those differences here: https://bugs.webkit.org/show_bug.cgi?id=169917#c2 I expect the differences when running on wayland to be bigger. I tried to run the tests with the virtualized weston (by passing --display-server=weston to run-webkit-tests script) and indeed I got a big diff: Regressions: Unexpected text-only failures (156) Regressions: Unexpected image-only failures (4) Regressions: Unexpected timeouts (8) My bet is that this is caused because the nested compositor is unable to run in that environment. I will watch our Wayland bot (that runs on real wayland with an Intel GPU) to check what differences are compared to X.
Committed r214223: <http://trac.webkit.org/changeset/214223>
Committed r214229: <http://trac.webkit.org/changeset/214229>