Running WebKit WPE tests locally, there are some tests that are marked as failing that pass, some that would pass if marked as slow, some that are flaky, and some that are marked as regressions, but just have out-of-date test expectations. It would be good to update all of these so we can have a new baseline, perhaps with an aim to enabling WPE tests in the future on EWS (though of course, there's a lot of work for non-WebGL tests to do before that happens). Side-bar: As a warning for anyone that wants to run these tests locally, both the WPE and Gtk ports are sensitive to the external shared-mime-info package. The Gtk port will build its own version in jhbuild, so the results can be expected to be reproducible (in this one regard), but WPE does not and so will rely on the distribution's version. In version 1.12 (latest release), it will falsely class some html documents as xhtml and cause failures due to parsing errors. This can be circumvented by adding a patched (or old) version of shared-mime-info to jhbuild using WEBKIT_EXTRA_MODULES / WEBKIT_EXTRA_MODULESETS, as documented here: https://trac.webkit.org/wiki/WebKitGtkExtendingJHBuild. I've submitted a patch upstream to fix this bug: https://gitlab.freedesktop.org/xdg/shared-mime-info/merge_requests/27 - this branch can also be used in the interim if one prefers that to downgrading to 1.10.
(In reply to Chris Lord from comment #0) > Side-bar: As a warning for anyone that wants to run these tests locally, > both the WPE and Gtk ports are sensitive to the external shared-mime-info > package. The Gtk port will build its own version in jhbuild, so the results > can be expected to be reproducible (in this one regard), but WPE does not > and so will rely on the distribution's version. In version 1.12 (latest > release), it will falsely class some html documents as xhtml and cause > failures due to parsing errors. This can be circumvented by adding a patched > (or old) version of shared-mime-info to jhbuild using WEBKIT_EXTRA_MODULES / > WEBKIT_EXTRA_MODULESETS, as documented here: I think it will be a good idea to also build a version of shared-mime-info in the WPE JHbuild.
(In reply to Carlos Alberto Lopez Perez from comment #1) > I think it will be a good idea to also build a version of shared-mime-info > in the WPE JHbuild. Filed bug 200768, will submit a patch soon.
Created attachment 376376 [details] Patch
I went through each unchecked failing/flaky test to verify if establishing a new baseline was reasonable, and in every case except oes-texture-half-float-with-image-data-expected, it corresponded to a similar Gtk or iOS failure (the two most similar back-ends). In that one exception, the test fails because of a missing extension (so it actually says passed, but it passes because it can't initialise the extension). I conducted these tests using Weston and Mesa 19.1.3 on Fedora 30 with Intel graphics, which I think is a very similar (and hopefully comparable) setup to the EWS build machine. Of course, some of the tests that are marked as failing could do with investigation, but none of the failures appear to be WPE-specific at this point.
Created attachment 376378 [details] Patch
Comment on attachment 376376 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=376376&action=review > LayoutTests/platform/wpe/TestExpectations:820 > +Bug(WPE) webgl/2.0.0/conformance/ogles/GL/swizzlers/swizzlers_049_to_056.html [ Timeout Pass ] This tests doesn't seem to timeout never on the WPE release bot, neither on the WPE debug bot (according to the tool wktesthunter). Does it timeout on your machine? > LayoutTests/platform/wpe/TestExpectations:837 > +webkit.org/b/169917 webgl/1.0.2/conformance/uniforms/out-of-bounds-uniform-array-access.html [ Slow ] This one doesn't seem to be timing out on the WPE Release or Debug bots > LayoutTests/platform/wpe/TestExpectations:841 > +Bug(WPE) webgl/2.0.0/conformance/uniforms/out-of-bounds-uniform-array-access.html [ Slow ] > +Bug(WPE) webgl/2.0.0/conformance2/buffers/one-large-uniform-buffer.html [ Slow ] None of this two seems to be timing out on the WPE Release or Debug bots > LayoutTests/platform/wpe/TestExpectations:842 > +Bug(WPE) webgl/2.0.0/conformance2/glsl3/bool-type-cast-bug-uint-ivec-uvec.html [ Slow ] This one was giving failures on the bots around r244498, but now seems to be passing fine (no timeouts) > LayoutTests/platform/wpe/TestExpectations:939 > +Bug(WPE) webgl/1.0.2/conformance/extensions/webgl-compressed-texture-size-limit.html [ Timeout ] Which test is this one? I can't find it on my webkit checkout > LayoutTests/platform/wpe/TestExpectations:944 > +Bug(WPE) webgl/1.0.2/conformance/textures/texture-size-limit.html [ Timeout ] This one is giving most of the time Failures on the WPE release test bot. I would put [ Timeout Failure ] > LayoutTests/platform/wpe/TestExpectations:965 > +Bug(WPE) webgl/2.0.0/conformance/glsl/misc/shaders-with-invariance.html [ Failure ] When reporting new failures, it is appreciated if you can open a new bug telling the test that is failing and any info about it that is known about that. - When it started failing? This tool can give hints about that: https://github.com/clopez/webkit-testhunter - What fails? Perhaps paste the text diff on the bug is a good idea. And then add that bug number to the expectations file instead of just "Bug(WPE)"
(In reply to Carlos Alberto Lopez Perez from comment #6) > > LayoutTests/platform/wpe/TestExpectations:820 > > +Bug(WPE) webgl/2.0.0/conformance/ogles/GL/swizzlers/swizzlers_049_to_056.html [ Timeout Pass ] > > This tests doesn't seem to timeout never on the WPE release bot, neither on > the WPE debug bot (according to the tool wktesthunter). > Does it timeout on your machine? > > > LayoutTests/platform/wpe/TestExpectations:837 > > +webkit.org/b/169917 webgl/1.0.2/conformance/uniforms/out-of-bounds-uniform-array-access.html [ Slow ] > > This one doesn't seem to be timing out on the WPE Release or Debug bots > > > LayoutTests/platform/wpe/TestExpectations:841 > > +Bug(WPE) webgl/2.0.0/conformance/uniforms/out-of-bounds-uniform-array-access.html [ Slow ] > > +Bug(WPE) webgl/2.0.0/conformance2/buffers/one-large-uniform-buffer.html [ Slow ] > > None of this two seems to be timing out on the WPE Release or Debug bots > > > LayoutTests/platform/wpe/TestExpectations:842 > > +Bug(WPE) webgl/2.0.0/conformance2/glsl3/bool-type-cast-bug-uint-ivec-uvec.html [ Slow ] > > This one was giving failures on the bots around r244498, but now seems to be > passing fine (no timeouts) These tests would all intermittently timeout unless marked as slow (and that still wasn't enough for swizzlers_049_to_056.html to consistently pass), what's the protocol in that case? My machine isn't weedy, but it's probably not up to buildbot standard :) > > LayoutTests/platform/wpe/TestExpectations:939 > > +Bug(WPE) webgl/1.0.2/conformance/extensions/webgl-compressed-texture-size-limit.html [ Timeout ] > > Which test is this one? I can't find it on my webkit checkout Huh, this is weird, must've been a copy/paste mistake I guess, thanks for spotting. > > LayoutTests/platform/wpe/TestExpectations:944 > > +Bug(WPE) webgl/1.0.2/conformance/textures/texture-size-limit.html [ Timeout ] > > This one is giving most of the time Failures on the WPE release test bot. I > would put [ Timeout Failure ] Okidokes. > > LayoutTests/platform/wpe/TestExpectations:965 > > +Bug(WPE) webgl/2.0.0/conformance/glsl/misc/shaders-with-invariance.html [ Failure ] > > When reporting new failures, it is appreciated if you can open a new bug > telling the test that is failing and any info about it that is known about > that. > - When it started failing? This tool can give hints about that: > https://github.com/clopez/webkit-testhunter > - What fails? Perhaps paste the text diff on the bug is a good idea. > > And then add that bug number to the expectations file instead of just > "Bug(WPE)" Will do, thanks.
Created attachment 376491 [details] Patch
Comment on attachment 376491 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=376491&action=review > LayoutTests/platform/wpe/TestExpectations:821 > +Bug(WPE) webgl/2.0.0/conformance/ogles/GL/swizzlers/swizzlers_049_to_056.html [ Timeout Pass ] > +Bug(WPE) webgl/2.0.0/conformance2/renderbuffers/multisampled-renderbuffer-initialization.html [ Failure Pass ] This are new failures reports, I would appreciate if next time you can open a bug for them and then associate the bug number here
Comment on attachment 376491 [details] Patch Clearing flags on attachment: 376491 Committed r248771: <https://trac.webkit.org/changeset/248771>
All reviewed patches have been landed. Closing bug.
*** Bug 200810 has been marked as a duplicate of this bug. ***