Bug 200765 – Update WebGL test expectations for WebKit WPE

• Status: RESOLVED
• Resolution: FIXED
• Component: WPE WebKit

[Chris Lord]

Running WebKit WPE tests locally, there are some tests that are marked as failing that pass, some that would pass if marked as slow, some that are flaky, and some that are marked as regressions, but just have out-of-date test expectations. It would be good to update all of these so we can have a new baseline, perhaps with an aim to enabling WPE tests in the future on EWS (though of course, there's a lot of work for non-WebGL tests to do before that happens).

Side-bar: As a warning for anyone that wants to run these tests locally, both the WPE and Gtk ports are sensitive to the external shared-mime-info package. The Gtk port will build its own version in jhbuild, so the results can be expected to be reproducible (in this one regard), but WPE does not and so will rely on the distribution's version. In version 1.12 (latest release), it will falsely class some html documents as xhtml and cause failures due to parsing errors. This can be circumvented by adding a patched (or old) version of shared-mime-info to jhbuild using WEBKIT_EXTRA_MODULES / WEBKIT_EXTRA_MODULESETS, as documented here: https://trac.webkit.org/wiki/WebKitGtkExtendingJHBuild. I've submitted a patch upstream to fix this bug: https://gitlab.freedesktop.org/xdg/shared-mime-info/merge_requests/27 - this branch can also be used in the interim if one prefers that to downgrading to 1.10.

[Carlos Alberto Lopez Perez]

(In reply to Chris Lord from comment #0)

> Side-bar: As a warning for anyone that wants to run these tests locally,
> both the WPE and Gtk ports are sensitive to the external shared-mime-info
> package. The Gtk port will build its own version in jhbuild, so the results
> can be expected to be reproducible (in this one regard), but WPE does not
> and so will rely on the distribution's version. In version 1.12 (latest
> release), it will falsely class some html documents as xhtml and cause
> failures due to parsing errors. This can be circumvented by adding a patched
> (or old) version of shared-mime-info to jhbuild using WEBKIT_EXTRA_MODULES /
> WEBKIT_EXTRA_MODULESETS, as documented here:

I think it will be a good idea to also build a version of shared-mime-info in the WPE JHbuild.

[Chris Lord]

(In reply to Carlos Alberto Lopez Perez from comment #1)
> I think it will be a good idea to also build a version of shared-mime-info
> in the WPE JHbuild.

Filed bug 200768, will submit a patch soon.

[Chris Lord]

Created attachment 376376 [details]
Patch

[Chris Lord]

I went through each unchecked failing/flaky test to verify if establishing a new baseline was reasonable, and in every case except oes-texture-half-float-with-image-data-expected, it corresponded to a similar Gtk or iOS failure (the two most similar back-ends). In that one exception, the test fails because of a missing extension (so it actually says passed, but it passes because it can't initialise the extension).

I conducted these tests using Weston and Mesa 19.1.3 on Fedora 30 with Intel graphics, which I think is a very similar (and hopefully comparable) setup to the EWS build machine.

Of course, some of the tests that are marked as failing could do with investigation, but none of the failures appear to be WPE-specific at this point.

[Chris Lord]

Created attachment 376378 [details]
Patch

[Carlos Alberto Lopez Perez]

Comment on attachment 376376 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=376376&action=review

> LayoutTests/platform/wpe/TestExpectations:820
> +Bug(WPE) webgl/2.0.0/conformance/ogles/GL/swizzlers/swizzlers_049_to_056.html [ Timeout Pass ]

This tests doesn't seem to timeout never on the WPE release bot, neither on the WPE debug bot (according to the tool wktesthunter).
Does it timeout on your machine?

> LayoutTests/platform/wpe/TestExpectations:837
> +webkit.org/b/169917 webgl/1.0.2/conformance/uniforms/out-of-bounds-uniform-array-access.html [ Slow ]

This one doesn't seem to be timing out on the WPE Release or Debug bots

> LayoutTests/platform/wpe/TestExpectations:841
> +Bug(WPE) webgl/2.0.0/conformance/uniforms/out-of-bounds-uniform-array-access.html [ Slow ]
> +Bug(WPE) webgl/2.0.0/conformance2/buffers/one-large-uniform-buffer.html [ Slow ]

None of this two seems to be timing out on the WPE Release or Debug bots

> LayoutTests/platform/wpe/TestExpectations:842
> +Bug(WPE) webgl/2.0.0/conformance2/glsl3/bool-type-cast-bug-uint-ivec-uvec.html [ Slow ]

This one was giving failures on the bots around r244498, but now seems to be passing fine (no timeouts)

> LayoutTests/platform/wpe/TestExpectations:939
> +Bug(WPE) webgl/1.0.2/conformance/extensions/webgl-compressed-texture-size-limit.html [ Timeout ]

Which test is this one? I can't find it on my webkit checkout

> LayoutTests/platform/wpe/TestExpectations:944
> +Bug(WPE) webgl/1.0.2/conformance/textures/texture-size-limit.html [ Timeout ]

This one is giving most of the time Failures on the WPE release test bot. I would put [ Timeout Failure ]

> LayoutTests/platform/wpe/TestExpectations:965
> +Bug(WPE) webgl/2.0.0/conformance/glsl/misc/shaders-with-invariance.html [ Failure ]

When reporting new failures, it is appreciated if you can open a new bug telling the test that is failing and any info about it that is known about that.
- When it started failing? This tool can give hints about that: https://github.com/clopez/webkit-testhunter
- What fails? Perhaps paste the text diff on the bug is a good idea.

And then add that bug number to the expectations file instead of just "Bug(WPE)"

[Chris Lord]

(In reply to Carlos Alberto Lopez Perez from comment #6)
> > LayoutTests/platform/wpe/TestExpectations:820
> > +Bug(WPE) webgl/2.0.0/conformance/ogles/GL/swizzlers/swizzlers_049_to_056.html [ Timeout Pass ]
> 
> This tests doesn't seem to timeout never on the WPE release bot, neither on
> the WPE debug bot (according to the tool wktesthunter).
> Does it timeout on your machine?
> 
> > LayoutTests/platform/wpe/TestExpectations:837
> > +webkit.org/b/169917 webgl/1.0.2/conformance/uniforms/out-of-bounds-uniform-array-access.html [ Slow ]
> 
> This one doesn't seem to be timing out on the WPE Release or Debug bots
> 
> > LayoutTests/platform/wpe/TestExpectations:841
> > +Bug(WPE) webgl/2.0.0/conformance/uniforms/out-of-bounds-uniform-array-access.html [ Slow ]
> > +Bug(WPE) webgl/2.0.0/conformance2/buffers/one-large-uniform-buffer.html [ Slow ]
> 
> None of this two seems to be timing out on the WPE Release or Debug bots
> 
> > LayoutTests/platform/wpe/TestExpectations:842
> > +Bug(WPE) webgl/2.0.0/conformance2/glsl3/bool-type-cast-bug-uint-ivec-uvec.html [ Slow ]
> 
> This one was giving failures on the bots around r244498, but now seems to be
> passing fine (no timeouts)

These tests would all intermittently timeout unless marked as slow (and that still wasn't enough for swizzlers_049_to_056.html to consistently pass), what's the protocol in that case? My machine isn't weedy, but it's probably not up to buildbot standard :)

> > LayoutTests/platform/wpe/TestExpectations:939
> > +Bug(WPE) webgl/1.0.2/conformance/extensions/webgl-compressed-texture-size-limit.html [ Timeout ]
> 
> Which test is this one? I can't find it on my webkit checkout

Huh, this is weird, must've been a copy/paste mistake I guess, thanks for spotting.

> > LayoutTests/platform/wpe/TestExpectations:944
> > +Bug(WPE) webgl/1.0.2/conformance/textures/texture-size-limit.html [ Timeout ]
> 
> This one is giving most of the time Failures on the WPE release test bot. I
> would put [ Timeout Failure ]

Okidokes.

> > LayoutTests/platform/wpe/TestExpectations:965
> > +Bug(WPE) webgl/2.0.0/conformance/glsl/misc/shaders-with-invariance.html [ Failure ]
> 
> When reporting new failures, it is appreciated if you can open a new bug
> telling the test that is failing and any info about it that is known about
> that.
> - When it started failing? This tool can give hints about that:
> https://github.com/clopez/webkit-testhunter
> - What fails? Perhaps paste the text diff on the bug is a good idea.
> 
> And then add that bug number to the expectations file instead of just
> "Bug(WPE)"

Will do, thanks.

[Chris Lord]

Created attachment 376491 [details]
Patch

[Carlos Alberto Lopez Perez]

Comment on attachment 376491 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=376491&action=review

> LayoutTests/platform/wpe/TestExpectations:821
> +Bug(WPE) webgl/2.0.0/conformance/ogles/GL/swizzlers/swizzlers_049_to_056.html [ Timeout Pass ]
> +Bug(WPE) webgl/2.0.0/conformance2/renderbuffers/multisampled-renderbuffer-initialization.html [ Failure Pass ]

This are new failures reports, I would appreciate if next time you can open a bug for them and then associate the bug number here

[WebKit Commit Bot]

Comment on attachment 376491 [details]
Patch

Clearing flags on attachment: 376491

Committed r248771: <https://trac.webkit.org/changeset/248771>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Chris Lord]

*** Bug 200810 has been marked as a duplicate of this bug. ***