Bug 153707 – [GTK] Some W3C DOM tests are failing

• Status: RESOLVED
• Resolution: CONFIGURATION CHANGED
• Component: WebKitGTK

[Michael Catanzaro]

The following W3C DOM tests are failing on the GTK port since they were moved in r189471-r189476:

imported/w3c/web-platform-tests/dom/events/EventTarget-dispatchEvent.html [ Failure ]
imported/w3c/web-platform-tests/dom/nodes/Document-characterSet-normalization.html [ Failure ]
imported/w3c/web-platform-tests/dom/nodes/Document-createEvent.html [ Failure ]
imported/w3c/web-platform-tests/html/dom/interfaces.html [ Failure ]

[Michael Catanzaro]

(In reply to comment #0)
> The following W3C DOM tests are failing on the GTK port since they were
> moved in r189471-r189476:
> 
> imported/w3c/web-platform-tests/dom/events/EventTarget-dispatchEvent.html [
> Failure ]

This one is fixed by r200309.

[Michael Catanzaro]

Updated expectations in r200929.

[Javier Fernandez]

(In reply to comment #1)
> (In reply to comment #0)
> > The following W3C DOM tests are failing on the GTK port since they were
> > moved in r189471-r189476:
> > 
> > imported/w3c/web-platform-tests/dom/events/EventTarget-dispatchEvent.html [
> > Failure ]
> 
> This one is fixed by r200309.

Somehow, broken again in r204370-r204376

[Ms2ger (he/him; ⌚ UTC+1/+2)]

(In reply to Michael Catanzaro from comment #0)
> imported/w3c/web-platform-tests/dom/nodes/Document-createEvent.html [
> Failure ]

Fwiw, this is because TOUCH_EVENTS is enabled by default on gtk, but not upstream.

[Michael Catanzaro]

The W3C tests do not expect touch events to be enabled, seriously? Then we should move these tests to the expected failures section of our TestExpectations file, mark them to be skipped, add a comment on top, remove the bug annotation, and close this bug.

[Ms2ger (he/him; ⌚ UTC+1/+2)]

(In reply to Michael Catanzaro from comment #5)
> The W3C tests do not expect touch events to be enabled, seriously? Then we
> should move these tests to the expected failures section of our
> TestExpectations file, mark them to be skipped, add a comment on top, remove
> the bug annotation, and close this bug.

The upstream -expected file marks the touch event subtests as failing, while they pass in gtk.

[Manuel Rego Casasnovas]

(In reply to Ms2ger from comment #6)
> (In reply to Michael Catanzaro from comment #5)
> > The W3C tests do not expect touch events to be enabled, seriously? Then we
> > should move these tests to the expected failures section of our
> > TestExpectations file, mark them to be skipped, add a comment on top, remove
> > the bug annotation, and close this bug.
> 
> The upstream -expected file marks the touch event subtests as failing, while
> they pass in gtk.

-expected.txt files for WPT imported tests usually contain PASS and FAIL messages,
so they can be used as regression tests to detect if any patch
modifies their behavior.

Regarding these particular tests it seems that they pass in GTK+
but fail in Mac.

So I guess we've differnt options here (not sure what's the preferred one,
so adding @youenn in CC for feedback):
1) Update the generic -expected.txt file to use the GTK+ results (PASS).
   Then mark the tests as failure in Mac TestExpectations file.
2) Update the generic -expected.txt file to use the GTK+ results (PASS).
   And add a specific Mac -expected.txt file with the results in Mac.
3) Keep current -expected file as is. And then add a specific
   GTK+ -expected.txt file, with the GTK+ results.

[youenn fablet]

In theory, I would prefer to keep the most compliant expectation as the baseline.
In practice, whenever w3c tests are refreshed, I usually use Mac-wk2 as the baseline and other bot results are put as specific expectations.

I would go with 2 and see whether things are going well when refreshing the tests.
And reverting to 3 or improving webkitpy if something does not go well after refreshing.
I usually dislike option 1 since we cannot catch any regression anymore.


(In reply to Manuel Rego Casasnovas from comment #7)
> (In reply to Ms2ger from comment #6)
> > (In reply to Michael Catanzaro from comment #5)
> > > The W3C tests do not expect touch events to be enabled, seriously? Then we
> > > should move these tests to the expected failures section of our
> > > TestExpectations file, mark them to be skipped, add a comment on top, remove
> > > the bug annotation, and close this bug.
> > 
> > The upstream -expected file marks the touch event subtests as failing, while
> > they pass in gtk.
> 
> -expected.txt files for WPT imported tests usually contain PASS and FAIL
> messages,
> so they can be used as regression tests to detect if any patch
> modifies their behavior.
> 
> Regarding these particular tests it seems that they pass in GTK+
> but fail in Mac.
> 
> So I guess we've differnt options here (not sure what's the preferred one,
> so adding @youenn in CC for feedback):
> 1) Update the generic -expected.txt file to use the GTK+ results (PASS).
>    Then mark the tests as failure in Mac TestExpectations file.
> 2) Update the generic -expected.txt file to use the GTK+ results (PASS).
>    And add a specific Mac -expected.txt file with the results in Mac.
> 3) Keep current -expected file as is. And then add a specific
>    GTK+ -expected.txt file, with the GTK+ results.

[Lauro Moura]

The last failing test, EventTarge-dispatchEvent has been passing since the last few rounds of imports:

glib pass: r274167
ios pass: r274196