Bug 153707 - [GTK] Some W3C DOM tests are failing
Summary: [GTK] Some W3C DOM tests are failing
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebKitGTK (show other bugs)
Version: Other
Hardware: PC Linux
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on: 175022
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-30 08:46 PST by Michael Catanzaro
Modified: 2017-08-01 15:38 PDT (History)
7 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Catanzaro 2016-01-30 08:46:49 PST
The following W3C DOM tests are failing on the GTK port since they were moved in r189471-r189476:

imported/w3c/web-platform-tests/dom/events/EventTarget-dispatchEvent.html [ Failure ]
imported/w3c/web-platform-tests/dom/nodes/Document-characterSet-normalization.html [ Failure ]
imported/w3c/web-platform-tests/dom/nodes/Document-createEvent.html [ Failure ]
imported/w3c/web-platform-tests/html/dom/interfaces.html [ Failure ]
Comment 1 Michael Catanzaro 2016-05-15 14:53:12 PDT
(In reply to comment #0)
> The following W3C DOM tests are failing on the GTK port since they were
> moved in r189471-r189476:
> 
> imported/w3c/web-platform-tests/dom/events/EventTarget-dispatchEvent.html [
> Failure ]

This one is fixed by r200309.
Comment 2 Michael Catanzaro 2016-05-15 15:06:43 PDT
Updated expectations in r200929.
Comment 3 Javier Fernandez 2016-09-15 09:35:49 PDT
(In reply to comment #1)
> (In reply to comment #0)
> > The following W3C DOM tests are failing on the GTK port since they were
> > moved in r189471-r189476:
> > 
> > imported/w3c/web-platform-tests/dom/events/EventTarget-dispatchEvent.html [
> > Failure ]
> 
> This one is fixed by r200309.

Somehow, broken again in r204370-r204376
Comment 4 Ms2ger (he/him; ⌚ UTC+1/+2) 2017-07-26 05:08:57 PDT
(In reply to Michael Catanzaro from comment #0)
> imported/w3c/web-platform-tests/dom/nodes/Document-createEvent.html [
> Failure ]

Fwiw, this is because TOUCH_EVENTS is enabled by default on gtk, but not upstream.
Comment 5 Michael Catanzaro 2017-07-26 09:18:28 PDT
The W3C tests do not expect touch events to be enabled, seriously? Then we should move these tests to the expected failures section of our TestExpectations file, mark them to be skipped, add a comment on top, remove the bug annotation, and close this bug.
Comment 6 Ms2ger (he/him; ⌚ UTC+1/+2) 2017-07-27 01:21:57 PDT
(In reply to Michael Catanzaro from comment #5)
> The W3C tests do not expect touch events to be enabled, seriously? Then we
> should move these tests to the expected failures section of our
> TestExpectations file, mark them to be skipped, add a comment on top, remove
> the bug annotation, and close this bug.

The upstream -expected file marks the touch event subtests as failing, while they pass in gtk.
Comment 7 Manuel Rego Casasnovas 2017-07-27 23:10:47 PDT
(In reply to Ms2ger from comment #6)
> (In reply to Michael Catanzaro from comment #5)
> > The W3C tests do not expect touch events to be enabled, seriously? Then we
> > should move these tests to the expected failures section of our
> > TestExpectations file, mark them to be skipped, add a comment on top, remove
> > the bug annotation, and close this bug.
> 
> The upstream -expected file marks the touch event subtests as failing, while
> they pass in gtk.

-expected.txt files for WPT imported tests usually contain PASS and FAIL messages,
so they can be used as regression tests to detect if any patch
modifies their behavior.

Regarding these particular tests it seems that they pass in GTK+
but fail in Mac.

So I guess we've differnt options here (not sure what's the preferred one,
so adding @youenn in CC for feedback):
1) Update the generic -expected.txt file to use the GTK+ results (PASS).
   Then mark the tests as failure in Mac TestExpectations file.
2) Update the generic -expected.txt file to use the GTK+ results (PASS).
   And add a specific Mac -expected.txt file with the results in Mac.
3) Keep current -expected file as is. And then add a specific
   GTK+ -expected.txt file, with the GTK+ results.
Comment 8 youenn fablet 2017-08-01 15:38:19 PDT
In theory, I would prefer to keep the most compliant expectation as the baseline.
In practice, whenever w3c tests are refreshed, I usually use Mac-wk2 as the baseline and other bot results are put as specific expectations.

I would go with 2 and see whether things are going well when refreshing the tests.
And reverting to 3 or improving webkitpy if something does not go well after refreshing.
I usually dislike option 1 since we cannot catch any regression anymore.


(In reply to Manuel Rego Casasnovas from comment #7)
> (In reply to Ms2ger from comment #6)
> > (In reply to Michael Catanzaro from comment #5)
> > > The W3C tests do not expect touch events to be enabled, seriously? Then we
> > > should move these tests to the expected failures section of our
> > > TestExpectations file, mark them to be skipped, add a comment on top, remove
> > > the bug annotation, and close this bug.
> > 
> > The upstream -expected file marks the touch event subtests as failing, while
> > they pass in gtk.
> 
> -expected.txt files for WPT imported tests usually contain PASS and FAIL
> messages,
> so they can be used as regression tests to detect if any patch
> modifies their behavior.
> 
> Regarding these particular tests it seems that they pass in GTK+
> but fail in Mac.
> 
> So I guess we've differnt options here (not sure what's the preferred one,
> so adding @youenn in CC for feedback):
> 1) Update the generic -expected.txt file to use the GTK+ results (PASS).
>    Then mark the tests as failure in Mac TestExpectations file.
> 2) Update the generic -expected.txt file to use the GTK+ results (PASS).
>    And add a specific Mac -expected.txt file with the results in Mac.
> 3) Keep current -expected file as is. And then add a specific
>    GTK+ -expected.txt file, with the GTK+ results.