Bug 148546 – Import W3C DOM test suite from github.com/w3c/web-platform-tests

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Chris Dumez]

Import W3C DOM test suite from github.com/w3c/web-platform-tests

[Chris Dumez]

Created attachment 260098 [details]
Patch

[Build Bot]

Comment on attachment 260098 [details]
Patch

Attachment 260098 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/114504

New failing tests:
http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/bare_mathml.svg
http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/minimal_html.svg
http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/bare_xhtml.svg
http/tests/w3c/dom/nodes/Document-createElement-namespace.html
http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/bare_svg.svg
http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/xhtml_ns_removed.svg
http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/xhtml.svg
http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/svg.svg
http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/xhtml_ns_changed.svg
http/tests/w3c/dom/nodes/Document-createElement-namespace-tests/mathml.svg

[Build Bot]

Created attachment 260104 [details]
Archive of layout-test-results from ews101 for mac-mavericks

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews101  Port: mac-mavericks  Platform: Mac OS X 10.9.5

[Build Bot]

Created attachment 260106 [details]
Archive of layout-test-results from ews107 for mac-mavericks-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews107  Port: mac-mavericks-wk2  Platform: Mac OS X 10.9.5

[Chris Dumez]

Created attachment 260111 [details]
Patch

[Build Bot]

Comment on attachment 260111 [details]
Patch

Attachment 260111 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/114975

New failing tests:
http/tests/w3c/dom/nodes/Document-createElement-namespace.html

[Build Bot]

Created attachment 260113 [details]
Archive of layout-test-results from ews100 for mac-mavericks

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews100  Port: mac-mavericks  Platform: Mac OS X 10.9.5

[Build Bot]

Created attachment 260114 [details]
Archive of layout-test-results from ews106 for mac-mavericks-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews106  Port: mac-mavericks-wk2  Platform: Mac OS X 10.9.5

[Chris Dumez]

Created attachment 260115 [details]
Patch

[Alexey Proskuryakov]

http://www.w3.org/Consortium/Legal/2008/04-testsuite-copyright.html says that this is available under three-clause BSD license, which is good.

Can't really review an 8-megabyte patch. What are the tests that need http there? Can others be run without http?

[Alexey Proskuryakov]

Also, how long do these tests take to run?

[Chris Dumez]

(In reply to comment #10)
> http://www.w3.org/Consortium/Legal/2008/04-testsuite-copyright.html says
> that this is available under three-clause BSD license, which is good.
> 
> Can't really review an 8-megabyte patch. What are the tests that need http
> there? Can others be run without http?

Yes, some of them do require HTTP, which is why I moved it there. I did not want to split the tests because it will make updating (re-importing) those tests more difficult.

[Chris Dumez]

(In reply to comment #11)
> Also, how long do these tests take to run?

Those tests are pretty fast, it takes less than 30 seconds locally. There ~250 tests if I remember correctly.

[Alexey Proskuryakov]

Is it 30 seconds in debug build, or in release, and is the machine a fast one? If it adds 30 seconds to a release test run on a 12-core Mac, then we may need to look into making these faster.

[Chris Dumez]

Created attachment 260228 [details]
Patch

[Chris Dumez]

(In reply to comment #14)
> Is it 30 seconds in debug build, or in release, and is the machine a fast
> one? If it adds 30 seconds to a release test run on a 12-core Mac, then we
> may need to look into making these faster.

I have skipped a few slow tests in debug. I have measured that the run time is now ~30 seconds for a debug build on a MacBookPro, including the http server bring up / tear down. Hopefully, this is acceptable? This brings ~200 tests with good coverage of the DOM. Also note that there are a lot of checks that we are currently failing and which I started to work on.

[Alexey Proskuryakov]

Comment on attachment 260228 [details]
Patch

rs=me

[WebKit Commit Bot]

Comment on attachment 260228 [details]
Patch

Clearing flags on attachment: 260228

Committed r189155: <http://trac.webkit.org/changeset/189155>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Alexey Proskuryakov]

One failure on bots:

http/tests/w3c/dom/nodes/Element-matches.html

@@ -171,7 +171,7 @@
 PASS In-document Element.matches: :empty pseudo-class selector, matching all empty elements (with no refNodes): #pseudo-empty :empty 
 PASS In-document Element.matches: :link and :visited pseudo-class selectors, matching a and area elements with href attributes (with no refNodes): #pseudo-link :link, #pseudo-link :visited 
 PASS In-document Element.matches: :link and :visited pseudo-class selectors, matching link elements with href attributes (with no refNodes): #head :link, #head :visited 
-PASS In-document Element.matches: :target pseudo-class selector, matching the element referenced by the URL fragment identifier (with no refNodes): :target 
+FAIL In-document Element.matches: :target pseudo-class selector, matching the element referenced by the URL fragment identifier (with no refNodes): :target assert_true: The element #target should match the selector. expected true got false
 PASS In-document Element.matches: :lang pseudo-class selector, matching inherited language (with no refNodes): #pseudo-lang-div1:lang(en) 
 PASS In-document Element.matches: :lang pseudo-class selector, matching specified language with exact value (with no refNodes): #pseudo-lang-div2:lang(fr) 
 PASS In-document Element.matches: :lang pseudo-class selector, matching specified language with partial value (with no refNodes): #pseudo-lang-div3:lang(en) 

EWS missed it because it treats flaky tests as passing.

[Chris Dumez]

(In reply to comment #20)
> One failure on bots:
> 
> http/tests/w3c/dom/nodes/Element-matches.html
> 
> @@ -171,7 +171,7 @@
>  PASS In-document Element.matches: :empty pseudo-class selector, matching
> all empty elements (with no refNodes): #pseudo-empty :empty 
>  PASS In-document Element.matches: :link and :visited pseudo-class
> selectors, matching a and area elements with href attributes (with no
> refNodes): #pseudo-link :link, #pseudo-link :visited 
>  PASS In-document Element.matches: :link and :visited pseudo-class
> selectors, matching link elements with href attributes (with no refNodes):
> #head :link, #head :visited 
> -PASS In-document Element.matches: :target pseudo-class selector, matching
> the element referenced by the URL fragment identifier (with no refNodes):
> :target 
> +FAIL In-document Element.matches: :target pseudo-class selector, matching
> the element referenced by the URL fragment identifier (with no refNodes):
> :target assert_true: The element #target should match the selector. expected
> true got false
>  PASS In-document Element.matches: :lang pseudo-class selector, matching
> inherited language (with no refNodes): #pseudo-lang-div1:lang(en) 
>  PASS In-document Element.matches: :lang pseudo-class selector, matching
> specified language with exact value (with no refNodes):
> #pseudo-lang-div2:lang(fr) 
>  PASS In-document Element.matches: :lang pseudo-class selector, matching
> specified language with partial value (with no refNodes):
> #pseudo-lang-div3:lang(en) 
> 
> EWS missed it because it treats flaky tests as passing.

Oh, I simply rebaselined it in https://trac.webkit.org/r189158 when I saw it failed on all bots. If you're saying it's flaky, I guess I should mark this test as flaky instead?

[Alexey Proskuryakov]

It seems more likely that it's not randomly flaky, but that it fails when run after a certain other test - meaning that there is a real WebCore bug.

[Chris Dumez]

(In reply to comment #22)
> It seems more likely that it's not randomly flaky, but that it fails when
> run after a certain other test - meaning that there is a real WebCore bug.

Agreed, I will investigate.

[Alexey Proskuryakov]

There are Windows test failures too: 

http/tests/w3c/dom/interfaces.html
http/tests/w3c/dom/traversal/NodeFilter-constants.html

https://build.webkit.org/results/Apple%20Win%207%20Release%20(Tests)/r189161%20(53771)/results.html

[Chris Dumez]

(In reply to comment #24)
> There are Windows test failures too: 
> 
> http/tests/w3c/dom/interfaces.html
> http/tests/w3c/dom/traversal/NodeFilter-constants.html
> 
> https://build.webkit.org/results/Apple%20Win%207%20Release%20(Tests)/
> r189161%20(53771)/results.html

I am working on at least the second one via https://bugs.webkit.org/show_bug.cgi?id=148602. This is an actual bug in the implementation :) I'll check out the second one.

[Chris Dumez]

rdar://problem/22543901

[youenn fablet]

(In reply to comment #12)
> (In reply to comment #10)
> > http://www.w3.org/Consortium/Legal/2008/04-testsuite-copyright.html says
> > that this is available under three-clause BSD license, which is good.
> > 
> > Can't really review an 8-megabyte patch. What are the tests that need http
> > there? Can others be run without http?
> 
> Yes, some of them do require HTTP, which is why I moved it there. I did not
> want to split the tests because it will make updating (re-importing) those
> tests more difficult.

https://bugs.webkit.org/show_bug.cgi?id=127095#c11 has some interesting information about how Firefox is doing.

Re-importing is important and should be made as easy as possible.
In general, I think all wpt tests should go to imported/w3c/web-platform-tests.

If at some point, we see that tests take too long to run due to being served by HTTP, we might want to evaluate how to improve things.

Also, we should try to import tests from the same github revision, hence using the import-w3c-tests script.
I will try to take some time to resync the different tests with a recent wpt revision (see bug 149645 and bug 149656).