Bug 149645 – Update web-platform-tests tools to the latest revision

• Component: Tools / Tests
• Status: RESOLVED FIXED

[youenn fablet]

web-platform-tests tools changed from as a submodule of submodules.
This should be reflected in WebKit test infrastructure.

[youenn fablet]

Created attachment 262103 [details]
Patch

[Build Bot]

Comment on attachment 262103 [details]
Patch

Attachment 262103 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/224663

New failing tests:
imported/w3c/web-platform-tests/html/dom/interfaces.html
imported/w3c/web-platform-tests/dom/interfaces.html

[Build Bot]

Created attachment 262105 [details]
Archive of layout-test-results from ews100 for mac-mavericks

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews100  Port: mac-mavericks  Platform: Mac OS X 10.9.5

[Build Bot]

Comment on attachment 262103 [details]
Patch

Attachment 262103 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/224687

New failing tests:
imported/w3c/web-platform-tests/html/dom/interfaces.html
imported/w3c/web-platform-tests/dom/interfaces.html

[Build Bot]

Created attachment 262108 [details]
Archive of layout-test-results from ews104 for mac-mavericks-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews104  Port: mac-mavericks-wk2  Platform: Mac OS X 10.9.5

[youenn fablet]

Created attachment 262148 [details]
Rebasing failing test expectations

[Chris Dumez]

Comment on attachment 262148 [details]
Rebasing failing test expectations

View in context: https://bugs.webkit.org/attachment.cgi?id=262148&action=review

> LayoutTests/TestExpectations:245
> +webkit.org/b/149656 imported/w3c/web-platform-tests/html/dom/interfaces.html [ Timeout ]

This seems really unfortunate. I wish this patch did not need to skip this test. We are working on improving our DOM / HTML compliance and rely on this test a lot.

[Chris Dumez]

Comment on attachment 262148 [details]
Rebasing failing test expectations

View in context: https://bugs.webkit.org/attachment.cgi?id=262148&action=review

> LayoutTests/imported/w3c/web-platform-tests/dom/interfaces-expected.txt:958
> +FAIL Element interface: operation before([object Object],[object Object]) assert_equals: property has wrong .length expected 0 but got 1

We should file a bug about this. Basically our length is 1 instead of 0 for operations taking a variadic arguments. This should be fixed.

[youenn fablet]

(In reply to comment #7)
> Comment on attachment 262148 [details]
> Rebasing failing test expectations
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=262148&action=review
> 
> > LayoutTests/TestExpectations:245
> > +webkit.org/b/149656 imported/w3c/web-platform-tests/html/dom/interfaces.html [ Timeout ]
> 
> This seems really unfortunate. I wish this patch did not need to skip this
> test. We are working on improving our DOM / HTML compliance and rely on this
> test a lot.

I'll postpone landing this patch until finding a solution for this one then.

[Chris Dumez]

(In reply to comment #9)
> (In reply to comment #7)
> > Comment on attachment 262148 [details]
> > Rebasing failing test expectations
> > 
> > View in context:
> > https://bugs.webkit.org/attachment.cgi?id=262148&action=review
> > 
> > > LayoutTests/TestExpectations:245
> > > +webkit.org/b/149656 imported/w3c/web-platform-tests/html/dom/interfaces.html [ Timeout ]
> > 
> > This seems really unfortunate. I wish this patch did not need to skip this
> > test. We are working on improving our DOM / HTML compliance and rely on this
> > test a lot.
> 
> I'll postpone landing this patch until finding a solution for this one then.

FYI, I think I had a similar problem when initially importing this test (would always timeout) and I worked around the issue by commenting out one of the interfaces being tested (Window iirc). Chances are you can do the same here.

[Chris Dumez]

(In reply to comment #8)
> Comment on attachment 262148 [details]
> Rebasing failing test expectations
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=262148&action=review
> 
> > LayoutTests/imported/w3c/web-platform-tests/dom/interfaces-expected.txt:958
> > +FAIL Element interface: operation before([object Object],[object Object]) assert_equals: property has wrong .length expected 0 but got 1
> 
> We should file a bug about this. Basically our length is 1 instead of 0 for
> operations taking a variadic arguments. This should be fixed.

I filed <https://bugs.webkit.org/show_bug.cgi?id=149714> and I am working on fixing the bug.

[youenn fablet]

(In reply to comment #11)
> (In reply to comment #8)
> > Comment on attachment 262148 [details]
> > Rebasing failing test expectations
> > 
> > View in context:
> > https://bugs.webkit.org/attachment.cgi?id=262148&action=review
> > 
> I filed <https://bugs.webkit.org/show_bug.cgi?id=149714> and I am working on
> fixing the bug.

Thanks for tackling this issue.
That said, I think that we will not be able to always resync and fix bugs at the same time.
This calls to two tools IMHO:
- Ensuring no regression: we have it with expected files
- Ensuring conformance: we do not have any really

Doing a quick grep, we have 185 tests out of 936 web-platform-tests that have "FAIL" in their expected files. There is a risk that we forget to fix them.
Having a tool to monitor expected files might be useful.

[youenn fablet]

(In reply to comment #10)
> (In reply to comment #9)
> > (In reply to comment #7)
> > > Comment on attachment 262148 [details]
> > > Rebasing failing test expectations
> > > 
> > > View in context:
> > > https://bugs.webkit.org/attachment.cgi?id=262148&action=review
> > > 
> > > > LayoutTests/TestExpectations:245
> > > > +webkit.org/b/149656 imported/w3c/web-platform-tests/html/dom/interfaces.html [ Timeout ]
> > > 
> > > This seems really unfortunate. I wish this patch did not need to skip this
> > > test. We are working on improving our DOM / HTML compliance and rely on this
> > > test a lot.
> > 
> > I'll postpone landing this patch until finding a solution for this one then.
> 
> FYI, I think I had a similar problem when initially importing this test
> (would always timeout) and I worked around the issue by commenting out one
> of the interfaces being tested (Window iirc). Chances are you can do the
> same here.

It might be good to keep track of these manual changes since they may get lost when resyncing.
Maybe by adding a comment so that it appears on the diff when resyncing?
Or uploading the changes to wpt directly?

[youenn fablet]

> > FYI, I think I had a similar problem when initially importing this test
> > (would always timeout) and I worked around the issue by commenting out one
> > of the interfaces being tested (Window iirc). Chances are you can do the
> > same here.

It seems commenting out showModalDialog method from Window interface is fixing the timeout. Any idea why it is doing so and how to fix that?

Not upgrading the testharness.js submodule revision also fixes that issue.
This upgrade should be done at the same time we update WebKit testharness.js file (currently we are using WebKit testharness.js file for WPT tests as well).

[youenn fablet]

Created attachment 263069 [details]
Patch

[youenn fablet]

(In reply to comment #15)
> Created attachment 263069 [details]
> Patch

Compared to r+ patch, the following changes have been made:
- Postpone testharness module upgrade to a later patch.
- Remove TestExpectations changes since they should not be needed anymore.

[Build Bot]

Comment on attachment 263069 [details]
Patch

Attachment 263069 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/285106

New failing tests:
streams/readable-stream-controller-error.html

[Build Bot]

Created attachment 263071 [details]
Archive of layout-test-results from ews103 for mac-mavericks

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-mavericks  Platform: Mac OS X 10.9.5

[Build Bot]

Comment on attachment 263069 [details]
Patch

Attachment 263069 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/285110

New failing tests:
streams/readable-stream-controller-error.html

[Build Bot]

Created attachment 263072 [details]
Archive of layout-test-results from ews106 for mac-mavericks-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews106  Port: mac-mavericks-wk2  Platform: Mac OS X 10.9.5

[youenn fablet]

Created attachment 263073 [details]
Cleaning patch

[youenn fablet]

Created attachment 263074 [details]
Patch for landing

[WebKit Commit Bot]

Comment on attachment 263074 [details]
Patch for landing

Clearing flags on attachment: 263074

Committed r191043: <http://trac.webkit.org/changeset/191043>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.