Bug 122747 – Introduce a WebIDL conformance test suite

• Component: Tools / Tests
• Status: RESOLVED INVALID

[Zan Dobersek]

The WebKit project could benefit from having a set of layout tests that would check that the WebIDL support is behaving in compliance with the standard.

[Zan Dobersek]

The current idea is to have test interfaces located under window.internals and to keep them under the new 'Internals' global context so nothing is leaked into the Web platform. For now I'm only using one interface, but more will most likely be required for a complete test effort. The constructor of that interface is hung on window.internals, along with a method to create new objects that implement that interface.

The new, testing-only interfaces would be added simply to have maximum control on what is tested and how it's tested. Testing on Web-facing APIs could be disrupted by any changes to the related interfaces, and it would also unlikely cover all the corner cases.

The layout tests would use the testharness.js[1] tools and the webidl2.js[2] parser. The WebIDL of the tested interface is placed into the test as text content that is then parsed and tested against an actual object implementing that interface.

The idlharness.js documentation outlines such testing, and the actual test case for User Timing interfaces makes it even clearer.
https://github.com/w3c/testharness.js/blob/master/idlharness.js
https://github.com/w3c/web-platform-tests/blob/master/user-timing/idlharness.html

[Zan Dobersek]

Created attachment 214142 [details]
Provisional patch

Here's the provisional implementation of that idea.

The WebIDLTest interface is kept simple and only defines one operation that's then tested. Of course more stuff needs to be added, even for starters. 
The text baseline lists all the tests and their outcome. The testharness.js scripts were adjusted a bit to properly utilize window.testRunner and to also prevent printing the complete stack for failed tests since this causes the filesystem paths to be written out, producing differing output on different systems.

Build targets for new files must also be added to other build systems.

[Zan Dobersek]

(In reply to comment #1)
> The layout tests would use the testharness.js[1] tools and the webidl2.js[2] parser. The WebIDL of the tested interface is placed into the test as text content that is then parsed and tested against an actual object implementing that interface.

[1] https://github.com/w3c/testharness.js
[2] https://github.com/darobin/webidl2.js

[Zan Dobersek]

Created attachment 214941 [details]
Patch

[Build Bot]

Comment on attachment 214941 [details]
Patch

Attachment 214941 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.appspot.com/results/5958030

[Build Bot]

Comment on attachment 214941 [details]
Patch

Attachment 214941 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.appspot.com/results/9648029

[Zan Dobersek]

Created attachment 214948 [details]
Patch

[Build Bot]

Comment on attachment 214948 [details]
Patch

Attachment 214948 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.appspot.com/results/9848001

[Build Bot]

Comment on attachment 214948 [details]
Patch

Attachment 214948 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.appspot.com/results/9668004

[Zan Dobersek]

Created attachment 215032 [details]
Patch

[Zan Dobersek]

Created attachment 216812 [details]
Patch

Updated test case.

[EFL EWS Bot]

Comment on attachment 216812 [details]
Patch

Attachment 216812 [details] did not pass efl-wk2-ews (efl-wk2):
Output: http://webkit-queues.appspot.com/results/22719962

[Brady Eidson]

Comment on attachment 216812 [details]
Patch

Assuming that patches for review since 2013 are stale, r-

[Anne van Kesteren]

If there's still interest in this please increase coverage of WPT and we'll import it.