Bug 134766

Summary: Add support for automated W3C conformance testing
Product: WebKit Reporter: youenn fablet <youennf>
Component: Tools / TestsAssignee: youenn fablet <youennf>
Status: NEW ---    
Severity: Normal CC: agouaillard, ap, bfulgham, bjonesbe, commit-queue, glenn, rhauck, rniwa
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on: 134764, 136848    
Bug Blocks: 134767    
Attachments:
Description Flags
Patch
none
Adding static files
none
Start adding support for ref tests
none
Support of all tests, result generated in WebKitBuild sub-folder
none
updated according 136848
none
Rebasing
none
Patch
none
Adding CSS testsuite support
none
Rebasing against new TestRepositories description
none
Refreshing and adding unit tests none

Description youenn fablet 2014-07-09 07:01:15 PDT
Automated W3C conformance testing based on run-webkit-tests would enable assessing how WebKit complies with W3C test suite. It could also serve as a tool to ease W3C test suite import witihn WebKit
Comment 1 youenn fablet 2014-07-09 07:07:13 PDT
Created attachment 234635 [details]
Patch
Comment 2 youenn fablet 2014-07-09 07:17:22 PDT
Created attachment 234637 [details]
Adding static files
Comment 3 youenn fablet 2014-07-16 11:35:47 PDT
Created attachment 235012 [details]
Start adding support for ref tests
Comment 4 youenn fablet 2014-09-16 02:47:02 PDT
Created attachment 238166 [details]
Support of all tests, result generated in WebKitBuild sub-folder
Comment 5 youenn fablet 2014-09-16 03:06:52 PDT
Created attachment 238169 [details]
updated according 136848
Comment 6 youenn fablet 2014-12-23 07:21:13 PST
Created attachment 243675 [details]
Rebasing
Comment 7 WebKit Commit Bot 2014-12-23 07:23:55 PST
Attachment 243675 [details] did not pass style-queue:


ERROR: Tools/Scripts/webkitpy/w3c/run_w3c_conformance_tests.py:43:  No name 'web_platform_test_importer' in module 'webkitpy.w3c'  [pylint/E0611] [5]
Total errors found: 1 in 7 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 8 youenn fablet 2014-12-23 15:37:10 PST
Created attachment 243703 [details]
Patch
Comment 9 Bem Jones-Bey 2015-01-06 08:03:49 PST
Comment on attachment 243703 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=243703&action=review

> LayoutTests/imported/w3c/resources/w3c_conformance_results.html:156
> +            (results.tests_directory ? results.tests_directory : "imported-tests")

Your indentation is all confused at a bunch of places in this file. Maybe you're mixing tabs and spaces?
Comment 10 youenn fablet 2015-01-06 13:13:45 PST
(In reply to comment #9)
> Comment on attachment 243703 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=243703&action=review
> 
> > LayoutTests/imported/w3c/resources/w3c_conformance_results.html:156
> > +            (results.tests_directory ? results.tests_directory : "imported-tests")
> 
> Your indentation is all confused at a bunch of places in this file. Maybe
> you're mixing tabs and spaces?

Woups, I'll clean that.
I will also add the possibility to filter tests according result and add bug-filing links
Comment 11 youenn fablet 2015-01-13 07:45:59 PST
Created attachment 244513 [details]
Adding CSS testsuite support
Comment 12 youenn fablet 2015-01-13 11:40:47 PST
(In reply to comment #11)
> Created attachment 244513 [details]
> Adding CSS testsuite support

This patch also generates two files, one to update the whitelist used to import W3C tests and one to update the test expectations (to skip infrastructure folders).
Comment 13 youenn fablet 2015-03-02 09:24:10 PST
Created attachment 247675 [details]
Rebasing against new TestRepositories description
Comment 14 youenn fablet 2015-03-30 06:27:03 PDT
Created attachment 249727 [details]
Refreshing and adding unit tests
Comment 15 Brent Fulgham 2016-03-10 10:41:32 PST
Comment on attachment 249727 [details]
Refreshing and adding unit tests

This seems like a really good tool!
Comment 16 Brent Fulgham 2016-03-10 10:42:37 PST
I can't get EWS to process this file. Youenn, would you mind rebaselining the patch and uploading it again?
Comment 17 youenn fablet 2016-03-10 12:01:52 PST
(In reply to comment #16)
> I can't get EWS to process this file. Youenn, would you mind rebaselining
> the patch and uploading it again?

The patch is old and webkitpy infrastructure changed a bit.
Time permitting, I will look at it again.