Bug 140934 – Import W3C web platform tests infrastructure

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[youenn fablet]

Now that wptserve can be activated within WebKit test infrastructure, we should populate LayoutTests/imported/w3c/web-platform-tests with wptserve implementation and related infrastructure folders.

[youenn fablet]

Created attachment 245439 [details]
Initial WPT infrastructure with domparsing tests

[Build Bot]

Comment on attachment 245439 [details]
Initial WPT infrastructure with domparsing tests

Attachment 245439 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.appspot.com/results/5906577227776000

New failing tests:
imported/w3c/web-platform-tests/domparsing/DOMParser-parseFromString-html.html
imported/w3c/web-platform-tests/domparsing/insert-adjacent.html

[Build Bot]

Created attachment 245443 [details]
Archive of layout-test-results from ews103 for mac-mavericks

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-mavericks  Platform: Mac OS X 10.9.5

[youenn fablet]

(In reply to comment #3)
> Created attachment 245443 [details]
> Archive of layout-test-results from ews103 for mac-mavericks
> 
> The attached test failures were seen while running run-webkit-tests on the
> mac-ews.
> Bot: ews103  Port: mac-mavericks  Platform: Mac OS X 10.9.5

WPT server seems to launch correctly on Mac-WK2 but not Mac WK1 bots.
The trace is the following:

Traceback (most recent call last):
  File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/servers/web_platform_test_launcher.py", line 14, in <module>
    import serve as WebPlatformTestServer
  File "./serve.py", line 16, in <module>
    from tools.scripts import _env
ImportError: No module named tools.scripts

I am not sure of the exact solution.

[Ryosuke Niwa]

Maybe due to different versions of Python being used?

[youenn fablet]

Created attachment 245470 [details]
Trying to fix wpt path

[Build Bot]

Comment on attachment 245470 [details]
Trying to fix wpt path

Attachment 245470 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.appspot.com/results/4989131411161088

New failing tests:
imported/w3c/web-platform-tests/domparsing/DOMParser-parseFromString-html.html
imported/w3c/web-platform-tests/domparsing/insert-adjacent.html

[Build Bot]

Created attachment 245486 [details]
Archive of layout-test-results from ews101 for mac-mavericks

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews101  Port: mac-mavericks  Platform: Mac OS X 10.9.5

[youenn fablet]

Created attachment 245531 [details]
Adding further logs

[Build Bot]

Comment on attachment 245531 [details]
Adding further logs

Attachment 245531 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.appspot.com/results/5311268990746624

New failing tests:
imported/w3c/web-platform-tests/domparsing/DOMParser-parseFromString-html.html
imported/w3c/web-platform-tests/domparsing/insert-adjacent.html

[Build Bot]

Created attachment 245533 [details]
Archive of layout-test-results from ews102 for mac-mavericks

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews102  Port: mac-mavericks  Platform: Mac OS X 10.9.5

[youenn fablet]

Created attachment 245537 [details]
Adding further logs

[Build Bot]

Comment on attachment 245537 [details]
Adding further logs

Attachment 245537 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.appspot.com/results/4780999443480576

New failing tests:
imported/w3c/web-platform-tests/domparsing/DOMParser-parseFromString-html.html
imported/w3c/web-platform-tests/domparsing/insert-adjacent.html

[Build Bot]

Created attachment 245543 [details]
Archive of layout-test-results from ews103 for mac-mavericks

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-mavericks  Platform: Mac OS X 10.9.5

[youenn fablet]

(In reply to comment #14)
> Created attachment 245543 [details]
> Archive of layout-test-results from ews103 for mac-mavericks
> 
> The attached test failures were seen while running run-webkit-tests on the
> mac-ews.
> Bot: ews103  Port: mac-mavericks  Platform: Mac OS X 10.9.5

According the WPT log below, the LayoutTests/imported/w3c/web-platform-tests/tools/__init__.py is missing for the Mac WK1 bot, hence the inability to import wpt modules.
Could it be that empty files are not created when applying the patch for this bot?

WPT log is:
DEBUG:web-platform-test-launcher:sys.path is: ['/Volumes/Data/EWS/WebKit/LayoutTests/imported/w3c/web-platform-tests', '/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/servers', '/Volumes/Data/EWS/WebKit/Tools/Scripts', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', '/Library/Python/2.7/site-packages']
DEBUG:web-platform-test-launcher:files in cwd are:['_certs', 'common', 'config.default.json', 'config.json', 'domparsing', 'fonts', 'images', 'resources', 'serve.py', 'serve.pyc', 'tools', 'w3c-import.log']
DEBUG:web-platform-test-launcher:files in cwd/tools are:['html5lib', 'pywebsocket', 'runner', 'scripts', 'six', 'sslutils', 'w3c-import.log', 'webdriver', 'wptserve']
Traceback (most recent call last):
  File "/Volumes/Data/EWS/WebKit/Tools/Scripts/webkitpy/layout_tests/servers/web_platform_test_launcher.py", line 18, in <module>
    logger.debug(imp.find_module('tools'))
ImportError: No module named tools

[youenn fablet]

Created attachment 245553 [details]
Checking __init__.py files

[youenn fablet]

Created attachment 245615 [details]
Cleaned-up version

[youenn fablet]

(In reply to comment #17)
> Created attachment 245615 [details]
> Cleaned-up version

See https://bugs.webkit.org/show_bug.cgi?id=141030 for a potentially cleaner handling of empty __init__.py files

[Ryosuke Niwa]

!?  Did you accidentally added all the files in the imported directory?  The patch is 5.7MB.

[youenn fablet]

(In reply to comment #19)
> !?  Did you accidentally added all the files in the imported directory?  The
> patch is 5.7MB.

No, I just use the import script that copies the whole tools folder.
It notably includes full html5lib and pywebsocket modules files.

If size is an issue, we could probably discard the tools/html5lib/html5lib/tests internal tests (2.5 MB) and the pywebsocket/src/tests internal tests (400Kb).

We could be more drastic by removing all modules except than wptserve, pywebsocket and sslutils, which would reduce the patch size to 1 MB probably.
Removing these modules may result in future unexpected test failures though.

[youenn fablet]

(In reply to comment #20)
> (In reply to comment #19)
> > !?  Did you accidentally added all the files in the imported directory?  The
> > patch is 5.7MB.
> 
> No, I just use the import script that copies the whole tools folder.
> It notably includes full html5lib and pywebsocket modules files.
> 
> If size is an issue, we could probably discard the
> tools/html5lib/html5lib/tests internal tests (2.5 MB) and the
> pywebsocket/src/tests internal tests (400Kb).
> 
> We could be more drastic by removing all modules except than wptserve,
> pywebsocket and sslutils, which would reduce the patch size to 1 MB probably.
> Removing these modules may result in future unexpected test failures though.

Or we auto-install the different tools as we do for thirdparty python modules before launching web-platform-test-server.
Some more scripting, but that would leave webkit repo slim.

[Ryosuke Niwa]

(In reply to comment #21)
> (In reply to comment #20)
> > (In reply to comment #19)
> > > !?  Did you accidentally added all the files in the imported directory?  The
> > > patch is 5.7MB.
> > 
> > No, I just use the import script that copies the whole tools folder.
> > It notably includes full html5lib and pywebsocket modules files.
> > 
> > If size is an issue, we could probably discard the
> > tools/html5lib/html5lib/tests internal tests (2.5 MB) and the
> > pywebsocket/src/tests internal tests (400Kb).
> > 
> > We could be more drastic by removing all modules except than wptserve,
> > pywebsocket and sslutils, which would reduce the patch size to 1 MB probably.
> > Removing these modules may result in future unexpected test failures though.
> 
> Or we auto-install the different tools as we do for thirdparty python
> modules before launching web-platform-test-server.
> Some more scripting, but that would leave webkit repo slim.

Yeah, can we do that instead?

[youenn fablet]

Created attachment 245661 [details]
Downloading web platform tests python modules on the fly

[youenn fablet]

Created attachment 245697 [details]
Downloading WPT python modules

[youenn fablet]

Created attachment 245700 [details]
Adding .gitignore for downloaded modules

[youenn fablet]

Created attachment 245703 [details]
Adding .gitignore for downloaded modules

[youenn fablet]

Size is reduced to 600 ko, hope this is small enough.

300 ko are due to tools/runner.
It is called by wpt server when some specific URLs are hit.
I doubt that this will ever happen in WebKit layout test context.

Note also that the _certs folder can be dynamically generated by wptserver.
It seems safer to have a repeatable test set-up hence why it is checked in.

[Ryosuke Niwa]

Comment on attachment 245703 [details]
Adding .gitignore for downloaded modules

View in context: https://bugs.webkit.org/attachment.cgi?id=245703&action=review

r=me for webkitpy change and rs=me for the import.

> Tools/Scripts/webkitpy/layout_tests/servers/web_platform_test_launcher.py:38
> +

Superflous change.

> LayoutTests/TestExpectations:234
> +imported/w3c/web-platform-tests/domparsing/DOMParser-parseFromString-html.html [ Pass ] # Some assertions failing

Shouldn't this have a failing expectation then?

[youenn fablet]

(In reply to comment #28)
> Comment on attachment 245703 [details]
> Adding .gitignore for downloaded modules
> > LayoutTests/TestExpectations:234
> > +imported/w3c/web-platform-tests/domparsing/DOMParser-parseFromString-html.html [ Pass ] # Some assertions failing
> 
> Shouldn't this have a failing expectation then?

This line just reminds that the expected.txt contains some FAIL assertions lines. It does not need to be there.

It serves more as a reminder that someone should investigate the inconsistency between WebKit and the test as TestImport file will do for failed tests.
It may also be useful when updating the revision of the test suite.

[youenn fablet]

Created attachment 245736 [details]
Patch for landing

[WebKit Commit Bot]

Attachment 245736 [details] did not pass style-queue:


ERROR: LayoutTests/TestExpectations:235:  Path does not exist.  [test/expectations] [5]
Total errors found: 1 in 93 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[WebKit Commit Bot]

Comment on attachment 245736 [details]
Patch for landing

Rejecting attachment 245736 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.appspot.com', '--bot-id=webkit-cq-02', 'land-attachment', '--force-clean', '--non-interactive', '--parent-command=commit-queue', 245736, '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit

Last 500 characters of output:
19ba9e139caa3210
r179424 = 7ceb91db1a07084c86cbd7997b85782e66ab8554
r179425 = e37d84c5b7056e35db5e224e50861aee8811eead
r179426 = fff116d7a3b48251caa7bf0a2fa92e847c39865d
Done rebuilding .git/svn/refs/remotes/origin/master/.rev_map.268f45cc-cd09-0410-ab3c-d52691b4dbfc
First, rewinding head to replay your work on top of it...
Fast-forwarded master to refs/remotes/origin/master.
ERROR: LayoutTests/TestExpectations:235:  Path does not exist.  [test/expectations] [5]
Total errors found: 1 in 1 files

Full output: http://webkit-queues.appspot.com/results/5570474293592064

[youenn fablet]

Created attachment 245776 [details]
Fixing style

[WebKit Commit Bot]

Comment on attachment 245776 [details]
Fixing style

Clearing flags on attachment: 245776

Committed r179439: <http://trac.webkit.org/changeset/179439>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.