Created attachment 54639 [details] translation script Some notes: - This contains both conformance tests that the web page runs, and other tests from source checkout. - It would be better to use run-javascriptcore-tests for these tests, but also more difficult to make a nice test runner. Someone can do that later. - Sputnik checkout contains tests that need to be processed before use. I've processed them to something like script-tests, but monolithic for better loading speed. - Sputnik tests often have several subtests, but bail out after the first error. There seems to be work in progress to distinguish hard failures from simple ones, and some tests use FAIL and ERROR distinctly - but since Google's own runner doesn't seem to distinguish those, I also didn't. - Several Unicode tests are disabled, because they are too slow due to iterating lots of code points. To import the tests, one needs to get a source checkout as below, edit sputnik.py, and run the attached script. - self.test_root = path.join(root, 'tests', 'Conformance') + self.test_root = path.join(root, 'tests')
Created attachment 54642 [details] partial patch The complete patch is 23 megabytes, uploading a part.
Comment on attachment 54642 [details] partial patch rs=me Is there a way to reduce the total amount of boilerplate? Since this suite was made by the V8 team, I assume that the failures are JavaScriptCore-specific, so we need to expect success on the Chromium platform. How will we do that?
> Is there a way to reduce the total amount of boilerplate? It was my intentional choice to have monolithic tests for faster loads (but I didn't measure how much that helps). But I don't feel very strongly about either approach - hopefully, these tests will be moved to run-javascriptcore-tests in foreseeable future. > we need to expect success on the Chromium platform. How will we do that? Someone from Chromium team usually updates expectations later, since build.webkit.org doesn't list Chromium results, as far as I can tell. I'm quite worried about timeouts on slower bots though - there are several tests that took noticeable time on my Mac Pro. V8 fails more of these tests than JSC does, according to <http://sputnik.googlelabs.com/compare>.
Committed <http://trac.webkit.org/changeset/58534>.
http://trac.webkit.org/changeset/58534 might have broken GTK Linux 64-bit Release and Qt Linux Release The following changes are on the blame list: http://trac.webkit.org/changeset/58533 http://trac.webkit.org/changeset/58534
Added the whole directory to Skipped list, since there are mysterious (and different) failures on all platforms besides SnowLeopard.
Paths in the test suite are way too long for svn on Windows. Subversion only supports up to 256 or 260 when used with relative paths, but up to 32K with absolute paths. I had to shorten the paths manually for checkout to work.
Filed http://code.google.com/p/sputniktests/issues/detail?id=25