Created attachment 54639 [details]
- This contains both conformance tests that the web page runs, and other tests from source checkout.
- Sputnik checkout contains tests that need to be processed before use. I've processed them to something like script-tests, but monolithic for better loading speed.
- Sputnik tests often have several subtests, but bail out after the first error. There seems to be work in progress to distinguish hard failures from simple ones, and some tests use FAIL and ERROR distinctly - but since Google's own runner doesn't seem to distinguish those, I also didn't.
- Several Unicode tests are disabled, because they are too slow due to iterating lots of code points.
To import the tests, one needs to get a source checkout as below, edit sputnik.py, and run the attached script.
- self.test_root = path.join(root, 'tests', 'Conformance')
+ self.test_root = path.join(root, 'tests')
Created attachment 54642 [details]
The complete patch is 23 megabytes, uploading a part.
Comment on attachment 54642 [details]
> Is there a way to reduce the total amount of boilerplate?
> we need to expect success on the Chromium platform. How will we do that?
Someone from Chromium team usually updates expectations later, since build.webkit.org doesn't list Chromium results, as far as I can tell. I'm quite worried about timeouts on slower bots though - there are several tests that took noticeable time on my Mac Pro.
V8 fails more of these tests than JSC does, according to <http://sputnik.googlelabs.com/compare>.
http://trac.webkit.org/changeset/58534 might have broken GTK Linux 64-bit Release and Qt Linux Release
The following changes are on the blame list:
Added the whole directory to Skipped list, since there are mysterious (and different) failures on all platforms besides SnowLeopard.
Paths in the test suite are way too long for svn on Windows. Subversion only supports up to 256 or 260 when used with relative paths, but up to 32K with absolute paths. I had to shorten the paths manually for checkout to work.