Bug 38296 - Import Sputnik JavaScript test suite
Summary: Import Sputnik JavaScript test suite
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: All All
: P2 Normal
Assignee: Alexey Proskuryakov
Depends on:
Reported: 2010-04-28 17:21 PDT by Alexey Proskuryakov
Modified: 2010-04-29 17:11 PDT (History)
4 users (show)

See Also:

translation script (3.82 KB, text/plain)
2010-04-28 17:21 PDT, Alexey Proskuryakov
no flags Details
partial patch (101.56 KB, patch)
2010-04-28 17:33 PDT, Alexey Proskuryakov
darin: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alexey Proskuryakov 2010-04-28 17:21:54 PDT
Created attachment 54639 [details]
translation script

Some notes:
- This contains both conformance tests that the web page runs, and other tests from source checkout.
- It would be better to use run-javascriptcore-tests for these tests, but also more difficult to make a nice test runner. Someone can do that later.
- Sputnik checkout contains tests that need to be processed before use. I've processed them to something like script-tests, but monolithic for better loading speed.
- Sputnik tests often have several subtests, but bail out after the first error. There seems to be work in progress to distinguish hard failures from simple ones, and some tests use FAIL and ERROR distinctly - but since Google's own runner doesn't seem to distinguish those, I also didn't.
- Several Unicode tests are disabled, because they are too slow due to iterating lots of code points.

To import the tests, one needs to get a source checkout as below, edit sputnik.py, and run the attached script.

-    self.test_root = path.join(root, 'tests', 'Conformance')
+    self.test_root = path.join(root, 'tests')
Comment 1 Alexey Proskuryakov 2010-04-28 17:33:30 PDT
Created attachment 54642 [details]
partial patch

The complete patch is 23 megabytes, uploading a part.
Comment 2 Darin Adler 2010-04-28 17:36:03 PDT
Comment on attachment 54642 [details]
partial patch


Is there a way to reduce the total amount of boilerplate? Since this suite was made by the V8 team, I assume that the failures are JavaScriptCore-specific, so we need to expect success on the Chromium platform. How will we do that?
Comment 3 Alexey Proskuryakov 2010-04-28 17:45:15 PDT
> Is there a way to reduce the total amount of boilerplate?

It was my intentional choice to have monolithic tests for faster loads (but I didn't measure how much that helps). But I don't feel very strongly about either approach - hopefully, these tests will be moved to run-javascriptcore-tests in foreseeable future.

> we need to expect success on the Chromium platform. How will we do that?

Someone from Chromium team usually updates expectations later, since build.webkit.org doesn't list Chromium results, as far as I can tell. I'm quite worried about timeouts on slower bots though - there are several tests that took noticeable time on my Mac Pro.

V8 fails more of these tests than JSC does, according to <http://sputnik.googlelabs.com/compare>.
Comment 4 Alexey Proskuryakov 2010-04-29 13:47:06 PDT
Committed <http://trac.webkit.org/changeset/58534>.
Comment 5 WebKit Review Bot 2010-04-29 14:07:48 PDT
http://trac.webkit.org/changeset/58534 might have broken GTK Linux 64-bit Release and Qt Linux Release
The following changes are on the blame list:
Comment 6 Alexey Proskuryakov 2010-04-29 14:36:22 PDT
Added the whole directory to Skipped list, since there are mysterious (and different) failures on all platforms besides SnowLeopard.
Comment 7 Alexey Proskuryakov 2010-04-29 17:01:46 PDT
Paths in the test suite are way too long for svn on Windows. Subversion only supports up to 256 or 260 when used with relative paths, but up to 32K with absolute paths. I had to shorten the paths manually for checkout to work.
Comment 8 Alexey Proskuryakov 2010-04-29 17:11:36 PDT
Filed http://code.google.com/p/sputniktests/issues/detail?id=25