Summary: | [Qt] Enable passing Sputnik tests | ||||||
---|---|---|---|---|---|---|---|
Product: | WebKit | Reporter: | Csaba Osztrogonác <ossy> | ||||
Component: | Tools / Tests | Assignee: | QtWebKit Unassigned <webkit-qt-unassigned> | ||||
Status: | RESOLVED FIXED | ||||||
Severity: | Normal | CC: | abecsi, ap, hausmann, kenneth, vestbo | ||||
Priority: | P3 | Keywords: | Qt | ||||
Version: | 528+ (Nightly build) | ||||||
Hardware: | All | ||||||
OS: | All | ||||||
Attachments: |
|
Description
Csaba Osztrogonác
2010-05-03 05:56:14 PDT
Created attachment 54919 [details]
enable tests
Comment on attachment 54919 [details]
enable tests
rs=me
+# Failing Sputnik tests
You could also consider landing platform specific results. Of course, these aren't all the tests that fail, we have many expected cross-platform failures checked in.
(In reply to comment #2) > (From update of attachment 54919 [details]) > rs=me Thx, landed: http://trac.webkit.org/changeset/58863 > You could also consider landing platform specific results. Of course, these > aren't all the tests that fail, we have many expected cross-platform failures > checked in. In general we don't land Qt specific expected results for failures, we prefer putting these tests into the skipped list and file a bug. I know disabling a complex test can be worse than check in couple of failures (eg. FAIL instead of PASS) into the expected file. But I don't see now how can we maintain which expected file is absolutely correct and which consists failures. We will think how can we do it. I cc-ed Simon, Tor Arne, Kenneth and András too. Guys, I think we should discuss check-in or not check-in failures policy sometime somewhere. If I am correct these tests are dumpAsText() tests, so my opinion is that we could check in results for tests which have any PASS hunks in the output. If someone fixes some issues he or she could easily decide if the resulting test failure is due to wrong expected results or some regression. In general I think this kind of policy would be a step forward, but I am categorically against checking in wrong DRT tree dumps, because to decide afterwards if the dump is correct or not is really hard. (In reply to comment #3) > (In reply to comment #2) > > (From update of attachment 54919 [details] [details]) > > rs=me > Thx, landed: http://trac.webkit.org/changeset/58863 > > > You could also consider landing platform specific results. Of course, these > > aren't all the tests that fail, we have many expected cross-platform failures > > checked in. > > In general we don't land Qt specific expected results for failures, > we prefer putting these tests into the skipped list and file a bug. > > I know disabling a complex test can be worse than check in couple of failures > (eg. FAIL instead of PASS) into the expected file. But I don't see now how can > we maintain which expected file is absolutely correct and which consists > failures. We will think how can we do it. > > I cc-ed Simon, Tor Arne, Kenneth and András too. Guys, I think we > should discuss check-in or not check-in failures policy sometime somewhere. (In reply to comment #4) > If I am correct these tests are dumpAsText() tests, so my opinion is that we > could check in results for tests which have any PASS hunks in the output. If > someone fixes some issues he or she could easily decide if the resulting test > failure is due to wrong expected results or some regression. > In general I think this kind of policy would be a step forward, but I am > categorically against checking in wrong DRT tree dumps, because to decide > afterwards if the dump is correct or not is really hard. I agree, for dumpAsText it makes sense. For render tree output we prefer to skip tests for now. Sputnik tests are weird - they silently run subtests until the first failure, at which time they bail out and print an error message. Only if there were no (unexpected) errors during the run, PASS is printed. |