Bug 85941 - nrwt: --skip-failing-tests should also skip flaky tests
Summary: nrwt: --skip-failing-tests should also skip flaky tests
Alias: None
Product: WebKit
Classification: Unclassified
Component: New Bugs (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Dirk Pranke
Depends on:
Reported: 2012-05-08 18:27 PDT by Dirk Pranke
Modified: 2012-05-09 12:59 PDT (History)
4 users (show)

See Also:

Patch (3.16 KB, patch)
2012-05-08 18:28 PDT, Dirk Pranke
no flags Details | Formatted Diff | Diff
write more tests (7.41 KB, patch)
2012-05-08 19:35 PDT, Dirk Pranke
ojan: review+
webkit.review.bot: commit-queue-
Details | Formatted Diff | Diff
Archive of layout-test-results from ec2-cr-linux-02 (742.35 KB, application/zip)
2012-05-08 20:16 PDT, WebKit Review Bot
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dirk Pranke 2012-05-08 18:27:51 PDT
nrwt: --skip-failing-tests should also skip flaky tests
Comment 1 Dirk Pranke 2012-05-08 18:28:22 PDT
Created attachment 140838 [details]
Comment 2 Dirk Pranke 2012-05-08 18:33:25 PDT
this should keep the chromium ews bots from running ~450 tests that are currently marked as flaky, and the only results you should see are from the tests that are expected to reliably PASS but actually fail.

Note that if you run a test and it fails, and then you retry and it passes, we should still see the failure, I think. I don't currently, and am investigating ...
Comment 3 Dirk Pranke 2012-05-08 19:35:08 PDT
Created attachment 140855 [details]
write more tests
Comment 4 WebKit Review Bot 2012-05-08 20:16:19 PDT
Comment on attachment 140855 [details]
write more tests

Attachment 140855 [details] did not pass chromium-ews (chromium-xvfb):
Output: http://queues.webkit.org/results/12656501

New failing tests:
Comment 5 WebKit Review Bot 2012-05-08 20:16:23 PDT
Created attachment 140858 [details]
Archive of layout-test-results from ec2-cr-linux-02

The attached test failures were seen while running run-webkit-tests on the chromium-ews.
Bot: ec2-cr-linux-02  Port: <class 'webkitpy.common.config.ports.ChromiumXVFBPort'>  Platform: Linux-2.6.35-28-virtual-x86_64-with-Ubuntu-10.10-maverick
Comment 6 Dirk Pranke 2012-05-09 12:59:03 PDT
Committed r116552: <http://trac.webkit.org/changeset/116552>