Bug 210880 - run-webkit-tests should not fail if all tests found to run are skipped.
Summary: run-webkit-tests should not fail if all tests found to run are skipped.
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Matt Lewis
Keywords: InRadar
Depends on:
Reported: 2020-04-22 15:18 PDT by Matt Lewis
Modified: 2020-05-13 11:37 PDT (History)
5 users (show)

See Also:

Patch (6.15 KB, patch)
2020-05-12 17:07 PDT, Matt Lewis
jbedard: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Matt Lewis 2020-04-22 15:18:24 PDT
run-webkit-tests should not fail if all tests found to run are skipped.
Currently if tests are found to run but all of those tests are skipped, we return a failing value of -1. We should still fail if there are no test to run, but we shouldn't if all tests are skipped.
Comment 1 Radar WebKit Bug Importer 2020-04-22 15:18:48 PDT
Comment 2 Jonathan Bedard 2020-04-22 15:28:37 PDT
I don't know about this.

Only selecting skipped tests seems like something we want to surface. The exit code really only matter in automation, and if we're skipping all tests, shouldn't we just avoid running that particular set of tests on that particular configuration?
Comment 3 Matt Lewis 2020-04-22 15:42:34 PDT
So I see your point here but after talking with Alexey, this was something we discussed as being a situation that we should't fail on. Perhaps he can provide a bit more behind his thoughts.
Comment 4 Alexey Proskuryakov 2020-04-22 15:52:13 PDT
Skipping and moving tests is a separate decision from what we run in automation. It is possible that we have some tests run one day, and they get moved to a different suite another day. I don't think that we'd want to change buildbot configuration in lockstep.
Comment 5 Matt Lewis 2020-05-12 17:07:27 PDT
Created attachment 399204 [details]
Comment 6 Jonathan Bedard 2020-05-13 08:06:26 PDT
Comment on attachment 399204 [details]

r+ing because this behavior is changed correctly, although I disagree that we should be changing this behavior. Since Alexey, Matt and our other bot watchers are for this change, so I'm not going to block it.
Comment 7 Matt Lewis 2020-05-13 11:37:53 PDT
Committed r261629: <https://trac.webkit.org/changeset/261629>