Bug 96664
| Summary: | Need an easy way to determine whether a test is skipped/expected to fail | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Alexey Proskuryakov <ap> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | NEW | ||
| Severity: | Enhancement | CC: | dpranke, slewis, thorton |
| Priority: | P2 | Keywords: | NRWT |
| Version: | 528+ (Nightly build) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
Alexey Proskuryakov
Oftentimes, there is a bug saying that a test crashes, and it's clear that this does not happen any more. But it's unclear whether someone quietly skipped it, or added a fail expectation.
It's very difficult to scan all the cascading text expectations and skip lists by hand.
Perhaps when running a test by name (which bypasses skip lists), NRWT should still say that the test is special?
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Dirk Pranke
I don't think I understand what you're asking for. What do you mean by "special" ?
webkit-patch print-expectations and run-webkit-tests --details will both print what the current expectation for a test is (but not where that expectation comes from). Are you looking for that information, or the actual filename and line number that the expectation comes from, or something else?
Alexey Proskuryakov
I tried running this on Mac, and it didn't tell me that the test was skipped:
run-webkit-tests compositing/animation/animated-composited-inside-hidden.html --details
webkit-patch pront-expectations said that the expectation is PASS, and never said that the test was skipped.
Alexey Proskuryakov
So looks like expectations can be easily determined, but not anything about Skipped lists.
Dirk Pranke
Sounds like a bug. I'm not sure why it wouldn't be looking at the Skipped files in those cases. I will take a look.
Dirk Pranke
Ah, I see what's going on. As an implementation detail, "Skip" isn't an expectation, it's a modifier. We expect all skipped tests to Pass (in fact, I think you were the person who made me change this), so that when we explicitly run them, failures are reported as "unexpected". But this is all kind of goofy.
Once I can land the new expectation syntax and get rid of the skipped files, I'll clean this all up. In the mean time these commands can at least print the modifiers as well so you get some sort of clue.
Dirk Pranke
Also, as an implementation detail, because of the way Skipped files were hacked into NRWT, we don't actually store the filename and line number information for entries in Skipped files. My strategy is to make Skipped files go away rather than fix this.