NEW 96664
Need an easy way to determine whether a test is skipped/expected to fail 
https://bugs.webkit.org/show_bug.cgi?id=96664
Summary Need an easy way to determine whether a test is skipped/expected to fail
Alexey Proskuryakov
Reported 2012-09-13 10:01:30 PDT
Oftentimes, there is a bug saying that a test crashes, and it's clear that this does not happen any more. But it's unclear whether someone quietly skipped it, or added a fail expectation. It's very difficult to scan all the cascading text expectations and skip lists by hand. Perhaps when running a test by name (which bypasses skip lists), NRWT should still say that the test is special?
Attachments
Add attachment proposed patch, testcase, etc.
Dirk Pranke
Comment 1 2012-09-13 11:59:58 PDT
I don't think I understand what you're asking for. What do you mean by "special" ? webkit-patch print-expectations and run-webkit-tests --details will both print what the current expectation for a test is (but not where that expectation comes from). Are you looking for that information, or the actual filename and line number that the expectation comes from, or something else?
Alexey Proskuryakov
Comment 2 2012-09-13 12:07:31 PDT
I tried running this on Mac, and it didn't tell me that the test was skipped: run-webkit-tests compositing/animation/animated-composited-inside-hidden.html --details webkit-patch pront-expectations said that the expectation is PASS, and never said that the test was skipped.
Alexey Proskuryakov
Comment 3 2012-09-13 12:08:29 PDT
So looks like expectations can be easily determined, but not anything about Skipped lists.
Dirk Pranke
Comment 4 2012-09-13 12:12:10 PDT
Sounds like a bug. I'm not sure why it wouldn't be looking at the Skipped files in those cases. I will take a look.
Dirk Pranke
Comment 5 2012-09-13 13:10:18 PDT
Ah, I see what's going on. As an implementation detail, "Skip" isn't an expectation, it's a modifier. We expect all skipped tests to Pass (in fact, I think you were the person who made me change this), so that when we explicitly run them, failures are reported as "unexpected". But this is all kind of goofy. Once I can land the new expectation syntax and get rid of the skipped files, I'll clean this all up. In the mean time these commands can at least print the modifiers as well so you get some sort of clue.
Dirk Pranke
Comment 6 2012-09-13 13:15:18 PDT
Also, as an implementation detail, because of the way Skipped files were hacked into NRWT, we don't actually store the filename and line number information for entries in Skipped files. My strategy is to make Skipped files go away rather than fix this.
Note You need to log in before you can comment on or make changes to this bug.