NEW 68123
run-webkit-tests on Lion ignores mac-lion/Skipped when specifying tests by name 
https://bugs.webkit.org/show_bug.cgi?id=68123
Summary run-webkit-tests on Lion ignores mac-lion/Skipped when specifying tests by name
Matthew Delaney
Reported 2011-09-14 15:15:41 PDT
Using run-webkit-tests with "--pixel" appears to not skip all the tests in platform/mac-lion/Skipped. For example, if you run the fast/canvas tests with and without the --pixel flag, it'll show different results for non-pixel tests.
Attachments
Add attachment proposed patch, testcase, etc.
Matthew Delaney
Comment 1 2011-09-14 15:20:00 PDT
I take that back. It appears to ignore the skipped list when you specify the tests by name on the command line such as: run-webkit-tests LayoutTests/fast/canvas/fillText-shadow.html LayoutTests/fast/canvas/shadow-huge-blur.html LayoutTests/fast/canvas/shadow-offset-1.html LayoutTests/fast/canvas/shadow-offset-2.html LayoutTests/fast/canvas/shadow-offset-3.html LayoutTests/fast/canvas/shadow-offset-4.html LayoutTests/fast/canvas/shadow-offset-5.html LayoutTests/fast/canvas/shadow-offset-6.html LayoutTests/fast/canvas/shadow-offset-7.html LayoutTests/fast/css/box-shadow-and-border-radius.html Is that intended behavior? If so, I was not aware. It seems a little odd to me though. We should instead just have a flag to ignore the skipped lists if that's what's needed in this case.
Adam Roben (:aroben)
Comment 2 2011-09-14 15:52:06 PDT
(In reply to comment #1) > I take that back. It appears to ignore the skipped list when you specify the tests by name on the command line such as: > > run-webkit-tests LayoutTests/fast/canvas/fillText-shadow.html LayoutTests/fast/canvas/shadow-huge-blur.html LayoutTests/fast/canvas/shadow-offset-1.html LayoutTests/fast/canvas/shadow-offset-2.html LayoutTests/fast/canvas/shadow-offset-3.html LayoutTests/fast/canvas/shadow-offset-4.html LayoutTests/fast/canvas/shadow-offset-5.html LayoutTests/fast/canvas/shadow-offset-6.html LayoutTests/fast/canvas/shadow-offset-7.html LayoutTests/fast/css/box-shadow-and-border-radius.html > > Is that intended behavior? Yes. Otherwise it would be hard to ever run skipped tests. > If so, I was not aware. It seems a little odd to me though. We should instead just have a flag to ignore the skipped lists if that's what's needed in this case. We could make the Skipped files override command-line arguments unless --skipped=ignore (an option we already support) is passed. But I suspect this might be a controversial change. You would probably want to raise it on webkit-dev.
Matthew Delaney
Comment 3 2011-09-14 16:43:38 PDT
Ah, I see. I wonder if there's a better way to do what I was just attempting to do... I made a change to shadow rendering code and wanted to just run all the shadow tests. They appear to be in many different directories. Since I didn't want to run ALL the layout tests, I just did a find one all tests with "shadow" in the name and piped it over to run-webkit-tests. For a few minutes, I got all worried when I saw a bunch of failing tests. But then I recalled one of them being on a skipped list, so that's who I figured it out. Can you think of a better way for me to have run just the "shadow" tests that aren't skipped?
Adam Roben (:aroben)
Comment 4 2011-09-14 22:54:28 PDT
(In reply to comment #3) > Can you think of a better way for me to have run just the "shadow" tests that aren't skipped? I don't think there's an easy way currently. I could imagine adding a new flag to run-webkit-tests, something like: run-webkit-tests --glob=*shadow* which would run all non-skipped tests with "shadow" in their name.
Note You need to log in before you can comment on or make changes to this bug.