Bug 89894 - run-webkit-test outputs the wrong number of tests executed when some are skipped.
Summary: run-webkit-test outputs the wrong number of tests executed when some are skip...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Mac (Intel) OS X 10.7
: P2 Normal
Assignee: Dirk Pranke
URL:
Keywords: NRWT
Depends on:
Blocks:
 
Reported: 2012-06-25 10:40 PDT by Alexis Menard (darktears)
Modified: 2012-07-13 20:46 PDT (History)
7 users (show)

See Also:


Attachments
Patch (11.04 KB, patch)
2012-07-13 18:36 PDT, Dirk Pranke
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alexis Menard (darktears) 2012-06-25 10:40:39 PDT
On Mac OS Lion if I run :

darktears@AlexisMenard:~/Development/troll/webkit:WEBKITOUTPUTDIR=/Users/darktears/Development/troll/webkit-mac/ Tools/Scripts/run-webkit-tests -2 --release --platform="mac-lion" --no-build LayoutTests/fast/box-decoration-break/
All 2 tests ran as expected.

The directory LayoutTests/fast/box-decoration-break/ contains 2 tests. One of them is skipped but it claims that two were run which is incorrect to me.

Running it with WEBKITOUTPUTDIR=/Users/darktears/Development/troll/webkit-mac/ Tools/Scripts/run-webkit-tests -2 --release --platform="mac-lion" --no-build --force LayoutTests/fast/box-decoration-break/

shows 1 test ran as expected, 1 didn't: which is expected.
Comment 1 Dirk Pranke 2012-06-25 11:13:30 PDT
(In reply to comment #0)
> On Mac OS Lion if I run :
> 
> darktears@AlexisMenard:~/Development/troll/webkit:WEBKITOUTPUTDIR=/Users/darktears/Development/troll/webkit-mac/ Tools/Scripts/run-webkit-tests -2 --release --platform="mac-lion" --no-build LayoutTests/fast/box-decoration-break/
> All 2 tests ran as expected.
> 
> The directory LayoutTests/fast/box-decoration-break/ contains 2 tests. One of them is skipped but it claims that two were run which is incorrect to me.
>

This is actually by design; the thinking was that it was "skipped as expected". That said, it was a dumb design :)

The output should be changed to make it clear that some tests are being skipped, not run.
 
> Running it with WEBKITOUTPUTDIR=/Users/darktears/Development/troll/webkit-mac/ Tools/Scripts/run-webkit-tests -2 --release --platform="mac-lion" --no-build --force LayoutTests/fast/box-decoration-break/
> 
> shows 1 test ran as expected, 1 didn't: which is expected.

I assume the skipped test failed here? i.e., both tests ran? That would be correct, agreed.
Comment 2 Alexis Menard (darktears) 2012-06-25 11:27:35 PDT
(In reply to comment #1)
> (In reply to comment #0)
> > On Mac OS Lion if I run :
> > 
> > darktears@AlexisMenard:~/Development/troll/webkit:WEBKITOUTPUTDIR=/Users/darktears/Development/troll/webkit-mac/ Tools/Scripts/run-webkit-tests -2 --release --platform="mac-lion" --no-build LayoutTests/fast/box-decoration-break/
> > All 2 tests ran as expected.
> > 
> > The directory LayoutTests/fast/box-decoration-break/ contains 2 tests. One of them is skipped but it claims that two were run which is incorrect to me.
> >
> 
> This is actually by design; the thinking was that it was "skipped as expected". That said, it was a dumb design :)

True.

> 
> The output should be changed to make it clear that some tests are being skipped, not run.

Something like "All 2 tests ran as expected (1 was Skipped)".

> 
> > Running it with WEBKITOUTPUTDIR=/Users/darktears/Development/troll/webkit-mac/ Tools/Scripts/run-webkit-tests -2 --release --platform="mac-lion" --no-build --force LayoutTests/fast/box-decoration-break/
> > 
> > shows 1 test ran as expected, 1 didn't: which is expected.
> 
> I assume the skipped test failed here? i.e., both tests ran? That would be correct, agreed.


Yes ;), I was trying to fix the failure.
Comment 3 Dirk Pranke 2012-07-13 18:36:19 PDT
Created attachment 152395 [details]
Patch
Comment 4 WebKit Review Bot 2012-07-13 20:46:01 PDT
Comment on attachment 152395 [details]
Patch

Clearing flags on attachment: 152395

Committed r122659: <http://trac.webkit.org/changeset/122659>
Comment 5 WebKit Review Bot 2012-07-13 20:46:06 PDT
All reviewed patches have been landed.  Closing bug.