Bug 89894 – run-webkit-test outputs the wrong number of tests executed when some are skipped.

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Alexis Menard (darktears)]

On Mac OS Lion if I run :

darktears@AlexisMenard:~/Development/troll/webkit:WEBKITOUTPUTDIR=/Users/darktears/Development/troll/webkit-mac/ Tools/Scripts/run-webkit-tests -2 --release --platform="mac-lion" --no-build LayoutTests/fast/box-decoration-break/
All 2 tests ran as expected.

The directory LayoutTests/fast/box-decoration-break/ contains 2 tests. One of them is skipped but it claims that two were run which is incorrect to me.

Running it with WEBKITOUTPUTDIR=/Users/darktears/Development/troll/webkit-mac/ Tools/Scripts/run-webkit-tests -2 --release --platform="mac-lion" --no-build --force LayoutTests/fast/box-decoration-break/

shows 1 test ran as expected, 1 didn't: which is expected.

[Dirk Pranke]

(In reply to comment #0)
> On Mac OS Lion if I run :
> 
> darktears@AlexisMenard:~/Development/troll/webkit:WEBKITOUTPUTDIR=/Users/darktears/Development/troll/webkit-mac/ Tools/Scripts/run-webkit-tests -2 --release --platform="mac-lion" --no-build LayoutTests/fast/box-decoration-break/
> All 2 tests ran as expected.
> 
> The directory LayoutTests/fast/box-decoration-break/ contains 2 tests. One of them is skipped but it claims that two were run which is incorrect to me.
>

This is actually by design; the thinking was that it was "skipped as expected". That said, it was a dumb design :)

The output should be changed to make it clear that some tests are being skipped, not run.
 
> Running it with WEBKITOUTPUTDIR=/Users/darktears/Development/troll/webkit-mac/ Tools/Scripts/run-webkit-tests -2 --release --platform="mac-lion" --no-build --force LayoutTests/fast/box-decoration-break/
> 
> shows 1 test ran as expected, 1 didn't: which is expected.

I assume the skipped test failed here? i.e., both tests ran? That would be correct, agreed.

[Alexis Menard (darktears)]

(In reply to comment #1)
> (In reply to comment #0)
> > On Mac OS Lion if I run :
> > 
> > darktears@AlexisMenard:~/Development/troll/webkit:WEBKITOUTPUTDIR=/Users/darktears/Development/troll/webkit-mac/ Tools/Scripts/run-webkit-tests -2 --release --platform="mac-lion" --no-build LayoutTests/fast/box-decoration-break/
> > All 2 tests ran as expected.
> > 
> > The directory LayoutTests/fast/box-decoration-break/ contains 2 tests. One of them is skipped but it claims that two were run which is incorrect to me.
> >
> 
> This is actually by design; the thinking was that it was "skipped as expected". That said, it was a dumb design :)

True.

> 
> The output should be changed to make it clear that some tests are being skipped, not run.

Something like "All 2 tests ran as expected (1 was Skipped)".

> 
> > Running it with WEBKITOUTPUTDIR=/Users/darktears/Development/troll/webkit-mac/ Tools/Scripts/run-webkit-tests -2 --release --platform="mac-lion" --no-build --force LayoutTests/fast/box-decoration-break/
> > 
> > shows 1 test ran as expected, 1 didn't: which is expected.
> 
> I assume the skipped test failed here? i.e., both tests ran? That would be correct, agreed.


Yes ;), I was trying to fix the failure.

[Dirk Pranke]

Created attachment 152395 [details]
Patch

[WebKit Review Bot]

Comment on attachment 152395 [details]
Patch

Clearing flags on attachment: 152395

Committed r122659: <http://trac.webkit.org/changeset/122659>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.