Bug 105531 – [WTR] Wrong totals in WebKitTestRunner.

• Component: Tools / Tests
• Status: UNCONFIRMED

[Mateusz Leszko]

When using  --debug-rwt-logging parameter total are not correct.
Example in attachment.

I see there two problems:
-	We got 229 test ran as expected, but in totals it’s 228.
-	We got 2 missing results – this shouldn’t be counted as expected if it's not expected.

I think this parameter use to result ratio like “passed as expected”/"all running" test in totals.

[Mateusz Leszko]

Created attachment 180333 [details]
test results

[Mateusz Leszko]

CC'ing owners of
 Tools/Scripts/new-run-webkit-tests
 Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py

[Zan Dobersek]

I don't think test result totals are related to the test runner (i.e. DRT or WTR), so this bug might be related to bug #105636.

[Mateusz Leszko]

Dirk, are you able to explain or say how to get ratio like “passed as expected”/"all running"?

[Dirk Pranke]

I think there's probably a couple bugs in NRWT. I've been away for a few days so haven't had a chance to look into it but will soon.

[Dirk Pranke]

Okay, sorry for the delay, I finally got time to look at this.

(In reply to comment #0)
> When using  --debug-rwt-logging parameter total are not correct.
> Example in attachment.
> 
> I see there two problems:
> -    We got 229 test ran as expected, but in totals it’s 228.

Here we have the problem of data being sliced and diced in different ways.

The "229" number comes from finding 232 tests and skipping 3 of them.

The "228" number comes from finding 228 tests and having 4 of them fail. In this case tests that are skipped are counted as having passed.

> -    We got 2 missing results – this shouldn’t be counted as expected if it's not expected.
>

I don't know what version you ran this against (so what the contents of your TestExpectations files were), but the lack of a "failed unexpectedly" in the log file probably means that you had the two tests marked as Missing, and, so, they were expected.

So, I don't see any actually incorrect computations here.

As I mentioned somewhere (email?) the problem with these routines is that there's lots of different ways one might want to slice and dice these statistics, and it's hard to come up with meaningful and clear buckets without being really, really verbose. That said, I'm definitely open to changes to the wording or how we are in fact bucketing things if that will help someone.