Bug 105636 – run-webkit-tests summary seems incomplete

• Status: NEW
• Component: Tools / Tests

[Ojan Vafai]

From http://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux/builds/3240/steps/webkit_tests/logs/stdio: 

=> Results: 33706/34160 tests passed (98.7%)

=> Tests to be fixed (894):
      6 crashes                  ( 0.7%)
     27 timeouts                 ( 3.0%)
      4 missing results          ( 0.4%)
     98 image-only failures      (11.0%)

=> Tests that will only be fixed if they crash (WONTFIX) (6753):


Expected to fail, but passed: (7)
  fast/forms/button-generated-content.html
  fast/forms/button-inner-block-reuse.html
  fast/text/hyphenate-character.html
  fast/text/hyphenate-first-word.html
  fast/text/hyphenate-limit-lines.html
  fast/text/hyphenate-locale.html
  fast/text/hyphens.html


The numbers under "Tests to be fixed" don't add up to 100%. Are we leaving out TEXT/IMAGE+TEXT failures? Do we leave out flaky passes there?

Also, 33706 + 894 != 34160. So, it's not clear to me what those numbers are supposed to be. Maybe this is due to tests listed as flaky that actually pass?

Some other nits I may as well mention as I look at this:
1) The colon after the WONTFIX line is confusing since nothing is printed after it.
2) The text for WONTFIX is no longer accurate since we skip these tests now. We should just say something like "Tests that will *never* be fixed".
3) I find the "Tests to be fixed" wording confusing. I'd prefer something like "Expected failures". At this point, everyone needs to know the concept of expected vs unexpected failures.

[Zan Dobersek]

(In reply to comment #0)
> From http://build.chromium.org/p/chromium.webkit/builders/WebKit%20Linux/builds/3240/steps/webkit_tests/logs/stdio: 
> 
> => Results: 33706/34160 tests passed (98.7%)
> 
> => Tests to be fixed (894):
>       6 crashes                  ( 0.7%)
>      27 timeouts                 ( 3.0%)
>       4 missing results          ( 0.4%)
>      98 image-only failures      (11.0%)
> 
> => Tests that will only be fixed if they crash (WONTFIX) (6753):
> 
> 
> The numbers under "Tests to be fixed" don't add up to 100%. Are we leaving out TEXT/IMAGE+TEXT failures? Do we leave out flaky passes there?

The numbers listed here are only for failures represented by the EXPECTATIONS_ORDER variable in test_expectations.py:

EXPECTATION_ORDER = (PASS, CRASH, TIMEOUT, MISSING, FAIL, IMAGE, SKIP)

So yeah, there are text, image+text and audio expectations that are being left out. Also, the skipped tests are not numbered here, but are counted in the total (which is 894 here).

[Dirk Pranke]

Harrumph. Yeah, Žan's diagnosis is correct. I think the thing to do here is to fix bug 97366 so that we can stop having all of these slightly different ways of mapping the failure types onto categories and having to worry about mapping between the "new" categories and the "old" categories. Anything else is just making cruft cruftier.

I'm planning to work on this this week now that I'm finally back, so hopefully we won't have to wait much longer.

[Dirk Pranke]

Of course, Ojan's other comments on the wording can be addressed independently.

[Dirk Pranke]

I don't expect to address this in the webkit.org repo. Resetting the 'assigned_to' field so that someone else can pick it up if they're interested.