Bug 103631

Summary:

Failures in ref tests should appear in results.html

Product:

WebKit

Reporter:

Yael <yael>

Component:

Tools / Tests

Assignee:

Nobody <webkit-unassigned>

Status:

RESOLVED INVALID

Severity:

Normal

CC:

dpranke, gyuyoung.kim, mrobinson, rakuco

Priority:

Keywords:

NRWT

Version:

528+ (Nightly build)

Hardware:

Unspecified

OS:

Unspecified

Attachments:

Description	Flags
Results file	none
json file	none
ref tests failures	none
printout of the run	none

Description Yael 2012-11-29 06:11:06 PST

When running run-webkit-tests, ref test failures do not show up in the results.html, and that is very misleading.
When I run it with --verbose, I see in the shell "failed (reference mismatch)", and I see the images in the results directory.
It would be much nicer if results.html would also show these failures.

Comment 1 Dirk Pranke 2012-11-29 10:50:04 PST

They show up for me ... there is a link for "ref html" next to the links for the images and the diff.

What command line are you using?

Comment 2 Yael 2012-11-29 10:55:27 PST

(In reply to comment #1)
> They show up for me ... there is a link for "ref html" next to the links for the images and the diff.
> 
> What command line are you using?
For example  
Tools/Scripts/run-webkit-tests --debug --efl -2  LayoutTests/fast/overflow/ LayoutTests/fast/parser/ LayoutTests/fast/regex/ LayoutTests/fast/regions/  --verbose

Comment 3 Dirk Pranke 2012-11-29 12:05:41 PST

Unfortunately, my attempts to build the efl port to reproduce this have stalled out temporarily. Do other EFL devs also see this ?

If someone can post the command line they use and the results.json they get from that command line to this bug (and maybe also the results.html, though that's just a static template), that would be helpful.

Comment 4 Yael 2012-11-29 12:15:31 PST

I am trying to recreate this problem, and I can't.
Maybe it happened because the flaky tests were passing on a second try and I did not notice that.

Comment 5 Yael 2012-11-29 12:17:08 PST

Created attachment 176781 [details]
Results file

Comment 6 Yael 2012-11-29 12:17:28 PST

Created attachment 176782 [details]
json file

Comment 7 Yael 2012-11-29 12:18:36 PST

Created attachment 176783 [details]
ref tests failures

Comment 8 Yael 2012-11-29 12:19:41 PST

Created attachment 176785 [details]
printout of the run

Comment 9 Yael 2012-11-29 12:20:45 PST

70 ref tests failed, but only 7 show in results.html

Comment 10 Dirk Pranke 2012-11-29 12:26:28 PST

We're logging both the tests that are failing as expected and the tests that are failing unexpectedly. results.html only shows the unexpected failures by default. If you click on the options link in the top right corner, and uncheck the "only unexpected results" box, you should see the other failures.

In the summary at the end of the run, we report 30 flaky failures and 7 consistent failures.

I think this is working correctly; does that make sense?

Comment 11 Yael 2012-11-29 12:30:35 PST

(In reply to comment #10)
> We're logging both the tests that are failing as expected and the tests that are failing unexpectedly. results.html only shows the unexpected failures by default. If you click on the options link in the top right corner, and uncheck the "only unexpected results" box, you should see the other failures.
> 
> In the summary at the end of the run, we report 30 flaky failures and 7 consistent failures.
> 
> I think this is working correctly; does that make sense?

I did not now about that option. Thanks for your help!