Bug 103631

Summary:

Failures in ref tests should appear in results.html

Product:

WebKit

Reporter:

Yael <yael>

Component:

Tools / Tests

Assignee:

Nobody <webkit-unassigned>

Status:

RESOLVED INVALID

Severity:

Normal

CC:

dpranke, gyuyoung.kim, mrobinson, rakuco

Priority:

Keywords:

NRWT

Version:

528+ (Nightly build)

Hardware:

Unspecified

OS:

Unspecified

Attachments:

Description	Flags
Results file	none
json file	none
ref tests failures	none
printout of the run	none

Yael

Reported 2012-11-29 06:11:06 PST

When running run-webkit-tests, ref test failures do not show up in the results.html, and that is very misleading. When I run it with --verbose, I see in the shell "failed (reference mismatch)", and I see the images in the results directory. It would be much nicer if results.html would also show these failures.

Attachments
Results file (39.53 KB, text/html) 2012-11-29 12:17 PST, Yael	no flags	Details
json file (37.76 KB, application/octet-stream) 2012-11-29 12:17 PST, Yael	no flags	Details
ref tests failures (5.51 KB, text/plain) 2012-11-29 12:18 PST, Yael	no flags	Details
printout of the run (670.45 KB, text/plain) 2012-11-29 12:19 PST, Yael	no flags	Details
View All Add attachment proposed patch, testcase, etc.

Dirk Pranke

Comment 1 2012-11-29 10:50:04 PST

They show up for me ... there is a link for "ref html" next to the links for the images and the diff. What command line are you using?

Yael

Comment 2 2012-11-29 10:55:27 PST

(In reply to comment #1) > They show up for me ... there is a link for "ref html" next to the links for the images and the diff. > > What command line are you using? For example Tools/Scripts/run-webkit-tests --debug --efl -2 LayoutTests/fast/overflow/ LayoutTests/fast/parser/ LayoutTests/fast/regex/ LayoutTests/fast/regions/ --verbose

Dirk Pranke

Comment 3 2012-11-29 12:05:41 PST

Unfortunately, my attempts to build the efl port to reproduce this have stalled out temporarily. Do other EFL devs also see this ? If someone can post the command line they use and the results.json they get from that command line to this bug (and maybe also the results.html, though that's just a static template), that would be helpful.

Yael

Comment 4 2012-11-29 12:15:31 PST

I am trying to recreate this problem, and I can't. Maybe it happened because the flaky tests were passing on a second try and I did not notice that.

Yael

Comment 5 2012-11-29 12:17:08 PST

Created attachment 176781 [details] Results file

Yael

Comment 6 2012-11-29 12:17:28 PST

Created attachment 176782 [details] json file

Yael

Comment 7 2012-11-29 12:18:36 PST

Created attachment 176783 [details] ref tests failures

Yael

Comment 8 2012-11-29 12:19:41 PST

Created attachment 176785 [details] printout of the run

Yael

Comment 9 2012-11-29 12:20:45 PST

70 ref tests failed, but only 7 show in results.html

Dirk Pranke

Comment 10 2012-11-29 12:26:28 PST

We're logging both the tests that are failing as expected and the tests that are failing unexpectedly. results.html only shows the unexpected failures by default. If you click on the options link in the top right corner, and uncheck the "only unexpected results" box, you should see the other failures. In the summary at the end of the run, we report 30 flaky failures and 7 consistent failures. I think this is working correctly; does that make sense?

Yael

Comment 11 2012-11-29 12:30:35 PST

(In reply to comment #10) > We're logging both the tests that are failing as expected and the tests that are failing unexpectedly. results.html only shows the unexpected failures by default. If you click on the options link in the top right corner, and uncheck the "only unexpected results" box, you should see the other failures. > > In the summary at the end of the run, we report 30 flaky failures and 7 consistent failures. > > I think this is working correctly; does that make sense? I did not now about that option. Thanks for your help!

Note You need to log in before you can comment on or make changes to this bug.