RESOLVED INVALID 103631
Failures in ref tests should appear in results.html 
https://bugs.webkit.org/show_bug.cgi?id=103631
Summary Failures in ref tests should appear in results.html
Yael
Reported 2012-11-29 06:11:06 PST
When running run-webkit-tests, ref test failures do not show up in the results.html, and that is very misleading. When I run it with --verbose, I see in the shell "failed (reference mismatch)", and I see the images in the results directory. It would be much nicer if results.html would also show these failures.
Attachments
Results file (39.53 KB, text/html)
2012-11-29 12:17 PST, Yael 		no flags
Details
json file (37.76 KB, application/octet-stream)
2012-11-29 12:17 PST, Yael 		no flags
Details
ref tests failures (5.51 KB, text/plain)
2012-11-29 12:18 PST, Yael 		no flags
Details
printout of the run (670.45 KB, text/plain)
2012-11-29 12:19 PST, Yael 		no flags
Details
View All Add attachment proposed patch, testcase, etc.
Dirk Pranke
Comment 1 2012-11-29 10:50:04 PST
They show up for me ... there is a link for "ref html" next to the links for the images and the diff. What command line are you using?
Yael
Comment 2 2012-11-29 10:55:27 PST
(In reply to comment #1) > They show up for me ... there is a link for "ref html" next to the links for the images and the diff. > > What command line are you using? For example Tools/Scripts/run-webkit-tests --debug --efl -2 LayoutTests/fast/overflow/ LayoutTests/fast/parser/ LayoutTests/fast/regex/ LayoutTests/fast/regions/ --verbose
Dirk Pranke
Comment 3 2012-11-29 12:05:41 PST
Unfortunately, my attempts to build the efl port to reproduce this have stalled out temporarily. Do other EFL devs also see this ? If someone can post the command line they use and the results.json they get from that command line to this bug (and maybe also the results.html, though that's just a static template), that would be helpful.
Yael
Comment 4 2012-11-29 12:15:31 PST
I am trying to recreate this problem, and I can't. Maybe it happened because the flaky tests were passing on a second try and I did not notice that.
Yael
Comment 5 2012-11-29 12:17:08 PST
Created attachment 176781 [details] Results file
Yael
Comment 6 2012-11-29 12:17:28 PST
Created attachment 176782 [details] json file
Yael
Comment 7 2012-11-29 12:18:36 PST
Created attachment 176783 [details] ref tests failures
Yael
Comment 8 2012-11-29 12:19:41 PST
Created attachment 176785 [details] printout of the run
Yael
Comment 9 2012-11-29 12:20:45 PST
70 ref tests failed, but only 7 show in results.html
Dirk Pranke
Comment 10 2012-11-29 12:26:28 PST
We're logging both the tests that are failing as expected and the tests that are failing unexpectedly. results.html only shows the unexpected failures by default. If you click on the options link in the top right corner, and uncheck the "only unexpected results" box, you should see the other failures. In the summary at the end of the run, we report 30 flaky failures and 7 consistent failures. I think this is working correctly; does that make sense?
Yael
Comment 11 2012-11-29 12:30:35 PST
(In reply to comment #10) > We're logging both the tests that are failing as expected and the tests that are failing unexpectedly. results.html only shows the unexpected failures by default. If you click on the options link in the top right corner, and uncheck the "only unexpected results" box, you should see the other failures. > > In the summary at the end of the run, we report 30 flaky failures and 7 consistent failures. > > I think this is working correctly; does that make sense? I did not now about that option. Thanks for your help!
Note You need to log in before you can comment on or make changes to this bug.