When running run-webkit-tests, ref test failures do not show up in the results.html, and that is very misleading. When I run it with --verbose, I see in the shell "failed (reference mismatch)", and I see the images in the results directory. It would be much nicer if results.html would also show these failures.
They show up for me ... there is a link for "ref html" next to the links for the images and the diff. What command line are you using?
(In reply to comment #1) > They show up for me ... there is a link for "ref html" next to the links for the images and the diff. > > What command line are you using? For example Tools/Scripts/run-webkit-tests --debug --efl -2 LayoutTests/fast/overflow/ LayoutTests/fast/parser/ LayoutTests/fast/regex/ LayoutTests/fast/regions/ --verbose
Unfortunately, my attempts to build the efl port to reproduce this have stalled out temporarily. Do other EFL devs also see this ? If someone can post the command line they use and the results.json they get from that command line to this bug (and maybe also the results.html, though that's just a static template), that would be helpful.
I am trying to recreate this problem, and I can't. Maybe it happened because the flaky tests were passing on a second try and I did not notice that.
Created attachment 176781 [details] Results file
Created attachment 176782 [details] json file
Created attachment 176783 [details] ref tests failures
Created attachment 176785 [details] printout of the run
70 ref tests failed, but only 7 show in results.html
We're logging both the tests that are failing as expected and the tests that are failing unexpectedly. results.html only shows the unexpected failures by default. If you click on the options link in the top right corner, and uncheck the "only unexpected results" box, you should see the other failures. In the summary at the end of the run, we report 30 flaky failures and 7 consistent failures. I think this is working correctly; does that make sense?
(In reply to comment #10) > We're logging both the tests that are failing as expected and the tests that are failing unexpectedly. results.html only shows the unexpected failures by default. If you click on the options link in the top right corner, and uncheck the "only unexpected results" box, you should see the other failures. > > In the summary at the end of the run, we report 30 flaky failures and 7 consistent failures. > > I think this is working correctly; does that make sense? I did not now about that option. Thanks for your help!