Bug 67268 – [NRWT] New tests without expected files are reported as flakey tests

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Csaba Osztrogonác]

I don't understand how can pass a test for the second run ...

http://build.webkit.org/results/Qt%20Linux%20Release/r94155%20%2836977%29/results.html
http://build.webkit.org/builders/Qt%20Linux%20Release/builds/36977/steps/layout-test/logs/stdio

Unexpected flakiness: no expected results found (3)
  fast/multicol/overflow-across-columns.html = MISSING PASS
  fast/multicol/overflow-unsplittable.html = MISSING PASS
  fast/multicol/positive-leading.html = MISSING PASS

A similar bug: https://bugs.webkit.org/show_bug.cgi?id=64812

[Csaba Osztrogonác]

More information: NRWT reports they are flakey when there are failing tests too, but it reports only missing when there aren't any failing test:

Regressions: Unexpected no expected results found : (3)
  fast/multicol/overflow-across-columns.html = MISSING
  fast/multicol/overflow-unsplittable.html = MISSING
  fast/multicol/positive-leading.html = MISSING

[Eric Seidel (no email)]

I could see how this might happen.  Sad times.

[Adam Barth]

Yep.  Pretty easy to fix.

[Zoltan Herczeg]

> Yep.  Pretty easy to fix.

Cool. When do you intend to?

[Adam Barth]

> Cool. When do you intend to?

Unfortunately, I've been dragged away to work on some other issues, but I hope to return to NRWT at some point.  Eric is more actively working on these issues.  Hopefully he'll be able to fix this issue soon.

[Csaba Osztrogonác]

Now

[Csaba Osztrogonác]

Very strange ... Missing tests can be flakey or missing
when there aren't any failing test:

missing: http://build.webkit.org/builders/Qt%20Linux%20Release/builds/37323/steps/layout-test/logs/stdio

flakey: http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20WebKit2/builds/11534/steps/layout-test/logs/stdio

[Csaba Osztrogonác]

I don't understand this crazy NRWT ... I'm really thinking that we should switch back to the stable ORWT ... NRWT is so slower and so flakey. 

-------
http://build.webkit.org/builders/Qt%20Linux%20Release/builds/37349 - 3 flakes, rc=0

Unexpected flakiness: tests timed out (2)
  storage/open-database-creation-callback-isolated-world.html = TIMEOUT PASS
  storage/open-database-creation-callback.html = TIMEOUT PASS

Unexpected flakiness: no expected results found (1)
  svg/custom/simple-text-double-shadow.svg = MISSING PASS
-------
http://build.webkit.org/builders/Qt%20Linux%20Release/builds/37350 - 1 missing results, rc=1

Regressions: Unexpected no expected results found : (1)
  svg/custom/simple-text-double-shadow.svg = MISSING

[Eric Seidel (no email)]

I fully support you switching back to ORWT for the Qt port.  It's my job to convince you that NRWT is better, not my job to force you to switch.

I'm happy to investigate this issue further on monday for you.  But again, you should also feel welcome to switch back to ORWT until the Pros list is longer than the Cons list for Qt.

[Ryosuke Niwa]

Created attachment 110785 [details]
Fixes the bug

[Ryosuke Niwa]

Created attachment 110786 [details]
Revert some change in expected result

[Dirk Pranke]

Comment on attachment 110786 [details]
Revert some change in expected result

This patch looks okay, but it's definitely a hack that we're counting MISSING tests as flaky, and I officially declare tthe logic in summarize_results and get_failures to be too complicated, so we need to clean this up. Can you file a separate bug for this effort?

[Ojan Vafai]

Comment on attachment 110786 [details]
Revert some change in expected result

View in context: https://bugs.webkit.org/attachment.cgi?id=110786&action=review

Are there any tests for this that you can update?

> Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:142
> +            num_flaky += 1

Why would we increment num_flaky if we're no longer considering this flaky?

[Ojan Vafai]

You might be able to test this through http://codesearch.google.com/codesearch#OAMlx_jo-ck/src/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/views/printing_unittest.py&l=416.

[Dirk Pranke]

(In reply to comment #13)
> (From update of attachment 110786 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=110786&action=review
> 
> Are there any tests for this that you can update?
> 

I believe rniwa is adding a patch w/ a test.

> > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:142
> > +            num_flaky += 1
> 
> Why would we increment num_flaky if we're no longer considering this flaky?

You may have missed the IRC conversation I had w/ rniwa ... the jist of it is that we increment num_flaky in order to turn the buildbot orange instead of red. This is a hack and that needs to be cleaned up in the follow-on bug I have asked for. This behavior was previously agreed to (in bug 64812); this bug is just making that behavior actually work correctly.

[Ojan Vafai]

(In reply to comment #15)
> (In reply to comment #13)
> > (From update of attachment 110786 [details] [details])
> > View in context: https://bugs.webkit.org/attachment.cgi?id=110786&action=review
> > > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:142
> > > +            num_flaky += 1
> > 
> > Why would we increment num_flaky if we're no longer considering this flaky?
> 
> You may have missed the IRC conversation I had w/ rniwa ... the jist of it is that we increment num_flaky in order to turn the buildbot orange instead of red. This is a hack and that needs to be cleaned up in the follow-on bug I have asked for. This behavior was previously agreed to (in bug 64812); this bug is just making that behavior actually work correctly.

In that case, there should be a comment + FIXME explaining this.

[Dirk Pranke]

(In reply to comment #16)
> (In reply to comment #15)
> > (In reply to comment #13)
> > > (From update of attachment 110786 [details] [details] [details])
> > > View in context: https://bugs.webkit.org/attachment.cgi?id=110786&action=review
> > > > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:142
> > > > +            num_flaky += 1
> > > 
> > > Why would we increment num_flaky if we're no longer considering this flaky?
> > 
> > You may have missed the IRC conversation I had w/ rniwa ... the jist of it is that we increment num_flaky in order to turn the buildbot orange instead of red. This is a hack and that needs to be cleaned up in the follow-on bug I have asked for. This behavior was previously agreed to (in bug 64812); this bug is just making that behavior actually work correctly.
> 
> In that case, there should be a comment + FIXME explaining this.

Yup, that's exactly what I asked for in comment #12.

[Ryosuke Niwa]

Created attachment 110794 [details]
Added a test

[WebKit Review Bot]

Comment on attachment 110794 [details]
Added a test

Clearing flags on attachment: 110794

Committed r97339: <http://trac.webkit.org/changeset/97339>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.