Bug 67268 - [NRWT] New tests without expected files are reported as flakey tests
Summary: [NRWT] New tests without expected files are reported as flakey tests
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: All All
: P1 Major
Assignee: Ryosuke Niwa
URL:
Keywords:
Depends on:
Blocks: 34984 64812 69990
  Show dependency treegraph
 
Reported: 2011-08-31 01:01 PDT by Csaba Osztrogonác
Modified: 2011-10-12 20:17 PDT (History)
12 users (show)

See Also:


Attachments
Fixes the bug (2.47 KB, patch)
2011-10-12 18:20 PDT, Ryosuke Niwa
no flags Details | Formatted Diff | Diff
Revert some change in expected result (2.12 KB, patch)
2011-10-12 18:21 PDT, Ryosuke Niwa
no flags Details | Formatted Diff | Diff
Added a test (4.13 KB, patch)
2011-10-12 19:15 PDT, Ryosuke Niwa
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Csaba Osztrogonác 2011-08-31 01:01:47 PDT
I don't understand how can pass a test for the second run ...

http://build.webkit.org/results/Qt%20Linux%20Release/r94155%20%2836977%29/results.html
http://build.webkit.org/builders/Qt%20Linux%20Release/builds/36977/steps/layout-test/logs/stdio

Unexpected flakiness: no expected results found (3)
  fast/multicol/overflow-across-columns.html = MISSING PASS
  fast/multicol/overflow-unsplittable.html = MISSING PASS
  fast/multicol/positive-leading.html = MISSING PASS

A similar bug: https://bugs.webkit.org/show_bug.cgi?id=64812
Comment 1 Csaba Osztrogonác 2011-08-31 07:33:33 PDT
More information: NRWT reports they are flakey when there are failing tests too, but it reports only missing when there aren't any failing test:

Regressions: Unexpected no expected results found : (3)
  fast/multicol/overflow-across-columns.html = MISSING
  fast/multicol/overflow-unsplittable.html = MISSING
  fast/multicol/positive-leading.html = MISSING
Comment 2 Eric Seidel (no email) 2011-08-31 09:27:55 PDT
I could see how this might happen.  Sad times.
Comment 3 Adam Barth 2011-08-31 13:46:14 PDT
Yep.  Pretty easy to fix.
Comment 4 Zoltan Herczeg 2011-09-01 04:22:29 PDT
> Yep.  Pretty easy to fix.

Cool. When do you intend to?
Comment 5 Adam Barth 2011-09-01 10:32:37 PDT
> Cool. When do you intend to?

Unfortunately, I've been dragged away to work on some other issues, but I hope to return to NRWT at some point.  Eric is more actively working on these issues.  Hopefully he'll be able to fix this issue soon.
Comment 6 Csaba Osztrogonác 2011-09-09 10:58:15 PDT
Now
Comment 7 Csaba Osztrogonác 2011-09-09 11:01:13 PDT
Very strange ... Missing tests can be flakey or missing
when there aren't any failing test:

missing: http://build.webkit.org/builders/Qt%20Linux%20Release/builds/37323/steps/layout-test/logs/stdio

flakey: http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20WebKit2/builds/11534/steps/layout-test/logs/stdio
Comment 8 Csaba Osztrogonác 2011-09-10 07:09:55 PDT
I don't understand this crazy NRWT ... I'm really thinking that we should switch back to the stable ORWT ... NRWT is so slower and so flakey. 

-------
http://build.webkit.org/builders/Qt%20Linux%20Release/builds/37349 - 3 flakes, rc=0

Unexpected flakiness: tests timed out (2)
  storage/open-database-creation-callback-isolated-world.html = TIMEOUT PASS
  storage/open-database-creation-callback.html = TIMEOUT PASS

Unexpected flakiness: no expected results found (1)
  svg/custom/simple-text-double-shadow.svg = MISSING PASS
-------
http://build.webkit.org/builders/Qt%20Linux%20Release/builds/37350 - 1 missing results, rc=1

Regressions: Unexpected no expected results found : (1)
  svg/custom/simple-text-double-shadow.svg = MISSING
Comment 9 Eric Seidel (no email) 2011-09-11 14:40:20 PDT
I fully support you switching back to ORWT for the Qt port.  It's my job to convince you that NRWT is better, not my job to force you to switch.

I'm happy to investigate this issue further on monday for you.  But again, you should also feel welcome to switch back to ORWT until the Pros list is longer than the Cons list for Qt.
Comment 10 Ryosuke Niwa 2011-10-12 18:20:46 PDT
Created attachment 110785 [details]
Fixes the bug
Comment 11 Ryosuke Niwa 2011-10-12 18:21:30 PDT
Created attachment 110786 [details]
Revert some change in expected result
Comment 12 Dirk Pranke 2011-10-12 18:39:53 PDT
Comment on attachment 110786 [details]
Revert some change in expected result

This patch looks okay, but it's definitely a hack that we're counting MISSING tests as flaky, and I officially declare tthe logic in summarize_results and get_failures to be too complicated, so we need to clean this up. Can you file a separate bug for this effort?
Comment 13 Ojan Vafai 2011-10-12 18:51:10 PDT
Comment on attachment 110786 [details]
Revert some change in expected result

View in context: https://bugs.webkit.org/attachment.cgi?id=110786&action=review

Are there any tests for this that you can update?

> Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:142
> +            num_flaky += 1

Why would we increment num_flaky if we're no longer considering this flaky?
Comment 15 Dirk Pranke 2011-10-12 18:57:27 PDT
(In reply to comment #13)
> (From update of attachment 110786 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=110786&action=review
> 
> Are there any tests for this that you can update?
> 

I believe rniwa is adding a patch w/ a test.

> > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:142
> > +            num_flaky += 1
> 
> Why would we increment num_flaky if we're no longer considering this flaky?

You may have missed the IRC conversation I had w/ rniwa ... the jist of it is that we increment num_flaky in order to turn the buildbot orange instead of red. This is a hack and that needs to be cleaned up in the follow-on bug I have asked for. This behavior was previously agreed to (in bug 64812); this bug is just making that behavior actually work correctly.
Comment 16 Ojan Vafai 2011-10-12 19:04:40 PDT
(In reply to comment #15)
> (In reply to comment #13)
> > (From update of attachment 110786 [details] [details])
> > View in context: https://bugs.webkit.org/attachment.cgi?id=110786&action=review
> > > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:142
> > > +            num_flaky += 1
> > 
> > Why would we increment num_flaky if we're no longer considering this flaky?
> 
> You may have missed the IRC conversation I had w/ rniwa ... the jist of it is that we increment num_flaky in order to turn the buildbot orange instead of red. This is a hack and that needs to be cleaned up in the follow-on bug I have asked for. This behavior was previously agreed to (in bug 64812); this bug is just making that behavior actually work correctly.

In that case, there should be a comment + FIXME explaining this.
Comment 17 Dirk Pranke 2011-10-12 19:12:05 PDT
(In reply to comment #16)
> (In reply to comment #15)
> > (In reply to comment #13)
> > > (From update of attachment 110786 [details] [details] [details])
> > > View in context: https://bugs.webkit.org/attachment.cgi?id=110786&action=review
> > > > Tools/Scripts/webkitpy/layout_tests/controllers/manager.py:142
> > > > +            num_flaky += 1
> > > 
> > > Why would we increment num_flaky if we're no longer considering this flaky?
> > 
> > You may have missed the IRC conversation I had w/ rniwa ... the jist of it is that we increment num_flaky in order to turn the buildbot orange instead of red. This is a hack and that needs to be cleaned up in the follow-on bug I have asked for. This behavior was previously agreed to (in bug 64812); this bug is just making that behavior actually work correctly.
> 
> In that case, there should be a comment + FIXME explaining this.

Yup, that's exactly what I asked for in comment #12.
Comment 18 Ryosuke Niwa 2011-10-12 19:15:22 PDT
Created attachment 110794 [details]
Added a test
Comment 19 WebKit Review Bot 2011-10-12 20:17:47 PDT
Comment on attachment 110794 [details]
Added a test

Clearing flags on attachment: 110794

Committed r97339: <http://trac.webkit.org/changeset/97339>
Comment 20 WebKit Review Bot 2011-10-12 20:17:54 PDT
All reviewed patches have been landed.  Closing bug.