Bug 73843 - nrwt exits early too frequently
: nrwt exits early too frequently
Status: RESOLVED FIXED
: WebKit
Tools / Tests
: 528+ (Nightly build)
: Unspecified Unspecified
: P2 Normal
Assigned To:
:
:
:
: 88680
  Show dependency treegraph
 
Reported: 2011-12-05 10:41 PST by
Modified: 2012-06-11 14:18 PST (History)


Attachments
Patch (1.37 KB, patch)
2011-12-05 10:47 PST, Ryosuke Niwa
no flags Review Patch | Details | Formatted Diff | Diff
Patch (2.86 KB, patch)
2012-06-08 17:52 PST, Dirk Pranke
no flags Review Patch | Details | Formatted Diff | Diff
change 0 to None (2.86 KB, patch)
2012-06-08 18:14 PST, Dirk Pranke
no flags Review Patch | Details | Formatted Diff | Diff


Note

You need to log in before you can comment on or make changes to this bug.


Description From 2011-12-05 10:41:04 PST
Exiting early after a mere 20+ tests timed out or crashed seems too disruptive. This has happened very frequently on Mac port for example:
http://build.webkit.org/builders/SnowLeopard%20Intel%20Release%20%28Tests%29?numbuilds=100

We probably need to tolerate more tests to time out / crash.
------- Comment #1 From 2011-12-05 10:47:33 PST -------
Created an attachment (id=117899) [details]
Patch
------- Comment #2 From 2011-12-05 11:02:07 PST -------
Part of the point of this policy was to avoid slowing down the bots too much when some low-level crash is introduced in WebKit. In many cases, all 20 tests will have crashed for the same reason, and any other tests that would have been run would also have crashed for the same reason, so running more tests doesn't give more diagnostic information. And since we have to wait for Crash Reporter for each crashed test, running more tests can significantly slow down the bots.
------- Comment #3 From 2011-12-05 11:03:57 PST -------
I don't think you want to change this.  At least not unless you fix https://bugs.webkit.org/show_bug.cgi?id=37797.  But I don't think we want that behavior on the bots anyway.
------- Comment #4 From 2011-12-05 11:05:00 PST -------
(In reply to comment #2)
> Part of the point of this policy was to avoid slowing down the bots too much when some low-level crash is introduced in WebKit. In many cases, all 20 tests will have crashed for the same reason, and any other tests that would have been run would also have crashed for the same reason, so running more tests doesn't give more diagnostic information. And since we have to wait for Crash Reporter for each crashed test, running more tests can significantly slow down the bots.

Right. However, in the world where we have multiple worker processes, 20 might be unrealistically low since many tests that get paged out temporarily will end up taking much longer time to run. This problem might go away once you add more RAM to Mac bots.

For example, I haven't had this issue on my MacPro (has 12GB of RAM).
------- Comment #5 From 2011-12-05 11:07:04 PST -------
It is possible we should re-evaluate this for the multi-proesss world.
------- Comment #6 From 2011-12-05 12:22:17 PST -------
I don't think we need to re-evaluate this. I think we need to fix the bugs that are causing these timeouts.

For comparison, we don't have these issues on the chromium bots, and no one has found the exit-early behavior to be a problem.
------- Comment #7 From 2012-06-08 13:23:08 PST -------
(In reply to comment #6)
> I don't think we need to re-evaluate this. I think we need to fix the bugs that are causing these timeouts.

If we can fix the underlying issues, that's clearly the best resolution.

> For comparison, we don't have these issues on the chromium bots, and no one has found the exit-early behavior to be a problem.

Is this just because we have a lot of tests listed as expected to timeout/crash flakily so they don't contribute to the count?
------- Comment #8 From 2012-06-08 17:52:30 PST -------
Created an attachment (id=146669) [details]
Patch
------- Comment #9 From 2012-06-08 17:53:03 PST -------
rniwa / ojan, please take a look?
------- Comment #10 From 2012-06-08 18:14:53 PST -------
Created an attachment (id=146673) [details]
change 0 to None
------- Comment #11 From 2012-06-08 18:26:24 PST -------
(From update of attachment 146673 [details])
setting cq- because we should update the chromium bots to pass these flags first.
------- Comment #12 From 2012-06-08 19:42:15 PST -------
(From update of attachment 146673 [details])
I'd prefer a test showing that None does what we think it does.
------- Comment #13 From 2012-06-09 20:17:37 PST -------
(In reply to comment #12)
> (From update of attachment 146673 [details] [details])
> I'd prefer a test showing that None does what we think it does.

I'm not entirely sure what you're looking for? It's the default argument, so the code path is being executed. However, I'm not sure how we would demonstrate that we never exit early? Given that there are many tests that include failures, is that sufficient? E.g., run_webkit_tests_integrationtest.MainTest.test_all shows that we run all of the tests and that N of them fail ...
------- Comment #14 From 2012-06-10 09:09:11 PST -------
(In reply to comment #13)
> (In reply to comment #12)
> > (From update of attachment 146673 [details] [details] [details])
> > I'd prefer a test showing that None does what we think it does.
> 
> I'm not entirely sure what you're looking for? It's the default argument, so the code path is being executed. However, I'm not sure how we would demonstrate that we never exit early? Given that there are many tests that include failures, is that sufficient? E.g., run_webkit_tests_integrationtest.MainTest.test_all shows that we run all of the tests and that N of them fail ...

You're right. I didn't think it through very well.
------- Comment #15 From 2012-06-11 14:18:15 PST -------
(From update of attachment 146673 [details])
Clearing flags on attachment: 146673

Committed r120007: <http://trac.webkit.org/changeset/120007>
------- Comment #16 From 2012-06-11 14:18:21 PST -------
All reviewed patches have been landed.  Closing bug.