Bug 30809 – buildbots should use --exit-after-N-failures

• Status: RESOLVED
• Resolution: WONTFIX
• Component: Tools / Tests

[Eric Seidel (no email)]

buildbots should use --exit-after-N-failures

http://build.webkit.org/builders/Leopard%20Intel%20Release%20%28Tests%29/builds/6544
has taken over an hour to run because every layout test is crashing!

http://build.webkit.org/builders/Leopard%20Intel%20Release%20%28Tests%29/builds/6544/steps/layout-test/logs/stdio

It should have exited after the first 10 failures or similar.

[Eric Seidel (no email)]

Created attachment 41936 [details]
Patch v1

[Eric Seidel (no email)]

Created attachment 41937 [details]
Upped limit to 20 per Mark's request.  Also added failure line to phase output.

[David Levin]

Comment on attachment 41937 [details]
Upped limit to 20 per Mark's request.  Also added failure line to phase output.

Looks good to me and it sounds like you got feedback from Mark as well.

I'll not cq+ this at the moment in case anyone else (Mark) wants a last chance to comment.

[Eric Seidel (no email)]

Comment on attachment 41937 [details]
Upped limit to 20 per Mark's request.  Also added failure line to phase output.

No comments.  Mark has to manually update the buildbots anyway, so he will see it again then.

[WebKit Commit Bot]

Comment on attachment 41937 [details]
Upped limit to 20 per Mark's request.  Also added failure line to phase output.

Clearing flags on attachment: 41937

Committed r50401: <http://trac.webkit.org/changeset/50401>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Csaba Osztrogonác]

Created attachment 43773 [details]
upped limit to 40

Yesterday master config was updated and it made QtBuildBot unhappy,
because we had 16 failing and 31 new tests. ( 31+16 > 20 ) On the
one hand failing tests counter shouldn't include new tests, that's why
I propose to modify it: https://bugs.webkit.org/show_bug.cgi?id=31829.
On the other hand if a patch committed make some layout tests fail,
and later an other patch make more failing tests, etc, we can easily 
lost which commit caused which fails. If we don't have this information,
debugging can be more difficult.

We make an effort to fix Qt related bugs ASAP, but quite a lot Qt-port 
maintainer sleep when american guys work. (because of timezone difference)

To make QtBuildbot and Qt guys happier, I suggest increasing 
the failure limit to 40.

[Adam Barth]

style-queue successfully ran check-webkit-style on attachment 43773 [details] without any errors

[Eric Seidel (no email)]

Comment on attachment 43773 [details]
upped limit to 40

I don't think we want this change.  I think this is fixing a symptom, not the root problem.  If I correctly overheard in #webkit, Mark Rowe feels similar.