Summary: | Add --exit-after-n-failures/crashes to NRWT | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | WebKit | Reporter: | Mihai Parparita <mihaip> | ||||||
Component: | Tools / Tests | Assignee: | Mihai Parparita <mihaip> | ||||||
Status: | RESOLVED FIXED | ||||||||
Severity: | Normal | CC: | commit-queue, dpranke, eric, mrobinson, ojan | ||||||
Priority: | P2 | ||||||||
Version: | 528+ (Nightly build) | ||||||||
Hardware: | All | ||||||||
OS: | All | ||||||||
Attachments: |
|
Description
Mihai Parparita
2010-12-15 18:41:04 PST
Created attachment 76726 [details]
Patch
Comment on attachment 76726 [details]
Patch
Looks OK. I'm sad that there are two of these. I guess I didn't know the "crashes" varient had been added. I wonder what itch that is scratching?
The ChangeLog from http://trac.webkit.org/changeset/62728 has the motivation (Ojan, can you elaborate?). Additionally, it looks like --exit-after-n-crashes was changed to --exit-after-n-crashes-or-timeouts with http://trac.webkit.org/changeset/62732. I guess I should mirror that here too? Huh? What's the difference between -failures and blah-blah-blah-timeouts? Looks like ojan would know. exit-after-n-crashes-or-timeouts lets the bots run with a ton of failures. This is important for being able to land a change that needs a ton of new platform-specific expectations. You need the bot to run the test in order to grab the new results. We exit after n crashes/timeouts because those are the ones that are really slow and thus cause the bot to take forever to cycle. But we don't care about other failure types because they don't cause cycle time to be slow. All of this is in the ChangeLog descriptions of those two patches. exit-after-n-failures is just there for the commit-queue I believe since it wants to bail ASAP. (In reply to comment #3) > Additionally, it looks like --exit-after-n-crashes was changed to --exit-after-n-crashes-or-timeouts with http://trac.webkit.org/changeset/62732. I guess I should mirror that here too? Yes. I wrote --exit-after-N-failures initially to get around waiting for a patch which caused every test to crash. It's true that the CQ likes exiting after any failure, but if every other use case is for the --exit-blah-blah-timeouts case, then maybe that should be the only one. Or maybe the "exit-after-timeouts" whatever thing should just be built in to both harnesses. I would want NRWT to bail after 30 crashes or timeouts regardless, I would think. (In reply to comment #7) > It's true that the CQ likes exiting after any failure, but if every other use case is for the --exit-blah-blah-timeouts case, then maybe that should be the only one. Fine with me. > Or maybe the "exit-after-timeouts" whatever thing should just be built in to both harnesses. I would want NRWT to bail after 30 crashes or timeouts regardless, I would think. I support getting rid of the commandline option and just always having it run. Although, in the case of chromium tests, we'd want to bail after 30 unexpected crash/timeouts Created attachment 76787 [details]
Patch
I found --exit-after-n-failures useful when used in combinations with --iterations for tracking down flaky tests (--iterations 100 --exit-after-n-failures 1 stops on the first failure). Adding --iterations support to NRWT is next on my todo list. I've updated the patch to support --exit-after-n-crashes-or-timeouts. Comment on attachment 76787 [details]
Patch
Change looks good to me. Thanks for doing this!
I have no real opinions on any of these flags, by the way. Whatever you guys decide is fine by me. Comment on attachment 76787 [details]
Patch
OK.
Comment on attachment 76787 [details] Patch Rejecting attachment 76787 [details] from commit-queue. Failed to run "['./WebKitTools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '--bot-id=cr-jail-3', 'apply-attachment', '--non-interactive', 76787]" exit_code: 2 Last 500 characters of output: ailed to merge in the changes. Patch failed at 0001 2010-12-16 Yury Semikhatsky <yurys@chromium.org> When you have resolved this problem run "git rebase --continue". If you would prefer to skip this patch, instead run "git rebase --skip". To restore the original branch and stop rebasing run "git rebase --abort". rebase refs/remotes/origin/master: command returned error: 1 Died at WebKitTools/Scripts/update-webkit line 132. Failed to run "['WebKitTools/Scripts/update-webkit']" exit_code: 2 Full output: http://queues.webkit.org/results/7176074 Comment on attachment 76787 [details]
Patch
Sorry, I'll go look at the bot.
Comment on attachment 76787 [details] Patch Rejecting attachment 76787 [details] from commit-queue. Failed to run "['./WebKitTools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '--bot-id=cr-jail-3', 'apply-attachment', '--non-interactive', 76787]" exit_code: 2 Last 500 characters of output: ailed to merge in the changes. Patch failed at 0001 2010-12-16 Yury Semikhatsky <yurys@chromium.org> When you have resolved this problem run "git rebase --continue". If you would prefer to skip this patch, instead run "git rebase --skip". To restore the original branch and stop rebasing run "git rebase --abort". rebase refs/remotes/origin/master: command returned error: 1 Died at WebKitTools/Scripts/update-webkit line 132. Failed to run "['WebKitTools/Scripts/update-webkit']" exit_code: 2 Full output: http://queues.webkit.org/results/7140100 OH. It's nto the bot. THe patch simply fails to apply. :) It also appears that git.webkit.org is down, which might be related? Comment on attachment 76787 [details]
Patch
No, the bot itself was wedged. I'm fixing it now. Sorry!
The commit-queue encountered the following flaky tests while processing attachment 76787 [details]: fast/workers/storage/use-same-database-in-page-and-workers.html bug 50995 (author: dumi@chromium.org) java/argument-to-object-type.html bug 51102 (author: ap@webkit.org) The commit-queue is continuing to process your patch. Comment on attachment 76787 [details] Patch Clearing flags on attachment: 76787 Committed r74223: <http://trac.webkit.org/changeset/74223> All reviewed patches have been landed. Closing bug. |