Bug 106945 - REGRESSION: Sheriffbot is failing to process rollout requests
Summary: REGRESSION: Sheriffbot is failing to process rollout requests
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Alan Cutter
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-01-15 14:45 PST by Levi Weintraub
Modified: 2013-02-19 11:12 PST (History)
4 users (show)

See Also:


Attachments
Patch (4.92 KB, patch)
2013-01-16 18:38 PST, Alan Cutter
no flags Details | Formatted Diff | Diff
Patch (8.96 KB, patch)
2013-01-16 22:28 PST, Alan Cutter
no flags Details | Formatted Diff | Diff
Patch (8.67 KB, patch)
2013-01-17 19:32 PST, Alan Cutter
no flags Details | Formatted Diff | Diff
Patch (8.23 KB, patch)
2013-02-18 01:40 PST, Alan Cutter
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Levi Weintraub 2013-01-15 14:45:16 PST
At least since Friday (when I started sheriffing), sheriffbot responds to all rollout requests with a message like this:
sheriffbot: Exception executing command: Failed to run "['git', 'svn', 'find-rev', 'r139779']" exit_code: 1 cwd: /mnt/git/webkit-sheriff-bot

The problem persists through restarts.
Comment 1 Eric Seidel (no email) 2013-01-15 14:47:39 PST
I assume that webkit-patch prepare-rollout works fine?  That's the command SB runs under the covers?
Comment 2 Levi Weintraub 2013-01-15 14:52:36 PST
Dunno? Haven't tried it!
Comment 3 Alan Cutter 2013-01-16 04:10:31 PST
This failure is caused by the migration of Sheriffbot to Google Compute Engine.

The git repository was not properly set up with SVN (manually fixed now on gce-feeder-01) however this is not the only issue.

Another problem that occurs when sheriffbot tries to do a rollout on gce-feeder-01 is this error:

> Running: webkit-patch --status-host=queues.webkit.org --bot-id=gce-feeder-01 create-rollout --force-clean --parent-command=sheriff-bot 139575 Rollout test (Requested by alancutter on #webkit).
> Failed to run "['/mnt/git/webkit-sheriff-bot/Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '--..." exit_code: 2 cwd: /mnt/git/webkit-sheriff-bot
> 
> Preparing rollout for bug 106747.
> Updating working directory
> Logging in as webkit.review.bot@gmail.com...
> Creating bug with title "REGRESSION(r139575): Rollout test (Requested by alancutter on #webkit)."
> Bug 106982 created.
> https://bugs.webkit.org/show_bug.cgi?id=106982
> Fetching: https://bugs.webkit.org/show_bug.cgi?id=106747&ctype=xml&excludefield=attachmentdata
> Re-opening bug 106747
> Re-opened since this is blocked by bug 106982
> Failed to run "['Tools/Scripts/prepare-ChangeLog']" exit_code: 1
> 
>   Running status to find changed, added, or removed files.
>   Reviewing diff to determine which lines changed.
>   Extracting affected function names from source files.
> Failed to determine ChangeLog name.
> Either:
>   set CHANGE_LOG_NAME in your environment
>   OR pass --name= on the command line
>   OR set REAL_NAME in your environment  OR git users can set 'git config user.name'
>                              
>   Failed to run "['Tools/Scripts/prepare-ChangeLog']" exit_code: 1
>   Updating OpenSource
>   Current branch master is up to date.
>     Running status to find changed, added, or removed files.
>     Reviewing diff to determine which lines changed.
>     Extracting affected function names from source files.
>   Failed to determine ChangeLog name.
>   Either:
>     set CHANGE_LOG_NAME in your environment
>     OR pass --name= on the command line
>     OR set REAL_NAME in your environment  OR git users can set 'git config user.name'
> 
>   No work item. Sleeping until 2013-01-16 07:20:26 (2 mins).
> 

Tools/Scripts/prepare-ChangeLog fails to execute (stand alone) when there are changes present in the repository on gce-feeder-01 because the git config doesn't contain a username or email address for Sheriffbot to create a ChangeLog entry.

These issues have been fixed on gce-feeder-01 and I've tested that Sheriffbot can perform rollouts again.

Patch to update the GCE build scripts to resolve these issues in the GCE build scripts coming (after I get some sleep). :)
Comment 4 Alan Cutter 2013-01-16 18:38:09 PST
Created attachment 183089 [details]
Patch
Comment 5 WebKit Review Bot 2013-01-16 18:48:22 PST
Attachment 183089 [details] did not pass style-queue:

Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'Tools/ChangeLog', u'Tools/EWSTools/GoogleC..." exit_code: 1
Tools/EWSTools/configure-git-user.sh:46:  Line contains tab character.  [whitespace/tab] [5]
Tools/EWSTools/configure-git-user.sh:47:  Line contains tab character.  [whitespace/tab] [5]
Total errors found: 2 in 4 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 6 Alan Cutter 2013-01-16 18:57:39 PST
(In reply to comment #5)
> Attachment 183089 [details] did not pass style-queue:
> 
> Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'Tools/ChangeLog', u'Tools/EWSTools/GoogleC..." exit_code: 1
> Tools/EWSTools/configure-git-user.sh:46:  Line contains tab character.  [whitespace/tab] [5]
> Tools/EWSTools/configure-git-user.sh:47:  Line contains tab character.  [whitespace/tab] [5]
> Total errors found: 2 in 4 files
> 
> 
> If any of these errors are false positives, please file a bug against check-webkit-style.

These style failures come from the git config file requiring tab characters instead of spaces. They are false positives.
Comment 7 Adam Barth 2013-01-16 19:34:00 PST
Comment on attachment 183089 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=183089&action=review

> Tools/EWSTools/build-repo.sh:45
> +git svn init --prefix=origin/ -T trunk http://svn.webkit.org/repository/webkit
> +git config --replace svn-remote.svn.fetch trunk:refs/remotes/origin/master
> +git svn fetch --local

It's not clear to me whether we want to configure svn on all the bots or just the ones that need to talk to svn.  Some of WebKit's other scripts behave differently when there's svn configured (e.g., they fetch data from the svn server).
Comment 8 Alan Cutter 2013-01-16 22:28:15 PST
Created attachment 183122 [details]
Patch
Comment 9 Alan Cutter 2013-01-16 22:32:20 PST
(In reply to comment #7)
> (From update of attachment 183089 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=183089&action=review
> 
> > Tools/EWSTools/build-repo.sh:45
> > +git svn init --prefix=origin/ -T trunk http://svn.webkit.org/repository/webkit
> > +git config --replace svn-remote.svn.fetch trunk:refs/remotes/origin/master
> > +git svn fetch --local
> 
> It's not clear to me whether we want to configure svn on all the bots or just the ones that need to talk to svn.  Some of WebKit's other scripts behave differently when there's svn configured (e.g., they fetch data from the svn server).

Split the git svn configuration commands into a separate script for Sheriffbot to use.
Comment 10 Adam Barth 2013-01-16 23:16:56 PST
Comment on attachment 183122 [details]
Patch

Thanks!
Comment 11 WebKit Review Bot 2013-01-16 23:19:13 PST
Comment on attachment 183122 [details]
Patch

Rejecting attachment 183122 [details] from commit-queue.

Failed to run "['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '-..." exit_code: 2 cwd: /mnt/git/webkit-commit-queue

Last 500 characters of output:
 with fuzz 1 (offset 3 lines).
1 out of 2 hunks FAILED -- saving rejects to file Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh.rej
patching file Tools/EWSTools/configure-git-svn.sh
patching file Tools/EWSTools/configure-git-user.sh
patching file Tools/EWSTools/create-webkit-git
rm 'Tools/EWSTools/create-webkit-git'

Failed to run "[u'/mnt/git/webkit-commit-queue/Tools/Scripts/svn-apply', '--force', '--reviewer', 'Adam Barth']" exit_code: 1 cwd: /mnt/git/webkit-commit-queue

Full output: http://queues.webkit.org/results/15910805
Comment 12 Eric Seidel (no email) 2013-01-16 23:22:06 PST
(In reply to comment #11)
> (From update of attachment 183122 [details])
> Rejecting attachment 183122 [details] from commit-queue.
> 
> Failed to run "['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '-..." exit_code: 2 cwd: /mnt/git/webkit-commit-queue
> 
> Last 500 characters of output:
>  with fuzz 1 (offset 3 lines).
> 1 out of 2 hunks FAILED -- saving rejects to file Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh.rej
> patching file Tools/EWSTools/configure-git-svn.sh
> patching file Tools/EWSTools/configure-git-user.sh
> patching file Tools/EWSTools/create-webkit-git
> rm 'Tools/EWSTools/create-webkit-git'
> 
> Failed to run "[u'/mnt/git/webkit-commit-queue/Tools/Scripts/svn-apply', '--force', '--reviewer', 'Adam Barth']" exit_code: 1 cwd: /mnt/git/webkit-commit-queue
> 
> Full output: http://queues.webkit.org/results/15910805

If you're looking for more infrastructure tasks, the CQ failure messages suck. :(  And they're certainly a highly visible part of the system.
Comment 13 Alan Cutter 2013-01-17 19:28:16 PST
(In reply to comment #12)
> (In reply to comment #11)
> > (From update of attachment 183122 [details] [details])
> > Rejecting attachment 183122 [details] [details] from commit-queue.
> > 
> > Failed to run "['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '-..." exit_code: 2 cwd: /mnt/git/webkit-commit-queue
> > 
> > Last 500 characters of output:
> >  with fuzz 1 (offset 3 lines).
> > 1 out of 2 hunks FAILED -- saving rejects to file Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh.rej
> > patching file Tools/EWSTools/configure-git-svn.sh
> > patching file Tools/EWSTools/configure-git-user.sh
> > patching file Tools/EWSTools/create-webkit-git
> > rm 'Tools/EWSTools/create-webkit-git'
> > 
> > Failed to run "[u'/mnt/git/webkit-commit-queue/Tools/Scripts/svn-apply', '--force', '--reviewer', 'Adam Barth']" exit_code: 1 cwd: /mnt/git/webkit-commit-queue
> > 
> > Full output: http://queues.webkit.org/results/15910805
> 
> If you're looking for more infrastructure tasks, the CQ failure messages suck. :(  And they're certainly a highly visible part of the system.

Haha definitely. Detecting these cases and hosting the .rej files would be ideal I think.

This would have told me what the problem was much faster:

> --- Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh
> +++ Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh
> @@ -1,5 +1,5 @@
>  #!/bin/sh
> -# Copyright (c) 2012 Google Inc. All rights reserved.
> +# Copyright (c) 2013 Google Inc. All rights reserved.
>  #
>  # Redistribution and use in source and binary forms, with or without
>  # modification, are permitted provided that the following conditions are
Comment 14 Alan Cutter 2013-01-17 19:32:36 PST
Created attachment 183352 [details]
Patch
Comment 15 WebKit Review Bot 2013-01-17 19:34:35 PST
Attachment 183352 [details] did not pass style-queue:

Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'Tools/ChangeLog', u'Tools/EWSTools/GoogleC..." exit_code: 1
Tools/EWSTools/configure-git-user.sh:46:  Line contains tab character.  [whitespace/tab] [5]
Tools/EWSTools/configure-git-user.sh:47:  Line contains tab character.  [whitespace/tab] [5]
Total errors found: 2 in 4 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 16 Alan Cutter 2013-02-18 01:21:57 PST
This issue cropped up again with gce-feeder-01 being reset, manually fixed the server git config.
The patch probably won't land anymore with the recent changes to the build scripts. Will need to repatch.
Comment 17 Alan Cutter 2013-02-18 01:40:20 PST
Created attachment 188822 [details]
Patch
Comment 18 WebKit Review Bot 2013-02-18 01:50:12 PST
Attachment 188822 [details] did not pass style-queue:

Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'Tools/ChangeLog', u'Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh', u'Tools/EWSTools/configure-git-svn.sh', u'Tools/EWSTools/configure-git-user.sh', u'Tools/EWSTools/create-webkit-git']" exit_code: 1
Tools/EWSTools/configure-git-user.sh:43:  Line contains tab character.  [whitespace/tab] [5]
Tools/EWSTools/configure-git-user.sh:44:  Line contains tab character.  [whitespace/tab] [5]
Total errors found: 2 in 4 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 19 Alan Cutter 2013-02-18 20:34:05 PST
(In reply to comment #18)
> Attachment 188822 [details] did not pass style-queue:
> 
> Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'Tools/ChangeLog', u'Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh', u'Tools/EWSTools/configure-git-svn.sh', u'Tools/EWSTools/configure-git-user.sh', u'Tools/EWSTools/create-webkit-git']" exit_code: 1
> Tools/EWSTools/configure-git-user.sh:43:  Line contains tab character.  [whitespace/tab] [5]
> Tools/EWSTools/configure-git-user.sh:44:  Line contains tab character.  [whitespace/tab] [5]
> Total errors found: 2 in 4 files
> 
> 
> If any of these errors are false positives, please file a bug against check-webkit-style.

These style failures can be ignored. The script writes to the .git/config file which uses tab characters for indentation.
Comment 20 WebKit Review Bot 2013-02-19 11:12:08 PST
Comment on attachment 188822 [details]
Patch

Clearing flags on attachment: 188822

Committed r143360: <http://trac.webkit.org/changeset/143360>
Comment 21 WebKit Review Bot 2013-02-19 11:12:14 PST
All reviewed patches have been landed.  Closing bug.