Bug 106945 – REGRESSION: Sheriffbot is failing to process rollout requests

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Levi Weintraub]

At least since Friday (when I started sheriffing), sheriffbot responds to all rollout requests with a message like this:
sheriffbot: Exception executing command: Failed to run "['git', 'svn', 'find-rev', 'r139779']" exit_code: 1 cwd: /mnt/git/webkit-sheriff-bot

The problem persists through restarts.

[Eric Seidel (no email)]

I assume that webkit-patch prepare-rollout works fine?  That's the command SB runs under the covers?

[Levi Weintraub]

Dunno? Haven't tried it!

[Alan Cutter]

This failure is caused by the migration of Sheriffbot to Google Compute Engine.

The git repository was not properly set up with SVN (manually fixed now on gce-feeder-01) however this is not the only issue.

Another problem that occurs when sheriffbot tries to do a rollout on gce-feeder-01 is this error:

> Running: webkit-patch --status-host=queues.webkit.org --bot-id=gce-feeder-01 create-rollout --force-clean --parent-command=sheriff-bot 139575 Rollout test (Requested by alancutter on #webkit).
> Failed to run "['/mnt/git/webkit-sheriff-bot/Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '--..." exit_code: 2 cwd: /mnt/git/webkit-sheriff-bot
> 
> Preparing rollout for bug 106747.
> Updating working directory
> Logging in as [e-mail]...
> Creating bug with title "REGRESSION(r139575): Rollout test (Requested by alancutter on #webkit)."
> Bug 106982 created.
> https://bugs.webkit.org/show_bug.cgi?id=106982
> Fetching: https://bugs.webkit.org/show_bug.cgi?id=106747&ctype=xml&excludefield=attachmentdata
> Re-opening bug 106747
> Re-opened since this is blocked by bug 106982
> Failed to run "['Tools/Scripts/prepare-ChangeLog']" exit_code: 1
> 
>   Running status to find changed, added, or removed files.
>   Reviewing diff to determine which lines changed.
>   Extracting affected function names from source files.
> Failed to determine ChangeLog name.
> Either:
>   set CHANGE_LOG_NAME in your environment
>   OR pass --name= on the command line
>   OR set REAL_NAME in your environment  OR git users can set 'git config user.name'
>                              
>   Failed to run "['Tools/Scripts/prepare-ChangeLog']" exit_code: 1
>   Updating OpenSource
>   Current branch master is up to date.
>     Running status to find changed, added, or removed files.
>     Reviewing diff to determine which lines changed.
>     Extracting affected function names from source files.
>   Failed to determine ChangeLog name.
>   Either:
>     set CHANGE_LOG_NAME in your environment
>     OR pass --name= on the command line
>     OR set REAL_NAME in your environment  OR git users can set 'git config user.name'
> 
>   No work item. Sleeping until 2013-01-16 07:20:26 (2 mins).
> 

Tools/Scripts/prepare-ChangeLog fails to execute (stand alone) when there are changes present in the repository on gce-feeder-01 because the git config doesn't contain a username or email address for Sheriffbot to create a ChangeLog entry.

These issues have been fixed on gce-feeder-01 and I've tested that Sheriffbot can perform rollouts again.

Patch to update the GCE build scripts to resolve these issues in the GCE build scripts coming (after I get some sleep). :)

[Alan Cutter]

Created attachment 183089 [details]
Patch

[WebKit Review Bot]

Attachment 183089 [details] did not pass style-queue:

Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'Tools/ChangeLog', u'Tools/EWSTools/GoogleC..." exit_code: 1
Tools/EWSTools/configure-git-user.sh:46:  Line contains tab character.  [whitespace/tab] [5]
Tools/EWSTools/configure-git-user.sh:47:  Line contains tab character.  [whitespace/tab] [5]
Total errors found: 2 in 4 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Alan Cutter]

(In reply to comment #5)
> Attachment 183089 [details] did not pass style-queue:
> 
> Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'Tools/ChangeLog', u'Tools/EWSTools/GoogleC..." exit_code: 1
> Tools/EWSTools/configure-git-user.sh:46:  Line contains tab character.  [whitespace/tab] [5]
> Tools/EWSTools/configure-git-user.sh:47:  Line contains tab character.  [whitespace/tab] [5]
> Total errors found: 2 in 4 files
> 
> 
> If any of these errors are false positives, please file a bug against check-webkit-style.

These style failures come from the git config file requiring tab characters instead of spaces. They are false positives.

[Adam Barth]

Comment on attachment 183089 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=183089&action=review

> Tools/EWSTools/build-repo.sh:45
> +git svn init --prefix=origin/ -T trunk http://svn.webkit.org/repository/webkit
> +git config --replace svn-remote.svn.fetch trunk:refs/remotes/origin/master
> +git svn fetch --local

It's not clear to me whether we want to configure svn on all the bots or just the ones that need to talk to svn.  Some of WebKit's other scripts behave differently when there's svn configured (e.g., they fetch data from the svn server).

[Alan Cutter]

Created attachment 183122 [details]
Patch

[Alan Cutter]

(In reply to comment #7)
> (From update of attachment 183089 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=183089&action=review
> 
> > Tools/EWSTools/build-repo.sh:45
> > +git svn init --prefix=origin/ -T trunk http://svn.webkit.org/repository/webkit
> > +git config --replace svn-remote.svn.fetch trunk:refs/remotes/origin/master
> > +git svn fetch --local
> 
> It's not clear to me whether we want to configure svn on all the bots or just the ones that need to talk to svn.  Some of WebKit's other scripts behave differently when there's svn configured (e.g., they fetch data from the svn server).

Split the git svn configuration commands into a separate script for Sheriffbot to use.

[Adam Barth]

Comment on attachment 183122 [details]
Patch

Thanks!

[WebKit Review Bot]

Comment on attachment 183122 [details]
Patch

Rejecting attachment 183122 [details] from commit-queue.

Failed to run "['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '-..." exit_code: 2 cwd: /mnt/git/webkit-commit-queue

Last 500 characters of output:
 with fuzz 1 (offset 3 lines).
1 out of 2 hunks FAILED -- saving rejects to file Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh.rej
patching file Tools/EWSTools/configure-git-svn.sh
patching file Tools/EWSTools/configure-git-user.sh
patching file Tools/EWSTools/create-webkit-git
rm 'Tools/EWSTools/create-webkit-git'

Failed to run "[u'/mnt/git/webkit-commit-queue/Tools/Scripts/svn-apply', '--force', '--reviewer', 'Adam Barth']" exit_code: 1 cwd: /mnt/git/webkit-commit-queue

Full output: http://queues.webkit.org/results/15910805

[Eric Seidel (no email)]

(In reply to comment #11)
> (From update of attachment 183122 [details])
> Rejecting attachment 183122 [details] from commit-queue.
> 
> Failed to run "['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '-..." exit_code: 2 cwd: /mnt/git/webkit-commit-queue
> 
> Last 500 characters of output:
>  with fuzz 1 (offset 3 lines).
> 1 out of 2 hunks FAILED -- saving rejects to file Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh.rej
> patching file Tools/EWSTools/configure-git-svn.sh
> patching file Tools/EWSTools/configure-git-user.sh
> patching file Tools/EWSTools/create-webkit-git
> rm 'Tools/EWSTools/create-webkit-git'
> 
> Failed to run "[u'/mnt/git/webkit-commit-queue/Tools/Scripts/svn-apply', '--force', '--reviewer', 'Adam Barth']" exit_code: 1 cwd: /mnt/git/webkit-commit-queue
> 
> Full output: http://queues.webkit.org/results/15910805

If you're looking for more infrastructure tasks, the CQ failure messages suck. :(  And they're certainly a highly visible part of the system.

[Alan Cutter]

(In reply to comment #12)
> (In reply to comment #11)
> > (From update of attachment 183122 [details] [details])
> > Rejecting attachment 183122 [details] [details] from commit-queue.
> > 
> > Failed to run "['/mnt/git/webkit-commit-queue/Tools/Scripts/webkit-patch', '--status-host=queues.webkit.org', '-..." exit_code: 2 cwd: /mnt/git/webkit-commit-queue
> > 
> > Last 500 characters of output:
> >  with fuzz 1 (offset 3 lines).
> > 1 out of 2 hunks FAILED -- saving rejects to file Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh.rej
> > patching file Tools/EWSTools/configure-git-svn.sh
> > patching file Tools/EWSTools/configure-git-user.sh
> > patching file Tools/EWSTools/create-webkit-git
> > rm 'Tools/EWSTools/create-webkit-git'
> > 
> > Failed to run "[u'/mnt/git/webkit-commit-queue/Tools/Scripts/svn-apply', '--force', '--reviewer', 'Adam Barth']" exit_code: 1 cwd: /mnt/git/webkit-commit-queue
> > 
> > Full output: http://queues.webkit.org/results/15910805
> 
> If you're looking for more infrastructure tasks, the CQ failure messages suck. :(  And they're certainly a highly visible part of the system.

Haha definitely. Detecting these cases and hosting the .rej files would be ideal I think.

This would have told me what the problem was much faster:

> --- Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh
> +++ Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh
> @@ -1,5 +1,5 @@
>  #!/bin/sh
> -# Copyright (c) 2012 Google Inc. All rights reserved.
> +# Copyright (c) 2013 Google Inc. All rights reserved.
>  #
>  # Redistribution and use in source and binary forms, with or without
>  # modification, are permitted provided that the following conditions are

[Alan Cutter]

Created attachment 183352 [details]
Patch

[WebKit Review Bot]

Attachment 183352 [details] did not pass style-queue:

Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'Tools/ChangeLog', u'Tools/EWSTools/GoogleC..." exit_code: 1
Tools/EWSTools/configure-git-user.sh:46:  Line contains tab character.  [whitespace/tab] [5]
Tools/EWSTools/configure-git-user.sh:47:  Line contains tab character.  [whitespace/tab] [5]
Total errors found: 2 in 4 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Alan Cutter]

This issue cropped up again with gce-feeder-01 being reset, manually fixed the server git config.
The patch probably won't land anymore with the recent changes to the build scripts. Will need to repatch.

[Alan Cutter]

Created attachment 188822 [details]
Patch

[WebKit Review Bot]

Attachment 188822 [details] did not pass style-queue:

Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'Tools/ChangeLog', u'Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh', u'Tools/EWSTools/configure-git-svn.sh', u'Tools/EWSTools/configure-git-user.sh', u'Tools/EWSTools/create-webkit-git']" exit_code: 1
Tools/EWSTools/configure-git-user.sh:43:  Line contains tab character.  [whitespace/tab] [5]
Tools/EWSTools/configure-git-user.sh:44:  Line contains tab character.  [whitespace/tab] [5]
Total errors found: 2 in 4 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Alan Cutter]

(In reply to comment #18)
> Attachment 188822 [details] did not pass style-queue:
> 
> Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'Tools/ChangeLog', u'Tools/EWSTools/GoogleComputeEngine/build-feeder-style-sheriffbot.sh', u'Tools/EWSTools/configure-git-svn.sh', u'Tools/EWSTools/configure-git-user.sh', u'Tools/EWSTools/create-webkit-git']" exit_code: 1
> Tools/EWSTools/configure-git-user.sh:43:  Line contains tab character.  [whitespace/tab] [5]
> Tools/EWSTools/configure-git-user.sh:44:  Line contains tab character.  [whitespace/tab] [5]
> Total errors found: 2 in 4 files
> 
> 
> If any of these errors are false positives, please file a bug against check-webkit-style.

These style failures can be ignored. The script writes to the .git/config file which uses tab characters for indentation.

[WebKit Review Bot]

Comment on attachment 188822 [details]
Patch

Clearing flags on attachment: 188822

Committed r143360: <http://trac.webkit.org/changeset/143360>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.