Bug 36193

Summary: git.webkit.org repository is missing svn revision r49890
Product: WebKit Reporter: Greg Bolsinga <bolsinga>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: RESOLVED FIXED    
Severity: Normal CC: abecsi, cjerdonek, ddkilzer, dpranke, gns, hamaji, mrowe, simon.fraser, wsiegrist
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: All   
OS: All   

Description Greg Bolsinga 2010-03-16 14:21:43 PDT
Here's my git config:
sonic:WebKit bolsinga$ cat .git/config 
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
	editor = ./WebKitTools/Scripts/commit-log-editor
	autocrlf = false
[remote "origin"]
	url = git://git.webkit.org/WebKit.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
	remote = origin
	merge = refs/heads/master
[svn-remote "svn"]
	url = http://svn.webkit.org/repository/webkit
	fetch = trunk:refs/remotes/trunk

When I git svn rebase, I see:
First, rewinding head to replay your work on top of it...
Applying: 2009-10-20  Mikhail Naganov  <mnaganov@chromium.org>
error: patch failed: WebCore/ChangeLog:1
error: WebCore/ChangeLog: patch does not apply
error: patch failed: WebCore/bindings/v8/V8Binding.h:61
error: WebCore/bindings/v8/V8Binding.h: patch does not apply
error: patch failed: WebCore/inspector/front-end/BottomUpProfileDataGridTree.js:29
error: WebCore/inspector/front-end/BottomUpProfileDataGridTree.js: patch does not apply
error: patch failed: WebCore/inspector/front-end/ProfileDataGridTree.js:126
error: WebCore/inspector/front-end/ProfileDataGridTree.js: patch does not apply
error: patch failed: WebCore/inspector/front-end/TopDownProfileDataGridTree.js:33
error: WebCore/inspector/front-end/TopDownProfileDataGridTree.js: patch does not apply
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging WebCore/ChangeLog
CONFLICT (content): Merge conflict in WebCore/ChangeLog
Auto-merging WebCore/bindings/v8/V8Binding.h
CONFLICT (content): Merge conflict in WebCore/bindings/v8/V8Binding.h
Auto-merging WebCore/inspector/front-end/BottomUpProfileDataGridTree.js
CONFLICT (content): Merge conflict in WebCore/inspector/front-end/BottomUpProfileDataGridTree.js
Auto-merging WebCore/inspector/front-end/ProfileDataGridTree.js
Auto-merging WebCore/inspector/front-end/TopDownProfileDataGridTree.js
CONFLICT (content): Merge conflict in WebCore/inspector/front-end/TopDownProfileDataGridTree.js
Failed to merge in the changes.
Patch failed at 0001 2009-10-20  Mikhail Naganov  <mnaganov@chromium.org>

When you have resolved this problem run "git rebase --continue".
If you would prefer to skip this patch, instead run "git rebase --skip".
To restore the original branch and stop rebasing run "git rebase --abort".

rebase refs/remotes/trunk: command returned error: 1

David Kilzer mentioned there was a missing svn commit. He said it was r49890. It is in remote/trunk, but not remotes/origin/master.

FWIW, the svn commit above is r50141.
Comment 1 David Kilzer (:ddkilzer) 2010-03-16 14:25:38 PDT
I will send a message to webkit-dev to ask how developers want to handle this issue.
Comment 2 Greg Bolsinga 2010-03-16 14:27:30 PDT
If I:

sonic:WebKit bolsinga$ git checkout -b NewMaster remotes/trunk
sonic:WebKit bolsinga$ git svn rebase

Things are good.
Comment 3 David Kilzer (:ddkilzer) 2010-03-16 14:35:54 PDT
I'm in favor of fixing git.webkit.org, even though it will cause issues when pulling from the repository again.  (Maybe we could move master to an old-master branch so the revisions are still there, but then fix master to include the missing revision again.)
Comment 4 David Kilzer (:ddkilzer) 2010-03-16 14:36:54 PDT
(In reply to comment #2)
> If I:
> 
> sonic:WebKit bolsinga$ git checkout -b NewMaster remotes/trunk
> sonic:WebKit bolsinga$ git svn rebase
> 
> Things are good.

This will break the next time you do a git-pull from git.webkit.org again.  (It will force a merge, I think, but I haven't tried it.)
Comment 5 Dirk Pranke 2010-03-16 16:07:47 PDT
+1 to fixing the repo; I'm not sure how having a git repo that is different from the svn repo would be acceptable?
Comment 6 Simon Fraser (smfr) 2010-03-16 17:05:52 PDT
We should fix the git repo.
Comment 7 Chris Jerdonek 2010-03-17 04:51:11 PDT
(In reply to comment #3)
> I'm in favor of fixing git.webkit.org, even though it will cause issues when
> pulling from the repository again.

I support fixing it even though I don't fully understand the down-side for doing so.  Will pulling for the first time after the fix be a time-intensive interruption for users (especially if we have many branches)?
Comment 8 Gustavo Noronha (kov) 2010-03-17 07:15:11 PDT
(In reply to comment #7)
> (In reply to comment #3)
> > I'm in favor of fixing git.webkit.org, even though it will cause issues when
> > pulling from the repository again.
> 
> I support fixing it even though I don't fully understand the down-side for
> doing so.  Will pulling for the first time after the fix be a time-intensive
> interruption for users (especially if we have many branches)?

The downside is that trees that use the git repository as a base will all be broken, and their current state will have to be recreated somehow. For instance, Debian uses the git repository to maintain webkitgtk packages, and we gtk+ porters maintain a git repository to track our "stable" branch.

It'll be a pain for me if this is done, but I guess we'll survive.
Comment 9 David Kilzer (:ddkilzer) 2010-03-17 07:59:48 PDT
(In reply to comment #8)
> The downside is that trees that use the git repository as a base will all be
> broken, and their current state will have to be recreated somehow. For
> instance, Debian uses the git repository to maintain webkitgtk packages, and we
> gtk+ porters maintain a git repository to track our "stable" branch.
> 
> It'll be a pain for me if this is done, but I guess we'll survive.

I'm planning on moving the current master to "broken-master", doing a hard reset of master back before the missing commit, then letting it catch up to svn trunk again.  The broken-master branch would not be updated after it was made.  I'll send a warning on IRC before making this change.

On the client side, you'd also want to create a local "broken-master" branch (maybe connecting it via remote tracking branch), do a hard reset of your master branch to before the missing commit, then re-pull from git.webkit.org.

After that comes the hard part of deciding whether to migrate existing branches from broken-master to master, or just leaving them as-is.  Please update this bug with any helpful techniques you find while doing this.
Comment 10 Chris Jerdonek 2010-03-17 10:56:14 PDT
(In reply to comment #9)
> I'm planning on moving the current master to "broken-master", doing a hard
> reset of master back before the missing commit, then letting it catch up to svn
> trunk again.  The broken-master branch would not be updated after it was made. 
> I'll send a warning on IRC before making this change.

Can you also comment on this report when it is done catching up?  Thanks.
Comment 11 David Kilzer (:ddkilzer) 2010-03-17 15:14:41 PDT
(In reply to comment #10)
> (In reply to comment #9)
> > I'm planning on moving the current master to "broken-master", doing a hard
> > reset of master back before the missing commit, then letting it catch up to svn
> > trunk again.  The broken-master branch would not be updated after it was made. 
> > I'll send a warning on IRC before making this change.
> 
> Can you also comment on this report when it is done catching up?  Thanks.

Sure.  Waiting to coordinate with Bill on the update.
Comment 12 David Kilzer (:ddkilzer) 2010-03-18 01:46:17 PDT
Bill and I will probably start working on this between 7 and 8 AM PDT (GMT-0700) today.

In the meantime, it appears git.webkit.org has developed another issue:

$ git pull origin && git svn fetch
remote: error: failed to unpack compressed delta at offset 167772124 from ./objects/pack/pack-48166dba7808af6c8cbc7b74dada2c456e2ba411.pack
remote: error: failed to read object be61563cc282b7c036d656ec05df6d5d66be4be7 at offset 167772121 from ./objects/pack/pack-48166dba7808af6c8cbc7b74dada2c456e2ba411.pack
remote: fatal: object be61563cc282b7c036d656ec05df6d5d66be4be7 is corrupted
remote: aborting due to possible repository corruption on the remote side.
fatal: protocol error: bad pack header

$ git show be61563cc282b7c036d656ec05df6d5d66be4be7
tree be61563cc282b7c036d656ec05df6d5d66be4be7

Mac-compatible-font-fallback.css
run-webkit-tests-epilogue.html
run-webkit-tests-prologue.html
Comment 13 David Kilzer (:ddkilzer) 2010-03-18 09:16:22 PDT
(In reply to comment #11)
> Sure.  Waiting to coordinate with Bill on the update.

We've started the process of fixing the history on the master.  When it's complete (in 1-2 hours), Bill will push a repaired git repository with a new master branch, and the old (broken) master branch renamed to "broken-master".  (He or I will post a comment here when that's done.)

If you have pulled from the broken master previously, you'll want to do the following:

1. Rename master to broken-master:
$ git branch -m master broken-master

2. Create a new master starting at r49889 (the commit before the missing r49890 commit):
$ git checkout -b master 81e1876cbf3bc239eca66e1a48367b9b277d66e1

3. Run your usual git-pull or git-fetch command to pull the new master branch down.

After that, it's up to you whether you want to rebase your local branches against the new master branch or leave them based on broken-master.

--

While we were working on this, we came up with a possible cause and fix for the occasional corruption that happens (see Comment #12) when the git.webkit.org repository is updated (via git-push using an svn post-commit hook).  Time will tell if we've fixed the issue.
Comment 14 Chris Jerdonek 2010-03-18 10:13:41 PDT
(In reply to comment #13)
> After that, it's up to you whether you want to rebase your local branches
> against the new master branch or leave them based on broken-master.

Thanks, David.  To be explicit, I assume the following command would work for each topic branch we want to rebase (assuming the difference between topic-branch and broken-master consists only of our local changes)?

> git rebase --onto master broken-master topic-branch
Comment 15 David Kilzer (:ddkilzer) 2010-03-18 11:29:40 PDT
(In reply to comment #14)
> (In reply to comment #13)
> > After that, it's up to you whether you want to rebase your local branches
> > against the new master branch or leave them based on broken-master.
> 
> Thanks, David.  To be explicit, I assume the following command would work for
> each topic branch we want to rebase (assuming the difference between
> topic-branch and broken-master consists only of our local changes)?
> 
> > git rebase --onto master broken-master topic-branch

The fixed repository is available on git.webkit.org.

I will test some of my local topic branches to see if the above command works as expected.
Comment 16 David Kilzer (:ddkilzer) 2010-03-18 11:36:43 PDT
(In reply to comment #15)
> (In reply to comment #14)
> > (In reply to comment #13)
> > > After that, it's up to you whether you want to rebase your local branches
> > > against the new master branch or leave them based on broken-master.
> > 
> > Thanks, David.  To be explicit, I assume the following command would work for
> > each topic branch we want to rebase (assuming the difference between
> > topic-branch and broken-master consists only of our local changes)?
> > 
> > > git rebase --onto master broken-master topic-branch
> 
> The fixed repository is available on git.webkit.org.

The corruption is still present.  Bill is going to repush the repository.
Comment 17 David Kilzer (:ddkilzer) 2010-03-18 12:18:09 PDT
(In reply to comment #13)
> If you have pulled from the broken master previously, you'll want to do the
> following:
> 
> 1. Rename master to broken-master:
> $ git branch -m master broken-master
> 
> 2. Create a new master starting at r49889 (the commit before the missing r49890
> commit):
> $ git checkout -b master 81e1876cbf3bc239eca66e1a48367b9b277d66e1

2.5.  Edit .git/config to change references to "broken-master" back to master, e.g.:

[branch "master"]
	remote = origin
	merge = refs/heads/master

> 3. Run your usual git-pull or git-fetch command to pull the new master branch
> down.

I ran "git pull origin" and it updated the master branch as expected.

More feedback to follow.
Comment 18 David Kilzer (:ddkilzer) 2010-03-18 12:26:56 PDT
(In reply to comment #14)
> Thanks, David.  To be explicit, I assume the following command would work for
> each topic branch we want to rebase (assuming the difference between
> topic-branch and broken-master consists only of our local changes)?
> 
> git rebase --onto master broken-master topic-branch

This is exactly how you want to merge topic branches from broken-master to master.  Works like a dream, especially with resolve-ChangeLogs as the merge driver.  :)
Comment 19 David Kilzer (:ddkilzer) 2010-03-18 12:55:38 PDT
Hopefully no one else did this, but I actually reset my svn history in my git repository so that it would match git.webkit.org.  (I did this by editing a ref file, deleting the git-svn rev map, then doing a git-svn-fetch to repopulate the rev map.)  In case you did that, too, here's how to undo it.

NOTE:  It may be quicker just to re-clone the git.webkit.org repository and follow the instructions here:  <http://trac.webkit.org/wiki/UsingGitWithWebKit#Checkout>

1. Look at your .git/config file to determine where the tracking information is stored for the svn repository:

[svn-remote "svn"]
	url = http://svn.webkit.org/repository/webkit
	fetch = trunk:refs/remotes/trunk

2. Remove the svn tracking directory (git will rebuild this on the next svn fetch).  Mine is "trunk" because that's what it's called in the svn-remote section above:

$ rm -rf .git/svn/trunk

3. If .git/refs/remotes/trunk exists, edit it and change its head commit back to r49889 (81e1876cbf3bc239eca66e1a48367b9b277d66e1).

If .git/info/refs exist, edit it there instead.

OPTIONAL: Instead of using the commitish for r49889, you could use the latest commitish on master.

4. Run "git svn fetch" to recreate the rev map and update back to trunk again.

NOTE: I edited other entries in .git/info/refs, created .git/refs/remotes/trunk (pasted the commitish at the head of master), and edited .git/refs/remotes/origin/HEAD and .git/refs/remotes/origin/master to contain master's latest commitish.  Not sure if all of that was necessary, but it didn't hurt.
Comment 20 David Kilzer (:ddkilzer) 2010-03-18 12:56:47 PDT
Marking this as RESOLVED/FIXED, but please continue to add comments as necessary.
Comment 21 Chris Jerdonek 2010-03-19 06:41:09 PDT
Here is a bit of info from my experience.

To see whether your branch has the commit for r49890, you can use any of the following techniques:

git log --before=2009-10-21 (and page down)
git log 3d3c0aff8a1d6bb6aca0dbe273d0559f870a9886 (everything <= 49891)
git log --grep="49890" (just the commit itself)

To see if your git-svn history has it, you can use:

git svn log -r 49889:49891

I ran into the following unusual behavior after running "git pull origin" in comment 17.

For some reason, every git-svn command became extremely slow for me.  The git-svn commands I was trying would give no output and then finish after on the order of 5 minutes.  Even something as simple as git-svn-log would take this long.  The process would consume around 40% of my machine's CPU during this time.  This made it difficult to figure out what was wrong!

My git-svn history was missing r49890 during this time.  This is probably because I set up my git checkout for the first time back in January, using the instructions here:

http://trac.webkit.org/wiki/UsingGitWithWebKit#Checkout

(January 2010 was after the missing commit in October 2009.)  Those suggestions say to "re-use the history that we've already cloned from git.webkit.org [which must have been missing the commit] rather than trying to fetch it all from Subversion" and then to use the SVN repository for subsequent commits.

Maybe the slowness was because git-svn was trying to compute the entire SVN history on the fly every time or something because it was out of synch.

Running "rm -rf .git/svn" did not speed it up.  What finally sped it up was running--

git svn reset -r 49889

This command may be a substitute for the manual edits suggested in comment 19.  Then, after running "git svn fetch" (which took quite a while), I was back to normal.
Comment 22 Gustavo Noronha (kov) 2010-03-19 07:10:10 PDT
I think it's important to send email to webkit-dev telling people what they should do, as reading through all comments here will be error prone =).
Comment 23 David Kilzer (:ddkilzer) 2010-03-19 20:12:11 PDT
(In reply to comment #22)
> I think it's important to send email to webkit-dev telling people what they
> should do, as reading through all comments here will be error prone =).

The wiki would be the best place to put the directions, but I don't have a feel for how many people are still having issues.  I don't want to spend time writing up instructions for a one-time fix if no one is going to use them.