Bug 243283 – [git-webkit] Update fork via API instead of local push

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Simon Fraser (smfr)]

LayoutTests % git webkit pr --no-update --issue 237812
Switched to a new branch 'eng/iOS-editingspellingediting-word-with-marker-1-html-causes-the-subsequent-test-to-time-out'
Created the local development branch 'eng/iOS-editingspellingediting-word-with-marker-1-html-causes-the-subsequent-test-to-time-out'
Branch 'main' set up to track remote branch 'main' from 'origin'.
  Running status to find changed, added, or removed files.
  Reviewing diff to determine which lines changed.
  Extracting affected function names from source files.
  Change author: Simon Fraser <[e-mail]>.
[eng/iOS-editingspellingediting-word-with-marker-1-html-causes-the-subsequent-test-to-time-out c1db7877e96e] [iOS] editing/spelling/editing-word-with-marker-1.html causes the subsequent test to time out https://bugs.webkit.org/show_bug.cgi?id=237812 <rdar://90529903>
 Date: Wed Jul 27 21:20:25 2022 -0700
 4 files changed, 26 insertions(+), 15 deletions(-)
 create mode 100644 LayoutTests/platform/ios-wk2/editing/spelling/editing-word-with-marker-1-expected.txt
    Found 1 commit...
Total errors found: 0 in 2 files
Enumerating objects: 3336, done.
Counting objects: 100% (3336/3336), done.
Delta compression using up to 20 threads
Compressing objects: 100% (1577/1577), done.
Writing objects: 100% (2672/2672), 6.47 MiB | 791.00 KiB/s, done.
Total 2672 (delta 1725), reused 1855 (delta 1025), pack-reused 0
[no more output]

Can't tell if this is a GitHub outage or something else.

[Simon Fraser (smfr)]

Interrupting gave:
^CTraceback (most recent call last):
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1764, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Volumes/Data/Development/system/webkit/Internal/Tools/Scripts/git-webkit", line 89, in <module>
    sys.exit(program.main(
  File "/Volumes/WebKit/OpenSource/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/__init__.py", line 148, in main
    return parsed.main(
  File "/Volumes/WebKit/OpenSource/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/pull_request.py", line 507, in main
    return cls.create_pull_request(repository, args, branch_point)
  File "/Volumes/WebKit/OpenSource/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/pull_request.py", line 382, in create_pull_request
    if run([repository.executable(), 'push', '-f', target, repository.branch], cwd=repository.root_path).returncode:
  File "/Volumes/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/subprocess_utils.py", line 51, in run
    return subprocess.run(*popenargs, timeout=timeout, **kwargs)
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 495, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1020, in communicate
    self.wait()
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1083, in wait
    return self._wait(timeout=timeout)
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1806, in _wait
    (pid, sts) = self._try_wait(0)
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1764, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt

[Simon Fraser (smfr)]

Reapting the same command gives:

LayoutTests % git webkit pr --no-update --issue 237812
Creating a pull-request for '237812' but we're on 'eng/iOS-editingspellingediting-word-with-marker-1-html-causes-the-subsequent-test-to-time-out'

[Simon Fraser (smfr)]

With verbose logging:

INFO:webkitscmpy:Pushing 'eng/iOS-editingspellingediting-word-with-marker-1-html-causes-the-subsequent-test-to-time-out' to 'fork'...
Enumerating objects: 3350, done.
Counting objects: 100% (3350/3350), done.
Delta compression using up to 20 threads
Compressing objects: 100% (1586/1586), done.
Writing objects: 100% (2681/2681), 6.47 MiB | 960.00 KiB/s, done.
Total 2681 (delta 1732), reused 1855 (delta 1025), pack-reused 0

[Simon Fraser (smfr)]

LayoutTests % GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push fork -f -v

[lots of logging)
21:55:46.877281 http.c:703              <= Recv header, 0000000023 bytes (0x00000017)
21:55:46.877284 http.c:715              <= Recv header: x-frame-options: DENY
21:55:46.877286 http.c:703              <= Recv header, 0000000055 bytes (0x00000037)
21:55:46.877288 http.c:715              <= Recv header: x-github-request-id: C0FF:9D5A:824128:997753:62E216D2
21:55:46.877291 http.c:703              <= Recv header, 0000000002 bytes (0x00000002)
21:55:46.877293 http.c:715              <= Recv header:
Writing objects: 100% (2672/2672), 6.47 MiB | 972.00 KiB/s, done.
Total 2672 (delta 1725), reused 1855 (delta 1025), pack-reused 0

and nothing more.

[Simon Fraser (smfr)]

I did a `git fetch fork` and it magically resolved this.

[Radar WebKit Bug Importer]

<rdar://problem/97741202>

[Simon Fraser (smfr)]

Still seeing this; seems worse when my fork is more out of date.

[Jonathan Bedard]

I'm going to use this bug to track removing of what I believe to be the offending push in most workflows.

What I believe is going on here is that we're attempting to update `main` in your personal fork by push hundreds or thousands of commits from your local checkout to your personal fork. According to GitHub's documentation, this is the correct way to update a local fork with raw `git` tools. Unsurprisingly, though, this is a slow operation. To make things worse, this operation gets slower as we add more protections in our pre-push hook. Pushing commits costs both local compute and network traffic which scales with the number of commits behind a fork is.

However, we don't actually need to push _any_ pre-existing commits in principle, because GitHub already has all these commits server-side because the pre-existing commits are originating from `WebKit/WebKit`. We should be able to invoke https://docs.github.com/en/rest/branches/branches#sync-a-fork-branch-with-the-upstream-repository before we do any pushes, then the only commits we're actually pushing are the ones a contributor has created locally.

[Jonathan Bedard]

Pull request: https://github.com/WebKit/WebKit/pull/12978

[EWS]

Committed 263251@main (556bf5c84d3e): <https://commits.webkit.org/263251@main>

Reviewed commits have been landed. Closing PR #12978 and removing active labels.