Bug 243283 - [git-webkit] Update fork via API instead of local push
Summary: [git-webkit] Update fork via API instead of local push
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: Safari Technology Preview
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Jonathan Bedard
URL:
Keywords: InRadar
Depends on:
Blocks: 239082
  Show dependency treegraph
 
Reported: 2022-07-27 21:28 PDT by Simon Fraser (smfr)
Modified: 2023-04-26 11:00 PDT (History)
6 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Simon Fraser (smfr) 2022-07-27 21:28:48 PDT
LayoutTests % git webkit pr --no-update --issue 237812
Switched to a new branch 'eng/iOS-editingspellingediting-word-with-marker-1-html-causes-the-subsequent-test-to-time-out'
Created the local development branch 'eng/iOS-editingspellingediting-word-with-marker-1-html-causes-the-subsequent-test-to-time-out'
Branch 'main' set up to track remote branch 'main' from 'origin'.
  Running status to find changed, added, or removed files.
  Reviewing diff to determine which lines changed.
  Extracting affected function names from source files.
  Change author: Simon Fraser <simon.fraser@apple.com>.
[eng/iOS-editingspellingediting-word-with-marker-1-html-causes-the-subsequent-test-to-time-out c1db7877e96e] [iOS] editing/spelling/editing-word-with-marker-1.html causes the subsequent test to time out https://bugs.webkit.org/show_bug.cgi?id=237812 <rdar://90529903>
 Date: Wed Jul 27 21:20:25 2022 -0700
 4 files changed, 26 insertions(+), 15 deletions(-)
 create mode 100644 LayoutTests/platform/ios-wk2/editing/spelling/editing-word-with-marker-1-expected.txt
    Found 1 commit...
Total errors found: 0 in 2 files
Enumerating objects: 3336, done.
Counting objects: 100% (3336/3336), done.
Delta compression using up to 20 threads
Compressing objects: 100% (1577/1577), done.
Writing objects: 100% (2672/2672), 6.47 MiB | 791.00 KiB/s, done.
Total 2672 (delta 1725), reused 1855 (delta 1025), pack-reused 0
[no more output]

Can't tell if this is a GitHub outage or something else.
Comment 1 Simon Fraser (smfr) 2022-07-27 21:31:45 PDT
Interrupting gave:
^CTraceback (most recent call last):
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1764, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Volumes/Data/Development/system/webkit/Internal/Tools/Scripts/git-webkit", line 89, in <module>
    sys.exit(program.main(
  File "/Volumes/WebKit/OpenSource/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/__init__.py", line 148, in main
    return parsed.main(
  File "/Volumes/WebKit/OpenSource/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/pull_request.py", line 507, in main
    return cls.create_pull_request(repository, args, branch_point)
  File "/Volumes/WebKit/OpenSource/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/pull_request.py", line 382, in create_pull_request
    if run([repository.executable(), 'push', '-f', target, repository.branch], cwd=repository.root_path).returncode:
  File "/Volumes/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/subprocess_utils.py", line 51, in run
    return subprocess.run(*popenargs, timeout=timeout, **kwargs)
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 495, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1020, in communicate
    self.wait()
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1083, in wait
    return self._wait(timeout=timeout)
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1806, in _wait
    (pid, sts) = self._try_wait(0)
  File "/AppleInternal/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1764, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt
Comment 2 Simon Fraser (smfr) 2022-07-27 21:32:41 PDT
Reapting the same command gives:

LayoutTests % git webkit pr --no-update --issue 237812
Creating a pull-request for '237812' but we're on 'eng/iOS-editingspellingediting-word-with-marker-1-html-causes-the-subsequent-test-to-time-out'
Comment 3 Simon Fraser (smfr) 2022-07-27 21:44:52 PDT
With verbose logging:

INFO:webkitscmpy:Pushing 'eng/iOS-editingspellingediting-word-with-marker-1-html-causes-the-subsequent-test-to-time-out' to 'fork'...
Enumerating objects: 3350, done.
Counting objects: 100% (3350/3350), done.
Delta compression using up to 20 threads
Compressing objects: 100% (1586/1586), done.
Writing objects: 100% (2681/2681), 6.47 MiB | 960.00 KiB/s, done.
Total 2681 (delta 1732), reused 1855 (delta 1025), pack-reused 0
Comment 4 Simon Fraser (smfr) 2022-07-27 21:57:27 PDT
LayoutTests % GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push fork -f -v

[lots of logging)
21:55:46.877281 http.c:703              <= Recv header, 0000000023 bytes (0x00000017)
21:55:46.877284 http.c:715              <= Recv header: x-frame-options: DENY
21:55:46.877286 http.c:703              <= Recv header, 0000000055 bytes (0x00000037)
21:55:46.877288 http.c:715              <= Recv header: x-github-request-id: C0FF:9D5A:824128:997753:62E216D2
21:55:46.877291 http.c:703              <= Recv header, 0000000002 bytes (0x00000002)
21:55:46.877293 http.c:715              <= Recv header:
Writing objects: 100% (2672/2672), 6.47 MiB | 972.00 KiB/s, done.
Total 2672 (delta 1725), reused 1855 (delta 1025), pack-reused 0

and nothing more.
Comment 5 Simon Fraser (smfr) 2022-07-27 22:15:41 PDT
I did a `git fetch fork` and it magically resolved this.
Comment 6 Radar WebKit Bug Importer 2022-07-28 13:30:20 PDT
<rdar://problem/97741202>
Comment 7 Simon Fraser (smfr) 2022-10-11 21:32:00 PDT
Still seeing this; seems worse when my fork is more out of date.
Comment 8 Jonathan Bedard 2023-04-20 09:03:59 PDT
I'm going to use this bug to track removing of what I believe to be the offending push in most workflows.

What I believe is going on here is that we're attempting to update `main` in your personal fork by push hundreds or thousands of commits from your local checkout to your personal fork. According to GitHub's documentation, this is the correct way to update a local fork with raw `git` tools. Unsurprisingly, though, this is a slow operation. To make things worse, this operation gets slower as we add more protections in our pre-push hook. Pushing commits costs both local compute and network traffic which scales with the number of commits behind a fork is.

However, we don't actually need to push _any_ pre-existing commits in principle, because GitHub already has all these commits server-side because the pre-existing commits are originating from `WebKit/WebKit`. We should be able to invoke https://docs.github.com/en/rest/branches/branches#sync-a-fork-branch-with-the-upstream-repository before we do any pushes, then the only commits we're actually pushing are the ones a contributor has created locally.
Comment 9 Jonathan Bedard 2023-04-20 10:16:13 PDT
Pull request: https://github.com/WebKit/WebKit/pull/12978
Comment 10 EWS 2023-04-21 11:07:21 PDT
Committed 263251@main (556bf5c84d3e): <https://commits.webkit.org/263251@main>

Reviewed commits have been landed. Closing PR #12978 and removing active labels.