flatpak-builder fails to clone recursive repository behind proxy > $ python3 ./Tools/Scripts/update-webkitgtk-flatpak > Updating Flatpak environment for GTK (Release) > Looking for updates... > Looking for updates... > Building org.webkit.GTK and dependencies in /home/fujii/work/webkit/ga/WebKitBuild/GTK/FlatpakTreeRelease > Downloading sources > Fetching git repo https://github.com/apache/httpd.git, ref refs/tags/2.4.33 > remote: Counting objects: 1, done. > remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0 > Unpacking objects: 100% (1/1), done. > Fetching git repo https://github.com/libevent/libevent.git, ref refs/tags/release-2.1.8-stable > remote: Counting objects: 1, done. > remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0 > Unpacking objects: 100% (1/1), done. > Fetching git repo https://chromium.googlesource.com/webm/libvpx, ref refs/tags/v1.7.0 > remote: Counting objects: 1, done > remote: Finding sources: 100% (1/1) > remote: Total 1 (delta 0), reused 1 (delta 0) > Unpacking objects: 100% (1/1), done. > fatal: unable to access 'https://anongit.freedesktop.org/git/gstreamer/gstreamer/': Received HTTP code 503 from proxy after CONNECT > Failed to download sources: module gstreamer: Child process exited with code 128 > Traceback (most recent call last): > File "./Tools/Scripts/update-webkitgtk-flatpak", line 28, in <module> > WebkitFlatpak.load_from_args(["--gtk", "--update"] + sys.argv[1:]).run() > File "./Tools/flatpak/flatpakutils.py", line 711, in run > return self.setup_dev_env() > File "./Tools/flatpak/flatpakutils.py", line 738, in setup_dev_env > subprocess.check_call(builder_args) > File "/usr/lib/python3.6/subprocess.py", line 291, in check_call > raise CalledProcessError(retcode, cmd) > subprocess.CalledProcessError: Command '['flatpak-builder', '--disable-rofiles-fuse', '--state-dir', '/home/fujii/work/webkit/ga/WebKitBuild/FlatpakCache', '--ccache', '/home/fujii/work/webkit/ga/WebKitBuild/GTK/FlatpakTreeRelease', '--force-clean', '/home/fujii/work/webkit/ga/WebKitBuild/FlatpakCache/org.webkit.GTK-generated.json', '--build-only', '--stop-at=org.webkit.GTK']' returned non-zero exit status 1. > $ Gstreamer and Gstreamer plugins have 'common' as a submodule.
I was wrong. This is not a problm flatpak-builder. Cloning repositories from https://anongit.freedesktop.org/ is unstable.
I think we should rely on tarballs whenever possible. Cloning over git takes more time and is less reliable and less proxy-friendly.
Actually, flatpak-builder does shallow clones, so they should be fast, and Thibault is using the https:// protocol here. (I agree that we should not use git:// anywhere.) So I think git is fine. Anyway, tarballs are fine too.
(In reply to Philippe Normand from comment #2) > I think we should rely on tarballs whenever possible. Cloning over git takes > more time and is less reliable and less proxy-friendly. As Michael said, we are doing shallow clones here, so the size should be very similar. Using git repos makes it simpler to update (only change the tag and you are done) so I am using git on purpose. (In reply to Fujii Hironori from comment #1) > I was wrong. This is not a problm flatpak-builder. > Cloning repositories from https://anongit.freedesktop.org/ is unstable. Let's close the bug then? I hope freedesktop get back to normal functioning but it is a bit out of ends.
(In reply to Thibault Saunier from comment #4) > Let's close the bug then? I hope freedesktop get back to normal functioning > but it is a bit out of ends. No. If the code hosting is unstable, then we can't depend on it. Surely the next step for the flatpak support is to work with Adrian to switch the bots from using jhbuild to flatpak. And then we will finally have consistent test results, can drop the jhbuild support, and can reenable tests on our EWS. None of that will be possible if the servers we rely on are unreliable. So if there are still problems with anongit.freedesktop.org, I would consider tarballs indeed. Or mirroring the git repo elsewhere. For GNOME, I have to host a few tarballs myself on people.gnome.org, since it's required for builds to be reliable. We can be happy that we probably won't need to do that.
(In reply to Michael Catanzaro from comment #5) > (In reply to Thibault Saunier from comment #4) > > Let's close the bug then? I hope freedesktop get back to normal functioning > > but it is a bit out of ends. > > No. If the code hosting is unstable, then we can't depend on it. Well the tarball are on the same server, so I would expect more or less the same instabilities, it really doesn't happen often. > Surely the next step for the flatpak support is to work with Adrian to > switch the bots from using jhbuild to flatpak. And then we will finally have > consistent test results, can drop the jhbuild support, and can reenable > tests on our EWS. None of that will be possible if the servers we rely on > are unreliable. > > So if there are still problems with anongit.freedesktop.org, I would > consider tarballs indeed. Or mirroring the git repo elsewhere. We could use the official github mirror too. I hope and expect the gitlab server to work properly once we move to it also (should happen soon I believe).
I tried update-webkitgtk-flatpak today, but failed with the same error again. I tried to invoke "git ls-remote https://anongit.freedesktop.org/git/gstreamer/common.git" 10 times. > $ for i in $(seq 10); do echo -n "try $i: "; git ls-remote https://anongit.freedesktop.org/git/gstreamer/common.git | wc;done > try 1: 258 516 18047 > try 2: 258 516 18047 > try 3: 258 516 18047 > try 4: fatal: unable to access 'https://anongit.freedesktop.org/git/gstreamer/common.git/': Received HTTP code 503 from proxy after CONNECT > 0 0 0 > try 5: 258 516 18047 > try 6: fatal: unable to access 'https://anongit.freedesktop.org/git/gstreamer/common.git/': Received HTTP code 503 from proxy after CONNECT > 0 0 0 > try 7: 258 516 18047 > try 8: 258 516 18047 > try 9: 258 516 18047 > try 10: 258 516 18047 > $ Yes, I'm using a proxy. This seems a problem of the proxy if other doesn't see this.
Umm, I guess the server randomly refuses connections. > $ curl --head --silent https://anongit.freedesktop.org/git/gstreamer/common.git/ > HTTP/1.1 503 Service Unavailable > Cache-Control: no-cache > Pragma: no-cache > Content-Type: text/html; charset=utf-8 > Proxy-Connection: close > Connection: close > Content-Length: 787 > > $
> $ curl --head --silent https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-1.14.1.tar.xz > HTTP/1.1 503 Service Unavailable > Cache-Control: no-cache > Pragma: no-cache > Content-Type: text/html; charset=utf-8 > Proxy-Connection: close > Connection: close > Content-Length: 1274 > > $ Downloading tarballs also fails sometimes. But, this hasn't been a problem since jhbuild caches tarballs in ~/.cache/jhbuild/downloads/.
(In reply to Fujii Hironori from comment #9) > > $ curl --head --silent https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-1.14.1.tar.xz > > HTTP/1.1 503 Service Unavailable > > Cache-Control: no-cache > > Pragma: no-cache > > Content-Type: text/html; charset=utf-8 > > Proxy-Connection: close > > Connection: close > > Content-Length: 1274 > > > > $ > > Downloading tarballs also fails sometimes. > But, this hasn't been a problem since jhbuild caches tarballs in > ~/.cache/jhbuild/downloads/. Flatpak-builder also caches everything in WebKitBuild/FlatpakCache.
(In reply to Thibault Saunier from comment #10) > Flatpak-builder also caches everything in WebKitBuild/FlatpakCache. Right. In my comment #0, git is fetching repository only the diff. But, it failed because it tried to connect the server. > remote: Counting objects: 1, done. > remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
I tested in my home and Goolge Clould Platform too. Both have no problem on connecting the server. This problem happens only from my office. Umm. > for i in $(seq 10); do echo try $i: ; curl --head --silent https://anongit.freedesktop.org/git/gstreamer/common.git/; done
I'm very sorry. This is a problem completely in my side. I can solve this problem by changing proxy. My company has three types of proxy, ipv6-only, ipv4-only, ipv6-prefer proxies. I was using ipv6-prefer proxy. This is the problem. Closed as INVALID.