Bug 148941 – update-webkitgtk-libs fails behind firewall

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Mark Salisbury]

Running update-webkitgtk-libs behind a firewall blocking git and active FTP connections fails.

 GIT failure:
  Updating GTK+ port dependencies using jhbuild...
  Cloning into 'jhbuild'...
  fatal: unable to connect to git.gnome.org:
  git.gnome.org[0: 209.132.180.184]: errno=Connection timed out

 FTP failure:
  --2015-09-01 10:40:08--  ftp://ftp.freedesktop.org/pub/mesa/10.5.4/mesa-10.5.4.tar.xz
    (try: 7) => \u2018/home/masali1/git/webkit-gtk/WebKitBuild/DependenciesGTK/Source/mesa-10.5.4.tar.xz\u2019
  Connecting to proxy.austin.hp.com (proxy.austin.hp.com)|xx.xx.xx.xx|:8080... connected.
  Logging in as anonymous ... 
  Error in server response, closing control connection.

Most package dependencies are already accessed using HTTP.  All packages/repos should be accessed using HTTP.

[Mark Salisbury]

Created attachment 260745 [details]
Proposed fix

[Mark Salisbury]

Ping... Any interest in this patch?

[Martin Robinson]

Comment on attachment 260745 [details]
Proposed fix

View in context: https://bugs.webkit.org/attachment.cgi?id=260745&action=review

> Tools/gtk/jhbuild.modules:276
>    <autotools id="libxml2"
> -             autogen-sh="configure"
> -             autogenargs="--without-python">
> -    <branch module="/libxml2/libxml2-2.9.1.tar.gz" version="2.9.1"
> -            repo="xmlsoft.org"
> -            hash="sha256:fd3c64cb66f2c4ea27e934d275904d92cec494a8e8405613780cbc8a71680fdb"
> -            md5sum="9c0cfef285d5c4a5c80d00904ddab380"/>
> +             autogen-sh="./autogen.sh; ./configure --with-python=no">
> +    <branch module="browse/libxml2" version="2.9.1" tag="v2.9.1"
> +            repo="git.gnome.org"/>
>    </autotools>

I think it would be better to keep fetching the tarball here if possible. Is it accessible via HTTP?

> Tools/gtk/jhbuild.modules:361
> -    <branch module="/pub/mesa/10.5.4/mesa-10.5.4.tar.xz" version="10.5.4"
> +    <branch module="git/mesa/mesa.git" version="10.5.4" tag="mesa-10.5.4"
>              checkoutdir="Mesa"
> -            repo="ftp.freedesktop.org"
> -            hash="sha256:b51e723f3a20d842c88a92d809435b229fc4744ca0dbec0317d9d4a3ac4c6803">
> +            repo="freedesktop-git">
>      </branch>

Ditto.

[Michael Catanzaro]

Comment on attachment 260745 [details]
Proposed fix

View in context: https://bugs.webkit.org/attachment.cgi?id=260745&action=review

> Tools/gtk/jhbuild.modules:48
> +      href="http://git.gnome.org/"/>

Better to use https://git.gnome.org/browse/ (note the https)

> Tools/gtk/jhbuild.modules:190
> +    <branch module="browse/libsoup" version="2.49.91.1"

Then you don't need the browse/ here

>> Tools/gtk/jhbuild.modules:276
>>    </autotools>
> 
> I think it would be better to keep fetching the tarball here if possible. Is it accessible via HTTP?

You can get it from download.gnome.org, but for some reason the last GNOME release was 2.6 from 2009. So seems like "no."

I think this is just a very broken firewall, but I know you're stuck with it. And I also prefer tarballs to git, but if the git clone is not too much slower than downloading the tarball, then I think using git is fine....

> Tools/jhbuild/jhbuild-wrapper:85
> +    process = subprocess.Popen(['git', 'clone', '--config', 'pack.threads=1', 'http://git.gnome.org/browse/jhbuild'], cwd=source_path)

https

[Mark Salisbury]

Comment on attachment 260745 [details]
Proposed fix

View in context: https://bugs.webkit.org/attachment.cgi?id=260745&action=review

>> Tools/gtk/jhbuild.modules:48
>> +      href="http://git.gnome.org/"/>
> 
> Better to use https://git.gnome.org/browse/ (note the https)

Sounds good.

>> Tools/gtk/jhbuild.modules:190
>> +    <branch module="browse/libsoup" version="2.49.91.1"
> 
> Then you don't need the browse/ here

OK.

>>> Tools/gtk/jhbuild.modules:276
>>>    </autotools>
>> 
>> I think it would be better to keep fetching the tarball here if possible. Is it accessible via HTTP?
> 
> You can get it from download.gnome.org, but for some reason the last GNOME release was 2.6 from 2009. So seems like "no."
> 
> I think this is just a very broken firewall, but I know you're stuck with it. And I also prefer tarballs to git, but if the git clone is not too much slower than downloading the tarball, then I think using git is fine....

I found libxml2 via http at http://xmlsoft.org/sources/libxml2-2.9.2.tar.gz so I can change that.

>> Tools/gtk/jhbuild.modules:361
>>      </branch>
> 
> Ditto.

I've not been able to find any way to access a mesa bundle using http... except on sourceforge but it is a few years old!

[Mark Salisbury]

Created attachment 262338 [details]
Proposed fix.

I believe I've addressed review feedback (thank you) except for mesa.  I haven't been able to find a good source for a mesa tarball.

I rebased too but it looks like dependencies haven't been updated in the last month; I didn't have any conflicts.

[WebKit Commit Bot]

Comment on attachment 262338 [details]
Proposed fix.

Clearing flags on attachment: 262338

Committed r190622: <http://trac.webkit.org/changeset/190622>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Michael Catanzaro]

OK, so the git clone of mesa is REALLY slow... Carlos, could this be a problem for the bots? If so, maybe we should host our own mesa tarball somewhere, or just accept that this isn't going to work if a firewall is blocking ftp.

[Michael Catanzaro]

We could also try patching jhbuild to do bare clones.

[Carlos Alberto Lopez Perez]

(In reply to comment #9)
> OK, so the git clone of mesa is REALLY slow... Carlos, could this be a
> problem for the bots? If so, maybe we should host our own mesa tarball
> somewhere, or just accept that this isn't going to work if a firewall is
> blocking ftp.

The bots have set the the JHBUILD_MIRROR environment variable that allows them to use a local directory for catching the download products <http://trac.webkit.org/r166061>. Works great for tarballs, for git repositories I think also works but i'm not 100% sure.

Just let me know if you notice some problem with the bots.


However I find odd that we switched the mesa source from a tarball to a git repo just to avoid a ftp download. Maybe we should request to the freedesktop.org people that they made available the files also via http or https?

[Mark Salisbury]

Not a bad suggestion.  I've contacted the webmaster for mesa3d.org.  His initial response seems positive.

[Mark Salisbury]

> However I find odd that we switched the mesa source from a tarball to a git
> repo just to avoid a ftp download. Maybe we should request to the
> freedesktop.org people that they made available the files also via http or
> https?

Brian Paul (ebmaster for mesa3d.org) has generously made mesa tarballs available over http at:
http://people.freedesktop.org/~brianp/mesa/

I've asked if it would be possible to make the page a more public/official looking URL...  something like http://www.freedesktop.org/mesa/download.

[Carlos Alberto Lopez Perez]

(In reply to comment #13)
> > However I find odd that we switched the mesa source from a tarball to a git
> > repo just to avoid a ftp download. Maybe we should request to the
> > freedesktop.org people that they made available the files also via http or
> > https?
> 
> Brian Paul (ebmaster for mesa3d.org) has generously made mesa tarballs
> available over http at:
> http://people.freedesktop.org/~brianp/mesa/
> 
> I've asked if it would be possible to make the page a more public/official
> looking URL...  something like http://www.freedesktop.org/mesa/download.

Great! thanks for handling this.

The xorg releases are available at http://xorg.freedesktop.org/releases/ so following the same naming scheme maybe http://mesa.freedesktop.org/releases/ is a good candidate

[Carlos Alberto Lopez Perez]

(In reply to comment #13)
> > However I find odd that we switched the mesa source from a tarball to a git
> > repo just to avoid a ftp download. Maybe we should request to the
> > freedesktop.org people that they made available the files also via http or
> > https?
> 
> Brian Paul (ebmaster for mesa3d.org) has generously made mesa tarballs
> available over http at:
> http://people.freedesktop.org/~brianp/mesa/
> 
> I've asked if it would be possible to make the page a more public/official
> looking URL...  something like http://www.freedesktop.org/mesa/download.

Any news regarding this?

[Mark Salisbury]

(In reply to comment #15)
> 
> Any news regarding this?

Unfortunately, no.  I haven't heard back.  I just asked again.

http://people.freedesktop.org/~brianp/mesa/ was set up and works.  I'm hesitant about using it though - will it go away overnight when some house cleaning is done?

When I first asked about getting a public download site, there was hesitancy as that was more involved - and I'm just one person that's asked for this.  I did my best to explain that these downloads are done automatically when building webkit-gtk so it's not really just one person.

[Carlos Alberto Lopez Perez]

(In reply to comment #16)
> (In reply to comment #15)
> > 
> > Any news regarding this?
> 
> Unfortunately, no.  I haven't heard back.  I just asked again.
> 
> http://people.freedesktop.org/~brianp/mesa/ was set up and works.  I'm
> hesitant about using it though - will it go away overnight when some house
> cleaning is done?
> 

If that happens we can switch back to git. I rather use this non very official URL than waiting lot of time for a more official page with the tarballs.

Actually, it seems that both ftp.freedesktop.org, and people.freedesktop.org are on the same machine, that is know as annarchy.freedesktop.org (looking at the DNS records), and developers have access to it according to http://www.freedesktop.org/wiki/AccountRequests/

So, I guess that what brianp did is just a symlink to his homedir from the FTP public subdirectory.

Should be fine meanwhile he don't deletes that.

> When I first asked about getting a public download site, there was hesitancy
> as that was more involved - and I'm just one person that's asked for this. 
> I did my best to explain that these downloads are done automatically when
> building webkit-gtk so it's not really just one person.

I think it should be simple to expose this files via http with a better URL, maybe we can open a bug requesting this on the freedesktop bugzilla if you don't get a positive response soon.


Thanks.

[Mark Salisbury]

(In reply to comment #17)
> 
> If that happens we can switch back to git. I rather use this non very
> official URL than waiting lot of time for a more official page with the
> tarballs.
> 

Thanks for your feedback.

I've opened https://bugs.webkit.org/show_bug.cgi?id=150470.

I'll submit a patch to change back to tarball download (over http).  If we get a better URL it'll be simple to start using it.

[Carlos Alberto Lopez Perez]

(In reply to Mark Salisbury from comment #18)
> (In reply to comment #17)
> > 
> > If that happens we can switch back to git. I rather use this non very
> > official URL than waiting lot of time for a more official page with the
> > tarballs.
> > 
> 
> Thanks for your feedback.
> 
> I've opened https://bugs.webkit.org/show_bug.cgi?id=150470.
> 
> I'll submit a patch to change back to tarball download (over http).  If we
> get a better URL it'll be simple to start using it.

Proposed to change this URL on bug 170431