Bug 57246 – Windows build instructions do not work if you are behind a proxy or have a Temp directory that you do not own directly.

• Component: WebKit Website
• Status: RESOLVED FIXED

[Carol Szabo]

I am using a new install of Windows 7, I am behind a proxy server and following the build instructions for the tools install and the windows build on webkit.org does not seem to work for me and many of my colleagues. This bug is for sharing what I had to do to get things to work:
1. Installing cygwin:
a. in order to be able to download the correct packages through the proxy, be sure to set the http_proxy environment variable to http[s]://[username[:password]@]proxyhost[:port] for cygwin-downloader.exe. (In the URL above square brackets indicate optional parts of the URL, not literal characters).
b. Once cygwin is downloaded and installed, be sure to select the git packages for install even if not using git, it appears that there are some dependencies on those.
c. The rebaseall script for cygwin does not seem to be enough, as it does not rebase libraries used by perl and python.
Thus from ash do this:
unset TMP
unset TEMP (For whatever reason ash thought it cannot write to the windows temp directory - likely due to Windows vs. cygwin path.
find /lib/perl5 -iname "*.dll">/tmp/DLLs.txt
find /lib/python2.6 -iname "*.dll">>/tmp/DLLs.txt
/bin/rebaseall -T /tmp/DLLs.txt
2. The platform SDK:
Apparently it is not enough to to add its bin diretory to the VC bin directory, but it also needs to be in the PATH environment variable in cygwin.

[Adam Roben (:aroben)]

(In reply to comment #0)
> I am using a new install of Windows 7, I am behind a proxy server and following the build instructions for the tools install and the windows build on webkit.org does not seem to work for me and many of my colleagues. This bug is for sharing what I had to do to get things to work:

Thanks for writing these down!

> 1. Installing cygwin:
> a. in order to be able to download the correct packages through the proxy, be sure to set the http_proxy environment variable to http[s]://[username[:password]@]proxyhost[:port] for cygwin-downloader.exe. (In the URL above square brackets indicate optional parts of the URL, not literal characters).

The source for that page is checked in to Websites/webkit.org/building/tools.html. Would you be willing to make a patch that adds these instructions?

> b. Once cygwin is downloaded and installed, be sure to select the git packages for install even if not using git, it appears that there are some dependencies on those.

This should be remedied by updating CygwinDownloader. Its sources are checked in to Tools/CygwinDownloader. Would you be willing to make a patch?

> c. The rebaseall script for cygwin does not seem to be enough, as it does not rebase libraries used by perl and python.
> Thus from ash do this:
> unset TMP
> unset TEMP (For whatever reason ash thought it cannot write to the windows temp directory - likely due to Windows vs. cygwin path.
> find /lib/perl5 -iname "*.dll">/tmp/DLLs.txt
> find /lib/python2.6 -iname "*.dll">>/tmp/DLLs.txt
> /bin/rebaseall -T /tmp/DLLs.txt

Our current "best practice" instructions can be found at <http://trac.webkit.org/wiki/BuildingOnWindows#ErrorlaunchingApachefromrun-webkit-testsonVista7orothererrorsthatmentionthewordremap>. I think they would have solved your problem. It would be great to have a patch that adds these instructions to the main webkit.org page. Would you be willing to do that?

> 2. The platform SDK:
> Apparently it is not enough to to add its bin diretory to the VC bin directory, but it also needs to be in the PATH environment variable in cygwin.

That sounds kind of like what's described here: <http://trac.webkit.org/wiki/BuildingOnWindows#VisualCExpressEdition>. Again, a patch to add relevant instructions to webkit.org would be much appreciated!

[Carol Szabo]

(In reply to comment #1)
> (In reply to comment #0)
> The source for that page is checked in to Websites/webkit.org/building/tools.html. Would you be willing to make a patch that adds these instructions?

I am working on that but I want to try them out on a couple of other machines before trying them out.

> > b. Once cygwin is downloaded and installed, be sure to select the git packages for install even if not using git, it appears that there are some dependencies on those.
> This should be remedied by updating CygwinDownloader. Its sources are checked in to Tools/CygwinDownloader. Would you be willing to make a patch?

I am not sure whether updating cygwinDownloader is the best thing. Maybe removing those dependencies is better (I would rather try to fix the dependencies first).
In anycase fixing the source of cygwinDownloader is not enough. The instructions have a link to a zip file containing the binaries, thus the fix needs to be built and the binary needs to be posted on the site. If it is under source control, then I could do that as well if it comes to it.

> > c. The rebaseall script for cygwin does not seem to be enough, as it does not rebase libraries used by perl and python.
> > Thus from ash do this:
> > unset TMP
> > unset TEMP (For whatever reason ash thought it cannot write to the windows temp directory - likely due to Windows vs. cygwin path.
> > find /lib/perl5 -iname "*.dll">/tmp/DLLs.txt
> > find /lib/python2.6 -iname "*.dll">>/tmp/DLLs.txt
> > /bin/rebaseall -T /tmp/DLLs.txt
> Our current "best practice" instructions can be found at <http://trac.webkit.org/wiki/BuildingOnWindows#ErrorlaunchingApachefromrun-webkit-testsonVista7orothererrorsthatmentionthewordremap>. I think they would have solved your problem. It would be great to have a patch that adds these instructions to the main webkit.org page. Would you be willing to do that?

As said above. I am trying to verify that I did not miss anything and that all steps are needed (maybe refine them a bit before I make a patch). I do not believe that the instructions that you linked would have fixed the patch. They appear to fix something about appache, but I still cannot see the perl5 and python2.6 directories that contain DLLs loaded by the build scripts.
Trully, I believe that a better option is to drop cygwin altogether, install Active State Perl and Python for Windows and use those if possible. The DLL fix, causes (at least as far as I understand it) huge memory fragmentation issues, as ALL listed DLLs are mapped into the process space and each is assigned a non overlapping address range, then all cygwin processes when they load any dll, it will be loaded in this address range, thus DLLs will be loaded here and there in memory, leaving gaps for the DLLs that are not needed.

> > 2. The platform SDK:
> > Apparently it is not enough to to add its bin diretory to the VC bin directory, but it also needs to be in the PATH environment variable in cygwin.
> That sounds kind of like what's described here: <http://trac.webkit.org/wiki/BuildingOnWindows#VisualCExpressEdition>. Again, a patch to add relevant instructions to webkit.org would be much appreciated!

Actually I believe that it does not have anythign to do with the VisualCExpressEdition problem, it is rather due to an error in my instalation of Platform SDK. This is why I want to doublecheck the instructions before I propose a patch. I posted them here so that people have access to them and can issue comments like yours, or can tell me whether they worked for them or not.

[Adam Roben (:aroben)]

(In reply to comment #2)
> (In reply to comment #1)
> > (In reply to comment #0)
> > The source for that page is checked in to Websites/webkit.org/building/tools.html. Would you be willing to make a patch that adds these instructions?
> 
> I am working on that but I want to try them out on a couple of other machines before trying them out.

Great!

> > > b. Once cygwin is downloaded and installed, be sure to select the git packages for install even if not using git, it appears that there are some dependencies on those.
> > This should be remedied by updating CygwinDownloader. Its sources are checked in to Tools/CygwinDownloader. Would you be willing to make a patch?
> 
> I am not sure whether updating cygwinDownloader is the best thing. Maybe removing those dependencies is better (I would rather try to fix the dependencies first).
> In anycase fixing the source of cygwinDownloader is not enough. The instructions have a link to a zip file containing the binaries, thus the fix needs to be built and the binary needs to be posted on the site. If it is under source control, then I could do that as well if it comes to it.

The binary is checked in, as is the script to generate it: <http://trac.webkit.org/browser/trunk/Tools/CygwinDownloader/make-zip.sh>.

> > > c. The rebaseall script for cygwin does not seem to be enough, as it does not rebase libraries used by perl and python.
> > > Thus from ash do this:
> > > unset TMP
> > > unset TEMP (For whatever reason ash thought it cannot write to the windows temp directory - likely due to Windows vs. cygwin path.
> > > find /lib/perl5 -iname "*.dll">/tmp/DLLs.txt
> > > find /lib/python2.6 -iname "*.dll">>/tmp/DLLs.txt
> > > /bin/rebaseall -T /tmp/DLLs.txt
> > Our current "best practice" instructions can be found at <http://trac.webkit.org/wiki/BuildingOnWindows#ErrorlaunchingApachefromrun-webkit-testsonVista7orothererrorsthatmentionthewordremap>. I think they would have solved your problem. It would be great to have a patch that adds these instructions to the main webkit.org page. Would you be willing to do that?
> 
> As said above. I am trying to verify that I did not miss anything and that all steps are needed (maybe refine them a bit before I make a patch). I do not believe that the instructions that you linked would have fixed the patch. They appear to fix something about appache, but I still cannot see the perl5 and python2.6 directories that contain DLLs loaded by the build scripts.

The instructions end up putting all DLLs in /lib into the rebase file, which should include the perl5 and python2.6 directories you mentioned.

> Trully, I believe that a better option is to drop cygwin altogether, install Active State Perl and Python for Windows and use those if possible. The DLL fix, causes (at least as far as I understand it) huge memory fragmentation issues, as ALL listed DLLs are mapped into the process space and each is assigned a non overlapping address range, then all cygwin processes when they load any dll, it will be loaded in this address range, thus DLLs will be loaded here and there in memory, leaving gaps for the DLLs that are not needed.

Moving off Cygwin is a long-term project tracked by bug 48166.

[Carol Szabo]

Created attachment 87904 [details]
Patch

[Carol Szabo]

Comment on attachment 87904 [details]
Patch

Missed changelog. Learning how to use webkit-patch.

[Carol Szabo]

Created attachment 87906 [details]
Patch

[Darin Adler]

Comment on attachment 87906 [details]
Patch

These instructions are complex; is there any way we can cover this without adding so many additional steps?! Any way to break it down, for example?

[Carol Szabo]

Created attachment 88088 [details]
Patch

[Carol Szabo]

(In reply to comment #7)
> (From update of attachment 87906 [details])
> These instructions are complex; is there any way we can cover this without adding so many additional steps?! Any way to break it down, for example?

There were so many steps in my previous patch, because I was trying to explain what to do at a Windows newbie level, just as the instructions for starting the cygwin_downloader were. I grouped the steps by objective via paragraph separators, maybe not clearly enough, but I was trying to add as little changes as possible to the complexity of the document structure and the requirements on the browser (that is why I stayed away from nested counters instead of list items).
I believe that this level of detail is unnecessary as the intended reader should be already familiar enough with windows to know how to extract a zip file and launch an executable contained in it, was well as to set an environment variable before starting an executable. I kept that level of detail in my first patch just for the sake of consistency.
In this second patch I have simplified the instructions, but I raised the prerequisite IQ and Windows/POSIX knowledge level.
Originally I started this bug to share some hard earned knowledge, but, by the time I finished writing this patch, I found that there are just a few warnings that may be worth sharing, such as the fact that ash -c /bin/rebaseall may fail due to lack of write access to the temp directory from cygwin (which is default for my Win 7 corporate install) and this would not be easy to detect since the window with the error is immediately closed and that if you are behind a proxy, configuring it in the usual Linux way helps.
This is what my latest patch does in essence, beside a little bit of rewording.

[Tor Arne Vestbø]

Comment on attachment 88088 [details]
Patch

lgtm

[WebKit Commit Bot]

Comment on attachment 88088 [details]
Patch

Clearing flags on attachment: 88088

Committed r85008: <http://trac.webkit.org/changeset/85008>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.