builtins-generator-tests and bindings-generation-tests are reporting "Failed to install configparser-4.0.2!" since r268267 on WinCairo Buildbot

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Fujii Hironori]

builtins-generator-tests and bindings-generation-tests are reporting "Failed to install configparser-4.0.2!" since r268267 on WinCairo Buildbot

https://build.webkit.org/builders/WinCairo-64-bit-WKL-Release-Tests/builds/8893

> Downloading configparser-4.0.2...
> Installing configparser-4.0.2...
> Failed to install configparser-4.0.2!
> Traceback (most recent call last):
>   File "./Tools/Scripts/run-bindings-tests", line 38, in <module>
>     import flatpakutils
>   File ".\Tools\flatpak\flatpakutils.py", line 21, in <module>
>     import configparser
>   File "C:\BW\wincairo-wkl-release-tests\build\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 481, in find_module
>     cls.install(name)
>   File "C:\BW\wincairo-wkl-release-tests\build\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 466, in install
>     return to_install.install()
>   File "C:\BW\wincairo-wkl-release-tests\build\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 267, in install
>     stderr=devnull,
>   File "C:\tools\python\lib\subprocess.py", line 190, in check_call
>     raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['C:\\tools\\python\\python.exe', 'c:\\users\\containeradministrator\\appdata\\local\\temp\\configparser\\configparser-4.0.2\\setup.py', 'install', '--home=C:\\BW\\wincairo-wkl-release-tests\\build\\Tools\\Scripts\\libraries\\autoinstalled\\python-2', '--root=/', '--prefix=', '--single-version-externally-managed', '--install-lib=C:\\BW\\wincairo-wkl-release-tests\\build\\Tools\\Scripts\\libraries\\autoinstalled\\python-2', '--install-scripts=C:\\BW\\wincairo-wkl-release-tests\\build\\Tools\\Scripts\\libraries\\autoinstalled\\python-2\\bin', '--install-data=C:\\BW\\wincairo-wkl-release-tests\\build\\Tools\\Scripts\\libraries\\autoinstalled\\python-2\\data', '--install-headers=C:\\BW\\wincairo-wkl-release-tests\\build\\Tools\\Scripts\\libraries\\autoinstalled\\python-2\\headers', '--old-and-unmanageable']' returned non-zero exit status 1
> program finished with exit code 1


See also: Bug 215895 – [webkitpy] Use webkitcorepy's auto installer for pytest packages

[Fujii Hironori]

It doesn't happen on my PC. (Windows 10 1903, Python 2.7.18)

[Fujii Hironori]

Bug 216948 – [Windows] Python module autoinstall fails if temp drive is different from source tree drive

Same issue with this?

[Stephan Szabo]

What I saw in 216948 was the install succeeding, but the modules being put in an incorrect location as it used the wrong root drive and used whatever drive was specified in TEMP rather than the one the source was on(and on some versions failing to be loaded, in others, the script continued but the files were still in the wrong place).

But the bots should have everything on the same drive, so I don't think it should be the same.

[Fujii Hironori]

Can you check what error is reporting on your bots by invoking setup.py manually or modifying libraries/webkitcorepy/webkitcorepy/autoinstall.py not to suppress stdout and stderr?

> C:\tools\python\python.exe c:\users\containeradministrator\appdata\local\temp\configparser\configparser-4.0.2\setup.py install --home=C:\BW\wincairo-wkl-release-tests\build\Tools\Scripts\libraries\autoinstalled\python-2 --root=/ --prefix= --single-version-externally-managed --install-lib=C:\BW\wincairo-wkl-release-tests\build\Tools\Scripts\libraries\autoinstalled\python-2 --install-scripts=C:\BW\wincairo-wkl-release-tests\build\Tools\Scripts\libraries\autoinstalled\python-2\bin --install-data=C:\BW\wincairo-wkl-release-tests\build\Tools\Scripts\libraries\autoinstalled\python-2\data --install-headers=C:\BW\wincairo-wkl-release-tests\build\Tools\Scripts\libraries\autoinstalled\python-2\headers --old-and-unmanageable

[Stephan Szabo]

A manual run of setup.py on the bot using the given command seems to succeed (as an "&& dir" showed the directory listing), although it didn't seem to make the bot happy.

[Stephan Szabo]

Looks like it might be a proxy issue:

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x0000000003A16288>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools-scm/


Going to try adding HTTP_PROXY and HTTPS_PROXY to the environment passed through locally on the bot and see if that makes it work.

[Stephan Szabo]

Created attachment 411248 [details]
Patch

[Fujii Hironori]

Is setuptools-scm installed while installing configparser?

Actually, configparser seems using setuptools-scm.
https://github.com/jaraco/configparser/blob/v4.0.2/setup.cfg#L30

However, there is no setuptools-scm in my 
Tools/Scripts/libraries/autoinstalled/python-2 directory.
There is configparser-4.0.2-py2.7.egg-info, setuptools and setuptools-44.1.1-py2.7.egg-info directories.

Why isn't setuptools-scm instlled on my PC?

How about idea installing setuptools-scm as well as other modules?

> AutoInstall.register(Package('setuptools-scm', Version(4, 0, 2)))
> import setuptools-scm

[Stephan Szabo]

(In reply to Fujii Hironori from comment #8)
> Is setuptools-scm installed while installing configparser?
> 
> Actually, configparser seems using setuptools-scm.
> https://github.com/jaraco/configparser/blob/v4.0.2/setup.cfg#L30
> 
> However, there is no setuptools-scm in my 
> Tools/Scripts/libraries/autoinstalled/python-2 directory.
> There is configparser-4.0.2-py2.7.egg-info, setuptools and
> setuptools-44.1.1-py2.7.egg-info directories.

It doesn't seem to be installed, which does seem strange, although the
setup does seem to work apart from this issue which seems to be network
related. So, why does the install seem to work even though it's apparently
not there?

[Fujii Hironori]

Created attachment 411304 [details]
Patch

Another approach.

[Fujii Hironori]

Comment on attachment 411304 [details]
Patch

OMG. WebKitPy-Tests-EWS failed.
> Failed to install setuptools-scm-4.1.2!

[Fujii Hironori]

I don't know why my approach failed on the bot. Let's take Stephan's approach. Stephan's patch LGTM.

[Jonathan Bedard]

(In reply to Fujii Hironori from comment #11)
> Comment on attachment 411304 [details]
> Patch
> 
> OMG. WebKitPy-Tests-EWS failed.
> > Failed to install setuptools-scm-4.1.2!

Might be a package only some ports require? In any case, forwarding more environment variables is definitely safe.

[Jonathan Bedard]

Comment on attachment 411248 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=411248&action=review

> Tools/ChangeLog:11
> +        * Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py:

Minor nit: can we bump the nano version of the library in Scripts/libraries/webkitcorepy/webkitcorepy/__init__.py?

[Stephan Szabo]

Created attachment 411330 [details]
Patch

[Stephan Szabo]

Created attachment 411333 [details]
Patch

[Jonathan Bedard]

Comment on attachment 411333 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=411333&action=review

> Tools/Scripts/libraries/webkitcorepy/webkitcorepy/__init__.py:38
> +version = Version(0, 4, 15, 0, 1)

Sorry, I communicated this poorly, should just be:
version = Version(0, 4, 16)

[Stephan Szabo]

Oh, sorry - I was wondering why nano rather than that. Will change.

[Stephan Szabo]

Created attachment 411336 [details]
Patch

[EWS]

Committed r268471: <https://trac.webkit.org/changeset/268471>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 411336 [details].

[Radar WebKit Bug Importer]

<rdar://problem/70298426>