Bug 215067 – [webkitcorepy] Use setup.py for the autoinstaller

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Jonathan Bedard]

Previously, we were just moving the Python files directly. This wouldn't work if the library in question has C dependencies (the webkitpy autoinstaller had a similar issue). setup.py is the blessed way to install in a custom location.

[Radar WebKit Bug Importer]

<rdar://problem/66432719>

[Jonathan Bedard]

Created attachment 405799 [details]
Patch

[Jonathan Bedard]

(In reply to Jonathan Bedard from comment #2)
> Created attachment 405799 [details]
> Patch

I've tested this with a few cryptography libraries that require compilation. So far, I haven't found a library where this approach does not work, assuming the libraries it depends on are also specified.

[Aakash Jain]

This patch seems to break api tests on mac.

e.g.: https://ews-build.webkit.org/#/builders/3/builds/29658

subprocess.CalledProcessError: Command 
'['/usr/bin/python', 
'/Volumes/Data/worker/API-Tests-macOS-EWS/build/Tools/Scripts/libraries/autoinstalled/python-2/urllib3.tmp/urllib3-1.25.10/setup.py', 
'install', 
'--home=/Volumes/Data/worker/API-Tests-macOS-EWS/build/Tools/Scripts/libraries/autoinstalled/python-2', 
'--root=/', 
'--single-version-externally-managed', 
'--install-lib=/Volumes/Data/worker/API-Tests-macOS-EWS/build/Tools/Scripts/libraries/autoinstalled/python-2', 
'--install-scripts=/Volumes/Data/worker/API-Tests-macOS-EWS/build/Tools/Scripts/libraries/autoinstalled/python-2', 
'--install-data=/Volumes/Data/worker/API-Tests-macOS-EWS/build/Tools/Scripts/libraries/autoinstalled/python-2/data', 
'--install-headers=/Volumes/Data/worker/API-Tests-macOS-EWS/build/Tools/Scripts/libraries/autoinstalled/python-2/headers', 
'--old-and-unmanageable']' 
returned non-zero exit status 1

[Jonathan Bedard]

(In reply to Aakash Jain from comment #4)
> This patch seems to break api tests on mac.
> 
> e.g.: https://ews-build.webkit.org/#/builders/3/builds/29658
> 
> subprocess.CalledProcessError: Command 
> '['/usr/bin/python', 
> '/Volumes/Data/worker/API-Tests-macOS-EWS/build/Tools/Scripts/libraries/
> autoinstalled/python-2/urllib3.tmp/urllib3-1.25.10/setup.py', 
> 'install', 
> '--home=/Volumes/Data/worker/API-Tests-macOS-EWS/build/Tools/Scripts/
> libraries/autoinstalled/python-2', 
> '--root=/', 
> '--single-version-externally-managed', 
> '--install-lib=/Volumes/Data/worker/API-Tests-macOS-EWS/build/Tools/Scripts/
> libraries/autoinstalled/python-2', 
> '--install-scripts=/Volumes/Data/worker/API-Tests-macOS-EWS/build/Tools/
> Scripts/libraries/autoinstalled/python-2', 
> '--install-data=/Volumes/Data/worker/API-Tests-macOS-EWS/build/Tools/Scripts/
> libraries/autoinstalled/python-2/data', 
> '--install-headers=/Volumes/Data/worker/API-Tests-macOS-EWS/build/Tools/
> Scripts/libraries/autoinstalled/python-2/headers', 
> '--old-and-unmanageable']' 
> returned non-zero exit status 1

Going to increase logging....bit weird that it broke API tests on Mac and nothing else, would have expected this to be all or nothing.

[Jonathan Bedard]

Created attachment 405831 [details]
Patch

[Jonathan Bedard]

(In reply to Jonathan Bedard from comment #5)
> (In reply to Aakash Jain from comment #4)
> > ....
> 
> Going to increase logging....bit weird that it broke API tests on Mac and
> nothing else, would have expected this to be all or nothing.

Fun! we have an outdated version of setuptools on these bots....which, rather ironically, means we need to install setup tools before installing anything else.

[Jonathan Bedard]

Created attachment 405834 [details]
Patch

[Jonathan Bedard]

Created attachment 405845 [details]
Patch

[Jonathan Bedard]

Created attachment 405850 [details]
Patch

[Jonathan Bedard]

Created attachment 405928 [details]
Patch

[Stephanie Lewis]

Talked over the patch with Jonathan and decided we should keep the egg info because it has a lot of dependencies which we can use to write an automatable dependency list

[EWS]

Committed r265254: <https://trac.webkit.org/changeset/265254>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 405928 [details].

[Fujii Hironori]

WinCairo buildbot is failing since this change.

https://build.webkit.org/builders/WinCairo%2064-bit%20WKL%20Release%20%28Tests%29/builds/7779

No handlers could be found for logger "webkitcorepy"
Traceback (most recent call last):
  File "C:\WebKit-BuildWorker\wincairo-wkl-release-tests\build\Tools\Scripts\webkitpy\layout_tests\run_webkit_tests.py", line 40, in <module>
    from webkitpy.layout_tests.controllers.manager import Manager
  File "C:\WebKit-BuildWorker\wincairo-wkl-release-tests\build\Tools\Scripts\webkitpy\layout_tests\controllers\manager.py", line 60, in <module>
    from webkitpy.results.upload import Upload
  File "C:\WebKit-BuildWorker\wincairo-wkl-release-tests\build\Tools\Scripts\webkitpy\results\upload.py", line 23, in <module>
    import webkitpy.thirdparty.autoinstalled.requests
  File "C:\WebKit-BuildWorker\wincairo-wkl-release-tests\build\Tools\Scripts\webkitpy\thirdparty\autoinstalled\requests\__init__.py", line 43, in <module>
    import urllib3
  File "C:\WebKit-BuildWorker\wincairo-wkl-release-tests\build\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 405, in find_module
    cls.install(name)
  File "C:\WebKit-BuildWorker\wincairo-wkl-release-tests\build\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 390, in install
    return to_install.install()
  File "C:\WebKit-BuildWorker\wincairo-wkl-release-tests\build\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 185, in install
    AutoInstall.install('setuptools')
  File "C:\WebKit-BuildWorker\wincairo-wkl-release-tests\build\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 390, in install
    return to_install.install()
  File "C:\WebKit-BuildWorker\wincairo-wkl-release-tests\build\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 238, in install
    stderr=devnull,
  File "C:\tools\python\lib\subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\tools\\python\\python.exe', 'C:\\WebKit-BuildWorker\\wincairo-wkl-release-tests\\build\\Tools\\Scripts\\libraries\\autoinstalled\\python-2\\setuptools.tmp\\setuptools-41.0.1\\setup.py', 'install', '--home=C:\\WebKit-BuildWorker\\wincairo-wkl-release-tests\\build\\Tools\\Scripts\\libraries\\autoinstalled\\python-2', '--root=/', '--single-version-externally-managed', '--install-lib=C:\\WebKit-BuildWorker\\wincairo-wkl-release-tests\\build\\Tools\\Scripts\\libraries\\autoinstalled\\python-2', '--install-scripts=C:\\WebKit-BuildWorker\\wincairo-wkl-release-tests\\build\\Tools\\Scripts\\libraries\\autoinstalled\\python-2', '--install-data=C:\\WebKit-BuildWorker\\wincairo-wkl-release-tests\\build\\Tools\\Scripts\\libraries\\autoinstalled\\python-2\\data', '--install-headers=C:\\WebKit-BuildWorker\\wincairo-wkl-release-tests\\build\\Tools\\Scripts\\libraries\\autoinstalled\\python-2\\headers', '--old-and-unmanageable']' returned non-zero exit status 1

[Jonathan Bedard]

Reopening to attach new patch.

[Jonathan Bedard]

Created attachment 405964 [details]
Patch

[Jonathan Bedard]

Comment on attachment 405964 [details]
Patch

Thanks to the Sony folks for finding the actual fix, this will not resolve the issue. Marking the patch as obsolete..

[Fujii Hironori]

Filed another ticket to fix the WinCairo issue.
Bug 215152 – [webkitcorepy] REGRESSION(r265254): autoinstaller is failing to execute setup.py on win32 Python

[Roy Reapor]

rdar://88521722