Bug 223609 – Update autoinstalled packages

• Status: ASSIGNED
• Component: Tools / Tests

[Sam Sneddon [:gsnedders]]

A lot of these are quite out of date, and we need more recent versions for Python 3.9 on macOS.

[Sam Sneddon [:gsnedders]]

Created attachment 423954 [details]
Patch

[Sam Sneddon [:gsnedders]]

Created attachment 424144 [details]
Patch

[Jonathan Bedard]

Comment on attachment 424144 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=424144&action=review

> Tools/Scripts/libraries/webkitcorepy/webkitcorepy/__init__.py:41
>  

Need to bump tiny version here and in setup.py.

> Tools/Scripts/libraries/webkitflaskpy/webkitflaskpy/__init__.py:46
>  version = Version(0, 1, 1)

Need to bump tiny version here and in setup.py.

> Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py:49
>  version = Version(0, 13, 6)

Need to bump tiny version here and in setup.py.

[Fujii Hironori]

I tested your patch with Windows Python 2 and 3.
It works fine with Python 3.
However, Python 2 fails.

PS C:\home\webkit\ga> git clean -dfx
PS C:\home\webkit\ga> python .\Tools\Scripts\test-webkitpy
Downloading wheel-0.36.2...
Failed to install wheel-0.36.2!
Traceback (most recent call last):
  File ".\Tools\Scripts\test-webkitpy", line 30, in <module>
    from webkitpy.common import multiprocessing_bootstrap
  File "C:\home\webkit\ga\Tools\Scripts\webkitpy\__init__.py", line 94, in <module>
    import webkitscmpy
  File "C:\home\webkit\ga\Tools\Scripts\libraries\webkitscmpy\webkitscmpy\__init__.py", line 62, in <module>
    from webkitscmpy import mocks
  File "C:\home\webkit\ga\Tools\Scripts\libraries\webkitscmpy\webkitscmpy\mocks\__init__.py", line 3, in <module>
    from webkitscmpy.mocks import local
  File "C:\home\webkit\ga\Tools\Scripts\libraries\webkitscmpy\webkitscmpy\mocks\local\__init__.py", line 4, in <module>
    from webkitscmpy.mocks.local.git import Git
  File "C:\home\webkit\ga\Tools\Scripts\libraries\webkitscmpy\webkitscmpy\mocks\local\git.py", line 30, in <module>
    from webkitscmpy.program.canonicalize.committer import main as committer_main
  File "C:\home\webkit\ga\Tools\Scripts\libraries\webkitscmpy\webkitscmpy\program\__init__.py", line 28, in <module>
    from webkitscmpy import local, log, remote
  File "C:\home\webkit\ga\Tools\Scripts\libraries\webkitscmpy\webkitscmpy\remote\__init__.py", line 24, in <module>
    from webkitscmpy.remote.svn import Svn
  File "C:\home\webkit\ga\Tools\Scripts\libraries\webkitscmpy\webkitscmpy\remote\svn.py", line 25, in <module>
    import fasteners
  File "C:\home\webkit\ga\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 560, in find_module
    cls.install(name)
  File "C:\home\webkit\ga\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 544, in install
    return all([to_install.install() for to_install in cls.register(package)])
  File "C:\home\webkit\ga\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 238, in install
    AutoInstall.install('wheel')
  File "C:\home\webkit\ga\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 544, in install
    return all([to_install.install() for to_install in cls.register(package)])
  File "C:\home\webkit\ga\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 259, in install
    archive.unpack(temp_location)
  File "C:\home\webkit\ga\Tools\Scripts\libraries\webkitcorepy\webkitcorepy\autoinstall.py", line 123, in unpack
    file.extractall(target)
  File "C:\Python27\lib\tarfile.py", line 2081, in extractall
    self.extract(tarinfo, path)
  File "C:\Python27\lib\tarfile.py", line 2118, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name))
  File "C:\Python27\lib\tarfile.py", line 2194, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "C:\Python27\lib\tarfile.py", line 2234, in makefile
    with bltn_open(targetpath, "wb") as target:
IOError: [Errno 22] invalid mode ('wb') or filename: 'c:\\users\\000013~1\\appdata\\local\\temp\\wheel\\wheel-0.36.2\\tests\\testdata\\unicode.dist\\unicodedist\\\xe5\xe4\xf6_???.py'

[Fujii Hironori]

test-webkitpy and test-webkitpy-python2 work nicely with Windows Python 2.7 if I downgrade the version of wheel.
> AutoInstall.register(Package('wheel', Version(0, 35, 1)))

[Sam Sneddon [:gsnedders]]

(In reply to Fujii Hironori from comment #5)
> test-webkitpy and test-webkitpy-python2 work nicely with Windows Python 2.7
> if I downgrade the version of wheel.
> > AutoInstall.register(Package('wheel', Version(0, 35, 1)))

That's super surprising that makes a difference!

Also had issues with mccabe having a setup_requires dependency on pytest-runner on Linux; potentially just lucky with ordering that it wasn't hit sooner.

[Sam Sneddon [:gsnedders]]

Created attachment 424309 [details]
Patch

[Fujii Hironori]

I tested the patch with Windows Python 2 and 3, and confirmed no autoinstalling issue.

> git clean -dfx
> python .\Tools\Scripts\test-webkitpy-python2
> C:\Python39\python.exe .\Tools\Scripts\test-webkitpy

Also, I got the same error (Comment 4) after modifying the wheel version to Version(0, 36, 2).

> python .\Tools\Scripts\test-webkitpy-python2

This is weired, but I don't want to debug Python 2 specific issue.
This problem can be fixed by migrating to Python 3.

[Jonathan Bedard]

Comment on attachment 424309 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=424309&action=review

> Tools/Scripts/webkitpy/autoinstalled/twisted.py:33
> +AutoInstall.register(Package('twisted', Version(20, 3, 0), pypi_name='Twisted'))

We need to check that this doesn't break performance testing, which is not well covered by unit tests

[Radar WebKit Bug Importer]

<rdar://problem/75977586>