Bug 285315 – 'git-webkit setup' fails with "ValueError: No archives for MarkupSafe-2.1.5 found"

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Michael Catanzaro]

It seems 'git-webkit setup' is broken:

$ git-webkit setup
Traceback (most recent call last):
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/git-webkit", line 63, in <module>
    sys.exit(program.main(
             ~~~~~~~~~~~~^
        identifier_template=is_webkit_filter('Canonical link: https://commits.webkit.org/{}'),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        classifier=is_webkit_filter(classifier()),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ))
    ^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/__init__.py", line 184, in main
    return parsed.main(
           ~~~~~~~~~~~^
        args=parsed,
        ^^^^^^^^^^^^
    ...<5 lines>...
        canonical_svn=canonical_svn,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/setup.py", line 621, in main
    result = cls.git(args, repository, **kwargs)
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/setup.py", line 432, in git
    result += InstallHooks.main(args, repository, hooks=hooks, **kwargs)
              ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/install_hooks.py", line 230, in main
    from jinja2 import Template
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1322, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1262, in _find_spec
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 697, in find_spec
    loader = cls.find_module(fullname, path=path)
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 711, in find_module
    cls.install(name)
    ~~~~~~~~~~~^^^^^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 684, in install
    return all([to_install.install() for to_install in packages])
                ~~~~~~~~~~~~~~~~~~^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 308, in install
    AutoInstall.install(dependency)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 684, in install
    return all([to_install.install() for to_install in packages])
                ~~~~~~~~~~~~~~~~~~^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 322, in install
    raise ValueError('No archives for {}-{} found'.format(self.pypi_name, self.version))
ValueError: No archives for MarkupSafe-2.1.5 found

I'm using python 3.13.0. No clue why it's not able to find MarkupSafe 2.1.5, which seems to be normally available from PyPi.

[Sam Sneddon [:gsnedders]]

I wish we gave better errors here.

https://pypi.org/project/MarkupSafe/2.1.5/#files shows there's no wheel for Python 3.13, and we seem to be forcing binary installs:

Tools/Scripts/libraries/webkitflaskpy/webkitflaskpy/__init__.py
57:AutoInstall.register(Package('markupsafe', Version(2, 1, 5), pypi_name='MarkupSafe', wheel=True))

Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py
56:AutoInstall.register(Package('markupsafe', Version(2, 1, 5), pypi_name='MarkupSafe', wheel=True))

Probably just need to bump these for 3.13, but that implies a major version upgrade — try and run the tests, I guess?

[Michael Catanzaro]

I tried running test-webkitpy but it fails:

Traceback (most recent call last):
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/test/main.py", line 392, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/test/main.py", line 115, in main
    return not tester.run()
               ~~~~~~~~~~^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/test/main.py", line 193, in run
    return self._run_tests(names)
           ~~~~~~~~~~~~~~~^^^^^^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/test/main.py", line 207, in _run_tests
    AutoInstall.install_everything()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 696, in install_everything
    package.install()
    ~~~~~~~~~~~~~~~^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 322, in install
    raise ValueError('No archives for {}-{} found'.format(self.pypi_name, self.version))
ValueError: No archives for hiredis-3 found

which is fixed by upgrading to hiredis-3.1.0, and also specifies wheel=true, so presumably that's the same problem. Next error:

Traceback (most recent call last):
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/test/main.py", line 392, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/test/main.py", line 115, in main
    return not tester.run()
               ~~~~~~~~~~^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/test/main.py", line 193, in run
    return self._run_tests(names)
           ~~~~~~~~~~~~~~~^^^^^^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/test/main.py", line 207, in _run_tests
    AutoInstall.install_everything()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 696, in install_everything
    package.install()
    ~~~~~~~~~~~~~~~^^
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 322, in install
    raise ValueError('No archives for {}-{} found'.format(self.pypi_name, self.version))
ValueError: No archives for lupa-2.2 found

But lupa is already at the latest version and there is just no wheel available. If I remove wheel=true, then our autoinstaller compiles it successfully instead of using the prebuilt wheel, so I guess we should do what works.

The webkitpy test suite fails, though. Several tests raise the following error:

602/2387] webkitpy.layout_tests.controllers.layout_test_finder_legacy_unittest.LayoutTestFinderTests.test_tempdir_relpath failed:              
  Traceback (most recent call last):
    File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder_legacy_unittest.py", line 1585, in test_tempdir_relpath
      with fs.mkdtemp() as path:
           ~~~~~~~~~~^^
    File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/common/system/filesystem.py", line 213, in __exit__
      self._filesystem.rmtree(self._directory_path)
      ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
    File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/common/system/filesystem.py", line 316, in rmtree
      shutil.rmtree(path, ignore_errors=True)
      ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib64/python3.13/shutil.py", line 763, in rmtree
      _rmtree_safe_fd(stack, onexc)
      ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
    File "/usr/lib64/python3.13/shutil.py", line 663, in _rmtree_safe_fd
      assert func is os.lstat
             ^^^^^^^^^^^^^^^^

This looks like a bug in the python standard library.

Then the test suite hangs after executing 2353 out of 2387 tests, but by this point I wasn't really expecting it to work, so OK. :)

My changes thus far:

diff --git a/Tools/Scripts/libraries/webkitflaskpy/webkitflaskpy/__init__.py b/Tools/Scripts/libraries/webkitflaskpy/webkitflaskpy/__init__.py
index f172250e7c3f..60d3ad9df042 100644
--- a/Tools/Scripts/libraries/webkitflaskpy/webkitflaskpy/__init__.py
+++ b/Tools/Scripts/libraries/webkitflaskpy/webkitflaskpy/__init__.py
@@ -52,11 +52,11 @@ version = Version(0, 7, 1)
 AutoInstall.register(Package('blinker', Version(1, 8, 2)))
 AutoInstall.register(Package('click', Version(8, 1, 7)))
 AutoInstall.register(Package('flask', Version(2, 3, 3)))
-AutoInstall.register(Package('hiredis', Version(3, 0, 0), wheel=True))
+AutoInstall.register(Package('hiredis', Version(3, 1, 0), wheel=True))
 AutoInstall.register(Package('itsdangerous', Version(1, 1, 0)))
-AutoInstall.register(Package('markupsafe', Version(2, 1, 5), pypi_name='MarkupSafe', wheel=True))
+AutoInstall.register(Package('markupsafe', Version(3, 0, 2), pypi_name='MarkupSafe', wheel=True))
 AutoInstall.register(Package('jinja2', Version(3, 1, 4), implicit_deps=['markupsafe']))
-AutoInstall.register(Package('lupa', Version(2, 2), wheel=True))
+AutoInstall.register(Package('lupa', Version(2, 2)))
 AutoInstall.register(Package('redis', Version(3, 5, 3)))
 AutoInstall.register(Package('sortedcontainers', Version(2, 4, 0)))
 AutoInstall.register(Package('werkzeug', Version(3, 0, 3)))
diff --git a/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py b/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py
index fd1dc5abed72..2bb03a2f05f3 100644
--- a/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py
+++ b/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py
@@ -53,7 +53,7 @@ except ImportError:
 version = Version(7, 0, 0)
 
 AutoInstall.register(Package('fasteners', Version(0, 15, 0)))
-AutoInstall.register(Package('markupsafe', Version(2, 1, 5), pypi_name='MarkupSafe', wheel=True))
+AutoInstall.register(Package('markupsafe', Version(3, 0, 2), pypi_name='MarkupSafe', wheel=True))
 AutoInstall.register(Package('jinja2', Version(3, 1, 4), implicit_deps=['markupsafe']))
 AutoInstall.register(Package('monotonic', Version(1, 5)))
 AutoInstall.register(Package('xmltodict', Version(0, 11, 0)))

[Sam Sneddon [:gsnedders]]

My suggestion would be to exclude reporelaypy, resultsdbpy, and webkitflaskpy from test-webkitpy (in Tools/Scripts/webkitpy/test/main.py); they're all server-side code and not that relevant in the short-term.

I thought we'd excluded them with some earlier release (3.12?), but I can't find such a thing in the history.

The failure in rmtree might be fixed by updating pyfakefs, because I think this will have been fixed as a result of https://github.com/pytest-dev/pyfakefs/commit/816d1142e1fb67bbc7526cd8f5f061e00ae8b5b4

[Radar WebKit Bug Importer]

<rdar://problem/142637619>

[Michael Catanzaro]

Pull request: https://github.com/WebKit/WebKit/pull/38804

[EWS]

Committed 288712@main (bef0ec308c65): <https://commits.webkit.org/288712@main>

Reviewed commits have been landed. Closing PR #38804 and removing active labels.