Bug 260877 – Cannot run test-webkitpy: errors when installing many pypi dependencies with python 3.12

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Michael Catanzaro]

I'd like to be able to run test-webkitpy locally using python 3.12, but can't because pylint fails to install:

Downloading pylint-2.6.0...
Installing pylint-2.6.0...
error in pylint setup command: 'python_requires' must be a string containing valid version specifiers; Invalid specifier: '>=3.5.*'
Failed to install pylint-2.6.0!

Fortunately, this is already fixed in the latest version of pylint. Then, next problem:

Downloading beautifulsoup4-4.9.3...
Installing beautifulsoup4-4.9.3...
error in beautifulsoup4 setup command: use_2to3 is invalid.
Failed to install beautifulsoup4-4.9.3!

This is also conveniently already fixed in the latest version of beautifulsoup4. Next problem:

Downloading Genshi-0.7.3...
Installing Genshi-0.7.3...
/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'features'
  warnings.warn(msg)
/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'convert_2to3_doctests'
  warnings.warn(msg)
/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'use_2to3_fixers'
  warnings.warn(msg)
error in Genshi setup command: use_2to3 is invalid.

If you guessed that it was already fixed in the latest version of Genshi, you'd unsurprisingly be correct.

Next problem occurs when compiling lupa. There are a bunch of GCC warnings and then many errors. The first error is:

lupa/_lupa.c: In function ‘__Pyx_PyInt_AddObjC’:
lupa/_lupa.c:34443:51: error: ‘PyLongObject’ {aka ‘struct _longobject’} has no member named ‘ob_digit’
34443 |         const digit* digits = ((PyLongObject*)op1)->ob_digit;
      |                                                   ^~

All of the errors are fixed in the latest version, lupa 2.0. However, it's not fixed in lupa 1.14.1. I was a little nervous about the major version increase, so I tried to find any documentation of incompatibilities between 1.x and 2.0, but couldn't immediately find anything. Anyway, it seems to work, so let's assume it's good if EWS is happy.

Next problem is a very long backtrace that ends in:

    File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/html5lib/_inputstream.py", line 4, in <module>
      from six.moves import http_client, urllib
      from six.moves import http_client, urllib
  ModuleNotFoundError: No module named 'six.moves'
  ModuleNotFoundError: No module named 'six.moves'

At first I feared this would be a bug that a simple version upgrade would not fix, but nope, just need to upgrade six and the problem goes away.

Next problem, a backtrace that ends in:

  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_finder_legacy_unittest.py", line 31, in <module>
    from pyfakefs.fake_filesystem_unittest import TestCaseMixin
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/pyfakefs/fake_filesystem_unittest.py", line 73, in <module>
    from pyfakefs import fake_pathlib
  File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/pyfakefs/fake_pathlib.py", line 139, in <module>
    flavour = pathlib._Flavour if pathlib else object
              ^^^^^^^^^^^^^^^^
AttributeError: module 'pathlib' has no attribute '_Flavour

This is fixed by upgrading pyfakefs.

Finally, I am able to run test-webkitpy! 10 tests fail due to some assertion functions that were renamed. They all look like one or the other of these two examples:

[1231/2225] webkitpy.port.xvfbdriver_unittest.XvfbDriverTest.test_xvfb_not_replying erred:                            
  Traceback (most recent call last):
    File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/port/xvfbdriver_unittest.py", line 99, in test_xvfb_not_replying
      self.assertRaisesRegexp(RuntimeError, 'Unable to start Xvfb display server', driver.start, False, [])
      ^^^^^^^^^^^^^^^^^^^^^^^
  AttributeError: 'XvfbDriverTest' object has no attribute 'assertRaisesRegexp'. Did you mean: 'assertRaisesRegex'?
  
[1760/2225] webkitpy.w3c.test_converter_unittest.W3CTestConverterTest.test_convert_for_webkit_harness_only erred:
  Traceback (most recent call last):
    File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/w3c/test_converter_unittest.py", line 111, in test_convert_for_webkit_harness_only
      self.verify_test_harness_paths(converted[2], 1, 1)
    File "/home/mcatanzaro/Projects/WebKit/Tools/Scripts/webkitpy/w3c/test_converter_unittest.py", line 368, in verify_test_harness_paths
      self.assertEquals(len(converted.findAll(src=orig_path_pattern)), num_src_paths, 'testharness src path should not have been converted')
      ^^^^^^^^^^^^^^^^^
  AttributeError: 'W3CTestConverterTest' object has no attribute 'assertEquals'. Did you mean: 'assertEqual'?

According to https://docs.python.org/3.12/whatsnew/3.12.html these were removed from python itself since they were deprecated since python 3.2. Maybe we had deprecation warnings disabled? Easy enough to fix as it only requires renaming things.

Finally, I see some warnings caused by bug #260729, which do not cause any test failures and which we can address separately. I also see one remaining unfixed warning:

[0/2225] reporelaypy.tests.checkout_unittest.CheckoutUnittest.test_constructor_no_sentinal/usr/lib/python3.12/site-packages/bs4/builder/__init__.py:545: XMLParsedAsHTMLWarning: It looks like you're parsing an XML document using an HTML parser. If this really is an HTML document (maybe it's XHTML?), you can ignore or filter this warning. If it's XML, you should know that using an XML parser will be more reliable. To parse this document as XML, make sure you have the lxml package installed, and pass the keyword argument `features="xml"` into the BeautifulSoup constructor.
  warnings.warn(

But I don't want to investigate this one, because it's not an error and all the tests pass, and I'm tired now. :)

[Michael Catanzaro]

Pull request: https://github.com/WebKit/WebKit/pull/17201

[Michael Catanzaro]

I decided to split the assertion fixes out into bug #260878.

[James Howard]

Looks like this change broke run-webkit-tests for people using 3.8 <= python < 3.12. At least, it broke it for me.

I get this error when running run-webkit-tests with a homebrew install of python3.11 (I also get the same error when using Xcode 14.3's built in python3):

jahoward@jhm1p ~/g/e/WebKit (main)> Tools/Scripts/run-webkit-tests 
Downloading setuptools-68.1.2...
Installing setuptools-68.1.2...
Installed setuptools-68.1.2!
Downloading setuptools-scm-5.0.2...
Installing setuptools-scm-5.0.2...
Installed setuptools-scm-5.0.2!
Downloading wheel-0.35.1...
Installing wheel-0.35.1...
Installed wheel-0.35.1!
Downloading packaging-21.3.0...
Installing packaging-21.3.0...
Installed packaging-21.3.0!
Downloading attrs-20.3.0...
Installing attrs-20.3.0...
Installed attrs-20.3.0!
Downloading html5lib-1.1.0...
Installing html5lib-1.1.0...
Installed html5lib-1.1.0!
Downloading six-1.15.0...
Installing six-1.15.0...
Installed six-1.15.0!
Downloading webencodings-0.5.1...
Installing webencodings-0.5.1...
Installed webencodings-0.5.1!
Downloading beautifulsoup4-4.9.3...
Installing beautifulsoup4-4.9.3...
error in beautifulsoup4 setup command: use_2to3 is invalid.
Failed to install beautifulsoup4-4.9.3!
Traceback (most recent call last):
  File "/Users/jahoward/git/ext/WebKit/Tools/Scripts/run-webkit-tests", line 35, in <module>
    from webkitpy.layout_tests.run_webkit_tests import main
  File "/Users/jahoward/git/ext/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 39, in <module>
    from webkitpy.layout_tests.controllers.manager import Manager
  File "/Users/jahoward/git/ext/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 54, in <module>
    from webkitpy.layout_tests.controllers.layout_test_runner import LayoutTestRunner
  File "/Users/jahoward/git/ext/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 40, in <module>
    from webkitpy.layout_tests.controllers import single_test_runner
  File "/Users/jahoward/git/ext/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 40, in <module>
    from webkitpy.w3c.test_parser import TestParser
  File "/Users/jahoward/git/ext/WebKit/Tools/Scripts/webkitpy/w3c/test_parser.py", line 34, in <module>
    from webkitpy.thirdparty.BeautifulSoup import BeautifulSoup as Parser
  File "/Users/jahoward/git/ext/WebKit/Tools/Scripts/webkitpy/thirdparty/BeautifulSoup.py", line 27, in <module>
    import bs4
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1138, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1078, in _find_spec
  File "/Users/jahoward/git/ext/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 646, in find_spec
    loader = cls.find_module(fullname, path=path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jahoward/git/ext/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 660, in find_module
    cls.install(name)
  File "/Users/jahoward/git/ext/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 635, in install
    return all([to_install.install() for to_install in packages])
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jahoward/git/ext/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 635, in <listcomp>
    return all([to_install.install() for to_install in packages])
                ^^^^^^^^^^^^^^^^^^^^
  File "/Users/jahoward/git/ext/WebKit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 339, in install
    subprocess.check_call(
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/homebrew/opt/python@3.11/bin/python3.11', '/var/folders/5t/czx4p3rs32v7lr4q31zy5pww0000gn/T/bs4-83705/beautifulsoup4-4.9.3/setup.py', 'install', '--home=/Users/jahoward/git/ext/WebKit/Tools/Scripts/libraries/autoinstalled/python-3-arm64', '--root=/', '--prefix=', '--install-lib=/Users/jahoward/git/ext/WebKit/Tools/Scripts/libraries/autoinstalled/python-3-arm64', '--install-scripts=/Users/jahoward/git/ext/WebKit/Tools/Scripts/libraries/autoinstalled/python-3-arm64/bin', '--install-data=/Users/jahoward/git/ext/WebKit/Tools/Scripts/libraries/autoinstalled/python-3-arm64/data', '--install-headers=/Users/jahoward/git/ext/WebKit/Tools/Scripts/libraries/autoinstalled/python-3-arm64/headers']' returned non-zero exit status 1.



---

If I revert cb4faa2a3ee38601f41c1d60fcac002a83cec4ff, all of the packages install correctly.

The issue seems to be the change to pin setuptools to version (68, 1, 2) for all python versions >= 3.8. Previously we were on setuptools (56, 0, 0). beautifulsoup4-4.9.3 can't install correctly if you have setuptools (58, 0, 0) or higher, because it depends on use_2to3, which was removed in setuptools 58 and newer.

[Michael Catanzaro]

Well you're in the wrong bug, because this change hasn't landed yet, but I created bug #260889 for this.

[James Howard]

Oh my bad, I thought I followed the link from cb4faa2a3ee38601f41c1d60fcac002a83cec4ff to ttps://bugs.webkit.org/show_bug.cgi?id=260726 but grabbed the wrong one. Will transfer my comment over.

[Sam Sneddon [:gsnedders]]

I already had a work-in-progress branch at https://github.com/gsnedders/WebKit/tree/py312 which starts to fix some of this. I realise you've started to land some of this which presumably conflicts with my branch,  but hopefully that provides some help for the rest of what needs done.

(Sorry for not actually filing a bug for this, and you ending up duplicating work here!)

[Michael Catanzaro]

If you had reported a bug, I almost certainly would not have noticed and wound up duplicating the work anyway. :P Oh well. At least it wasn't too hard as basically every problem I faced was solved by upgrading dependencies (except for the assertions that were renamed).

[Radar WebKit Bug Importer]

<rdar://problem/115010539>

[Sam Sneddon [:gsnedders]]

Pull request: https://github.com/WebKit/WebKit/pull/18565

[EWS]

Committed 268856@main (46e33611bc5c): <https://commits.webkit.org/268856@main>

Reviewed commits have been landed. Closing PR #18565 and removing active labels.