Bug 261113 – Fix (gone-in-Python-3.12) distutils calls in setup.py from AutoInstaller (migrate to a venv?)

• Component: Tools / Tests
• Status: NEW

[Sam Sneddon [:gsnedders]]

In Python 3.12, distutils is gone.

Well, sorta. setuptools nowadays provides a distutils package to provide backwards compatibility—but `import distutils` will fail if `setuptools` hasn't been imported first, which means the AutoInstaller will fail to install a variety of packages.

Specifically:

Installing MarkupSafe-1.1.1...
Traceback (most recent call last):
  File "/tmp/markupsafe-311492/MarkupSafe-1.1.1/setup.py", line 6, in <module>
    from distutils.errors import CCompilerError
ModuleNotFoundError: No module named 'distutils'

Installing zope-interface-5.1.0...
Traceback (most recent call last):
  File "/tmp/zope.interface-311492/zope.interface-5.1.0/setup.py", line 25, in <module>
    from distutils.errors import CCompilerError
ModuleNotFoundError: No module named 'distutils'

Installing selenium-3.141.0...
Traceback (most recent call last):
  File "/tmp/selenium-311492/selenium-3.141.0/setup.py", line 20, in <module>
    from distutils.command.install import INSTALL_SCHEMES
ModuleNotFoundError: No module named 'distutils'

Installing jeepney-0.7.1...
Traceback (most recent call last):
  File "/tmp/jeepney-311492/jeepney-0.7.1/setup.py", line 4, in <module>
    from distutils.core import setup
ModuleNotFoundError: No module named 'distutils'

Installing cffi-1.15.1...
Traceback (most recent call last):
  File "/tmp/cffi-311492/cffi-1.15.1/setup.py", line 148, in <module>
    ask_supports_thread()
  File "/tmp/cffi-311492/cffi-1.15.1/setup.py", line 75, in ask_supports_thread
    config = get_config()
             ^^^^^^^^^^^^
  File "/tmp/cffi-311492/cffi-1.15.1/setup.py", line 68, in get_config
    from distutils.core import Distribution
ModuleNotFoundError: No module named 'distutils'

Installing entrypoints-0.3.0...
Traceback (most recent call last):
  File "/tmp/entrypoints-311492/entrypoints-0.3/setup.py", line 4, in <module>
    from distutils.core import setup
ModuleNotFoundError: No module named 'distutils'

Some of these have upstream releases which fix this, but some don't. And with setup.py deprecated, many of these packages are moving further away from what we support in the AutoInstaller today (c.f. bug 261082).

There's a few possible paths forward, in decreasing levels of hackiness:

 * Move to relying on pip to download/build/install packages, and get out of all of those games entirely.
 * Implement a PEP 517 compliant builder (i.e., fix bug 261082).
 * Hardcode usage of (the default PEP 517 builder) setuptools.build_meta.__legacy__ to build all packages (which will solve the above problems, as the setuptools legacy builder keeps them working).
 * Tokenize/untokenize setup.py files to add a setuptools import at the top.

[Sam Sneddon [:gsnedders]]

(In reply to Sam Sneddon [:gsnedders] from comment #0)
> There's a few possible paths forward, in decreasing levels of hackiness:

I did, of course, mean "increasing levels of hackiness".

[Radar WebKit Bug Importer]

<rdar://problem/115287945>

[Sam Sneddon [:gsnedders]]

Ahhhh, I'd failed to realised how this worked with setuptools's vendored distutils (via _distutils_hack).

So setuptools installs a `distutils-precedence.pth`, which makes setuptools' provided distutils get exposed as distutils.

Thus, naturally, you'd expect this to work, but pth files are only processed for site directories (i.e., site-packages directories), and _not_ for any paths added to sys.path via the PYTHONPATH environment variable (as we use to invoke setup.py).

This feels like another reason to move to using a venv for the AutoInstaller.

See https://github.com/web-platform-tests/wpt/blob/9eb816722861f5f12f46738c9591f4c88aa0e995/tools/wpt/virtualenv.py#L99-L137 for code that roughly self-activates a venv within the running interpreter.

I guess there's no reason why we can't just do this on Python 3 in the near future (where the venv module already exists), though it will trigger reinstalls (again). Might be worth to land alongside Bug 224669 to avoid multiple cases of reinstalling the world.

[Fujii Hironori]

I have no problem with Windows Python 3.12.6 today.
If I recall correctly, I have the same problem before.
Did something have changed?

[Sam Sneddon [:gsnedders]]

(In reply to Fujii Hironori from comment #4)
> I have no problem with Windows Python 3.12.6 today.
> If I recall correctly, I have the same problem before.
> Did something have changed?

If you have an appropriate setuptools version installed at the system level it will work, I believe.

[Sam Sneddon [:gsnedders]]

One other obvious option for setup.py specifically is to just move to using a shim to invoke setup.py, similar to https://github.com/pypa/pip/blob/23.0.1/src/pip/_internal/utils/setuptools_build.py

This would also provide us to start invoking bdist_wheel by default, thus unifying our build/install pipeline, because we'd then always be installing from a wheel.