Bug 228628 – [WPE][GTK] Clean build broken with python3 after r280382

• Component: CMake
• Status: RESOLVED FIXED

[Carlos Alberto Lopez Perez]

Not sure still what causes this, it looks like the autoinstall python magic is not working as expected with python3 here.

This was detected on the EWS: https://ews-build.webkit.org/#/builders/36/builds/41166

Steps to reproduce:

$ rm -fr WebKitBuild/GTK/Release/
$ rm -fr Tools/Scripts/libraries/autoinstalled/
$ Tools/Scripts/build-webkit --gtk --cmakeargs='-DPYTHON_EXECUTABLE=/usr/bin/python3'


Output:

FAILED: Source/JavaScriptCore/CMakeFiles/JavaScriptCore-build-revision 
cd /app/webkit && /usr/bin/python3 /app/webkit/Tools/glib/apply-build-revision-to-files.py /app/webkit/WebKitBuild/Release/Source/JavaScriptCore/javascriptcoregtk-4.1.pc
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 908, in _find_spec
AttributeError: type object 'AutoInstall' has no attribute 'find_spec'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/usr/lib/python3.8/http/client.py", line 1417, in connect
    super().connect()
  File "/usr/lib/python3.8/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/usr/lib/python3.8/socket.py", line 787, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/webkit/Tools/glib/apply-build-revision-to-files.py", line 25, in <module>
    from webkitpy.common.checkout.scm.detection import SCMDetector  # nopep8
  File "/app/webkit/Tools/Scripts/webkitpy/__init__.py", line 78, in <module>
    import webkitscmpy
  File "/app/webkit/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 57, in <module>
    from webkitscmpy.commit import Commit
  File "/app/webkit/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/commit.py", line 24, in <module>
    import six
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 910, in _find_spec
  File "<frozen importlib._bootstrap>", line 884, in _find_spec_legacy
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 598, in find_module
    cls.install(name)
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 582, in install
    return all([to_install.install() for to_install in packages])
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 582, in <listcomp>
    return all([to_install.install() for to_install in packages])
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 261, in install
    AutoInstall.install('setuptools')
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 582, in install
    return all([to_install.install() for to_install in packages])
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 582, in <listcomp>
    return all([to_install.install() for to_install in packages])
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 272, in install
    if not self.archives():
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 181, in archives
    response = AutoInstall._request('https://{}/{}'.format(AutoInstall.index, path))
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 414, in _request
    return urlopen(url, timeout=cls.timeout, context=context)
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1397, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.8/urllib/request.py", line 1357, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
[1172/6979] Generating ../../JavaScriptCore/DerivedSources/LLIntDesiredSettings.h
ninja: build stopped: subcommand failed.

[Carlos Alberto Lopez Perez]

I attached a patch on bug 228629 to fix the build with python2, which should be enough for the moment to have the EWS working back

[Carlos Alberto Lopez Perez]

As far as I can see, the issue (apparently) comes because the class AutoInstall in file Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py does not implement the method find_spec() which is new in python 3.4 

See: https://docs.python.org/3/library/importlib.html#importlib.abc.MetaPathFinder.find_spec

What puzzles me is that the script works if I call it directly with python3, it only fails if is called from the build :?

[Philippe Normand]

The network issue happens because the build sandbox has no network access, unless either sccache or icecc is enabled. That would be easy to fix, but then...

... the webkitpy auto install black magic doesn't seem to work well in case it's triggered from concurrent processes. I tried to debug this for a few hours, gave up...

... I'll submit a patch getting rid of webkitpy in the glib/apply-build-revision-to-files.py. We don't need webkitpy for this, we can call svnversion, git svn info, or git log -1... depending on the kind of checkout...

[Philippe Normand]

Created attachment 435583 [details]
Patch

[EWS]

Committed r281077 (240537@main): <https://commits.webkit.org/240537@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 435583 [details].

[WebKit Commit Bot]

Re-opened since this is blocked by bug 229137

[Philippe Normand]

Created attachment 435598 [details]
[fast-cq] Patch

[EWS]

Committed r281089 (240549@main): <https://commits.webkit.org/240549@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 435598 [details].