Bug 230211 – [webkitpy] AutoInstall not loading default OpenSSL certificates

• Status: NEW
• Component: Tools / Tests

[Philippe Normand]

The issue:

Downloading setuptools-56.2.0...
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>
Failed to download setuptools, retrying
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>
Failed to download setuptools, retrying
Failed to install setuptools-56.2.0!
Traceback (most recent call last):
  File "Tools/Scripts/git-webkit", line 28, in <module>
    from webkitpy.common.config.committers import CommitterList
  File "/home/buildbot/igalia-jsc32-armv7-ews/JSC-ARMv7-32bits-Build-EWS/build/Tools/Scripts/webkitpy/__init__.py", line 78, in <module>
    import webkitscmpy
  File "/home/buildbot/igalia-jsc32-armv7-ews/JSC-ARMv7-32bits-Build-EWS/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 57, in <module>
    from webkitscmpy.commit import Commit
  File "/home/buildbot/igalia-jsc32-armv7-ews/JSC-ARMv7-32bits-Build-EWS/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/commit.py", line 24, in <module>
    import six
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 963, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 906, in _find_spec
  File "/home/buildbot/igalia-jsc32-armv7-ews/JSC-ARMv7-32bits-Build-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 602, in find_spec
    loader = cls.find_module(fullname, path=path)
  File "/home/buildbot/igalia-jsc32-armv7-ews/JSC-ARMv7-32bits-Build-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 616, in find_module
    cls.install(name)
  File "/home/buildbot/igalia-jsc32-armv7-ews/JSC-ARMv7-32bits-Build-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 591, in install
    return all([to_install.install() for to_install in packages])
  File "/home/buildbot/igalia-jsc32-armv7-ews/JSC-ARMv7-32bits-Build-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 591, in <listcomp>
    return all([to_install.install() for to_install in packages])
  File "/home/buildbot/igalia-jsc32-armv7-ews/JSC-ARMv7-32bits-Build-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 273, in install
    AutoInstall.install('setuptools')
  File "/home/buildbot/igalia-jsc32-armv7-ews/JSC-ARMv7-32bits-Build-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 591, in install
    return all([to_install.install() for to_install in packages])
  File "/home/buildbot/igalia-jsc32-armv7-ews/JSC-ARMv7-32bits-Build-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 591, in <listcomp>
    return all([to_install.install() for to_install in packages])
  File "/home/buildbot/igalia-jsc32-armv7-ews/JSC-ARMv7-32bits-Build-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 296, in install
    archive.unpack(temp_location)
  File "/home/buildbot/igalia-jsc32-armv7-ews/JSC-ARMv7-32bits-Build-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/autoinstall.py", line 126, in unpack
    raise IOError('Failed to find archive at {}'.format(self.path))
OSError: Failed to find archive at /home/buildbot/igalia-jsc32-armv7-ews/JSC-ARMv7-32bits-Build-EWS/build/Tools/Scripts/libraries/autoinstalled/python-3/setuptools-56.2.tar.gz

[Philippe Normand]

I landed r282322 as an emergency fix for EWS/etc bots. I am not sure why this custom cacert is needed in the first place.

Is it due to a Python limitation on Apple platforms? In linux we can usually assume the host has valid root certificates, otherwise, maybe we could bootstrap 'certifi' before installing setuptools? That would likely avoid the need for a custom cert in the first place.

[Jonathan Bedard]

A large number of Apple's bots rely on a different pypi source for their packages, and we have a bit of a chicken-egg problem with getting that cacert. Basically, the cacert in webkitcorepy isn't just for pypi.org.

Will get it updated, this is the cost of owning the cacert....

[Michael Catanzaro]

Might want to rename the file to avoid confusion with http://www.cacert.org/ ;)

[Radar WebKit Bug Importer]

<rdar://problem/83298684>

[Jonathan Bedard]

Created attachment 438742 [details]
Move Linux special case to webkitpy

[Jonathan Bedard]

Created attachment 438746 [details]
Add root CA

[Jonathan Bedard]

I think we'd prefer the second one, but if we aren't going to do that, we should do the first so that the Linux special case is in webkitpy and doesn't apply to other callers of webkitpy

[Philippe Normand]

I'll try to test tomorrow.

[Sam Sneddon [:gsnedders]]

265530@main further changed the code here.

What we should do is probably _always_ load the default OpenSSL certificates (via SSLContext.load_default_certs) and then load our own (via SSLContext.load_verify_locations).

Longer-term, especially on Python 3.10 and later, we may well want to consider using truststore (https://pypi.org/project/truststore/) to create a context using the system trust store, thus avoiding the problems entirely.