Bug 232403 – AutoInstall should have separate install directories per CPU architecture

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Alexey Proskuryakov]

Would like to make it possible to run scripts on macOS under Rosetta without deleting the auto installed directory (binaries there aren't FAT).

[Alexey Proskuryakov]

Created attachment 442635 [details]
proposed patch

[Jonathan Bedard]

Comment on attachment 442635 [details]
proposed patch

View in context: https://bugs.webkit.org/attachment.cgi?id=442635&action=review

> Tools/Scripts/webkitpy/__init__.py:35
> +AutoInstall.set_directory(os.path.join(libraries, 'autoinstalled', 'python-{}-{}'.format(sys.version_info[0], platform.machine())))

This looks good, the only issue is that we're going to force a re-install of everything because python-3 will now be python-3-arm64, but that seems like an acceptable cost.

[Alexey Proskuryakov]

Comment on attachment 442635 [details]
proposed patch

Looks like this broke the style bot somehow.

[Alexey Proskuryakov]

I'm told that this happened on this machine before, and a retry cured it (not just on EWS which re-ran on a different bot, but on webkit-misc itself).

[EWS]

Downloading setuptools-56.2.0...
Installing setuptools-56.2.0...
Installed setuptools-56.2.0!
Downloading setuptools-scm-5.0.2...
Installing setuptools-scm-5.0.2...
Installed setuptools-scm-5.0.2!
Downloading wheel-0.35.1...
Installing wheel-0.35.1...
Installed wheel-0.35.1!
Downloading six-1.15.0...
Installing six-1.15.0...
Installed six-1.15.0!
Downloading fasteners-0.15.0...
Installing fasteners-0.15.0...
Installed fasteners-0.15.0!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 66, in <module>
    from webkitscmpy import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/__init__.py", line 4, in <module>
    from webkitscmpy.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py", line 38, in <module>
    from webkitscmpy import remote, Commit, Contributor, Version
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/__init__.py", line 24, in <module>
    from webkitscmpy.remote.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py", line 25, in <module>
    import fasteners
ModuleNotFoundError: No module named 'fasteners'

[Alexey Proskuryakov]

Comment on attachment 442635 [details]
proposed patch

Same thing here on webkit-cq-03 apparently, and I feel like it will happen in other places. Not new to this patch though, it just stress tests existing behavior.

[EWS]

Downloading monotonic-1.5.0...
Installing monotonic-1.5.0...
Installed monotonic-1.5.0!
Downloading requests-2.24.0...
Installing requests-2.24.0...
Installed requests-2.24.0!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 66, in <module>
    from webkitscmpy import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/__init__.py", line 4, in <module>
    from webkitscmpy.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py", line 38, in <module>
    from webkitscmpy import remote, Commit, Contributor, Version
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/__init__.py", line 24, in <module>
    from webkitscmpy.remote.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py", line 29, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

[EWS]

Downloading setuptools-56.2.0...
Installing setuptools-56.2.0...
Installed setuptools-56.2.0!
Downloading setuptools-scm-5.0.2...
Installing setuptools-scm-5.0.2...
Installed setuptools-scm-5.0.2!
Downloading wheel-0.35.1...
Installing wheel-0.35.1...
Installed wheel-0.35.1!
Downloading six-1.15.0...
Installing six-1.15.0...
Installed six-1.15.0!
Downloading fasteners-0.15.0...
Installing fasteners-0.15.0...
Installed fasteners-0.15.0!
Downloading monotonic-1.5.0...
Installing monotonic-1.5.0...
Installed monotonic-1.5.0!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 66, in <module>
    from webkitscmpy import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/__init__.py", line 4, in <module>
    from webkitscmpy.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py", line 38, in <module>
    from webkitscmpy import remote, Commit, Contributor, Version
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/__init__.py", line 24, in <module>
    from webkitscmpy.remote.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py", line 25, in <module>
    import fasteners
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/fasteners/__init__.py", line 23, in <module>
    from fasteners.lock import locked  # noqa
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/fasteners/lock.py", line 24, in <module>
    from fasteners import _utils
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/fasteners/_utils.py", line 39, in <module>
    from monotonic import monotonic as now  # noqa
ModuleNotFoundError: No module named 'monotonic'

[EWS]

Downloading requests-2.24.0...
Installing requests-2.24.0...
Installed requests-2.24.0!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 66, in <module>
    from webkitscmpy import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/__init__.py", line 4, in <module>
    from webkitscmpy.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py", line 38, in <module>
    from webkitscmpy import remote, Commit, Contributor, Version
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/__init__.py", line 24, in <module>
    from webkitscmpy.remote.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py", line 29, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

[EWS]

Downloading urllib3-1.25.10...
Installing urllib3-1.25.10...
Installed urllib3-1.25.10!
Downloading chardet-3.0.4...
Installing chardet-3.0.4...
Installed chardet-3.0.4!
Downloading certifi-2020.6.20...
Installing certifi-2020.6.20...
Installed certifi-2020.6.20!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 66, in <module>
    from webkitscmpy import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/__init__.py", line 4, in <module>
    from webkitscmpy.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py", line 38, in <module>
    from webkitscmpy import remote, Commit, Contributor, Version
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/__init__.py", line 24, in <module>
    from webkitscmpy.remote.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py", line 29, in <module>
    import requests
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/requests/__init__.py", line 120, in <module>
    from . import utils
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/requests/utils.py", line 25, in <module>
    from . import certs
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/requests/certs.py", line 15, in <module>
    from certifi import where
ModuleNotFoundError: No module named 'certifi'

[EWS]

Downloading idna-2.10.0...
Installing idna-2.10.0...
Installed idna-2.10.0!
Downloading PySocks-1.7.1...
Installing PySocks-1.7.1...
Installed PySocks-1.7.1!
Downloading xmltodict-0.11.0...
Installing xmltodict-0.11.0...
Installed xmltodict-0.11.0!
Downloading mock-4.0.3...
Installing mock-4.0.3...
Installed mock-4.0.3!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 67, in <module>
    from webkitscmpy import mocks
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/__init__.py", line 3, in <module>
    from webkitscmpy.mocks import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/__init__.py", line 3, in <module>
    from webkitscmpy.mocks.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/svn.py", line 28, in <module>
    from mock import patch
ModuleNotFoundError: No module named 'mock'

[EWS]

Downloading whichcraft-0.6.1...
Installing whichcraft-0.6.1...
Installed whichcraft-0.6.1!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 67, in <module>
    from webkitscmpy import mocks
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/__init__.py", line 3, in <module>
    from webkitscmpy.mocks import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/__init__.py", line 4, in <module>
    from webkitscmpy.mocks.local.git import Git
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/git.py", line 34, in <module>
    from webkitscmpy.program.canonicalize.committer import main as committer_main
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/__init__.py", line 28, in <module>
    from .blame import Blame
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/blame.py", line 28, in <module>
    from webkitscmpy.program.command import FilteredCommand
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/command.py", line 32, in <module>
    from whichcraft import which
ModuleNotFoundError: No module named 'whichcraft'

[EWS]

Downloading urllib3-1.25.10...
Installing urllib3-1.25.10...
Installed urllib3-1.25.10!
Downloading chardet-3.0.4...
Installing chardet-3.0.4...
Installed chardet-3.0.4!
Downloading certifi-2020.6.20...
Installing certifi-2020.6.20...
Installed certifi-2020.6.20!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 66, in <module>
    from webkitscmpy import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/__init__.py", line 4, in <module>
    from webkitscmpy.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py", line 38, in <module>
    from webkitscmpy import remote, Commit, Contributor, Version
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/__init__.py", line 24, in <module>
    from webkitscmpy.remote.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py", line 29, in <module>
    import requests
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/requests/__init__.py", line 120, in <module>
    from . import utils
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/requests/utils.py", line 25, in <module>
    from . import certs
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/requests/certs.py", line 15, in <module>
    from certifi import where
ModuleNotFoundError: No module named 'certifi'

[EWS]

Committed r284964 (243615@main): <https://commits.webkit.org/243615@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 442635 [details].

[Radar WebKit Bug Importer]

<rdar://problem/84733102>