Bug 232403

Summary: AutoInstall should have separate install directories per CPU architecture
Product: WebKit Reporter: Alexey Proskuryakov <ap>
Component: Tools / TestsAssignee: Alexey Proskuryakov <ap>
Status: RESOLVED FIXED    
Severity: Normal CC: jbedard, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=232573
Attachments:
Description Flags
proposed patch none

Description Alexey Proskuryakov 2021-10-27 14:06:04 PDT 
Would like to make it possible to run scripts on macOS under Rosetta without deleting the auto installed directory (binaries there aren't FAT).
Comment 1 Alexey Proskuryakov 2021-10-27 14:07:12 PDT 
Created attachment 442635 [details]
proposed patch
Comment 2 Jonathan Bedard 2021-10-27 14:15:46 PDT 
Comment on attachment 442635 [details]
proposed patch

View in context: https://bugs.webkit.org/attachment.cgi?id=442635&action=review

> Tools/Scripts/webkitpy/__init__.py:35
> +AutoInstall.set_directory(os.path.join(libraries, 'autoinstalled', 'python-{}-{}'.format(sys.version_info[0], platform.machine())))

This looks good, the only issue is that we're going to force a re-install of everything because python-3 will now be python-3-arm64, but that seems like an acceptable cost.
Comment 3 Alexey Proskuryakov 2021-10-27 14:17:42 PDT 
Comment on attachment 442635 [details]
proposed patch

Looks like this broke the style bot somehow.
Comment 4 Alexey Proskuryakov 2021-10-27 14:33:53 PDT 
I'm told that this happened on this machine before, and a retry cured it (not just on EWS which re-ran on a different bot, but on webkit-misc itself).
Comment 5 EWS 2021-10-27 14:34:03 PDT 
Downloading setuptools-56.2.0...
Installing setuptools-56.2.0...
Installed setuptools-56.2.0!
Downloading setuptools-scm-5.0.2...
Installing setuptools-scm-5.0.2...
Installed setuptools-scm-5.0.2!
Downloading wheel-0.35.1...
Installing wheel-0.35.1...
Installed wheel-0.35.1!
Downloading six-1.15.0...
Installing six-1.15.0...
Installed six-1.15.0!
Downloading fasteners-0.15.0...
Installing fasteners-0.15.0...
Installed fasteners-0.15.0!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 66, in <module>
    from webkitscmpy import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/__init__.py", line 4, in <module>
    from webkitscmpy.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py", line 38, in <module>
    from webkitscmpy import remote, Commit, Contributor, Version
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/__init__.py", line 24, in <module>
    from webkitscmpy.remote.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py", line 25, in <module>
    import fasteners
ModuleNotFoundError: No module named 'fasteners'
Comment 6 Alexey Proskuryakov 2021-10-27 14:58:57 PDT 
Comment on attachment 442635 [details]
proposed patch

Same thing here on webkit-cq-03 apparently, and I feel like it will happen in other places. Not new to this patch though, it just stress tests existing behavior.
Comment 7 EWS 2021-10-27 15:00:01 PDT 
Downloading monotonic-1.5.0...
Installing monotonic-1.5.0...
Installed monotonic-1.5.0!
Downloading requests-2.24.0...
Installing requests-2.24.0...
Installed requests-2.24.0!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 66, in <module>
    from webkitscmpy import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/__init__.py", line 4, in <module>
    from webkitscmpy.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py", line 38, in <module>
    from webkitscmpy import remote, Commit, Contributor, Version
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/__init__.py", line 24, in <module>
    from webkitscmpy.remote.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py", line 29, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'
Comment 8 EWS 2021-10-27 15:06:42 PDT 
Downloading setuptools-56.2.0...
Installing setuptools-56.2.0...
Installed setuptools-56.2.0!
Downloading setuptools-scm-5.0.2...
Installing setuptools-scm-5.0.2...
Installed setuptools-scm-5.0.2!
Downloading wheel-0.35.1...
Installing wheel-0.35.1...
Installed wheel-0.35.1!
Downloading six-1.15.0...
Installing six-1.15.0...
Installed six-1.15.0!
Downloading fasteners-0.15.0...
Installing fasteners-0.15.0...
Installed fasteners-0.15.0!
Downloading monotonic-1.5.0...
Installing monotonic-1.5.0...
Installed monotonic-1.5.0!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 66, in <module>
    from webkitscmpy import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/__init__.py", line 4, in <module>
    from webkitscmpy.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py", line 38, in <module>
    from webkitscmpy import remote, Commit, Contributor, Version
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/__init__.py", line 24, in <module>
    from webkitscmpy.remote.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py", line 25, in <module>
    import fasteners
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/fasteners/__init__.py", line 23, in <module>
    from fasteners.lock import locked  # noqa
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/fasteners/lock.py", line 24, in <module>
    from fasteners import _utils
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/fasteners/_utils.py", line 39, in <module>
    from monotonic import monotonic as now  # noqa
ModuleNotFoundError: No module named 'monotonic'
Comment 9 EWS 2021-10-27 15:28:33 PDT 
Downloading requests-2.24.0...
Installing requests-2.24.0...
Installed requests-2.24.0!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 66, in <module>
    from webkitscmpy import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/__init__.py", line 4, in <module>
    from webkitscmpy.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py", line 38, in <module>
    from webkitscmpy import remote, Commit, Contributor, Version
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/__init__.py", line 24, in <module>
    from webkitscmpy.remote.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py", line 29, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'
Comment 10 EWS 2021-10-27 15:31:21 PDT 
Downloading urllib3-1.25.10...
Installing urllib3-1.25.10...
Installed urllib3-1.25.10!
Downloading chardet-3.0.4...
Installing chardet-3.0.4...
Installed chardet-3.0.4!
Downloading certifi-2020.6.20...
Installing certifi-2020.6.20...
Installed certifi-2020.6.20!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 66, in <module>
    from webkitscmpy import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/__init__.py", line 4, in <module>
    from webkitscmpy.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py", line 38, in <module>
    from webkitscmpy import remote, Commit, Contributor, Version
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/__init__.py", line 24, in <module>
    from webkitscmpy.remote.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py", line 29, in <module>
    import requests
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/requests/__init__.py", line 120, in <module>
    from . import utils
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/requests/utils.py", line 25, in <module>
    from . import certs
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/requests/certs.py", line 15, in <module>
    from certifi import where
ModuleNotFoundError: No module named 'certifi'
Comment 11 EWS 2021-10-27 16:10:08 PDT 
Downloading idna-2.10.0...
Installing idna-2.10.0...
Installed idna-2.10.0!
Downloading PySocks-1.7.1...
Installing PySocks-1.7.1...
Installed PySocks-1.7.1!
Downloading xmltodict-0.11.0...
Installing xmltodict-0.11.0...
Installed xmltodict-0.11.0!
Downloading mock-4.0.3...
Installing mock-4.0.3...
Installed mock-4.0.3!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 67, in <module>
    from webkitscmpy import mocks
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/__init__.py", line 3, in <module>
    from webkitscmpy.mocks import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/__init__.py", line 3, in <module>
    from webkitscmpy.mocks.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/svn.py", line 28, in <module>
    from mock import patch
ModuleNotFoundError: No module named 'mock'
Comment 12 EWS 2021-10-27 16:12:14 PDT 
Downloading whichcraft-0.6.1...
Installing whichcraft-0.6.1...
Installed whichcraft-0.6.1!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 67, in <module>
    from webkitscmpy import mocks
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/__init__.py", line 3, in <module>
    from webkitscmpy.mocks import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/__init__.py", line 4, in <module>
    from webkitscmpy.mocks.local.git import Git
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/git.py", line 34, in <module>
    from webkitscmpy.program.canonicalize.committer import main as committer_main
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/__init__.py", line 28, in <module>
    from .blame import Blame
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/blame.py", line 28, in <module>
    from webkitscmpy.program.command import FilteredCommand
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program/command.py", line 32, in <module>
    from whichcraft import which
ModuleNotFoundError: No module named 'whichcraft'
Comment 13 EWS 2021-10-27 16:17:52 PDT 
Downloading urllib3-1.25.10...
Installing urllib3-1.25.10...
Installed urllib3-1.25.10!
Downloading chardet-3.0.4...
Installing chardet-3.0.4...
Installed chardet-3.0.4!
Downloading certifi-2020.6.20...
Installing certifi-2020.6.20...
Installed certifi-2020.6.20!
Traceback (most recent call last):
  File "Tools/Scripts/webkit-patch", line 41, in <module>
    from webkitpy.common.system.logutils import configure_logging
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/webkitpy/__init__.py", line 85, in <module>
    import webkitscmpy
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py", line 66, in <module>
    from webkitscmpy import local
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/__init__.py", line 4, in <module>
    from webkitscmpy.local.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py", line 38, in <module>
    from webkitscmpy import remote, Commit, Contributor, Version
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/__init__.py", line 24, in <module>
    from webkitscmpy.remote.svn import Svn
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py", line 29, in <module>
    import requests
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/requests/__init__.py", line 120, in <module>
    from . import utils
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/requests/utils.py", line 25, in <module>
    from . import certs
  File "/Volumes/Data/worker/Commit-Queue/build/Tools/Scripts/libraries/autoinstalled/python-3-x86_64/requests/certs.py", line 15, in <module>
    from certifi import where
ModuleNotFoundError: No module named 'certifi'
Comment 14 EWS 2021-10-27 16:27:38 PDT 
Committed r284964 (243615@main): <https://commits.webkit.org/243615@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 442635 [details].
Comment 15 Radar WebKit Bug Importer 2021-10-27 16:28:16 PDT 
<rdar://problem/84733102>