262369 2023-09-29 07:48:42 -0700 [webkitpy] Update twisted and pyopenssl autoinstalled versions (fix issues with python-3.11 and WPE ARM32 perf bots) 2023-10-17 19:51:14 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://bugs.webkit.org/show_bug.cgi?id=262512 https://bugs.webkit.org/show_bug.cgi?id=263299 InRadar P2 Normal --- 1 clopez clopez gsnedders jbedard philn webkit-bug-importer oldest_to_newest 1981270 0 clopez 2023-09-29 07:48:42 -0700 The WPE Perf bots for ARM-32 are having issues with the WebKit python autoinstall modules. See: https://build.webkit.org/#/builders/892/builds/1912 The are two issues there: 1) The current versions of twisted auto-installed from Tools/Scripts/webkitpy/autoinstalled/twisted.py fail to build on a system with python-3.11 See: creating build/temp.linux-armv7l-3.11 creating build/temp.linux-armv7l-3.11/src creating build/temp.linux-armv7l-3.11/src/twisted creating build/temp.linux-armv7l-3.11/src/twisted/test arm-poky-linux-gnueabi-gcc -mthumb -mfpu=neon-vfpv4 -mfloat-abi=hard -mcpu=cortex-a7 -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -Wsign-compare -DNDEBUG -g -O3 -Wall -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/python3/3.11.2-r0/Python-3.11.2=/usr/src/debug/python3/3.11.2-r0 -fdebug-prefix-map=/python3/3.11.2-r0/Python-3.11.2=/usr/src/debug/python3/3.11.2-r0 -fmacro-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.11.2-r0 -fdebug-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.11.2-r0 -fdebug-prefix-map== -fmacro-prefix-map== -fdebug-prefix-map== -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/python3/3.11.2-r0/Python-3.11.2=/usr/src/debug/python3/3.11.2-r0 -fdebug-prefix-map=/python3/3.11.2-r0/Python-3.11.2=/usr/src/debug/python3/3.11.2-r0 -fmacro-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.11.2-r0 -fdebug-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.11.2-r0 -fdebug-prefix-map== -fmacro-prefix-map== -fdebug-prefix-map== -fPIC -I/usr/include/python3.11 -c src/twisted/test/raiser.c -o build/temp.linux-armv7l-3.11/src/twisted/test/raiser.o src/twisted/test/raiser.c:198:12: fatal error: longintrepr.h: No such file or directory 198 | #include "longintrepr.h" | ^~~~~~~~~~~~~~~ This is related to https://github.com/python/cpython/pull/28968 and https://github.com/cython/cython/pull/4428 (also related: https://github.com/aio-libs/aiohttp/issues/6600 ) And has been fixed in newer releases of twisted. 2) pyopenssl gives a failure when starting the benchmark $ Tools/Scripts/run-benchmark --count 1 --plan sunspider 2023-09-29 14:26:06,949 - INFO - Ensure dependencies of http server is satisfied Traceback (most recent call last): File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/run-benchmark", line 13, in <module> sys.exit(main()) ^^^^^^ File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/run_benchmark.py", line 177, in main return start(parse_args(config_argument_parser())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/run_benchmark.py", line 165, in start run_benchmark_plan(args, args.plan) File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/run_benchmark.py", line 99, in run_benchmark_plan runner = benchmark_runner_class(plan, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/webserver_benchmark_runner.py", line 27, in __init__ self._http_server_driver = HTTPServerDriverFactory.create(platform) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/generic_factory.py", line 13, in create return cls.products[description]() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/http_server_driver/simple_http_server_driver.py", line 27, in __init__ self._ensure_http_server_dependencies() File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/http_server_driver/simple_http_server_driver.py", line 118, in _ensure_http_server_dependencies from webkitpy.autoinstalled import twisted File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/autoinstalled/twisted.py", line 42, in <module> from twisted.protocols.tls import TLSMemoryBIOFactory File "/usr/lib/python3.11/site-packages/twisted/protocols/tls.py", line 42, in <module> from OpenSSL.SSL import Connection, Error, SysCallError, WantReadError, ZeroReturnError File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/libraries/autoinstalled/python-3-armv7l/OpenSSL/__init__.py", line 8, in <module> from OpenSSL import crypto, SSL File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/libraries/autoinstalled/python-3-armv7l/OpenSSL/crypto.py", line 1556, in <module> class X509StoreFlags(object): File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/libraries/autoinstalled/python-3-armv7l/OpenSSL/crypto.py", line 1577, in X509StoreFlags CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK' This is also fixed by upgrading pyopenssl 1981275 1 clopez 2023-09-29 07:55:15 -0700 Pull request: https://github.com/WebKit/WebKit/pull/18412 1981729 2 ews-feeder 2023-10-02 06:04:17 -0700 Committed 268723@main (4ae17ef2311b): <https://commits.webkit.org/268723@main> Reviewed commits have been landed. Closing PR #18412 and removing active labels. 1981730 3 webkit-bug-importer 2023-10-02 06:05:14 -0700 <rdar://problem/116342308> 1981867 4 jbedard 2023-10-02 14:50:13 -0700 We're going to have to conditionalize 268723@main on Python 3.11. The previous specified versions of Twisted and OpenSSL were picked because they match what ews-build.webkit.org and build.webkit.org are running. 1981873 5 gsnedders 2023-10-02 15:02:41 -0700 I don't understand how the patch fixed things? The AutoInstaller _cannot_ install Twisted 23.8.0, because we don't support PEP 517 builders (Bug 261082). I think 22.10.0 is the latest version on PyPI the AutoInstaller should be able to install? The commit message is also wrong to say "there aren't much pre-built pip packages for ARM32", because unless wheel=True is passed we _don't use pre-built packages_. This has, resultantly, broken certain internal CI jobs with a traceback ending in: File "/Volumes/Data/worker/ews-tools-trybot/build/OpenSource/Tools/Scripts/webkitpy/autoinstalled/buildbot.py", line 27, in <module> from webkitpy.autoinstalled import twisted File "/Volumes/Data/worker/ews-tools-trybot/build/OpenSource/Tools/Scripts/webkitpy/autoinstalled/twisted.py", line 43, in <module> from twisted.protocols.tls import TLSMemoryBIOFactory ModuleNotFoundError: No module named 'twisted' In general, we would rather the versions match those we use on our buildbot instances, but clearly that doesn't work for all configurations.