Bug 262369 – [webkitpy] Update twisted and pyopenssl autoinstalled versions (fix issues with python-3.11 and WPE ARM32 perf bots)

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Carlos Alberto Lopez Perez]

The WPE Perf bots for ARM-32 are having issues with the WebKit python autoinstall modules.

See: https://build.webkit.org/#/builders/892/builds/1912


The are two issues there:

1)

The current versions of twisted auto-installed from Tools/Scripts/webkitpy/autoinstalled/twisted.py fail to build on a system with python-3.11

See:

creating build/temp.linux-armv7l-3.11
creating build/temp.linux-armv7l-3.11/src
creating build/temp.linux-armv7l-3.11/src/twisted
creating build/temp.linux-armv7l-3.11/src/twisted/test
arm-poky-linux-gnueabi-gcc -mthumb -mfpu=neon-vfpv4 -mfloat-abi=hard -mcpu=cortex-a7 -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -Wsign-compare -DNDEBUG -g -O3 -Wall -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/python3/3.11.2-r0/Python-3.11.2=/usr/src/debug/python3/3.11.2-r0 -fdebug-prefix-map=/python3/3.11.2-r0/Python-3.11.2=/usr/src/debug/python3/3.11.2-r0 -fmacro-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.11.2-r0 -fdebug-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.11.2-r0 -fdebug-prefix-map== -fmacro-prefix-map== -fdebug-prefix-map== -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/python3/3.11.2-r0/Python-3.11.2=/usr/src/debug/python3/3.11.2-r0 -fdebug-prefix-map=/python3/3.11.2-r0/Python-3.11.2=/usr/src/debug/python3/3.11.2-r0 -fmacro-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.11.2-r0 -fdebug-prefix-map=/build/path/unavailable/=/usr/src/debug/python3/3.11.2-r0 -fdebug-prefix-map== -fmacro-prefix-map== -fdebug-prefix-map== -fPIC -I/usr/include/python3.11 -c src/twisted/test/raiser.c -o build/temp.linux-armv7l-3.11/src/twisted/test/raiser.o
src/twisted/test/raiser.c:198:12: fatal error: longintrepr.h: No such file or directory
  198 |   #include "longintrepr.h"
      |            ^~~~~~~~~~~~~~~


This is related to  https://github.com/python/cpython/pull/28968 and https://github.com/cython/cython/pull/4428 (also related: https://github.com/aio-libs/aiohttp/issues/6600 )
And has been fixed in newer releases of twisted.



2) pyopenssl gives a failure when starting the benchmark

$ Tools/Scripts/run-benchmark --count 1 --plan sunspider
2023-09-29 14:26:06,949 - INFO - Ensure dependencies of http server is satisfied
Traceback (most recent call last):
  File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/run-benchmark", line 13, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/run_benchmark.py", line 177, in main
    return start(parse_args(config_argument_parser()))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/run_benchmark.py", line 165, in start
    run_benchmark_plan(args, args.plan)
  File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/run_benchmark.py", line 99, in run_benchmark_plan
    runner = benchmark_runner_class(plan,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/webserver_benchmark_runner.py", line 27, in __init__
    self._http_server_driver = HTTPServerDriverFactory.create(platform)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/generic_factory.py", line 13, in create
    return cls.products[description]()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/http_server_driver/simple_http_server_driver.py", line 27, in __init__
    self._ensure_http_server_dependencies()
  File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/benchmark_runner/http_server_driver/simple_http_server_driver.py", line 118, in _ensure_http_server_dependencies
    from webkitpy.autoinstalled import twisted
  File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/webkitpy/autoinstalled/twisted.py", line 42, in <module>
    from twisted.protocols.tls import TLSMemoryBIOFactory
  File "/usr/lib/python3.11/site-packages/twisted/protocols/tls.py", line 42, in <module>
    from OpenSSL.SSL import Connection, Error, SysCallError, WantReadError, ZeroReturnError
  File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/libraries/autoinstalled/python-3-armv7l/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/libraries/autoinstalled/python-3-armv7l/OpenSSL/crypto.py", line 1556, in <module>
    class X509StoreFlags(object):
  File "/opt/worker/WPE-Linux-RPi4-32bits-Mesa-Release-Perf-Tests/build/Tools/Scripts/libraries/autoinstalled/python-3-armv7l/OpenSSL/crypto.py", line 1577, in X509StoreFlags
    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'


This is also fixed by upgrading pyopenssl

[Carlos Alberto Lopez Perez]

Pull request: https://github.com/WebKit/WebKit/pull/18412

[EWS]

Committed 268723@main (4ae17ef2311b): <https://commits.webkit.org/268723@main>

Reviewed commits have been landed. Closing PR #18412 and removing active labels.

[Radar WebKit Bug Importer]

<rdar://problem/116342308>

[Jonathan Bedard]

We're going to have to conditionalize 268723@main on Python 3.11. The previous specified versions of Twisted and OpenSSL were picked because they match what ews-build.webkit.org and build.webkit.org are running.

[Sam Sneddon [:gsnedders]]

I don't understand how the patch fixed things?

The AutoInstaller _cannot_ install Twisted 23.8.0, because we don't support PEP 517 builders (Bug 261082). I think 22.10.0 is the latest version on PyPI the AutoInstaller should be able to install?

The commit message is also wrong to say "there aren't much pre-built pip packages for ARM32", because unless wheel=True is passed we _don't use pre-built packages_.

This has, resultantly, broken certain internal CI jobs with a traceback ending in:

  File "/Volumes/Data/worker/ews-tools-trybot/build/OpenSource/Tools/Scripts/webkitpy/autoinstalled/buildbot.py", line 27, in <module>
    from webkitpy.autoinstalled import twisted
  File "/Volumes/Data/worker/ews-tools-trybot/build/OpenSource/Tools/Scripts/webkitpy/autoinstalled/twisted.py", line 43, in <module>
    from twisted.protocols.tls import TLSMemoryBIOFactory
ModuleNotFoundError: No module named 'twisted'

In general, we would rather the versions match those we use on our buildbot instances, but clearly that doesn't work for all configurations.