RESOLVED FIXED308641
[WKCI][Tools] Bots WPE-Linux-RPi4-64bits-Mesa-Release-Perf-Build and WPE-Linux-ARM64-bit-Release-Build are uploading the release.zip file to the same S3 directory, causing conflicts and problems on the testers 
https://bugs.webkit.org/show_bug.cgi?id=308641
Summary [WKCI][Tools] Bots WPE-Linux-RPi4-64bits-Mesa-Release-Perf-Build and WPE-Linu...
Carlos Alberto Lopez Perez
Reported 2026-02-25 06:48:32 PST
We were having some unexplained crashes on the new WPE ARM64 bots that we have deployed. Upon investigation it seems that it was because the WPE binaries were linking with some libraries that don't exist (neither on the builder, neither on the tester) (checked for this libraries on /jhbuild and all the whole filesystem.. no trace of those) $ ldd bin/WebKitTestRunner |grep not libbacktrace.so.0 => not found libjpeg.so.62 => not found libjxl.so.0.8 => not found libavif.so.15 => not found For example, there is no trace of libjxl.so.0.8 anywhere on the filesystem, there is only libjxl.so.0.7 I also checked if any other library could have a NEEDED entry for libjxl.so.0.8 but it is not the case, only for libjxl.so.0.7 Also it shouldn't link with libacktrace.so.0 because we build libbacktrace as a .a library (not as a .so) so it should get embedded into the binary instead of linking against it (we don't even have the .so available) ``` $ find / 2>/dev/null|grep "libbacktrace\." /usr/lib/gcc/aarch64-linux-gnu/14/libbacktrace.a /usr/lib/gcc/aarch64-linux-gnu/13/libbacktrace.a /usr/lib/gcc/aarch64-linux-gnu/13/libstdc++_libbacktrace.a /usr/lib/gcc-cross/arm-linux-gnueabihf/13/libbacktrace.a /usr/lib/gcc-cross/arm-linux-gnueabihf/13/libstdc++_libbacktrace.a /jhbuild/install/lib/libbacktrace.a ``` The issue doesn't happen always, some builds are fine but not others. Finally i reduced the problem to an issue with the uploads to S3 is *NOT THE SAME* file that the testers download. Check this example: - Builder WPE-Linux-ARM64-bit-Release-Build builds 308198@main and triggers a run for the tester at WPE-Linux-ARM64-bit-Release-Tests/28 https://build.webkit.org/#/builders/1882/builds/28 - Now look in the builder the details of the release.zip file created (step archive-built-product) and write down all the files that go into the release.zip file - Now do the same on the tester (step extract-built-product) - Compare the list of files.. should be the same right? but is not $ diff -u <(curl -s https://build.webkit.org/api/v2/logs/49216383/raw_inline | grep adding: | awk '{print $2}' | grep -v /$ | sort) <(curl -s https://build.webkit.org/api/v2/logs/49216931/raw_inline | grep -P '(inflating|extracting|linking):' | awk '{print $2}' | grep -v /$ | sort) --- /dev/fd/63 2026-02-25 15:38:29.978933995 +0100 +++ /dev/fd/62 2026-02-25 15:38:29.978933995 +0100 @@ -4,7 +4,6 @@ bin/LLIntSettingsExtractor bin/MallocBench bin/MiniBrowser -bin/qt-wpe-mini-browser bin/testair bin/testapi bin/testapiScripts/basic.js @@ -34,10 +33,6 @@ bin/TestWebKitAPI/WPEPlatform/TestDisplayWayland bin/TestWebKitAPI/WPEPlatform/TestSettings bin/TestWebKitAPI/WPEPlatform/TestView -bin/TestWebKitAPI/WPEQt/TestLoad -bin/TestWebKitAPI/WPEQt/TestLoadHtml -bin/TestWebKitAPI/WPEQt/TestLoadRequest -bin/TestWebKitAPI/WPEQt/TestRunJavaScript bin/TestWebKitAPI/WPE/resources/webkitglib-tests-resources.gresource bin/TestWebKitAPI/WPE/resources/webkitglib-tests-resources.gresource.deps bin/TestWebKitAPI/WPE/TestAuthentication @@ -81,16 +76,16 @@ bin/WPEWebProcess lib/libgtest.so lib/libmbmalloc.so -lib/libqtwpe.so lib/libsigaction-disabler.so lib/libTestRunnerInjectedBundle.so lib/libTestWebKitAPIInjectedBundle.so lib/libWPEInjectedBundle.so +lib/libWPEPlatform-2.0.so +lib/libWPEPlatform-2.0.so.0 +lib/libWPEPlatform-2.0.so.0.0.0 lib/libWPEWebKit-2.0.so lib/libWPEWebKit-2.0.so.1 lib/libWPEWebKit-2.0.so.1.10.0 -lib/qt6/qml/org/wpewebkit/qtwpe/libqtwpe.so -lib/qt6/qml/org/wpewebkit/qtwpe/qmldir lib/system/libmbmalloc.so lib/WPEAPITests/libWebProcessExtensionTest.so lib/WPEAPITests/libWebProcessTest.so Crazy.. so the release.zip file downloaded on the tester lacks several files that were added to the release.zip file on the builder, like for example lib/libqtwpe.so or bin/qt-wpe-mini-browser and includes some files that were not on the builder like lib/libWPEPlatform-2.0.so.0.0.0 It seems the problem is that we are colliding with the bot WPE-Linux-RPi4-64bits-Mesa-Release-Perf-Build at https://build.webkit.org/#/builders/894/builds/32447 We can double-check by running the diff of files with this builder $ diff -u <(curl -s https://build.webkit.org/api/v2/logs/49216387/raw_inline | grep adding: | awk '{print $2}' | grep -v /$ | sort) <(curl -s https://build.webkit.org/api/v2/logs/49216931/raw_inline | grep -P '(inflating|extracting|linking):' | awk '{print $2}' | grep -v /$ | sort) $ md5sum <(curl -s https://build.webkit.org/api/v2/logs/49216387/raw_inline | grep adding: | awk '{print $2}' | grep -v /$ | sort) <(curl -s https://build.webkit.org/api/v2/logs/49216931/raw_inline | grep -P '(inflating|extracting|linking):' | awk '{print $2}' | grep -v /$ | sort) 3c5c753d7471e1df3a9b0b4c608a3f2a /dev/fd/63 3c5c753d7471e1df3a9b0b4c608a3f2a /dev/fd/62 match! We can also check on the tester that triggered from this WPE-Linux-RPi4-64bits-Mesa-Release-Perf-Build (which is WPE-Linux-RPi4-64bits-Mesa-Release-Perf-Tests/23792 https://build.webkit.org/#/builders/895/builds/23792/steps/10/logs/stdio ) has the same URL than the downloaded file on the step download-built-product at WPE-Linux-ARM64-bit-Release-Tests/28 https://build.webkit.org/#/builders/1882/builds/28/steps/10/logs/stdio argv: [b'python3', b'Tools/CISupport/download-built-product', b'--platform=wpe', b'--release', b'https://s3-us-west-2.amazonaws.com/archives.webkit.org/wpe-arm64-release/308198@main.zip'] argv: [b'python3', b'Tools/CISupport/download-built-product', b'--platform=wpe', b'--release', b'https://s3-us-west-2.amazonaws.com/archives.webkit.org/wpe-arm64-release/308198@main.zip']
Attachments
Add attachment proposed patch, testcase, etc.
Carlos Alberto Lopez Perez
Comment 1 2026-02-25 10:23:44 PST
Pull request: https://github.com/WebKit/WebKit/pull/59428
EWS
Comment 2 2026-02-26 03:47:01 PST
Committed 308273@main (0ed5c0a3be8f): <https://commits.webkit.org/308273@main> Reviewed commits have been landed. Closing PR #59428 and removing active labels.
Note You need to log in before you can comment on or make changes to this bug.