[Flatpak SDK] Move toolchains to UserFlatpak and improve SDK upgrades

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Philippe Normand]

.

[Philippe Normand]

Created attachment 437988 [details]
Patch

[Lauro Moura]

I had an issue with the Toolchain folders inside the source UserFlatpak being invisible while inside flatpak (maybe a flatpak thing to avoid some kind of recursion?)

To work, I used the patch below to point ICECC_VERSION to the mapped (/app/webkit) toolchain file in the sandbox.

Other than that, it seems to be working fine (build still running, though).

diff --git a/Tools/flatpak/flatpakutils.py b/Tools/flatpak/flatpakutils.py
index fbcfd3bdc12c..cf86e817dfbd 100644
--- a/Tools/flatpak/flatpakutils.py
+++ b/Tools/flatpak/flatpakutils.py
@@ -887,13 +887,16 @@ class WebkitFlatpak:
             except KeyError:
                 Console.error_message("Toolchains configuration not found. Please run webkit-flatpak -r")
                 return 1
+            if not os.path.isfile(toolchain_path):
+                Console.error_message("%s is not a valid IceCC toolchain. Please run webkit-flatpak -r", toolchain_path)
+                return 1
+            # Normalize toolchain path so it's visible inside Flatpak
+            toolchain_dir, _ = os.path.split(toolchain_path)
+            if toolchain_dir.startswith(self.flatpak_build_path):
+                toolchain_path = toolchain_path.replace(self.source_root, self.sandbox_source_root)
             if "ICECC_VERSION_APPEND" in os.environ:
                 toolchain_path += ","
                 toolchain_path += os.environ["ICECC_VERSION_APPEND"]
-            native_toolchain = toolchain_path.split(",")[0]
-            if not os.path.isfile(native_toolchain):
-                Console.error_message("%s is not a valid IceCC toolchain. Please run webkit-flatpak -r", native_toolchain)
-                return 1
             sandbox_environment.update({
                 "CCACHE_PREFIX": "icecc",
                 "ICECC_TEST_SOCKET": "/run/icecc/iceccd.socket",

[Carlos Alberto Lopez Perez]

Comment on attachment 437988 [details]
Patch

r=me
Please check suggestion from Lauro before landing

[Philippe Normand]

(In reply to Lauro Moura from comment #2)
> I had an issue with the Toolchain folders inside the source UserFlatpak
> being invisible while inside flatpak (maybe a flatpak thing to avoid some
> kind of recursion?)
> 
> To work, I used the patch below to point ICECC_VERSION to the mapped
> (/app/webkit) toolchain file in the sandbox.
> 

Strange, I didn't notice this with sccache. Thanks for raising this issue, I'll check it here as well.

[Lauro Moura]

(In reply to Philippe Normand from comment #4)
> (In reply to Lauro Moura from comment #2)
> > I had an issue with the Toolchain folders inside the source UserFlatpak
> > being invisible while inside flatpak (maybe a flatpak thing to avoid some
> > kind of recursion?)
> > 
> > To work, I used the patch below to point ICECC_VERSION to the mapped
> > (/app/webkit) toolchain file in the sandbox.
> > 
> 
> Strange, I didn't notice this with sccache. Thanks for raising this issue,
> I'll check it here as well.

For context, I tested with icecc driven by local ccache (network still too slow for sccache).

[Philippe Normand]

Yes, I understood your initial comment, it was clear. I was mentioning sccache because it behaves similarly to icecc and uses the same toolchains.

[Philippe Normand]

Actually there was an issue with sccache too ;) As this patch moves toolchains deeper, in a path that is not directly mapped in the sandbox, we now need to have icecc/sccache configs use sandbox paths.

Can you try this?

http://sprunge.us/IB3zTn

[Philippe Normand]

Created attachment 438479 [details]
[fast-cq] Patch

[EWS]

Committed r282742 (241879@main): <https://commits.webkit.org/241879@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 438479 [details].

[Radar WebKit Bug Importer]

<rdar://problem/83297236>

[Sergio Villar Senin]

This has broken the icecc builds, everytime I try to build I get

  Building flatpak based environment

  /app/webkit/WebKitBuild/UserFlatpak/Toolchains/webkit-sdk-gcc-86e1cecb502cd13a11a8fb18b58071fd.tar.gz is not a valid IceCC toolchain. Please run webkit-flatpak -r

over and over again even if I run webkit-flatpak -r

[Philippe Normand]

webkit-flatpak -c ls /app/webkit/WebKitBuild/UserFlatpak/Toolchains/

shows nothing?

[Philippe Normand]

Workaround is to comment this out: https://github.com/WebKit/WebKit/blob/main/Tools/flatpak/flatpakutils.py#L896..L898

Follow-up patch coming soon.