Bug 230201 - [Flatpak SDK] Move toolchains to UserFlatpak and improve SDK upgrades
Summary: [Flatpak SDK] Move toolchains to UserFlatpak and improve SDK upgrades
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Philippe Normand
URL:
Keywords: InRadar
Depends on:
Blocks: 229919
  Show dependency treegraph
 
Reported: 2021-09-12 07:39 PDT by Philippe Normand
Modified: 2021-09-21 04:01 PDT (History)
5 users (show)

See Also:


Attachments
Patch (7.83 KB, patch)
2021-09-12 07:47 PDT, Philippe Normand
no flags Details | Formatted Diff | Diff
[fast-cq] Patch (9.84 KB, patch)
2021-09-17 09:00 PDT, Philippe Normand
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Philippe Normand 2021-09-12 07:39:44 PDT
.
Comment 1 Philippe Normand 2021-09-12 07:47:16 PDT
Created attachment 437988 [details]
Patch
Comment 2 Lauro Moura 2021-09-16 06:07:51 PDT
I had an issue with the Toolchain folders inside the source UserFlatpak being invisible while inside flatpak (maybe a flatpak thing to avoid some kind of recursion?)

To work, I used the patch below to point ICECC_VERSION to the mapped (/app/webkit) toolchain file in the sandbox.

Other than that, it seems to be working fine (build still running, though).

diff --git a/Tools/flatpak/flatpakutils.py b/Tools/flatpak/flatpakutils.py
index fbcfd3bdc12c..cf86e817dfbd 100644
--- a/Tools/flatpak/flatpakutils.py
+++ b/Tools/flatpak/flatpakutils.py
@@ -887,13 +887,16 @@ class WebkitFlatpak:
             except KeyError:
                 Console.error_message("Toolchains configuration not found. Please run webkit-flatpak -r")
                 return 1
+            if not os.path.isfile(toolchain_path):
+                Console.error_message("%s is not a valid IceCC toolchain. Please run webkit-flatpak -r", toolchain_path)
+                return 1
+            # Normalize toolchain path so it's visible inside Flatpak
+            toolchain_dir, _ = os.path.split(toolchain_path)
+            if toolchain_dir.startswith(self.flatpak_build_path):
+                toolchain_path = toolchain_path.replace(self.source_root, self.sandbox_source_root)
             if "ICECC_VERSION_APPEND" in os.environ:
                 toolchain_path += ","
                 toolchain_path += os.environ["ICECC_VERSION_APPEND"]
-            native_toolchain = toolchain_path.split(",")[0]
-            if not os.path.isfile(native_toolchain):
-                Console.error_message("%s is not a valid IceCC toolchain. Please run webkit-flatpak -r", native_toolchain)
-                return 1
             sandbox_environment.update({
                 "CCACHE_PREFIX": "icecc",
                 "ICECC_TEST_SOCKET": "/run/icecc/iceccd.socket",
Comment 3 Carlos Alberto Lopez Perez 2021-09-16 10:20:07 PDT
Comment on attachment 437988 [details]
Patch

r=me
Please check suggestion from Lauro before landing
Comment 4 Philippe Normand 2021-09-16 11:33:49 PDT
(In reply to Lauro Moura from comment #2)
> I had an issue with the Toolchain folders inside the source UserFlatpak
> being invisible while inside flatpak (maybe a flatpak thing to avoid some
> kind of recursion?)
> 
> To work, I used the patch below to point ICECC_VERSION to the mapped
> (/app/webkit) toolchain file in the sandbox.
> 

Strange, I didn't notice this with sccache. Thanks for raising this issue, I'll check it here as well.
Comment 5 Lauro Moura 2021-09-16 18:50:12 PDT
(In reply to Philippe Normand from comment #4)
> (In reply to Lauro Moura from comment #2)
> > I had an issue with the Toolchain folders inside the source UserFlatpak
> > being invisible while inside flatpak (maybe a flatpak thing to avoid some
> > kind of recursion?)
> > 
> > To work, I used the patch below to point ICECC_VERSION to the mapped
> > (/app/webkit) toolchain file in the sandbox.
> > 
> 
> Strange, I didn't notice this with sccache. Thanks for raising this issue,
> I'll check it here as well.

For context, I tested with icecc driven by local ccache (network still too slow for sccache).
Comment 6 Philippe Normand 2021-09-17 00:59:44 PDT
Yes, I understood your initial comment, it was clear. I was mentioning sccache because it behaves similarly to icecc and uses the same toolchains.
Comment 7 Philippe Normand 2021-09-17 04:16:34 PDT
Actually there was an issue with sccache too ;) As this patch moves toolchains deeper, in a path that is not directly mapped in the sandbox, we now need to have icecc/sccache configs use sandbox paths.

Can you try this?

http://sprunge.us/IB3zTn
Comment 8 Philippe Normand 2021-09-17 09:00:10 PDT
Created attachment 438479 [details]
[fast-cq] Patch
Comment 9 EWS 2021-09-20 01:08:57 PDT
Committed r282742 (241879@main): <https://commits.webkit.org/241879@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 438479 [details].
Comment 10 Radar WebKit Bug Importer 2021-09-20 01:09:20 PDT
<rdar://problem/83297236>
Comment 11 Sergio Villar Senin 2021-09-20 10:51:20 PDT
This has broken the icecc builds, everytime I try to build I get

  Building flatpak based environment

  /app/webkit/WebKitBuild/UserFlatpak/Toolchains/webkit-sdk-gcc-86e1cecb502cd13a11a8fb18b58071fd.tar.gz is not a valid IceCC toolchain. Please run webkit-flatpak -r

over and over again even if I run webkit-flatpak -r
Comment 12 Philippe Normand 2021-09-20 10:56:35 PDT
webkit-flatpak -c ls /app/webkit/WebKitBuild/UserFlatpak/Toolchains/

shows nothing?
Comment 13 Philippe Normand 2021-09-20 11:54:31 PDT
Workaround is to comment this out: https://github.com/WebKit/WebKit/blob/main/Tools/flatpak/flatpakutils.py#L896..L898

Follow-up patch coming soon.