Bug 212086

Summary: [FlatPak SDK] Missing ruby gems json and highline
Product: WebKit Reporter: Carlos Alberto Lopez Perez <clopez>
Component: WebKitGTKAssignee: Philippe Normand <pnormand>
Status: RESOLVED FIXED    
Severity: Normal CC: aperez, bugs-noreply, pnormand, zan
Priority: P2    
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch
none
Patch none

Description Carlos Alberto Lopez Perez 2020-05-19 09:57:47 PDT
From https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/13771/steps/jscore-test/logs/stdio


Running: /usr/bin/env ruby Tools/Scripts/run-jsc-stress-tests -j /app/webkit/WebKitBuild/Release/bin/jsc -o /app/webkit/WebKitBuild/Release/bin/jsc-stress-results PerformanceTests/SunSpider/tests/sunspider-1.0 PerformanceTests/JetStream/cdjs/cdjs-tests.yaml PerformanceTests/ARES-6/Air/airjs-tests.yaml PerformanceTests/ARES-6/Basic/basic-tests.yaml JSTests/executableAllocationFuzz.yaml JSTests/exceptionFuzz.yaml PerformanceTests/SunSpider/no-architecture-specific-optimizations.yaml PerformanceTests/SunSpider/shadow-chicken.yaml PerformanceTests/SunSpider/tests/v8-v6 JSTests/stress JSTests/microbenchmarks JSTests/slowMicrobenchmarks.yaml PerformanceTests/SunSpider/profiler-test.yaml LayoutTests/jsc-layout-tests.yaml JSTests/typeProfiler.yaml JSTests/controlFlowProfiler.yaml JSTests/es6.yaml JSTests/modules.yaml JSTests/complex.yaml JSTests/ChakraCore.yaml JSTests/wasm.yaml JSTests/mozilla/mozilla-tests.yaml --memory-limited --
Warning: did not find json or highline; some features will be disabled.
Run "sudo gem install json highline" to fix the issue.
Error: #<LoadError: cannot load such file -- highline>
Comment 1 Philippe Normand 2020-05-19 10:25:56 PDT
Seems like buildstream doesn't handle "ruby gems" yet.
Comment 2 Philippe Normand 2020-05-20 05:40:53 PDT
Created attachment 399830 [details]
Patch
Comment 3 Carlos Alberto Lopez Perez 2020-05-20 06:12:06 PDT
Comment on attachment 399830 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=399830&action=review

> Tools/buildstream/elements/sdk/ruby-highline.bst:22
> +      mkdir /tmp/gem-tmp
> +      gem install --local --install-dir /tmp/gem-tmp/ *.gem
> +      mkdir -p %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/
> +      mv /tmp/gem-tmp/gems/*/lib/* %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/

I suggest to add a final step to ensure of cleaning the temp-directory, so in can start from a clean state in case of rebuild.
rm -fr /tmp/gem-tmp

> Tools/buildstream/elements/sdk/ruby-json.bst:19
> +      mkdir /tmp/gem-tmp
> +      gem install --local --install-dir /tmp/gem-tmp/ *.gem
> +      mkdir -p %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/
> +      mv /tmp/gem-tmp/gems/*/lib/* %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/

ditto
Comment 4 Adrian Perez 2020-05-20 06:19:28 PDT
Comment on attachment 399830 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=399830&action=review

> Tools/buildstream/elements/sdk/ruby-highline.bst:22
> +      mv /tmp/gem-tmp/gems/*/lib/* %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/

Why not asking Ruby itself where it wants libraries installed? Also, the
“gem install“ command could be instructed to copy files directly inside
the target installation directory instead of manually moving files around.
This command should be enough:

  gem install --local --ignore-dependencies --no-user-install --no-document \
      --install-dir "$(ruby -e'puts Gem.default_dir')" \
      --bindir /usr/bin --build-root '%{install-root}' \
      file.gem

(This is basically what GNU/Linux distributions do for Ruby packages.)
Comment 5 Adrian Perez 2020-05-20 06:20:36 PDT
(In reply to Carlos Alberto Lopez Perez from comment #3)
> Comment on attachment 399830 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=399830&action=review
> 
> > Tools/buildstream/elements/sdk/ruby-highline.bst:22
> > +      mkdir /tmp/gem-tmp
> > +      gem install --local --install-dir /tmp/gem-tmp/ *.gem
> > +      mkdir -p %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/
> > +      mv /tmp/gem-tmp/gems/*/lib/* %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/
> 
> I suggest to add a final step to ensure of cleaning the temp-directory, so
> in can start from a clean state in case of rebuild.
> rm -fr /tmp/gem-tmp

That should not be needed with Buildstream, AFAIU the sandbox is always
recreated afresh on every rebuild. Anyway that won't be needed using the
alternative that I have proposed in my review.
Comment 6 Carlos Alberto Lopez Perez 2020-05-20 06:29:22 PDT
(In reply to Adrian Perez from comment #4)
> Comment on attachment 399830 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=399830&action=review
> 
> > Tools/buildstream/elements/sdk/ruby-highline.bst:22
> > +      mv /tmp/gem-tmp/gems/*/lib/* %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/
> 
> Why not asking Ruby itself where it wants libraries installed? Also, the
> “gem install“ command could be instructed to copy files directly inside
> the target installation directory instead of manually moving files around.
> This command should be enough:
> 
>   gem install --local --ignore-dependencies --no-user-install --no-document \
>       --install-dir "$(ruby -e'puts Gem.default_dir')" \
>       --bindir /usr/bin --build-root '%{install-root}' \
>       file.gem
> 
> (This is basically what GNU/Linux distributions do for Ruby packages.)

It seems this will still leave some "trash" in the install dir. Other arch packages do a manual step of cleaning after that. Check for example: https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/ruby-json#n44
But maybe that cache files are needed to keep the "gem" command working so it can recognize installed gems and can uninstall them. Given that we don't have a package manager inside flatpak, it can be a good idea to leave that gem cache installed so the gem command can work inside flatpak and be able to detect installed gems.

No strong opinion on any of this, just sharing what i found.
Comment 7 Philippe Normand 2020-05-20 07:02:48 PDT
Created attachment 399836 [details]
Patch
Comment 8 Philippe Normand 2020-05-20 07:03:26 PDT
Thanks for the reviews. If you like this new version let's land it :)
Comment 9 Adrian Perez 2020-05-20 07:38:11 PDT
Comment on attachment 399836 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=399836&action=review

All looking nice and tidy now! =)

> Tools/buildstream/elements/sdk/ruby-highline.bst:19
> +      gem install --local --ignore-dependencies --no-user-install --no-document --install-dir "$(ruby -e'puts Gem.default_dir')" --bindir %{install-root}%{prefix}/bin --build-root '%{install-root}' *.gem

Oh, using %{prefix} here is even an improvement over what I suggested. Neat.
Comment 10 EWS 2020-05-20 08:20:15 PDT
Committed r261918: <https://trac.webkit.org/changeset/261918>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 399836 [details].