Bug 212086 – [FlatPak SDK] Missing ruby gems json and highline

• Status: RESOLVED
• Resolution: FIXED
• Component: WebKitGTK

[Carlos Alberto Lopez Perez]

From https://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release%20%28Tests%29/builds/13771/steps/jscore-test/logs/stdio


Running: /usr/bin/env ruby Tools/Scripts/run-jsc-stress-tests -j /app/webkit/WebKitBuild/Release/bin/jsc -o /app/webkit/WebKitBuild/Release/bin/jsc-stress-results PerformanceTests/SunSpider/tests/sunspider-1.0 PerformanceTests/JetStream/cdjs/cdjs-tests.yaml PerformanceTests/ARES-6/Air/airjs-tests.yaml PerformanceTests/ARES-6/Basic/basic-tests.yaml JSTests/executableAllocationFuzz.yaml JSTests/exceptionFuzz.yaml PerformanceTests/SunSpider/no-architecture-specific-optimizations.yaml PerformanceTests/SunSpider/shadow-chicken.yaml PerformanceTests/SunSpider/tests/v8-v6 JSTests/stress JSTests/microbenchmarks JSTests/slowMicrobenchmarks.yaml PerformanceTests/SunSpider/profiler-test.yaml LayoutTests/jsc-layout-tests.yaml JSTests/typeProfiler.yaml JSTests/controlFlowProfiler.yaml JSTests/es6.yaml JSTests/modules.yaml JSTests/complex.yaml JSTests/ChakraCore.yaml JSTests/wasm.yaml JSTests/mozilla/mozilla-tests.yaml --memory-limited --
Warning: did not find json or highline; some features will be disabled.
Run "sudo gem install json highline" to fix the issue.
Error: #<LoadError: cannot load such file -- highline>

[Philippe Normand]

Seems like buildstream doesn't handle "ruby gems" yet.

[Philippe Normand]

Created attachment 399830 [details]
Patch

[Carlos Alberto Lopez Perez]

Comment on attachment 399830 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=399830&action=review

> Tools/buildstream/elements/sdk/ruby-highline.bst:22
> +      mkdir /tmp/gem-tmp
> +      gem install --local --install-dir /tmp/gem-tmp/ *.gem
> +      mkdir -p %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/
> +      mv /tmp/gem-tmp/gems/*/lib/* %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/

I suggest to add a final step to ensure of cleaning the temp-directory, so in can start from a clean state in case of rebuild.
rm -fr /tmp/gem-tmp

> Tools/buildstream/elements/sdk/ruby-json.bst:19
> +      mkdir /tmp/gem-tmp
> +      gem install --local --install-dir /tmp/gem-tmp/ *.gem
> +      mkdir -p %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/
> +      mv /tmp/gem-tmp/gems/*/lib/* %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/

ditto

[Adrian Perez]

Comment on attachment 399830 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=399830&action=review

> Tools/buildstream/elements/sdk/ruby-highline.bst:22
> +      mv /tmp/gem-tmp/gems/*/lib/* %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/

Why not asking Ruby itself where it wants libraries installed? Also, the
“gem install“ command could be instructed to copy files directly inside
the target installation directory instead of manually moving files around.
This command should be enough:

  gem install --local --ignore-dependencies --no-user-install --no-document \
      --install-dir "$(ruby -e'puts Gem.default_dir')" \
      --bindir /usr/bin --build-root '%{install-root}' \
      file.gem

(This is basically what GNU/Linux distributions do for Ruby packages.)

[Adrian Perez]

(In reply to Carlos Alberto Lopez Perez from comment #3)
> Comment on attachment 399830 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=399830&action=review
> 
> > Tools/buildstream/elements/sdk/ruby-highline.bst:22
> > +      mkdir /tmp/gem-tmp
> > +      gem install --local --install-dir /tmp/gem-tmp/ *.gem
> > +      mkdir -p %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/
> > +      mv /tmp/gem-tmp/gems/*/lib/* %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/
> 
> I suggest to add a final step to ensure of cleaning the temp-directory, so
> in can start from a clean state in case of rebuild.
> rm -fr /tmp/gem-tmp

That should not be needed with Buildstream, AFAIU the sandbox is always
recreated afresh on every rebuild. Anyway that won't be needed using the
alternative that I have proposed in my review.

[Carlos Alberto Lopez Perez]

(In reply to Adrian Perez from comment #4)
> Comment on attachment 399830 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=399830&action=review
> 
> > Tools/buildstream/elements/sdk/ruby-highline.bst:22
> > +      mv /tmp/gem-tmp/gems/*/lib/* %{install-root}/%{prefix}/lib/%{gcc_triplet}/ruby/2.6.0/
> 
> Why not asking Ruby itself where it wants libraries installed? Also, the
> “gem install“ command could be instructed to copy files directly inside
> the target installation directory instead of manually moving files around.
> This command should be enough:
> 
>   gem install --local --ignore-dependencies --no-user-install --no-document \
>       --install-dir "$(ruby -e'puts Gem.default_dir')" \
>       --bindir /usr/bin --build-root '%{install-root}' \
>       file.gem
> 
> (This is basically what GNU/Linux distributions do for Ruby packages.)

It seems this will still leave some "trash" in the install dir. Other arch packages do a manual step of cleaning after that. Check for example: https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/ruby-json#n44
But maybe that cache files are needed to keep the "gem" command working so it can recognize installed gems and can uninstall them. Given that we don't have a package manager inside flatpak, it can be a good idea to leave that gem cache installed so the gem command can work inside flatpak and be able to detect installed gems.

No strong opinion on any of this, just sharing what i found.

[Philippe Normand]

Created attachment 399836 [details]
Patch

[Philippe Normand]

Thanks for the reviews. If you like this new version let's land it :)

[Adrian Perez]

Comment on attachment 399836 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=399836&action=review

All looking nice and tidy now! =)

> Tools/buildstream/elements/sdk/ruby-highline.bst:19
> +      gem install --local --ignore-dependencies --no-user-install --no-document --install-dir "$(ruby -e'puts Gem.default_dir')" --bindir %{install-root}%{prefix}/bin --build-root '%{install-root}' *.gem

Oh, using %{prefix} here is even an improvement over what I suggested. Neat.

[EWS]

Committed r261918: <https://trac.webkit.org/changeset/261918>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 399836 [details].