Bug 188731 – [Flatpak] Update to GNOME master runtime

• Component: WebKitGTK
• Status: RESOLVED FIXED

[Michael Catanzaro]

The 3.28 runtime is pretty stale at this point. We could wait a couple weeks for the 3.30 runtime, but I don't think there's any need to stick with the stable runtime when we have the ability to pin to a particular version of the master runtime.

Notable changes:

 * Freedesktop 1.8 base, instead of Freedesktop 1.6. Huge improvements, e.g. it's now multiarch.
 * This fixes address sanitizer builds, which Philippe and Patrick both noticed were broken last week. Problem was that there's no asan in the stable runtime. This was my primary motivation for working on this upgrade.
 * All GStreamer packages removed. Let's use upstream GStreamer instead. Turns out all of our current patches are already upstream (largely thanks to Thibault, good job!) so we don't need to build our own anymore. Since everything is built from git master, it's a simple matter to change the SDK hashes whenever we need new versions.
 * OpenH264 is removed, since it was bunched in with the GStreamer plugins. Not sure if that was right, though. Is it needed separately?
 * I wasn't able to figure out how to build apache from git, since autoreconf doesn't work anymore for some reason. I switched it to a tarball build.

Known regressions:

 * This exposes bug #188064, which is probably a good thing since we don't want to hide regressions like that.
 * Weird cosmetic issue https://gitlab.com/freedesktop-sdk/freedesktop-sdk/issues/320 (should be fixed soon)
 * ** (gst-plugin-scanner:16): CRITICAL **: 12:00:00.438: Couldn't g_module_open libpython. Reason: /usr/lib/libpython3.7m.so: cannot open shared object file: No such file or directory. I don't understand this one. Well, it's looking in the old non-multiarch install location, while the new SDK is multiarch. But I don't know why, because we aren't building gstreamer ourselves anymore. It might be a bug in the upstream SDK?

[Michael Catanzaro]

(In reply to Michael Catanzaro from comment #0)
> Since everything is built from git master, it's a simple matter to change the SDK hashes whenever we need new versions.

BTW, I hope this will help encourage us to maintain our upstream first policy for GStreamer patches. It should only take a day or two for upstream changes to land in the master runtime, so I don't think this will be much inconvenience. If there are exceptional circumstances, we can always reverse course and add some GStreamer stuff back, but hopefully we can avoid that.

BTW #2: the upstream ffmpeg extension has been broken for several months now. I think that's https://gitlab.com/freedesktop-sdk/freedesktop-sdk/issues/306 but maybe there are other issues too. Again, I think it's desirable to expose issues like this that are currently hidden by us building our own custom GStreamer.

[Michael Catanzaro]

Created attachment 347458 [details]
Patch

[Thibault Saunier]

Interesting approach to just upgrade to master, but thinking about it, it does make sense.

Checked the patch and I add an informal r+ for it.

[Thibault Saunier]

>  * Freedesktop 1.8 base, instead of Freedesktop 1.6. Huge improvements, e.g. it's now multiarch.

fwiw, 3.28 was already multi arch, I am working on a branch where I am building targeting aarch64 and arm simply using binfmt and qemu-user. it works like a charm (patch yet to be finalized: https://github.com/thiblahute/webkit/commit/29d20bd9251507578d4bbd87054e8954aff8339c) and makes the worflow to target embedded devices very neat as I simply have a flatpak repo on the desktop and update org.webkit.WPE on the RPI :-)

[Michael Catanzaro]

(In reply to Thibault Saunier from comment #4)
> >  * Freedesktop 1.8 base, instead of Freedesktop 1.6. Huge improvements, e.g. it's now multiarch.
> 
> fwiw, 3.28 was already multi arch, I am working on a branch where I am
> building targeting aarch64 and arm simply using binfmt and qemu-user. it
> works like a charm (patch yet to be finalized:
> https://github.com/thiblahute/webkit/commit/
> 29d20bd9251507578d4bbd87054e8954aff8339c) and makes the worflow to target
> embedded devices very neat as I simply have a flatpak repo on the desktop
> and update org.webkit.WPE on the RPI :-)

"multiarch" has a very specific meaning: it means Debian-style libdirs, e.g. /usr/lib/x86_64-linux-gnu and /usr/lib/i386-linux-gnu, as opposed to "multilib" which is the /usr/lib64 and /usr/lib (or sometimes /usr/lib32) used everywhere else. multiarch is a bit nicer. The 1.6/3.28 SDKs weren't even multilib, they just put everything under /usr/lib. (Note that /app/lib is still not multiarch, and probably never will be, because apps are built for just one architecture.)

So when I see this problem:

(gst-plugin-scanner:16): CRITICAL **: 12:00:00.438: Couldn't g_module_open libpython. Reason: /usr/lib/libpython3.7m.so: cannot open shared object file: No such file or directory.

I wonder why gst-plugin-scanner is looking in the old location, rather than the new multiarch location.

[Philippe Normand]

Comment on attachment 347458 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=347458&action=review

> Tools/flatpak/org.webkit.WebKit.yaml:-114
> -  - name: libvpx

Isn't this needed for WebRTC?

> Tools/flatpak/org.webkit.WebKit.yaml:-187
> -  - name: gst-libav

Without this you loose another chance to run layout tests, quite a few use h264 files.

[Thibault Saunier]

(In reply to Philippe Normand from comment #6)
> Comment on attachment 347458 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=347458&action=review
> 
> > Tools/flatpak/org.webkit.WebKit.yaml:-114
> > -  - name: libvpx
> 
> Isn't this needed for WebRTC?

It is in the gnome master Sdk.

> > Tools/flatpak/org.webkit.WebKit.yaml:-187
> > -  - name: gst-libav
> 
> Without this you loose another chance to run layout tests, quite a few use
> h264 files.

The flatpak FFMPEG extension should work, afaiu it doesn't right now, so I maybe we want to workaround that for now?

[Michael Catanzaro]

(In reply to Thibault Saunier from comment #7)
> The flatpak FFMPEG extension should work, afaiu it doesn't right now, so I
> maybe we want to workaround that for now?

Instead of working around it, I would rather we fix it.

The extension was renamed, btw, to html5-codecs. But I do not seem to have it installed, which is concerning. The old extension org.freedesktop.Platform.ffmpeg/x86_64/1.6 was installed automatically somehow; not sure how. The other problem is that gstreamer-libav is currently not being built by freedesktop SDK, so not sure how the extension could work. Another question is how overlaps work: the GNOME runtime builds its own gstreamer-libav which gets installed after (overlaps on disk) the freedesktop one, and that's not in any extension. So theoretically, I would expect it to already work. Not sure why it doesn't; it will need debugging by GStreamer experts. (When we switch the GNOME runtime generation to gnome-build-meta, the freedesktop version will no longer be installed, eliminating the overlap.) Anyway, how the extension is supposed to work when we rebuild all the gstreamer elements is beyond me.

The final problem is that if we do fix it, I am going to start complaining about autoplay video again. :P I'm tired of being the only browser that still allows autoplay videos. But that is a separate bug; let's not block on that.

[Michael Catanzaro]

Committed r235114: <https://trac.webkit.org/changeset/235114>

[Michael Catanzaro]

I think the commits I had tagged were pruned from the server, so perhaps master is going to be a no-go. We'll probably need to revert this or switch to 3.30?

[Michael Catanzaro]

Reverted r235114 for reason:

ostree server deleted our commits?

Committed r235500: <https://trac.webkit.org/changeset/235500>

[Thibault Saunier]

It looks like the same happened on the 3.28 branch :| Should we check with GNOME sysadmins how for how long they keep history in the ostree repo? Maybe ask them to keep them for more time? fmpov it should try to keep as much of history as possible so we can bisect!

[Michael Catanzaro]

(In reply to Thibault Saunier from comment #12)
> It looks like the same happened on the 3.28 branch :|

Then I don't know what we can do, I guess our flatpak build scripts are just not usable right now. Good thing we failed to switch over the bots....

The GNOME sysadmins won't be able to answer this. Please ask Alex; he runs sdk.gnome.org.

[Michael Catanzaro]

Reverted r235500 for reason:

Time to switch back to master runtime

Committed r236195: <https://trac.webkit.org/changeset/236195>