I686 in Fedora: problem statement

[decathorpe (Fabio Valentini)]

@chapien

Creating something like a container to run 32 bit software, or something like Apple’s Rosetta, would be astronomically more difficult than maintaining the packages as is.

I’m not so sure about that.

Because keeping around and maintaining the i686 package set is definitely not free - the cost is just very distributed, and thus, largely invisible …

[chapien (Claire Robsahm)]

@decathorpe

I’m not so sure about that.

Because keeping around and maintaining the i686 package set is definitely not free - the cost is just very distributed, and thus, largely invisible …

I don’t doubt that it is costly and difficult, which is why something needs to be done. But making something new, a whole new way to run 32 bit software, would be rather difficult, it always is.

[decathorpe (Fabio Valentini)]

Difficult, yes - but it won’t get more difficult. But keeping i686 arch support around is going to get more difficult. So at some point in the future, those two lines will intersect, and while “keep i686 stuff around” is the “cheaper” option for now, I doubt that it will remain this way forever.

[jspaleta (Jef Spaleta)]

I’m not sure its the cheapest option now.

Assuming a linux container populated with some number of 32bit libs is sufficient to satify a particular use case.

How difficult is it really to start from a Fedora 64bit host and populate those libs by rebuilding source, avoiding all the rpm buildrequirisms that force having a full 32bit build root.

The rpmisms that we seem to struggle with from an infra resource burn pov seem to be to be centered on our self-consistent policy that what Fedora is producing be from a coherent build root.

A container solution meant to be use for legacy use cases where the application of interest isnt even part of that self-consistent stack that Fedora, could break some of those assumptions baked into Fedora infra because the use case itself breaks the very assumptions that Fedora infra mandates.

Well at least to me I don’t see a problem with that, it just might not be hostable as Fedora even though it could be derived entirely from Fedora corresponding source.

What I’m least convinced of is that a container solution for 32bit runtime would be more work. If anything it might be less work.

[decathorpe (Fabio Valentini)]

This sounds similar to how the FEX image for running x86 software on aarch64 is assembled - though that is (to my knowledge) still put together from RPM packages built in koji, so I don’t think that buys us anything …

[dmtaylor (Dylan M Taylor)]

What is pulling in hundreds of Rust packages? I don’t recall any users mentioning that they need to run i686 Rust bins in the larger discussion. Are these just build deps? Can we use cross-compilation instead?

[decathorpe (Fabio Valentini)]

What is pulling in hundreds of Rust packages?

There’s a few things that are in the dependency tree of what’s going to be needed on i686 - including rpm-sequoia (OpenPGP signature verification of RPM packages), librsvg2 (SVG rendering in GTK and other places), possibly also the rav1e AV1 encoder and some GStreamer plugins that are written in Rust …

Can we use cross-compilation instead?

Cross-compilation is pretty much a no-go in general, as far as I can tell.

[ferdnyc (Frank Dana (FeRD))]

@dmtaylor

What is pulling in hundreds of Rust packages? I don’t recall any users mentioning that they need to run i686 Rust bins in the larger discussion.

Keep in mind that, due to the nature of Rust crates, you can need 500 Rust packages just to build one binary.

@decathorpe

rpm-sequoia (OpenPGP signature verification of RPM packages), librsvg2 (SVG rendering in GTK and other places), possibly also the rav1e AV1 encoder and some GStreamer plugins that are written in Rust …

GDKPixbuf as of GNOME 49 also uses a Rust backend called Glycin for all of its image loading/saving functionality. (Which is great as it’s brought support for a lot of things that were previously lacking, like EXIF rotation in PNG files. But it’s also broken things, like the Graphviz GDKPixbuf export plugin — which is (or has been) used by default for JPG, BMP, TIF, and ICO. We’re still trying to track down that bug. There are other options for Graphviz, with the right subpackages installed, but they need to be made required and default to avoid F43 Graphviz users getting empty image exports.)

[tarulia (Tarulia Dulacre)]

@bookwar

Simply lack of awareness among existing and new packagers about the initiatives like:

Changes/EncourageI686LeafRemoval - Fedora Project Wiki

This is partially solved by the recent discussion I guess

Opt-out vs opt-in

New packages are created with i686 builds enabled which grows the 32-bit dependency tree. Which goes against the efforts to trim it.

I’m sorry I’m somewhat(?) necroing this thread, but this one popped up in the related threads after reading the entirety (yes) of the alluded to removal proposal thread, so now I’m here.

I was actually aware of the LeafRemoval proposal for a while now. However, I always assumed that when this was implemented a switch to an opt-in model was already made, or that it always had been opt-in in the first place. As someone who’s not deeply involved in the core Fedora packaging process, it just made sense to me that a packager would have to choose supported architectures explicitly (outside of some “core” arches like x86_64). I am building a handful of packages on COPR, and seeing as it is explicit opt in there, I assumed it would be the same for core packages.

I think that might well also be a source of a “lack of awareness”. Users might have built packages on COPR before the same way I did and came to the same conclusion.

---

Side tangent regarding the “drop i686 proposal”:

(sorry this ended up longer than intended )

(If the technical background is already known or not you’re not interested, skip to “The Fedora packaging conundrum”)

Fast forward to 12h ago when I learned that not only does the core Fedora repo provide 32 bit builds of obs-studio-plugin-vkcapture (which makes sense to a certain extent which I will come back to in a bit), but it also provides 32 bit builds of obs-studio itself. This… caught me by surprise, to say the least, because OBS officially dropped support for all 32 bit Operating Systems over 3 years ago:

Additionally, Qt 6 has dropped support for Windows 7 & 8, macOS 10.13 & 10.14, Ubuntu 18.04 and all 32-bit operating systems. As such, OBS will no longer be supported on these platforms.

So, to me it seems like a miracle that this even still compiles, but I also seriously question who would be using OBS Studio on a 32 bit OS. (And yes I’m aware that at that point in time OBS was still built with Qt5 as long as OBS provided a period where it could be built with both, but that has long since passed).

Anyway, coming back to obs-studio-plugin-vkcapture: This also came up in the i686 removal proposal:

F44 Change Proposal: Drop i686 support (system wide)

This also would break OBS’ gamecapture of games since that requires a 32-bit userspace graphics stack.

And while this statement is not entirely incorrect, it’s also not without nuance, and later in the thread seems to have been misunderstood as keeping OBS from capturing games at all:

@kylegospo

As a user I should be able to install Steam (Which as of the time of writing is 32-bit only), install any 32-bit game (The vast majority of the game library on Steam), and launch it without any end-user applied workarounds. OBS should be able to capture the game without issue.

@dmtaylor

This change is the better part of a year out, so perhaps the packaging system could be figured out so that deps for Steam, OBS to work on 32-bit could continue to be built by then.

@adamkafei

Personally, Steam is a dealbreaker for me, as is recording stuff in OBS.

@chapien

gaming is not the only use case here; it’s just the most visible. OBS, legacy games, legacy enterprise software, etc. are all software that would become problematic.

I wanted to provide some clarifications and clear some misconception on this. For transparency I’ll say I’m not a contributor to OBS, however I am someone who has been active in OBS’ Linux support channel for the better part of 6 years. I also maintain my own build of obs-studio (historically forked from the RPM Fusion package, but long since departed from it), and a COPR with a handful of OBS plugins.
So let’s say I have some experience with users wanting to use this “gamecapture” (not least of which because certain distros (not Fedora) felt it was a good idea to ship this by default without telling users they need to set it up, but I digress) and when it is and isn’t needed.

So, first things first:
obs-studio itself has no 32 bit dependencies at all whatsoever - as also evident by dropping 32 bit support entirely as per release notes above.

Secondly:
OBS is perfectly capable of capturing any window - including games - with no issues just using natively provided APIs, and in fact many of our users (including myself) do so on a daily basis. Xcomposite is available on X11 sessions, and PipeWire capture through xdg-desktop-portal’s screencast on Wayland (and in theory X11 sessions, but to my knowledge only GNOME implemented this).
Neal also mentioned that the portal had a history of locking up the desktop when capturing games, however I can’t say having heard of any such behaviour. This might have been the case on select early versions, but it is certainly not the case now.

So, what is this “gamecapture” we’re talking about then? Well, that is actually a 2-part answer:

1. A third-party(!) plugin for obs-studio (obs-plugins/linux-vkcapture.so), which on its own simply provides a source called “gamecapture”, analogous - but not equivalent - to OBS’ native “Game Capture” source on Windows. This source alone doesn’t actually provide any capture capability.
   It should be noted that this plugin is not arch dependant and only need be available in OBS’ arch, i.e. 64 bit on a 64 bit installation. While Fedora builds obs-studio-plugin-vkcapture on 32 bit, this would only be required if the user was actually running 32 bit OBS.
2. A set of userland executables and libraries (/usr/bin/obs-gamecapture) through which the game (or technically any executable) is run. The executable injects a Vulkan layer (/usr/lib/libVkLayer_obs_vkcapture.so, /usr/lib64/libVkLayer_obs_vkcapture.so), which can also be achieved via an env var. This - to my understanding - provides a shared memory buffer that the aforementioned obs-studio plugin can access to then provide to OBS for scene rendering.
   Since this essentially injects foreign code into the game’s rendering pipeline, this is required in both a 32 bit and 64 bit variety (depending on the game in question).

For context: On Windows there are measurable performance penalties associated with capturing a game as a Window, or even the entire desktop. Through unspeakable horrors of dark magic OBS hooks into the game process’ graphics rendering pipeline and grabs the framebuffer directly for further use in compositing the scene. This provides essentially 0-latency capture because OBS doesn’t need to wait for frames to be composited by Windows and then passed over to OBS.

This performance penalty however doesn’t exist to a measurable degree on Linux. It wasn’t an issue on X11 (for window capture, XSHM for screen capture is discouraged), and it isn’t an issue on Wayland.

So what’s the point then? Well, there are some edge cases where Xcomposite and the screencast portal fall short:

• One example is any kind of capture that involves an alpha channel, i.e. transparency - this is notably important for VTubers. While both X11 as well as the screencast portal are technically capable of alpha channel capture, users are essentially at the mercy of their compositor on whether this is implemented correctly - or indeed at all. On the pipewire side this is also fairly recent, so this also still needs work on the OBS side.
  • This can be worked around by using a distinct background color behind the character and using a color-key filter in OBS (i.e. “green screen”)
• A second example is DEs/Window Manager for which simply no screencast portal with Window capture support exists, such as (most) wlroots based compositors
• A third example is automated capture of any game. The Windows screencapture works in a way that it (very simplified) finds any process that is using a Direct3D context, and tries to hook it. This kind of capture is not possible on Linux, hence the required setup using either obs-gamecapture or the env var
  • While on X11 users could simply add an Xcomposite capture for every game they tend to play, and the capture would start as the games are started, the same is architecturally not possible on Wayland. The way the screencast portal is designed is that the requesting application (i.e. OBS) never gets to know what it is actually capturing, it’s simply receiving frames from PipeWire. The entire user interaction for choosing the capture is handled through the screencast portal. The screencast portal does provide a way to restore a previous capture by way of a session restore token, however this has 2 drawbacks:
    • The user is once again at the mercy of their compositor on whether this session restore is implemented at all, and how well
    • The token is single-use, so if no window corresponding to the token is found, the user is presented with another capture selector. Meaning if a user has say 5 games in rotation, they are met with at minimum 4 capture selectors provided one of the games was actually open before OBS was started

That is unfortunately not an issue with the implementation on OBS’ side, but simply a consequence of the fundamental screencast portal design. So this is not “fixable” without the screencast portal seeing major changes, which I don’t see happening anytime soon given the stated goal of Wayland to not allow this kind of automated capture in the first place.

All of this is to say, while vkcapture certainly has its uses, and it would certainly be annoying for users if the 32 bit Vulkan layer were to be rendered non-functional, it is not the case that it would “break” OBS in its entirety. This is after all a third-party plugin, i.e. not an integral part of the software, and in fact in most cases in the support channel we recommend against its usage unless the user has a) the technical knowledge to set it up properly and b) an actual need for it. This is - for the most part - simply a convenience after all.

---

The Fedora packaging conundrum

However, it should also be noted that the entire premise in the drop proposal is also flawed. From my understanding the concern boiled down to “I won’t be able to gamecapture 32 bit games in a [presumably] 64 bit Fedora install”.

However (and someone feel free to correct me), from what I can tell the obs-studio-plugin-vkcapture.i686 package has a hard dependency on obs-studio.i686:

# For directory ownership
Requires:       vulkan-loader%{?_isa}
Requires:       obs-studio%{?_isa}

Which means users would have to reinstall and swap between the obs-studio’s i686 and x64_86 packages anytime they wanted to rotate between a 32 bit and 64 bit game, because the obs-studio-plugin-vkcapture packages only contain the Vulkan layers for their architecture.

Even if that weren’t the case however, the packages would still not install, because this would result in a file conflict on /usr/bin/obs-gamecapture since both packages contain one for their arch each.

This means that realistically, this would require 4 packages: 2 with the game capture binaries that conflict one another (and depend on their respective obs-studio), and 2 packages with the Vulkan layer files that don’t.

See also for comparison the AUR package (maintained by the upstream plugin author):

PKGBUILD - aur.git - AUR Package Repositories

They delete the obs-gamecapture binary for the 32 bit package because they opted simply not to ship one. But this is the only way to have both Vulkan layers installed.

[ngompa (Neal Gompa)]

@tarulia

However (and someone feel free to correct me), from what I can tell the obs-studio-plugin-vkcapture.i686 package has a hard dependency on obs-studio.i686:

# For directory ownership
Requires:       vulkan-loader%{?_isa}
Requires:       obs-studio%{?_isa}

Which means users would have to reinstall and swap between the obs-studio’s i686 and x64_86 packages anytime they wanted to rotate between a 32 bit and 64 bit game, because the obs-studio-plugin-vkcapture packages only contain the Vulkan layers for their architecture.

Even if that weren’t the case however, the packages would still not install, because this would result in a file conflict on /usr/bin/obs-gamecapture since both packages contain one for their arch each.

No, RPM automatically filters secondary arch binaries out on install provided the package versions exactly match. So this actually does work.

That said, there is a pending pull request to fix the unnecessary dependency on i686 OBS with the vkcapture plugin package.

[tarulia (Tarulia Dulacre)]

@ngompa

No, RPM automatically filters secondary arch binaries out on install provided the package versions exactly match. So this actually does work.

Hm weird, when I tesed this in a Toolbox earlier I was met with all sorts of conflicts and using --allowerasing wanted to replace the OBS package

edit:
Just tried again, this time installing both plugin packages separately instead of together:

$ sudo dnf install obs-studio -y
…
$ sudo dnf install obs-studio-plugin-vkcapture.x86_64 -y
…
[3/3] Installing obs-studio-plugin-vkcapture-0:1.5.1-2.fc42.x86_64                                                                                                                                 100% | 976.5 KiB/s | 171.9 KiB |  00m00s
Complete!
⬢ [tarulia@toolbx ~]$ sudo dnf install obs-studio-plugin-vkcapture.i686
Updating and loading repositories:
Repositories loaded.
Failed to resolve the transaction:
Problem: package obs-studio-libs-31.0.2-2.fc42.i686 from fedora requires libavcodec.so.61, but none of the providers can be installed
  - package obs-studio-libs-31.0.2-2.fc42.i686 from fedora requires libavcodec.so.61(LIBAVCODEC_61), but none of the providers can be installed
  - package libavcodec-free-7.1.1-3.fc42.i686 from fedora requires libopenh264.so.7, but none of the providers can be installed
  - package libavcodec-free-7.1.2-1.fc42.i686 from updates requires libopenh264.so.7, but none of the providers can be installed
  - problem with installed package
  - installed package openh264-2.5.1-1.fc42.x86_64 conflicts with openh264 provided by noopenh264-2.5.0-2.fc42.i686 from fedora
  - installed package openh264-2.5.1-1.fc42.x86_64 obsoletes noopenh264 < 1:0 provided by noopenh264-2.5.0-2.fc42.i686 from fedora
  - package noopenh264-2.5.0-2.fc42.i686 from fedora conflicts with openh264 provided by openh264-2.5.1-1.fc42.x86_64 from fedora-cisco-openh264
  - package openh264-2.5.1-1.fc42.x86_64 from fedora-cisco-openh264 obsoletes noopenh264 < 1:0 provided by noopenh264-2.5.0-2.fc42.i686 from fedora
  - package obs-studio-31.0.2-2.fc42.i686 from fedora requires obs-studio-libs(x86-32) = 31.0.2-2.fc42, but none of the providers can be installed
  - package obs-studio-plugin-vkcapture-1.5.1-2.fc42.i686 from fedora requires obs-studio(x86-32), but none of the providers can be installed
  - conflicting requests
You can try to add to command line:
  --allowerasing to allow removing of installed packages to resolve problems
  --skip-broken to skip uninstallable packages

Which I guess is a good demonstration of the problems with packaging i686

(don’t go out of your way to fix this, I think obs-studio.i686 is due for removal anyway)