F44 Change Proposal: Drop i686 support (system wide)

Project Discussion Change Proposals
,
325

More I read this discussion more I scratch my head. I am not deeply familiar with decision making and governance processes in Fedora project, so maybe it is why the situation feels surreal.

I see a lot of pathos regarding “burden”, “actionability” and “Sword of Damocles”.

There should be no need of any kind of sword to:

  1. Conduct a survey among people doing “official” packaging to:

    • get tangible feedback of people who are doing actual job,
    • identify which packages are experiencing most/least trouble with 32-bit variants and human burnout,
    • determine application domains affected by problematic packages,
    • prepare application domain deprecation roadmap based on severity of related package problems.

  2. For dependencies having 32-bit builds deprecated by upstream, determine if these are used by widespread software and how it is handled.

  3. Do a research on how other distributions handle(d) affected application domain issues (and look at Statement on 32-bit i386 packages for Ubuntu 19.10 and 20.04 LTS story among other things).

  4. Consult with build infrastructure and packaging to see how deprecation roadmap and “success stories” of other distributions can be implemented. Estimate efforts required per path forward.

  5. Present this to the community and ask for necessary help (why is Canonical handling this and Red Hat isn’t?)

So far I have seen some (at times vocal, and at times rude) people saying or implying here and in the mailing list, “We don’t want to do it, people who need it shall do it instead”. Who are “we”? What specific packages are “we” maintaining? Are these packages worth keeping?

“Here is deadline, some intentional vagueness, go figure something out” is not what any sane person wants to hear from leadership.

This mess is not motivating me to join and help; instead I feel motivated to seek other options and keep my sanity.

4 Likes
326

Third party is an over simplification of what’s going on here. Bazzite is downstream from Fedora, and its project lead is involved in Fedora’s community. There’s more cross entanglement than you think. Same with Asahi Linux. People who are maintainers of both distros have posted in this very thread. Asahi Linux is outright a “sister project”.

As for “borderline proprietary tech”, there’s nothing proprietary about Proton. You can run proton with umu outside of Steam perfectly fine. You can fork it (see proton-ge). It does not require the Steam runtime to funciton anymore. You need Steam to run games purchased on steam, which are proprietary (generally, I know Wesnoth and CDDA can be acquired on Steam). Steam is proprietary, no doubt about it, but Proton, DXVK, and other Valve-related projects are definitionally not proprietary. Not in spirit, and not in letter.

Now I’m considering putting Bazzite on my deck, lol.

3 Likes
327

The Change process is a formal Fedora project process that leads to a vote by FESCo via a ticketing system. I believe the change proposer can retract the proposal before the FESCo ticket is opened and that signifies that the discussion topic can be closed as its no longer a germane discussion.

And while chat moderators have power to close this discussion without consent of the proposer, I believe that because this is a formal process with a clearly defined discussion window, it may not be appropriate to close discussion early simply because its repetitive or participants are tired of it.

I would be estatic if we could all agree that the gaming use-case has been sufficiently represented in this discussion and we could create a space for other use cases to be discussed. One of my concerns so far in this discussion is that the gaming community has perhaps dominated discussion a little too strongly and we’ve missed talking more fully about some other use cases that resonate differently with slightly different considerations.

10 Likes
328

Valve provides a Linux runtime for native games on Steam, you can safely use the requires of the Steam RPM in RPM Fusion to generate this list.

7 Likes
330

Hi all,

On Flatpaks / Containers as a solution

I have seen some back and forth on this, however I have not seen anyone mention the following point specifically.

I’m not sure that Flatpaks will particularly solve the key issue at play here - that being the burden on maintainers to make 32bit libraries work. While Flatpaks / Containers are a great option to use runtimes and utilities that would otherwise conflict with each other on a single system, they do not really remove the maintenance burden. Someone still has to build them.

Now there may be a question of who that would be, and perhaps there is an assumption that it would not be the Fedora maintainers, but that leaves the question open: who would it be? Valve, through their absence of support for Flatpaks so far, appear to have made it clear that they wouldn’t in the case of Steam, at the very least.

Perhaps the burden would continue to fall on the community maintainers of the existing Flatpaks, but that doesn’t really feel like “solving the issue”, more just moving it onto someone else. Given one of the core (and quite legitimate!) concerns of the maintainers here is that they aren’t getting enough 32-bit support from upstreams, turning around and doing a similar thing to downstream Flatpak maintainers does not feel like a sustainable solution.

On Steam

I am aware that Steam is not the only concern in this thread - users have mentioned OBS, as well as some CI / Enterprise software requirements. However Steam is one that keeps coming up, and given its complexity and the lack of viable alternatives, that is the issue I will focus on here.

It seems clear to me that we need to understand what Steam’s requirements actually are a bit better, however I fear there may be a lot of truth to the fear that this is a bigger issue than just the Steam client.

As an experiment, I ran Portal 2 on my (linux) laptop via Steam just now. Portal 2 is one of Valve’s flagship games, played by millions. It’s a 32 bit binary. I’d wager a guess that by that logic most of their games are. Asking Valve to find the time to rewrite the Steam Client might happen, but asking them to do the same for all of their games is a bigger problem.

Portal 2 is a native game, and therefore doesn’t run via Proton or Wine, so there is no opportunity for WOW64 style 32 ↔ 64 translations there.

I also looked at what libraries Portal 2 had open during runtime. Interestingly, the vast majority of these appear to be libraries installed by Steam into my local games directory. One presumes that these were provided / managed by Valve in that case, and aren’t the distro versions at all?

We desperately need clarity from Valve on how this works and what they actually need from distributions.

On the strategy as a whole

There are two disparate points I’ve seen in this thread that I’d like to bring together:

  • Fedora does not have an “in” with Valve to get the information / form a plan going forward
  • Ubuntu / RHEL are having to make similar considerations given their much longer LTS windows which are now at risk because of the 2038 problem.

These two points feel very linked to me. I wonder if there is something to be gained in reaching out to the other distributions, and perhaps the Linux Foundation itself, to form a joint response / action plan over this.

Every distribution is going to want to know how to handle the EOL of lib32s, every distribution is going to want to know what to do with Steam, etc. This is a very worth while discussion, but feels a waste to do it alone.

4 Likes
331

I think, whilst I am an avid proponent and consumer of the gaming use case; steam and/or likewise relying on any 3rd party controlled/packed i686 container is a non starter.

Those i686 deps are integral to cross compilation, and even native compilation in some cases. They need to be able to be tracked/versioned/maintained and fixable by the distro. In many ways steam/gaming is a ‘red herring’ , it’s just the most end user visible.

To solve this issue you also have to solve cross-compilation/native deps problem in parallel.

3 Likes
332

It may help to have context.. this is a long journey towards this point.

The last step happend in the Fedora 37 timeframe.

And for people who love reading and irony…
we probably could have just cut and pasted the F37 change discussion here and noone would have really thought it was out of place. Many of the same people, many of the same arguments.

Nothing has changed since F37 till now really. Its the same multlib problem.. just 2 1/2 years later. The steam usecase discussion is not materially different in any way.

What is different now is that we are more misaligned than in F37… meaning somehow the interest in the continued output of 32bit packages seems to have grown while the interest/ability/access in doing the work to maintain those compiled binary artifacts has dimished. That’s a problem {tm}.

My biggest concern at this point, and it should be everyone’s concern reading this, is if there are things in the steam i686 dep stack where the upstream maintainers are no longer even interested in taking in reports about i686 specific bugs any longer that “an untenable problem”{tm} that goes beyond just Fedora but calls in the question the sustainability of the use-case ecosystem wide. That’s the sort of thing we can’t hide from or put off. And I’m kinda concerned about it.

3 Likes
333

Indeed, as @jspaleta has made clear, gaming is not the only use case here; it’s just the most visible. OBS, legacy games, legacy enterprise software, etc. are all software that would become problematic. More than that, it’s not just legacy software; some groups (ie valve, obs) are still using 32 bit in actively maintained software.

Some stuff is only just now finally adopting 64 bit, too. WASM (web assembly) only provided a 32 bit address space until January of this year. The rust WASM64 target is still experimental, with WASM32 being the default web target for rustc. GCC’s people have indicated there’s problems on their end too in the mailing list.

The way I see it, the main usecases in terms of end-user visibility are gaming and some legacy software. However, there’s also concerns from developers of said software, and there are still 32 bit actively maintained libraries in the wild for one reason or another.

The entire x86-64 bit architecture is itself technically a hack, and the fact the entire desktop world operates on it has led to this problem. But it is what it is. As long as 64 bit processors can still run 32 bit software, 32 bit stuff will still be expected to work by many, both users and developers. The only reason Valve even released a 64 bit Steam client for Apple was because of Apple Silicon and the EOL of Rosetta.

1 Like
I686 in Fedora: problem statement
334

I agree with this, but the steam container is an enabled-by-default and tested-by-valve solution. It’ll continue to be deployed regardless of anything we do here. If it means we only need i686 glibc/nm/Mesa (not a complete list), then that’s the bare minimum burden for 32bit games to continue working on Steam.

More generic compatibility outside of that would need long form discussion in something like the proposed gaming SIG.

2 Likes
335

Yeah, exactly. If unpaid volunteers continue to provide 32-bit multilib libraries for them to use, they will logically feel no pressure to change anything.

As @jspaleta mentioned, this is not a new discussion, nor do I think this is the latest incarnation of it.

The better question is: is this right?

3 Likes
336

I want to reiterate that Valve was and is reluctant to do so not for financial reasons but for user experience reasons. Valve makes clear that users should be able to run their entire Steam library so long as they are using the appropriate OS. The reason they haven’t done away with 32bit on Windows and indeed Linux is because there are legacy 32 bit games, and Valve considers it an unacceptable tradeoff to prevent their customers from playing the games they paid for on a supported platform.

Contrast this with Mac, which has a far smaller game library on Steam than even Linux (and no Proton support at all). Any games that were/are for Mac, had to be rebuilt for Apple Silicon to work because Apple is a company that is willing to say “Screw you, do it our way” to both developers and customers. If your program doesn’t run on Apple Silicon, tough luck. So any games that would not work on Silicon are just removed from Steam as being “supported” on Mac period.

But there were far fewer 32 bit Mac supported games than there are Windows and Linux ones. So while Valve was reluctant to make the change, it was a tradeoff they ultimately had to accept.

To be clear, I and many others believe Apple is in the wrong in this story. While Apple Silicon is an interesting piece of hardware, discontinuing existing x86-64 built software was deeply unpopular among developers who use the Apple ecosystem.

Imagine you sold dozens of games to someone, and promised they would always work. To do this, you ensure that you provide a 32 bit runtime – which Steam does do. Additionally, you continue to be a 32 bit program because porting it is time consuming and because your runtime needs 32 bit compatibility anyway to continue to run games you sold to people. Dropping 32 bit on Windows would be a PR disaster for valve for this reason!

There are many, many reasons to criticize Valve, including with regards to this situation (lack of communication, unwillingness to compromise, etc). But to say it is solely a “paid program not wanting to use their resources because they’re greedy” is simply incorrect. If I suddenly lost access to games I bought from Valve because they dropped their 32 bit runtime, I’d be pretty upset at them, as would anyone.

All that aside, there’s more to this conversation than gaming, as has been covered. People expect backwards compatibility. Providing minimal libraries to do so (like Ubuntu’s LTS 32 bit support or Arch Linux’s multilib repository) is not unreasonable. Expecting Fedora to maintain every package as 32 bit is unreasonable, I agree. But this proposal is a sledgehammer to an issue better solved with a screwdriver.

8 Likes
337

Before we get into the details, let me say I appreciate your civil response and different perspective.

Imagine you sold dozens of games to someone and promised they would always work.

If I ever made that promise, then I should be the one keeping it (and I would never make such a promise for my software). It is a costly endeavor to promise that your software will work forever. It is borderline false advertising if you don’t continue to provide support for it, as operating systems, runtimes, and hardware will continue to change. I know this happens, but I do not find it right. Customers should know exactly what they are buying, and vendors should be transparent. But this is a very different discussion.

To be clear, I and many others believe Apple is in the wrong in this story. While Apple Silicon is an interesting piece of hardware, discontinuing existing x86-64 built software was deeply unpopular among developers who use the Apple ecosystem.

This is a fair point. I was affected as well, but I do not find Apple at fault because I never paid them (or had any contract with them) to keep my legacy software running forever. Let me address this bit later, as I don’t want to repeat my point.

All that aside, there’s more to this conversation than gaming, as has been covered. People expect backward compatibility. Providing minimal libraries to do so (like Ubuntu’s LTS 32-bit support or Arch Linux’s multilib repository) is not unreasonable.

And here I think we are getting to the gist of the issue: what is or is not reasonable. I am grateful to the developers who contribute to making my favorite operating system. But I do not take their work for granted, and I try to contribute back if possible. If they think this is a reasonable proposal, then all is well, and we can close this discussion. If they do not, then somebody else (like the people and companies benefiting from this work) should step up and do the work.

It is really a very simple situation—nobody is entitled to anyone’s work, especially when it is voluntary.

3 Likes
338

Both Asahi Linux and Bazzite are contributors back to and active in upstream maintenance and ongoing developments. In Asahi’s case, their work is directly used for the Fedora Asahi spins.

A very good way to drive off contributors is to tell them that the technical requirements that they are unable to do without for the work they are providing is “entitlement”, and that if they don’t have the resource or funds for it (like Fedora, majority backed by Red Hat, has more of), they should just pack up bags, close up shop, and leave.

2 Likes
339

Has anyone stepped up yet to be a contributor for 32-bit support on Fedora? Is this something contributors can handle?

340

Conversely, have we actually verified that the packagers for the small number libraries needed for gaming are actually opposed to continuing to package the 32-bit versions?

2 Likes
341

Several people have expressed interest in assisting with maintenance of 32-bit support in some capacity:

For many, it’s not a sense of urgency, it’s a sense of doom - less “we need to act quickly to solve this problem” and moreso “we’re about to lose everything we’ve built”.

And sentiment from several members of the detractor side seems to be outsized and thinly veiled contempt towards downstream consumers, and possibly even their continued existence in the Fedora ecosystem.

Just a year out is also very soon, especially for smaller downstreams. Only one buffer release between now and having to completely replace and supplement the entirety of the i686 build infrastructure, for teams perhaps 5% of the size of Fedora, at the best.

This is likely why some have said this proposal as it currently stands threatens to do damage to the community. The technical discussion becomes secondary to the survival anxiety.

1 Like
342

It’s not clear to me this is a fair question to ask of anyone yet.

Don’t get me wrong I would love to be able to directly challenge individuals to step up and point them to clearly communicated opportunities. for individual packages that need an i686 focused maintainer.

But because we dont have an orphan process for i686 builds that looks like the orphan process for source packages. Its not clear to me that before this discussion, that the project has communicated the opportunity for i686 specific maintainers as a thing people could stand up for. And because we havent communicated theses opportunities, I don’t feel comfortable guilting people about failing to step up before now.

Historically at the src package level the package orphan announcements have sort of created an urgency in the same way this discussion has for i686. The package orphan process has let us communicate opportunties for people to get involved. Sometimes the external tech press has picked up on it and it brought heat when the orphaned package was notable or widely used. See a pattern? Our communicated intent to drop something creates the ncessary urgency for self-interested people to step up and take on the work.

We just dont have anything like that for i686 maintainence yet. If we had i686 builds be its own thing, somhow (TBD) , then I would expect some sort of orphan notification process would be spun up as a useful practise to advertse opportunity for involvement within the scope of that set of i686 builds.

3 Likes
343

I want to be very clear.

I LOVE the downstream consumer ecosystem. LOVE IT.

It is unfortunate for all of us that one of the most notable outputs in that space is dependent on the i686 stuff, which is a bit of a mess.

But just because i value the donstream work, it doesn’t replace the need for contributors who can do the work to maintain the things built in fedora that the downstrems depend on.

Maybe I have a different perspective, but my version of doom is a lot different than others. I think its far far worse an outcome that we end up in a world a year from now where we find a i686 specific security issue and we don’t have a maintainer or even an upstream that is willing to deal with but we are still just generating i686 packages because people expect them.

If we get to that sort of future, you don’t necessarily even get the luxury of a change process with a 1 year execution timeline. I’m trying to avoid that future. Do you feel the urgency yet?

4 Likes
344

So why not focus on clearly communicating the intent to orphan 32-bit packages due to the maintenance burden? This would allow interested parties to step up to maintain them and provide the associated technical solutions to make it possible, such as separating 32-bit and 64-bit packages. Who knows, there may be enough interest from corporations or individuals to step up and help maintain the packages.

I still personally believe that the majority of these cases should be handled by container-based technologies. This issue is not specific to Fedora; container-based solutions like Flatpak and Podman address the problem for everyone, even if they do not resolve all the issues, as was pointed out to me with Gamescope. However, this particular case is something Valve can address with a native 64-bit Steam client, but that is ultimately a different discussion.

345

It’s fairly unlikely that maintainers will complain so long as the set of packages is reasonable. Maintaining the small set of i686 packages required by Wine and Steam, and by the GNU and LLVM toolchains, is just not a very big risk to Fedora.

I would draw the line at supporting native 32-bit Linux games with dependencies on arbitrary system libraries. To the extent those still miraculously work, it would be more realistic to use the Windows version of the game under Proton, or use a container.

Change proposals get deferred all the time, so don’t worry too much about the date. If people form an i686 SIG and are actively working on this problem, the proposal owner will surely just push back the date as needed.

3 Likes