I686 in Fedora: problem statement

Project Discussion
,
1

Let me try to summarize the understanding of the problem with 32-bit libraries in Fedora.

The top post might be updated later Corrections or additions are welcome.

Q: Why keeping the status quo is bad?

Overhead on maintainers

Maintainers of the i686 libraries currently are forced to keep building and maintaining them even when the upstream disappears or no longer supports the bug resolution, because someone else might depend on them for the unknown reason.

A failure to build on the i686 task blocks the entire build from succeeding. If the upstream is unwilling/unable to assist with the build failure, that puts quite a bit of stress on the maintainer to figure out the problem themselves so they can keep updating the package for all other architectures. This is even more stressful when CVEs are involved. They either have to solve the problem themselves, or exclude the i686 architecture. The latter can be quite disruptive if it isn’t a leaf package.

Overhead on infrastructure

We generate a lot of load on infrastructure and people maintaining it simply by building the packages. It roughly can be guesstimated to up to 20%(corrections welcome!) of the load of the infra, not counting the effort of engineering resources needed to perform a successful mass-rebuild every release.

Overhead on infrastructure tooling

The current implementation of the multilib creates constant overhead on infrastructure tooling (test tools, analytics, admin tools,..) which need to handle special cases and keep additional logic for i686 packages.

Q: Why it is hard to phase out 32-bit libraries gradually in Fedora

Inertia

Simply lack of awareness among existing and new packagers about the initiatives like:

This is partially solved by the recent discussion I guess :slight_smile:

Opt-out vs opt-in

New packages are created with i686 builds enabled which grows the 32-bit dependency tree. Which goes against the efforts to trim it.

The possible solutions here:

  • introduce IncludeArch tag in the RPM spec file - requires work in RPM upstream OR abuse one of the existing tags, for example, Group to store the opt-in information;
  • AND introduce a check for explicit mention of i686 in the rpm spec file in Koji - requires patch in Koji upstream like:
if not 'i686' in koji.get_header_field(h, 'group'):
  archlist.remove('i686')

Lack of tooling and data to track workloads and their dependencies

It is not easy to figure out which packages are the dependency tree of the 32-bit libraries and which workloads depend on them. Especially when we talk about libraries which might be useful for tools and apps not packaged to Fedora.

This issue is unresolved. The attempts to solve it include:

“no multilib support in koji”

noarch dependencies may resolve to i686 dependencies in the background increasing the 32-bit dependency tree.

If an arch-dependent package has a dependency on a tool like asciidoctor or pytest to build docs or test a package during build time, these dependencies will be resolved according to i686 architecture and pull in ruby.i686 or python.i686 even though ruby.x86_64 would be enough for the use case.

6 Likes
2

This is not true. Since Changes/EncourageI686LeafRemoval - Fedora Project Wiki leaf packages are ENCOURAGED to exclude i686. And if upstream does not support i686 you are free to exclude it too per Fedora Packaging Guidelines :: Fedora Docs (note that “does not work” is for me the same as “upstream does not support it”)

3

Focusing on the packages is premature optimization. We should identify workloads first. Vote (in FESCO?) which we want to support (i.e., keep) in Fedora Linux. And only then resolve the package set.

4

I disagree. I think that the size of the dependency tree is an important part of the conversation. Decision might be quite different if we choose between 5000 or 50 packages.

But collecting workloads with proper descriptions and reasoning is a good entry point anyway.

5

Can we not coordinate with major players? Canonical, Valve, Arch, Linux Foundation, etc, to pool resources to transition things off I686? While distros may not have the power to transition proprietary software, surely we can do PRs or forks for open source packages? Better yet, in terms of maintainer time, merge down related dependencies into more manageable projects?

If it really is impossible to transition everything over, then maybe coming together as a Linux community, we can make arrangements to maintain compatibility into the future without massive overhead? It seems to me that solving this kind of problem comes with the “cutting edge” territory that Fedora seems to want to occupy and if we don’t have the resources to do it alone then collaboration seems the logical move.

2 Likes
6

For better or worse, I noticed that being “forced” to build on i686 exposed (C type) problems in upstream’s source whenever (OK: often when) we update gcc and gcc becomes stricter.

On the other hand, whenever fixing upstream is too difficult, we use a compiler switch and just move on because noone runs this stuff on i686 anyways. So this might be a non-argument.

7

“Forced” is probably a too strong word here. But the “encouraging” is also too optimistic. Maintainers are expected to act responsibly and not simply remove the brick from the foundation of someone else’s work unknowingly. Thus we are encouraging to do the research and build the dependency graph themselves before considering the removal. And it is not really working at the moment, afaik.

3 Likes
8

To elaborate on this point, a failure to build on the i686 task blocks the entire build from succeeding. If the upstream is unwilling/unable to assist with the build failure, that puts quite a bit of stress on the maintainer to figure out the problem themselves so they can keep updating the package for all other architectures. This is even more stressful when CVEs are involved. They either have to solve the problem themselves, or exclude the i686 architecture. The latter can be quite disruptive if it isn’t a leaf package.

How would IncludeArch differ from the existing ExclusiveArch?

9

Added to the top post.

The idea is that we need to specify that package is opting in for i686 architecture additionally to all the other default architectures.

ExclusiveArch: i686 will build on i686 only.

IncludeArch: i686 would build on x86, aarch64, s390, ppc (as by default) AND on i686

10

I’m worried about the tag approach for two reasons.

Introducing a new RPM tag inevitably means that there will be earlier versions of RPM that don’t recognize that tag. The way this usually affects packagers is they try to merge changes from Fedora branches to EPEL branches, and then realize that introduces an unrecognized tag which fails the build in EPEL. This could be avoided with the abuse of an existing tag like Group, but that feels wrong as well.

Introducing koji code changes to handle this sounds messy. The code suggested in the first post sounds like the koji tag would still need to define i686 as an architecture in the build tag, but that would lead to surprising behavior for a koji admin who configures i686 as a build tag architecture but then builds a spec file that doesn’t have the IncludeArch tag for i686. For this tag to be viable, I think koji would need deeper changes. The current build tag architecture list is both the allowed architectures and the default architectures. I think koji would need to split those concepts, so that koji admins could configure an architecture as “allowed and default” or “allowed and non-default”.

11

Considering we only need to “tell Koji” something and Koji already builds from distgit, could we maybe introduce a file in distgit that serves as a configuration for i686 inclusion?

For example, an empty i686.enabled file in the root of the distgit repository[1].

Benefits of this approach:

  • Requires no changes on the RPM level.
  • Such file is trivially added/removed by code. No need to parse spec files and figure out where to add a new tag, or where to remove it from (especially if the specfile has a lot of %if spaghetti).

Problems:

  • If we need to include a package in i686 set on f44 but not on f45 we need to diverge dist git branches. Some maintainers prefer not to do that, and some will inevitably merge back and forth and break this. Perhpas an empty file is too simple and we need proper configuration (e.g. something like package.cfg with i686_brnaches key).

  1. But feel free to paint this bikeshed any colour you like[2] ↩︎

  2. As long as it’s black, of course. ↩︎

1 Like
12

I see three broad use cases for 32-bit packages that don’t have a time limit. One important use case that was mentioned in the big discussion, but only a few times, so it was easy to miss is compiler and library development. glibc and llvm development was mentioned… Similarly, I want/need to test systemd compiled for 32 bits as part of upstream development. I do this either using the full 32 root, or with -m32 and multiarch packages. I very much want to do this with Fedora, using normal packages, so that I have the latest&greatest dependencies. This use case is not going to go away in the foreseeable future, because there are and will be 32-bit users of the software.

The second use case that is not going away is 32-bit hardware (and 16-bit too!), incl. various embedded environments. People will want and need to do native and cross compilations for that hardware. One of the primary goals of Fedora is to provide a great development environment, and this means that the basic 32-bit userspace is needed. And as long as 32-bit hardware is in play, we’ll need libraries and tools for 32-bit, i.e. we’re back to item #1.

The third case, already discussed extensively, is legacy 32-bit games and binaries. While some games and tools might be updated, there is a long tail of binaries that will never be recompiled. I don’t accept the solution of “use a 3rd party container” or “use packages from Fedora 41” as a solution. It is much better to be able to run those programs in an environment that is maintained and updated. No matter what Valve does, this use case is not going to go away.

Those three cases mean that demand for 32-bit environments is not going away, not in the near future and probably not for at least a decade or two. There are good reasons to maintain it and spend resources, because if is an important part of the “developer story”. Thus, I think we need to figure out a way to sustainably provide a subset of 32-bit libraries and tools, i.e. the solution should not be geared towards eliminating 32-bit packages, but significantly pruning them.

Lack of tooling and data to track workloads and their dependencies

The efforts to improve tooling are great. With some reasonable tooling we can probably cut down the number of 32-bit packages to ~500, i.e. by 95%. If we do that, I think the load on the infrastructure will be acceptable and not something to worry about. The figuring out of cases and the development of content resolver tooling can happen in parallel. We’ll should keep refining both based on feedback.

4 Likes
13

I’d say, personally, I like using the Group RPM tag more than adding a new config file in dist-git.

Having a dedicated group for i686 packages makes sense to me even generally, without Koji logic attached to it. So it would be just using the group tag correctly to figure out which package belong to it.

But in the end IncludeArch, Group or i686.enabled - all of those three approaches do the same - they put the configuration into a dist-git per package per branch, done by maintainer or proven packager.

The significantly different approach would be to have the externally managed repository of this data, one per distro. The benefit would be that one could manage entire trees of dependencies adding or removing large groups of packages to the i686 opt-in list via a single commit.

The downside would be that it will require someone to take active ownership of this data, as opposed to asking each maintainer to manage it themselves.

14

I think I have a similar take away.

Yes, we can more or less agree that 32-bit is staying for a while, so we need to deal with it, but ideally to address at least some of its pain points .

The goal of the thread is to move past that part. Let’s list those specific pain points, and let’s figure out if we can choose one where the low-effort high-gain change is possible.

I don’t have a way to help with infra tooling overhead, it is painful, but the “opt-out vs opt-in” item, which was highlighted multiple times in the previous discussion, seems to be solvable.

Yes, with a hack, but still, can we maybe tolerate a tiny simple hack :slight_smile: for the big win it brings?

1 Like
15

That seems to be rather optimistic - based on the content-resolver test run, we’ll need at least 500-1000 Rust packages. And there’s not really a good way to cut down that number (other than disable test suites etc. on i686, which kind of defeats the point of having them).

16

This also has the downsides of 1) adding more moving parts, 2) adding more responsible people, and 3) adding more round-trip-time between identifying an issue and fixing it.

The information which architectures a packages should be built for is already stored in each package (via ExcludeArch / ExclusiveArch) - treating i686 differently would be a bit strange. And it would mean that package maintainers can’t on their own re-enable building a package to build on i686 (and they would need to go through changing that external list instead).

I’m not saying it’s a bad idea, but it will add more friction to the process than “just edit your spec file”.

1 Like
17

Btw, can someone elaborate a bit more on this. Because I don’t quite understand how it works.

If we have a package which has BuildRequires on let’s say asciidoctor to build docs, or pytest to run tests.

And then I try to build it for i686.

Will it fetch the asciidoctor.i686 and pytest.i686 to complete the build?

18

Those two are kind of a bad example since they’re both noarch packages (the first is Ruby, the second one Python, both without native extensions). The only i686 arch packages getting pulled in by those two are 1) the Ruby interpreter (and any native Ruby extensions) and 2) the Python interpreter (and any native Python extensions).

But yes, all dependencies for i686 arch builds need to be available from the i686 arch repositories (which also includes noarch packages).

19

So if I am building a C++ library for i686, and it has docs in asciidoc format, then during the library build for i686 I will fetch asciidoctor.noarch package, but it will in turn fetch ruby.i686 as a dependency.

And if ruby.i686 is not available, it will not fallback to ruby.x86_64, it will rather fail.

Right?

20

That’s correct. The build runs in a pure 32-bit / i686 build environment and doesn’t even have access to packages from any other architecture.

1 Like