F40 Change Proposal: Build With DNF5 (Self-Contained)

Project Discussion Change Proposals
,
1

Build Fedora with DNF 5

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

https://fedoraproject.org/wiki/Changes/BuildWithDNF5
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/5B3TF7LQIWSOVTMTT2B3ISEIH7RLOBDI/

:link: Summary

We are proposing to change the Mock configuration in Mock (mock-core-configs), Koji, and Copr to use DNF 5 as Mock’s package manager instead of DNF 4. DNF 5 would be used by Mock to install build dependencies into chroots for package builds. This change is related to the build infrastructure and is distinct from changing the default package manager in Fedora.

:link: Owner

:link: Current status

  • Targeted release: Fedora Linux 40
  • Last updated: 2023-10-23
  • [ devel thread]
  • FESCo issue:
  • Tracker bug:
  • Release notes tracker:

:link: Detailed Description

DNF 5 is a new package manager intended to replace DNF: About — dnf5 documentation. It offers significant performance improvements over DNF while achieving lower memory usage and disk footprint. The switch to DNF 5 was originally planned for Fedora 39, but it’s been postponed to (likely) Fedora 41: Issue #3039: RFC: Roadmap for DNF5 in Fedora 39 / invoking the Contingency Mechanism - fesco - Pagure.io.

In the meantime, we would like to start building Fedora with DNF 5. The set of package management features that Mock needs for setting up buildroots is small compared to the full capabilities of DNF, so it’s a good place to start deploying DNF 5. We will be able to test the stability of DNF 5 at a large scale and gather data about its performance.

The Mock developers have been working alongside the DNF 5 developers for a while to ensure Mock can use DNF 5, per this tracking issue: Mock compatibility with DNF5 · Issue #894 · rpm-software-management/mock · GitHub. The two remaining items on that issue are “optional” items that are not blocking this proposed change.

:link: Feedback

:link: Benefit to Fedora

With the switch to DNF 5 as the default package manager on the horizon, the build infrastructure offers an opportunity to subject a crucial subset of DNF 5’s features to heavy testing. This change will let us verify that every build dependency in the distribution is installable by DNF 5.

In addition, we expect a substantial performance improvement for package builds that spend a significant portion of their time installing build dependencies. Larger, compilation-heavy packages likely won’t see much improvement; the difference will be most apparent when building many smaller packages. Switching the build system over to DNF 5 will let us measure the performance improvement over DNF across a wide variety of install transactions.

:link: Scope

  • Proposal owners:

The work to support DNF 5 in Mock is done already. This change should be as simple as setting the Mock option config_opts['package_manager'] = 'dnf5' in Mock, Koji, and Copr for F40+ builds (Koji config option exists from the yum -> dnf4 era). The dnf5 doesn’t necessarily have to be installed on building hosts - in such a case, Mock will automatically use /bin/dnf-3 (DNF4) from the host to install DNF5 into the bootstrap chroot, to further use that DNF5 for build chroot installation.

:link: Upgrade/compatibility impact

:link: How To Test

There are no special steps needed to test the change after it happens (updated mock-core-configs package is installed on your host), just enjoy the installation speedup.

There’s a way to test this on Fedora 37+ or EPEL8+ host (builder) in advance. Considering you want to build SRCRPM like mock -r fedora-40-x86_64 your.src.rpm, you can do this instead:

  1. mock -r fedora-40-x86_64 --scrub=all (mandatory step to cleanup DNF4 from all caches)

  2. mock -r fedora-40-x86_64 --config-opts=package_manager=dnf5 your.src.rpm (DNF5 is installed and cached in bootstrap)

  3. mock -r fedora-40-x86_64 --scrub=all (to invalidate caches again)

:link: User Experience

This change will mostly be invisible to users. The builds, namely the buildroot preparation, will be much faster.

:link: Dependencies

:link: Contingency Plan

  • Contingency mechanism: (What to do? Who will do it?)

Revert the F40 Mock configuration in Koji and Copr back to using dnf (5-minute work).

  • Contingency deadline:

F40 Beta freeze

  • Blocks release? Yes

:link: Documentation

:link: Release Notes

2

So, does this change imply that mock-core-configs enabling dnf5 for f40+ builds would be pushed to all fedora/epel releases? Or just rawhide?

ie, will end developers building in mock on their local stable fedoras also see this config change? Or only if they are running rawhide?

If yes, it might be worth doing a slightly staggered roll out: ie, change in copr for a short while, then change in koji, then change in end users mock config?

Otherwise as far as can recall we fixed all the issues in koji for dnf5 building, so I like this plan. :slight_smile:

3

Yes, updated mock-core-configs would be pushed to all Fedora/EPEL releases. So developers would use it locally, too.

If accepted, I think we can make the change in Copr first. However, to simplify the process, I’d prefer to do the Copr change via an updated mock-core-configs package (installed to Copr from updates-testing/infra repos), which would imply wrapping a new mock-core-configs release first.

Perhaps we could keep the package a little longer in updates-testing if we prefer Koji doing the switch strictly before do the switch for users.

4

Yes, updated mock-core-configs would be pushed to all Fedora/EPEL releases. So developers would use it locally, too.

great!

If accepted, I think we can make the change in Copr first. However, to simplify the process, I’d prefer to do the Copr change via an updated mock-core-configs package (installed to Copr from updates-testing/infra repos), which would imply wrapping a new mock-core-configs release first.

Perhaps we could keep the package a little longer in updates-testing if we prefer Koji doing the switch strictly before do the switch for users.

Sure. It might be nice to test in staging before we switch koji in prod,
but we can do that anytime before…

5

This change proposal has now been submitted to FESCo with ticket #3096 for voting.

To find out more, please visit our Changes Policy documentation.

6

Is the plan to make this change before the F40 mass rebuild?

7

Does this really count as a Self-Contained Change? Changes to the buildsystem impact the entire distribution.

8

There was no planning WRT mass rebuilding, we were rather thinking about “do the change as soon as possible”. What’s your preference?

Since “no issues are expected” (trademark), I think we could do it before F40 rebuild. One reason is to speed the mass-rebuild up (DNF5 is reasonably faster) and the second reason is to get extra testing of DNF5.

9

I agree that it makes sense to do this with enough time before the mass rebuild. It’s a lot easier to revert it if it turns out to break things before the mass rebuild than during it.

10

@gotmax23 seems self-contained, dunno? The change should be mostly invisible to users, and there’s an easy way to revert if something gets broken.

11

There’s also an option to do it after the mass-rebuild…?

12

A self-contained change is a change to isolated package(s), or a general change with limited scope and impact on the rest of the distribution/project. […]

vs.

System-wide changes involve system-wide defaults, critical path components, or other changes that are not eligible as self-contained changes. […]

This does not sounds like a self-contained change.

13

I don’t have a preference, but since it could potentially have a major impact on the mass rebuild, I think it would be good to make a decision now about doing it before or after the mass rebuild and add that to the proposal.

14

It should not affect the shape of the built packages because DNF5 is intended to directly replace DNF4 (for the installation of BuildRequires, etc.). However, we probably have to consider potential bugs while picking the change category… Let’s switch then, and let’s propose doing the change before mass-rebuild.

1 Like
15

I’ve added the following statement into the contingency plan section:

We plan to implement the change at least a month before the F40 Mass Rebuild event starts, ideally as soon as possible. The builds conducted prior to the mass rebuild will offer us added assurance that the mass rebuild won’t be affected, and will provide us with time to address any potential issues in advance.

In the context of ‘system-wide’ vs. ‘self-contained’ categorization: The anticipated (let’s say ‘expected’) issues are potential build failures. The scenario is such that either the builds will continue to succeed (build deps installed by dnf5), or they might start failing (missing packages, dnf5 segfaults, …), necessitating a prompt fix or a worst-case scenario of reverting the change (which would take approximately a few minutes in case of Koji). This alteration shouldn’t significantly impact the structure of F40 or cause miscompilations. Given this, I believe it might be preferable to maintain this as self-contained. Due to time constraints, I won’t be able to update the wiki page before today’s FESCo meeting. FESCo may decide to switch the proposal to ‘system-wide’, I’ll dump a note into the issue.

1 Like
16

To me, the crucial thing isn’t necessarily scope of impact to the distribution[1]. It’s potential impact to people.

  1. although that’s important ↩︎

17

Changes to the build system affect every single package in the distribution and should thus be considered System-Wide. Anyways, it seems FESCo already decided to make this a System-Wide Change.

18

I agree with you. Just note that, in this sense, the change is similar to any routine change in Mock, OR in the current DNF implementation on the build host, OR in the bootstrap chroot (more likely causing some problems) OR in any library or tool (e.g. RPM) transitively affecting the DNF’s behavior. The level of transitivity in the build systems is insane, but - at least in this case - we have a very powerful braking mechanism.

it seems FESCo already decided to make this a System-Wide Change

No, I think there are no decisions so far actually. However part of FESCo stated their preference, and so we switched this to a system-wide change.

Matt wrote: To me, the crucial thing isn’t necessarily scope of impact to the distribution[1]. It’s potential impact to people.

ACK. I appreciate that we all care! And yeah, let’s give this change a bit more visibility.

19

So, the first step → there are (autokarma disabled) updates of mock-core-configs, https://bodhi.fedoraproject.org/updates/?search=mock-core-configs

Anyone is welcome to give the update a try. Next week (maybe Monday), I’ll try to push the update to Fedora Copr and we’ll start testing there.