Hi, all,
Tl;dr
Let’s change the way we version rpms
Current: dnf-4.9.0-12.fc36.noarch.rpm
Proposed: dnf-4.9.0-12.fc36.34897715.noarch.rpm
Intro
I want to start Fedora part of the conversation which we are having in various places now.
I’ve been running with this idea for quite some time. I think I mentioned it couple of times in Fedora Nest hallway tracks, devconf’s, and council meetings, but didn’t get very far with it.
But now I think we have an opportunity to turn this proposal into a real work item, and I would like to know your opinions and thoughts on it.
Note: I am going to update this top-post based on the feedback I get.
Concept
The main underlying principles driving the idea are:
- Build environment is a third equal player in the build process.
We have application sources, which are versioned by the Version tag of an rpm. We have spec files, which are versioned by the Release tag of an rpm. The third item you need to define the outcome of the build process is Build environment, and therefore we need visibility and versioning for it.
- Rpm sources can have multiple applications, and the same sources can be build into different binary artifacts.
This goes in line with conversations we had regarding Fedora ELN, USE flags, Minimization activities and so on.
If we consider Fedora packages as building blocks, which one can mix and match to achieve their own goals, like building Fedora ELN with el-like macro, or a custom Fedora Remix with a minimalistic approach to dependencies, or a module with a different set of build tools, we don’t want every such use case to generate a fork of the dist-git repository and diverge form the official Fedora upstream.
If we stop encoding vendor-specific data (like, for example, mass-rebuild events) in dist-git sources we can start treating dist-git as a shared space, where people can collaborate on features required for packaging a certain app in various forms.
What others do
-
in case of mass-rebuilds or soname bumps in Fedora we create an “empty” commit to the dist-git repository which bumps a release number in the spec file
smth-1.2.3-5.fc36
→smth-1.2.3-6.fc36
. -
in case of Fedora ELN we bump the disttag macro for the entire buildroot (
smth-1.2.3-5.eln101
→smth-1.2.3-5.eln102
), which allows us exactly one rebuild of any package. -
in case of Packit the release field is filled using the current date.
-
ostree uses build time
-
OBS uses counters provided by the build system
Goals
- Provide a possibility to change build environment and rebuild rpm packages without changing their content: neither sources nor spec files.
- Set a common standard for the RPM-based ecosystem, which can be used not just within Fedora, but also by Remixes, downstreams, SIGs and other distributions.
Possible benefits for Fedora
Here we collect some possible applications of this change in Fedora.
Note that while this gives us some practical examples for how the Build tag can be used in the future, each of this applications requires a discussion of its own and it is not a part of the Build tag proposal.
Implementing Build tag in rpm doesn’t immediately bring a change to Fedora workflows. It is essentially a refactoring of our approach to rpm versioning. The goal is to remove certain obstacles, so that the future changes can be designed and implemented without this limitation.
Replace mass-rebuild by a targeted continuous rebuild
We can uses analysis provided by the Koschei system, to implement a auto-rebuild loop so that there wil be no need for a scheduled mass-rebuild.
Deal with soname bumps without proven packager access to all Fedora sources
Whenever there is a soname bump or any other change in the buildroot, rebuild of a dependent package can be triggered automatically without a version bump in the sources.
- This simplifies packaging for various toolchains and libraries.
- Possibly aids in quicker resolution of CVEs for components that use statically linked language stacks (golang/rust)
Fix build order issues without dist-git commits
We can rebuild packages in the right order, without spending time on creating new pull requests.
Real pull-request workflow
Possibility to build real rpms on pull-request, put them in a sidetag and rebuild easily on clean rebases and updates before merging.
Proposal for implementation
Current: dnf-4.9.0-12.el9_0.noarch.rpm
Proposed: dnf-4.9.0-12.el9_0.34897715.noarch.rpm
Now → Compatibility mode
- Introduce
Build
tag in the rpm metadata - Introduce “build reason” to be added to rpm changelog as a top entry
- Enable passing Build tag value to the build in Koji. The value of the Build tag will be set from Koji build id.
- Introduce macro in
Release
field of the rpm spec files, which adds Build tag after the usual disttag- Release: 12.%{?dist}.%{?build}
- Introduce option to pass “build reason” to a Koji build via koji cli and fedpkg/centpkg tooling.
Compatiility mode → Final
- Implement support for the upgrade path on the rpm side in a compatible way. So that NV(R’=R+B) and NVRB are treated the same.
- Remove %{build} part from Release tag and use it as independent tag
Now | Compat | Final | |
---|---|---|---|
Name | smth | smth | smth |
Version | 1.2.5 | 1.2.5 | 1.2.5 |
Build | 3175428 | 3175428 | |
Release | 5.{dist} | 5.{dist}.{build} | 5.{dist} |
Release | 5.fc36 | 5.fc36.3175428 | 5.fc36 |
Filename | {name}-{version}-{release} | {name}-{version}-{release} | {name}-{version}-{release}.{build} |
Filename | smth-1.2.5-5.fc36 | smth-1.2.5-5.fc36.3175428 | smth-1.2.5-5.fc36.3175428 |
Why not rpmautospec
rpmautospec is a shortcut which allows you to not fill Release tag in the dist-git manually. But to rebuild a package you still need at least one additional commit to dist-git, which will change the value that macro calculates. In the end the number is tied to a dist-git history.
rpmautospec automates the Release tag itself (as a version for spec files), it doesn’t have a Build component in it. Thus, while rpmautospec is a valuable tool for working on dist-git sources, its development doesn’t overlap with the Build tag topic.
And, for example, rpmautospec will not help in the case we need to update a build on pull request update: When you work with pull-requests you don’t necessarily add commits, you rework the history of a branch from which you run a PR. Sometimes even reducing the number of commits in it.
Other links
- Upstream Build version (build tag?) in RPM · rpm-software-management/rpm · Discussion #2031 · GitHub
- Mailing list [RFC] Build tag in RPM: from NVR to NVRB - devel - Fedora Mailing-Lists
Open questions
Build macro
{build}
is a bad name for a macro as it overlaps with %build section in the rpm spec- Must
{build}
contain the dot at the beginning? Or can it be avoided- Update: Looks like
%{?build:.%{build}}
is the solution for this case
Is there a simpler way thanRelease: 12.%{?dist}%{?build:.%{build}}
- Update: Looks like
Build reason
- Does it need to be in the binary artifact?
- How can we store and navigate build reasons if we have three builds of the same sources
- Any alternatives?
Cross-distribution upgrade paths
- If remixes build their own binaries from the same sources with their own vendor-specific implementation of a build id, should there be upgrade path between such builds?
Source of a build tag value
Current proposal suggest to use Koji Build id as a value for Build tag. But it is not required. The only mandatory requirement for the Build tag is that it monotonically increases within the context of a specific Name-Version-Release combination. As soon as we bump the Release tag, Build tag can be reset to 0.
- What are the alternatives for the Build value in case of Fedora using Koji build system?