[RFC] Build tag in rpms: NVR -> NVRB

[bookwar (Aleksandra Fedorova)]

Hi, all,

Tl;dr

Let’s change the way we version rpms

Current: dnf-4.9.0-12.fc36.noarch.rpm
Proposed: dnf-4.9.0-12.fc36.34897715.noarch.rpm

Intro

I want to start Fedora part of the conversation which we are having in various places now.

I’ve been running with this idea for quite some time. I think I mentioned it couple of times in Fedora Nest hallway tracks, devconf’s, and council meetings, but didn’t get very far with it.

But now I think we have an opportunity to turn this proposal into a real work item, and I would like to know your opinions and thoughts on it.

Note: I am going to update this top-post based on the feedback I get.

Concept

The main underlying principles driving the idea are:

1. Build environment is a third equal player in the build process.

We have application sources, which are versioned by the Version tag of an rpm. We have spec files, which are versioned by the Release tag of an rpm. The third item you need to define the outcome of the build process is Build environment, and therefore we need visibility and versioning for it.

2. Rpm sources can have multiple applications, and the same sources can be build into different binary artifacts.

This goes in line with conversations we had regarding Fedora ELN, USE flags, Minimization activities and so on.

If we consider Fedora packages as building blocks, which one can mix and match to achieve their own goals, like building Fedora ELN with el-like macro, or a custom Fedora Remix with a minimalistic approach to dependencies, or a module with a different set of build tools, we don’t want every such use case to generate a fork of the dist-git repository and diverge form the official Fedora upstream.

If we stop encoding vendor-specific data (like, for example, mass-rebuild events) in dist-git sources we can start treating dist-git as a shared space, where people can collaborate on features required for packaging a certain app in various forms.

What others do

1. in case of mass-rebuilds or soname bumps in Fedora we create an “empty” commit to the dist-git repository which bumps a release number in the spec file smth-1.2.3-5.fc36 → smth-1.2.3-6.fc36.

2. in case of Fedora ELN we bump the disttag macro for the entire buildroot (smth-1.2.3-5.eln101 → smth-1.2.3-5.eln102), which allows us exactly one rebuild of any package.

3. in case of Packit the release field is filled using the current date.

4. ostree uses build time

5. OBS uses counters provided by the build system

Goals

• Provide a possibility to change build environment and rebuild rpm packages without changing their content: neither sources nor spec files.
• Set a common standard for the RPM-based ecosystem, which can be used not just within Fedora, but also by Remixes, downstreams, SIGs and other distributions.

Possible benefits for Fedora

Here we collect some possible applications of this change in Fedora.

Note that while this gives us some practical examples for how the Build tag can be used in the future, each of this applications requires a discussion of its own and it is not a part of the Build tag proposal.

Implementing Build tag in rpm doesn’t immediately bring a change to Fedora workflows. It is essentially a refactoring of our approach to rpm versioning. The goal is to remove certain obstacles, so that the future changes can be designed and implemented without this limitation.

Replace mass-rebuild by a targeted continuous rebuild

We can uses analysis provided by the Koschei system, to implement a auto-rebuild loop so that there wil be no need for a scheduled mass-rebuild.

Deal with soname bumps without proven packager access to all Fedora sources

Whenever there is a soname bump or any other change in the buildroot, rebuild of a dependent package can be triggered automatically without a version bump in the sources.

• This simplifies packaging for various toolchains and libraries.
• Possibly aids in quicker resolution of CVEs for components that use statically linked language stacks (golang/rust)

Fix build order issues without dist-git commits

We can rebuild packages in the right order, without spending time on creating new pull requests.

Real pull-request workflow

Possibility to build real rpms on pull-request, put them in a sidetag and rebuild easily on clean rebases and updates before merging.

Proposal for implementation

Current: dnf-4.9.0-12.el9_0.noarch.rpm
Proposed: dnf-4.9.0-12.el9_0.34897715.noarch.rpm

Now → Compatibility mode

1. Introduce Build tag in the rpm metadata
2. Introduce “build reason” to be added to rpm changelog as a top entry
3. Enable passing Build tag value to the build in Koji. The value of the Build tag will be set from Koji build id.
4. Introduce macro in Release field of the rpm spec files, which adds Build tag after the usual disttag
   • Release: 12.%{?dist}.%{?build}
5. Introduce option to pass “build reason” to a Koji build via koji cli and fedpkg/centpkg tooling.

Compatiility mode → Final

6. Implement support for the upgrade path on the rpm side in a compatible way. So that NV(R’=R+B) and NVRB are treated the same.
7. Remove %{build} part from Release tag and use it as independent tag

	Now	Compat	Final
Name	smth	smth	smth
Version	1.2.5	1.2.5	1.2.5
Build		3175428	3175428
Release	5.{dist}	5.{dist}.{build}	5.{dist}
Release	5.fc36	5.fc36.3175428	5.fc36
Filename	{name}-{version}-{release}	{name}-{version}-{release}	{name}-{version}-{release}.{build}
Filename	smth-1.2.5-5.fc36	smth-1.2.5-5.fc36.3175428	smth-1.2.5-5.fc36.3175428

Why not rpmautospec

rpmautospec is a shortcut which allows you to not fill Release tag in the dist-git manually. But to rebuild a package you still need at least one additional commit to dist-git, which will change the value that macro calculates. In the end the number is tied to a dist-git history.

rpmautospec automates the Release tag itself (as a version for spec files), it doesn’t have a Build component in it. Thus, while rpmautospec is a valuable tool for working on dist-git sources, its development doesn’t overlap with the Build tag topic.

And, for example, rpmautospec will not help in the case we need to update a build on pull request update: When you work with pull-requests you don’t necessarily add commits, you rework the history of a branch from which you run a PR. Sometimes even reducing the number of commits in it.

Other links

• Upstream Build version (build tag?) in RPM · rpm-software-management/rpm · Discussion #2031 · GitHub
• Mailing list [RFC] Build tag in RPM: from NVR to NVRB - devel - Fedora Mailing-Lists

Open questions

Build macro

• {build} is a bad name for a macro as it overlaps with %build section in the rpm spec
• Must {build} contain the dot at the beginning? Or can it be avoided
  • Update: Looks like %{?build:.%{build}} is the solution for this case
    Is there a simpler way than Release: 12.%{?dist}%{?build:.%{build}}

Build reason

• Does it need to be in the binary artifact?
• How can we store and navigate build reasons if we have three builds of the same sources
• Any alternatives?

Cross-distribution upgrade paths

• If remixes build their own binaries from the same sources with their own vendor-specific implementation of a build id, should there be upgrade path between such builds?

Source of a build tag value

Current proposal suggest to use Koji Build id as a value for Build tag. But it is not required. The only mandatory requirement for the Build tag is that it monotonically increases within the context of a specific Name-Version-Release combination. As soon as we bump the Release tag, Build tag can be reset to 0.

• What are the alternatives for the Build value in case of Fedora using Koji build system?

[ngompa (Neal Gompa)]

@bookwar

If we stop encoding vendor-specific data (like, for example, mass-rebuild events) in dist-git sources we can start treating dist-git as a shared space, where people can collaborate on features required for packaging a certain app in various forms.

I definitely want this.

@bookwar

OBS uses counters provided by the build system

Is there a reason we can’t do this? We could extend %dist to have a prefix %{?buildcount:.%{buildcount}} in the macro definition. That would be backwards compatible and work reasonably well. Knowing how many rebuilds of a commit has is useful information, IMO.

@bookwar

Enable passing Build tag value to the build in Koji. The value of the Build tag will be set from Koji build id.

This certainly could work, my only concern is that it’s essentially a kind of random number. This is a problem particularly when packages shift Koji instances or across different build systems. It isn’t a particularly strong concern, I suppose, since the normal release field values would supersede anyway. I would prefer a simpler build counter, though, since it’s more comprehensible to contributors and users at a glance.

[bookwar (Aleksandra Fedorova)]

@ngompa

Is there a reason we can’t do this? We could extend %dist to have a prefix %{?buildcount:.%{buildcount}} in the macro definition. That would be backwards compatible and work reasonably well. Knowing how many rebuilds of a commit has is useful information, IMO.

As I understand it, in Fedora we traditionally avoid dependency on external services and resources, and want a self-contained artifact which can be reproduced to a certain extent on the local environment.

In the proposal we are trying to move the “binary” part out of the srpm and dist-git. Which creates a question of how and where we manage the build reason history. But OBS also offloads management of the Release tag to the external database. And we probably don’t want to go that far.

That’s being said I think %{?buildcount:.%{buildcount}} is actually equivalent to %{build} thing I have in the proposal for the compatibility stage. I was just not aware you can write it like this, so thank you for the tip.

The difference is that

1. OBS also has a “checkin count” before build count. That is the role which Release tag takes in Fedora and I think we don’t want to remove it.

2. We don’t want to stop just at overriding the Release tag. It is definitely an option, but seeing that so many people “abuse” Release tag for this purpose, we better make build tag into its own thing. So not only it makes it possible to build things with different build ids, it also makes it possible to parse them later and manage the builds from the high-level libraries, without parsing Release tag with regular expressions every time on the client side.

Therefore after the compatibility stage which implements the “OBS variant” of sorts, we propose the final variant with a dedicated Build tag.

@ngompa

This is a problem particularly when packages shift Koji instances or across different build systems.

This is resolved by the fact that this number only needs to monotonically increase within the scope of a fixed Version-Release pair. So to reset the Koji counter globally, you would only need to bump a Release tag once, and then you are going to be again in the area of clean monotonic build versions for that new V-and-R

[mattdm (Matthew Miller)]

My preference here is to go all-in on rpmautospec, which replaces Release with a macro and generates the changelog from the dist-git changelog.

This cleanly accomplishes the first of the two goals (rebuild without changing spec or source files). And I think this could become a common approach as well — it’s reasonably similar to what SUSE does as I understand it.

I’d like to see it go a step further, where:

• Release: becomes optional, and only used for what are now flags to the %autorelease macro, and
• %changelog becomes obsolete and produces a warning.

This does mean that official rebuilds triggered automatically would need to be able to make a git commit, and I see the appeal of not giving bot accounts commit access — but on the other hand, as it stands right now, every official build corresponds directly to a git commit, and I think that’s good. (It also has some obvious flaws, though — it can’t record the state of the buildroot because commit time does not correspond to build time. So I’m open to doing this another way.)

I’m also not fond of making the names longer, for three reasons:

• Longer names risk getting elided in output, making diagnostics, qa, and support harder
• It adds cognitive load parsing the name visually.
• Okay, partly it is because it’s … ugly?

[bookwar (Aleksandra Fedorova)]

@mattdm

This cleanly accomplishes the first of the two goals (rebuild without changing spec or source files).

I don’t think it works this way. rpmautospec is a shortcut which allows you to not fill Release tag in the dist-git manually. But to rebuild a package you still need at least one additional commit to dist-git, which will change the value that macro calculates. In the end the number is tied to a dist-git history.

So rpmautospec automates the Release tag itself (as a version for spec files), it doesn’t have a Build component in it.

And, for example, rpmautospec will not help in the case we need to update a build on pull request update. Because when you work with pull-requests you don’t necessarily add commits, you rework the history of a branch from which you run a PR. Sometimes even reducing the commits in the branch.

@mattdm

I’m also not fond of making the names longer, for three reasons:

• Longer names risk getting elided in output, making diagnostics, qa, and support harder
• It adds cognitive load parsing the name visually.
• Okay, partly it is because it’s … ugly?

This is where “OBS”-like approach could help us. Because it would use build counter instead of Koji Build-id.

Honestly speaking the only reason I am suggesting Koji Build id is that it is easy for the implementation to rely on a number, which already exists in the build system. Proposal would work fine with any number as long as it is increasing within the scope of one NVR. So in the upstream conversation I actually propose to not enforce any kind of special meaning to it and leave it to a build-system to implement it however they like.

Thus, if we find an alternative way, with Koji or (maybe, one day) without it, we should be able to switch to that different form very easily, just bumping Release tag once.

[bookwar (Aleksandra Fedorova)]

@mattdm

This does mean that official rebuilds triggered automatically would need to be able to make a git commit

And this is the key issue.

Currently we are using the dist-git repository not as a storage of the RPM sources, but also as a “control panel” to some very distro-specific, release-specific actions, like mass rebuilds.

Which means that dist-git repo becomes a definition not for the rpm, but for the distribution build process.

And it becomes much harder to share and reuse rpm sources between projects, because they also need to place their distro-specific operational logic somewhere, and we can not put it all in one repo.

We sometimes try though, through distro-specific conditionals in the spec files. But if you look into USE_FLAGS proposal which got stale for now, it tries to address that by switching the approach from having distro-specific conditionals, to feature-specific logic, and again moving distro-specific and build-specific things outside of dist-git.

We are far from implementing it for real, but I think that this direction, separating sources from the ways we build them has a lot of potential.

Thus the proposal suggests that Release tag, and rpmautospec which generates it, stay with the sources. Because it tracks real changes in the code of the package itself. While infra events, like rebuilds due to soname bumps, or mass-rebuilds, are managed by the build system. And every build system can do it in its own way.

[ngompa (Neal Gompa)]

@mattdm

My preference here is to go all-in on rpmautospec, which replaces Release with a macro and generates the changelog from the dist-git changelog.

Originally, rpmautospec was going to help solve this problem. But at the last minute, that was removed. Now we’re back to discussing this specific aspect that was part of rpmautospec in the first place.

@bookwar’s ask is not mutually exclusive, but if we want it to succeed, it needs to not actually require conversion to work, because the mechanics of the process should not require manipulating Dist-Git to do the right thing.

If we require Dist-Git to change, all kinds of requirements and tricky problems come up, which is why we still don’t have this kind of automation.

[ngompa (Neal Gompa)]

@bookwar

We sometimes try though, through distro-specific conditionals in the spec files. But if you look into USE_FLAGS proposal which got stale for now, it tries to address that by switching the approach from having distro-specific conditionals, to feature-specific logic, and again moving distro-specific and build-specific things outside of dist-git.

Portions of the build control logic have been able to be implemented in the build system for a while now. Koji now supports setting macros in build tags, so you can do things like flip bconds and other things through it.

@bookwar

• OBS also has a “checkin count” before build count. That is the role which Release tag takes in Fedora and I think we don’t want to remove it.

So, here’s how I would structure it:

Release: 1%{?dist}

The %dist macro would be set as follows:

%%{!?distprefix0:%%{?distprefix}}%%{expand:%%{lua:for i=0,9999 do print("%%{?distprefix" .. i .."}") end}}.fc%%{fedora}%%{?buildcount:.%%{buildcount}}%%{?with_bootstrap:%{__bootstrap}}

On a local system, this would resolve to:

Release: 1.fc36

When making an official build, it would resolve to:

Release: 1.fc36.1

This would make it higher than everything else. Rebuilds of the same commit would bump the counter, like so.

Release: 1.fc36.2

This structure is important because it doesn’t violate our guidelines for how we handle things like Obsoletes+Provides and such.

I do generally support the idea of extending the EVR to an EVRB, so I’m fine with putting the build count at the end and then splitting it out eventually.

For my Fedora package builds in OBS, I configure the build system to set the release field like so:

Release: <SPEC_REL>+obs<CI_CNT>.<B_CNT>

The <SPEC_REL> statement means to preserve what the spec file has, then I append to it in the rest of the statement. <CI_CNT> is the OBS checkin counter, <B_CNT> is the OBS build counter.

So suppose a package has Release: 1%{?dist} and was checked in to OBS for the first time? It would generate the following Release:

Release:  1.fc36+obs1.1

Obviously, if I could, I’d move the build information to its own tag. And I think if we had such a field, I could get the OBS folks to use it…

[bookwar (Aleksandra Fedorova)]

Thanks, this is very helpful.

@ngompa

Release: 1%{?dist}

The %dist macro would be set as follows:

%%{!?distprefix0:%%{?distprefix}}%%{expand:%%{lua:for i=0,9999 do print("%%{

I assume that you try to achieve minimal disruption on the developer side. If we change the dist macro, we won’t need to change anything in majority of the spec-files.

But I am a bit concerned with long-term implications of that.

1. While introducing build id, I don’t want to redefine what {dist} tag represents. I thing it has its own meaning, and the overall goal would be to untangle various pieces of RPM metadata, rather than to bundle them.
2. I also would like to have build tag as a separate macro next to {dist} rather than inside it, because
   • it will be clear that {build} macro goes last in the Release tag. It is a must have for the future, since we want NVRB to be sorted the same way as NV(R’=R+B)
   • I also want to make build tag recognized as a thing, both in metadata and in macro language. It has no immediate use right now, but by making it visible under a proper name we teach people what it is about. And we provide more clarity to the whole construction of the Release tag .

[ngompa (Neal Gompa)]

@bookwar

• While introducing build id, I don’t want to redefine what {dist} tag represents. I thing it has its own meaning, and the overall goal would be to untangle various pieces of RPM metadata, rather than to bundle them.

Sure, but doing it this way means we can use it right away, rather than waiting for everyone to adapt/adopt, which I guarantee you won’t happen.

@bookwar

• I also would like to have build tag as a separate macro next to {dist} rather than inside it, because
  • it will be clear that {build} macro goes last in the Release tag. It is a must have for the future, since we want NVRB to be sorted the same way as NV(R’=R+B)
  • I also want to make build tag recognized as a thing, both in metadata and in macro language. It has no immediate use right now, but by making it visible under a proper name we teach people what it is about. And we provide more clarity to the whole construction of the Release tag .

This is laudable, but basically makes it impossible for us to adopt it. As I noted earlier, if you require people to change their spec files, we can effectively never use it. If you’re truly serious about wanting this capability, you need it to exist basically automatically in 100% of packages built in Fedora immediately.

Splitting the build counter into its own field absolutely makes sense, and we can maintain the rough semantics by ensuring it’s at the end for now. When we switch to NVRB, then we can remove this compatibility measure from Fedora releases that have the capability.

We will require this for EPEL anyway, so it makes sense to design in a way that people don’t have to think about it.

[mcdanlj (Michael K Johnson)]

I’d like to contribute some practical experience with this.

One of the decisions we never regretted in Conary was including build IDs in defining metadata for binary packaging. We recognized that source:binary was a 1:n relationship in practical semantics — that is, that building the same source in a new context was a different output. Combined with the ability to not create a new version if the build output was sufficiently identical with the prior version, it did in practice enable targeted continuous rebuild capability.

Rather, this was part of why continuous rebuild worked in practice for us, but I should share other aspects that contributed to success there.

We used the dependency graph between packages to kick off builds of dependent packages (we recorded the complete binary versions of every package in the transitive closure of the build requirements, which was everything installed in the build root), and commit the entire set of (potentially many) built packages in a transaction with the (potentially few or single) source changes that triggered it; that kept the state of the repository always consistent. Furthermore, Conary groups being versioned, compiled references to specific package versions, with those changes also committed in the same transaction, also contributed to continuous rebuild working as a practice.

I think that having and using a record of the complete build root content so that we could automatically find all the packages that ought to be affected by a rebuild was also key in the Conary/rMake universe. It is not obvious to me that continuous rebuild can be stable and scalable without that. But you can’t really get there without starting with build identifiers to represent that state.

[ferdnyc (Frank Dana (FeRD))]

@bookwar

Build macro

• {build} is a bad name for a macro as it overlaps with %build section in the rpm spec

Is it a bad name, or an invalid name? Does rpmbuild actually differentiate between %build and %{build}, or will trying to use both end up breaking something?

If %{build} is an invalid name for a macro, then I think it would be better to stop talking about it, even as a placeholder — the actual macro is going to have to have some other name, so it’d be best to figure out what that is sooner rather than later, so we can talk about this feature in the right terms.

(Even if %{build} would technically work, I think it’s an unnecessarily confusing choice for a name. So IMHO it needs to be called something else regardless. Let’s talk about %{buildnum} or %{buildtag} or whatever, not “build” which already has a completely different meaning in RPM context. The longer we keep using the wrong name, the harder and more confusing it will be for everyone to un-learn it in favor of some other name.)

[ngompa (Neal Gompa)]

@ferdnyc

Is it a bad name, or an invalid name? Does rpmbuild actually differentiate between %build and %{build}, or will trying to use both end up breaking something?

%build in rpmbuild is actually a macro, so it’d be a terrible idea to use that for this. It’d blow away the %build section.

@ferdnyc

(Even if %{build} would technically work, I think it’s an unnecessarily confusing choice for a name. So IMHO it needs to be called something else regardless. Let’s talk about %{buildnum} or %{buildtag} or whatever, not “build” which already has a completely different meaning in RPM context. The longer we keep using the wrong name, the harder and more confusing it will be for everyone to un-learn it in favor of some other name.)

That’s why I suggested %{buildcount} earlier.

[mcdanlj (Michael K Johnson)]

As soon as you want to use a hash of the buildroot contents to construct it, %{buildcount} will feel like an anachronism. %{buildtag} will cover a count, a hash, or a hash+count.

[mattdm (Matthew Miller)]

@bookwar

Which means that dist-git repo becomes a definition not for the rpm, but for the distribution build process.

I’ve been thinking about this. Neal gave an intro talk about making rpms at the Hatch event in Rochester NY this week, and it got me thinking again about how I wish the minor skunkworks project from Flock 2013 to sneak Conary into Koji had actually gotten traction. (Sorry, @mcdanlj!) There was nothing in Neal’s presentation about that specifically, but he demonstrated rust2rpm, and that got me thinking “why do we use those tools only as initial conversion utilities, and then hand-maintain spec files after”?

It would be nice if instead of a lot of boilerplate and individualized spec files that grow apart over time, most packages could be just defined as:

• link to the source
• automatic detection of the type of thing that package is
• optional exceptions for anything strange about this package in particular

I think the best shot we have currently at moving in that direction is the packit / source-git concept. This gives a split between tracking the packaging and tracking the builds — but a linked split, which satisfies some of the concerns I have. ^[1]

So… maybe I’m coming around to this? I’d still like:

• official builds tied 1:1 to commits in dist-git (not all official builds need be shipped, though)
• the filename not to contain long random strings

---

1. The way it’s tied to Github’s non-git-based “release” descriptions maybe needs some tweaks, though? ↩︎

[mcdanlj (Michael K Johnson)]

@mattdm

most packages could be just defined as:

What about build dependencies?

Add those and you basically get what the Conary packaging experience was like.

If I were doing Conary over from scratch today, instead of using Python as the DSL for packaging, it would be a YAML file, and anything that needs to run as code (e.g. the per-package policies that were possible for really weird packages) would be just another source file referenced in a build policy list in the YAML file. Also you wouldn’t check source into Conary, I would instead use git as canonical for the source.

That would give you an even better packaging experience than we managed in Conary, where we perpetually confused people by using a general-purpose language (Python) to encode a declarative package model. (In our defense, it was at least easier than JSON or XML…)

If you could then use the build tags to compile groups from source lists of names to binary objects that reference particular builds, you could use those for stable dependency resolution.

[mattdm (Matthew Miller)]

@mcdanlj

What about build dependencies?

For a lot of modern languages, those are enumerated in an upstream build config file of some sort. Maybe even C/C++. Hopefully we can get to the point where listing dependencies falls under what I called “optional exceptions for anything strange about this package in particular”.

If I were doing Conary over from scratch today, instead of using Python as the DSL for packaging, it would be a YAML file, and anything that needs to run as code (e.g. the per-package policies that were possible for really weird packages) would be just another source file referenced in a build policy list in the YAML file. Also you wouldn’t check source into Conary, I would instead use git as canonical for the source.

Just need to build on this a little bit …

[mcdanlj (Michael K Johnson)]

@mattdm

For a lot of modern languages, those are enumerated in an upstream build config file of some sort

Sure, that makes sense; we inherited and introspect build dependencies in Conary. Inasmuch as that’s successful, that’s great, and the more successful it is, the more successful it will become. Sometimes success begets success!

[praiskup (Pavel Raiskup)]

I missed this thread, I am sorry I am late here. FTR, a variant of the build tag was implemented in Copr if anyone wanted to experiment.

[bookwar (Aleksandra Fedorova)]

@ngompa

The %dist macro would be set as follows:

%%{!?distprefix0:%%{?distprefix}}%%{expand:%%{lua:for i=0,9999 do print("%%{?distprefix" .. i .."}") end}}.fc%%{fedora}%%{?buildcount:.%%{buildcount}}%%{?with_bootstrap:%{__bootstrap}}

I want to resurrect this topic and turn it into an actual Fedora 43 Change Proposal.

Looking at impact and risks, I think to play on the safe side we need the ability to restrict permissions in Koji for people to trigger builds without changes in dist-git. And then initially we would restrict this ability to releng or proven packagers group.

@ngompa Do you have ideas on how it can be implemented on the Koji side? Some check on whether the specified git commit has already been built successfully?