Question about package pending before pushed to testing

Ask Fedora
, ,
1

Hello,

I’ve noticed already several times that usually there is a delay between a new version of any package is submitted for testing and actually pushed to the testing repository.

For example now I see it in the following upcoming update of NSS:

https://bodhi.fedoraproject.org/updates/FEDORA-2026-d604bc1848

This update has been submitted for testing by fkrenzel.          8 hours ago
This update's test gating status has been changed to 'waiting'.  8 hours ago
This update's test gating status has been changed to 'waiting'.  8 hours ago
This update's test gating status has been changed to 'passed'.   7 hours ago

And its current status is shown as:
image

At the same time I see in the Test Gating section: “All required tests passed”.

Then why this package is still pending and not pushed into the testing repository automatically?

Is it because not all tests are green and there is some issue with rpminspect?

image
image1408×604 69.8 KB

If I go to the test results and enable “Show AI summary” there it prints the following:

Based on the testout log, the test failed due to the following root cause:

Missing rpminspect configuration file in the nss repository. When the test cloned https://src.fedoraproject.org/rpms/nss.git (commit 619c2513c472943059fb6537904f77f6f9ab67b8), it found none of the expected rpminspect configuration files (rpminspect.yaml, rpminspect.json, or rpminspect.dson) in the repository:

No rpminspect.yaml in the repository...
No rpminspect.json in the repository...
No rpminspect.dson in the repository...

Without a project-specific configuration file, rpminspect ran with default/fallback settings, which likely caused one or more inspection checks to report failures that would otherwise have been suppressed or tuned via the missing configuration. The results were written to results.yaml, and the overall test outcome was marked as failed.

I remember very similar issues with rpminspect during publishing updates of other packages too.

If there were no problems with rpminspect, the new versions of packages would have been pushed to the testing repository automatically right after all automatic tests are passed?

Will this issue be resolved some day?

I think it would help to test those updates better and quicker because otherwise those updates are much harder to install and try. I mean a command like sudo dnf upgrade --refresh --advisory=FEDORA-2026-d604bc1848 doesn’t run properly and isn’t advertised in the Bodhi page of the new package version before the update is actually pushed into the testing repository.

To this must be added additional delays due to the synchronization of mirrors after the update actually pushed to the testing repositories.

2

I believe updates are pushed to the updates-testing repository one or two times a day—not immediately. So, that’s usually when our updates stay in this state. I’d check on the update in 24 hours and see how it goes? One can contact infra/releng if it’s still stuck.

3

Why to do so? It just slows down the testing process. Think about git push that happens not instantly but with a delay of 12 or 24 hours.

4

There are practical considerations around how packages are signed and published to be synced by the mirrors. For one, mirrors around the world don’t sync “all the time”, they only sync periodically. You can speak to releng/infra for more details. The SOP is here:

PS: packages updates are not like git commits, they’re more like git tags/releases which are also not published by developers for every commit.

One can get the packages directly from koji if they want to test updates immediately, but this isn’t a general use case.

5

I’m not talking about mirrors synchronization. This synchronization requires an additional waiting time after an update is already pushed for testing. I’m talking exclusively about the pending stage before pushing for testing. For example FEDORA-2026-d604bc1848 is pending before testing already more than 22 hours for no practical reason. Such a long pending maybe have some reason before pushing packages to the update repository, but what is the rationale of such pending before testing? To do some meditations and pray?

6

As I understand it the Fedora tooling needed to create a “compose” of the RPMs that are available and check that they do not conflict.
That is not a fast process and is run periodically.

For testing I tend to do a fedpkg mock build and use the RPMs that produces for my testing. Then when I’m happy get koji to build them.

7

For testing I tend to do a fedpkg mock build and use the RPMs that produces for my testing. Then when I’m happy get koji to build them.

All the RPM files are already built:

But installing or updating those packages is less useful. Instead of a simple command like:

sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2026-d604bc1848

one need to run something monstrous like:

sudo dnf upgrade \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nspr-4.39.0-3.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nspr-debuginfo-4.39.0-3.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nspr-devel-4.39.0-3.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-debuginfo-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-debugsource-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-devel-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-pkcs11-devel-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-softokn-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-softokn-debuginfo-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-softokn-devel-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-softokn-freebl-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-softokn-freebl-debuginfo-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-softokn-freebl-devel-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-sysinit-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-sysinit-debuginfo-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-tools-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-tools-debuginfo-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-util-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-util-debuginfo-3.126.0-1.fc44.x86_64.rpm \
https://kojipkgs.fedoraproject.org//packages/nss/3.126.0/1.fc44/x86_64/nss-util-devel-3.126.0-1.fc44.x86_64.rpm
8

Is this about testing the packages or easy of installed the built packages?

The testing I do is in two step:

  1. test the local mock built packages
  2. test the koji built packages
9

This is about ease of installing and testing already built packages before they reach the update repository. This is not about building those packages locally or privately.

10

And when the testing shows the packages are bad you add bad karma to stop the promotion to updates?

11

People already do that. Is there any problem? Why are you asking and how your question relates to the topic?

12

I’m curious about your workflow, that’s all.

13

How you test updates in the pending before testing stage?

14

I’m not sure tbh—I haven’t been observing my pushes and have not yet needed to have something go to testing "immediately.

Can we continue to please be “excellent to each other here”? We’re here discussing this with you, and if there are improvements to be made we can all discuss this with releng/infra together. Negative tone and use of sarcasm etc. are not welcome. If you are frustrated and feeling so strongly about this that your feelings are bleeding into your comments, I please request you to walk away until you have cooled down and can resume the discussion while being “excellent to others”.

15

There’s koji download-build. Try: koji download-build --help.

16

I use fedpk mockbuild to create the RPMs locally and then install and test them. Once I’m happy they pass my testing then I get fedpkg build to send them to koji. Once they show up in updates-testing I confirm that my test still work.

As I use mockbuild I can iterate quickly to fix any issues that building or testing finds.

17

I found the docs:

So, it’s compose related. Info on composes here:

Unless it’s a CVE (for which infra manually runs composes etc. to push them out quickly), I don’t. If I have to, I fetch the packages from the koji-build using koji download-build, but that’s rare.

Just to clarify: “testing” means the packages to go “updates-testing”, not to “updates”. Only when they get positive karma etc. are they pushed from “updates-testing” to “updates”. So, “pending” means “not yet in updates-testing”. Once they get to “testing”, there is plenty of time to test them to provide karma and decide if they go to “updates”.

18

koji download-build -a x86_64 3071764 just downloaded 13 RPM files. They need to be additionally treated by sudo dnf up anyway. Does sudo dnf up support something like buildID=3071764 in its arguments to do the whole job by itself?

19

All this is obvious without any additional clarification. My point is that pending before updates-testing is just a waste of time, at least after all required automatic tests are passed. This is why I used the analogy with “some meditations and pray”. This analogy is definitely not about “being not excellent to each other here”.

Quote from the Update states — bodhi 26.4.0 documentation article that you mentioned:

Pending

Once an update is submitted to Bodhi it will enter the pending state. Bodhi will perform some sanity checks on the update request. Is the package built and available in koji? Is it tagged as an updates candidate for the release it is intended for? The update path for the package will also be checked at this point to make sure you don’t release a newer version of a package on an older release. The update must not break the upgrade path or it will be rejected.

Bodhi will send e-mail notifications when the update has been signed and pushed.

Pending/Testing

The update is ready to be pushed into the testing repositories, which will happen at the next compose run.

Only the stage called there as “Pending” makes sense. The “Pending/Testing” stage is just waiting.

20

Perhaps it’s best you speak to infra/releng via their channels to understand why this is and whether this can be improved? I could move this topic to the Project Discussion category to get it more attention, or you could file a ticket with infra here: