This morning - like every Monday - I started my day with reviewing the automated dependency updates for all the projects that I contribute to or help maintain. That is when I came across the failing tests[1] on the MDAPI[2] that had been failing for some weeks now. I have been putting it aside given that I was occupied with the Git Forge migration endeavours until today.
I started looking into the problem as to why the tests were continually failing and it dawned upon me that the Fedora Rawhide branch[3][4] did not have the SQLite3 based metadata anymore. As MDAPI depends on the SQLite3-based metadata for supplying its endpoints, this was a breaking change for the MDAPI’s functioning that was not duly notified in the tracker[5].
On further digging deeper into the changes that led to this happening, I came across the change titled “Upgrade systems to createrepo_c 1.0 and change repositories metadata settings”[6] from Fedora Linux 41 changeset[7]. With the createrepo_c versions greater than 1.0, SQLite3-based metadata would no longer be generated as those were used only with YUM uses[8].
While there has been a proposal ticket associated with the deprecation of SQLite3-based metadata[9] (which is unresolved as of the day of writing this) and a request for feedback in the original change proposal[10], MDAPI was not ready for the change to happen by the time it did. I had to dig up the aftermath from the conversations from the Pulp discussions thread[11].
This was proposed to the project a couple of years back[12] by @dalley but the proposed approaches did not have any updates since then. Back then, this was not that pressing of an issue, so even I did not give it much of a thought until today, when I was working on patching up the test suite[13], when we are one release past[14] the one where the breaking change took place.
Among the list of things that I could have wished for, better communication and better activity remain on the top. That goes on both the side of the change proposers to better communicate with the stakeholders that would be affected by the change (especially when they have been repeatedly mentioned of[15][16][17][18]) but also on maintainer’s side in terms of attentiveness.
MDAPI needs to be ported for future releases that will not have SQLite3-based metadata. But let us get a temperature check first, shall we? Who here depends on MDAPI and would be willing to put in some work into helping me out with the port? How can we make the port more resilient to changes that happen down the road? Should we rewrite it in Rust or Go for fun?
I came across the change titled “Upgrade systems to createrepo_c 1.0 and change repositories metadata settings”[6] from Fedora Linux 41 changeset[7]. With the createrepo_c versions greater than 1.0, SQLite3-based metadata would no longer be generated as those were used only with YUM uses[8].
sqlite metadata support hasn’t been removed from createrepo_c or anything, it’s just not in the defaults anymore. It can be turned back on with --database
And Pungi ought to be enabling this already - it doesn’t rely on defaults for this behavior, it uses flags. Tree - pungi - Pagure.io
But, it looks like some recent changes to Pungi may have changed the default for Pungi, so actually I don’t think the createrepo_c change is even the real culprit here, it just happened around the same time as the Pungi changes.
If the Pungi default is now false, then all that needs to happen is a configuration change to turn it back on, for Pungi, or to change the default back until all the other infra is in position to turn it off forever.
As for mdapi, I can try to help write a more permanent fix again if I get some free time. I was working on it before, but, obviously never finished. I’ve never looked at packages.fedoraproject.org , but hopefully a similar solution can be applied there also.
I’m sorry communication wasn’t better here. ;( Probibly my fault for not realizing that mdapi needed that sqlite data at least in it’s current form. ;(
I think perhaps @james suggested there might be a way using the yum code to convert the xml? perhaps that would be a short term fix?
@james, (responding here for including in the bigger discussion here), The “rawhide” branch situation is current. As time goes on, I suppose, this problem would arise for all newer releases beyond Fedora Linux 42. The work around that I have provided for now mirrors the Koji’s “rawhide” branch and I am not sure just how long they would keep things that way, before things require changing again.
@kevin, it goes both ways - I suppose I could have been a bit more attentive towards the Fedora Linux 41 changeset and the possible repercussions on MDAPI. The fact that the original release engineering proposal ticket[1] is left dangling without a resolution, although, is a massive lacuna in communication that I would have addressed (as I referred to that ticket initially when @dalley pointed to it before).
@jnsamyak, would you know as to why the SQLite3-based metadata is no longer generated by Pungi? Could you please look into it[2] and turn it back on until we are at a position to go back to porting MDAPI to support XML-based metadata? If packages.fedoraproject.org no longer uses the SQLite3-based metadata, maybe we can work on something together @dalley for the port - it just has to be a bit later.
@mattia, for now, I have sent a pull request under @jednorozec’s direction to reverse the SQLite3 metadata creation[1]. I have also created the feature tracking ticket for adding the XML-based metadata support to MDAPI here[2]. Until someone from Fedora Infrastructure[3] takes it up and implements it, I would like to have the SQLite3-based metadata created for at least the next couple of releases.
Yesterday, I also worked on a “workaround release” v3.1.7 that tracks Rawhide information from kojipkgs.fedoraproject.org instead of the built composes on dl.fedoraproject.org. The same has been published on GitHub[4] and PyPI[5] as well as deployed on the staging environment[6]. Should we get the SQLite3-based metadata back, I should be able to reverse these changes in a new release.
I merged the sqlite re-enable for rawhide. ;( We need to decide though if we are doing that for the other places: f42 branched, f41 updates/updates-testing.
On kojipkgs, you mean the buildroot repos? I’m not sure if it will be a problem for mdapi, but those repos aren’t the same as the composed/released ones:
no multilib (ie, no i686 rpms in the x86_64 repo)
If maintainers add a builroot override, the package will be in the buildroot, but not in the composed/released repo
The buildroot repos regenerate all the time, so they will get packages that will only appear in the end compose when it runs later in the day.
None of that might be too big a problem, but is not ideal…