How can we include Upstream URLs in Fedora Packages?

Project Discussion
1

Hello Fedora,

It’s been a little while since my internship working with Fedora CommOps ended. My current work mainly centers around CHAOSS, where I’m involved in a great many things, but most notably the CollectOSS project. This project collects data from the public github API to help open source communities analyze things like the health of their upstreams and is the data source that powers 8Knot (public instance: https://eightknot.osci.io), a self-serve data visualization tool for various community insights surrounding contributors, issues, velocity, and many other forge-based metrics that are heavily based on the CHAOSS metrics.

I wanted to make a discussion post because I have run into a metadata shortcoming that I wanted to ask the community for help navigating (since I have virtually no experience as a Fedora Packager, short of some incidental messing around with COPR).

My proposal

Essentially, I would really like to find a way to link fedora packages to their canonical upstream URLs.

These URLs aren’t hard to find, and can often be found on a one-off or manual basis for a single repo. For example, take libvirt, whose specfile is here. I was able to find that the upstream URL is very likely to be libvirt / libvirt · GitLab with just a couple clicks using the URL already present in the specfile. Package maintainers also very likely already know what these urls are for their packages.

Issues
The issue is more that this data is not standardized and generally available, which causes several issues:

  • Different teams working on different things may need this same information, resorting to independently finding it for themselves - a highly inefficient setup

  • Putting automated tools at a disadvantage for being able to automatically find this information in bulk and act on it. This includes things like:

    • tooling for detecting whether a CVE has already been fixed upstream
    • tooling to proactively monitor the health of all the packages a particular system depends on (tracking the underlying health of a community like Xz before burnout can create more CVEs)
    • fuzzers and tooling designed to search for bugs and vulnerabilities
    • Tooling that montors for declared Licenses (potentially for compliance with Fedora policy or to catch license changes upstream sooner)

  • (im still working on gathering the various usecases and examples for this)

Benefits
In addition to the opposite of all of the above points, I think there some additional benefits to being able to link packages to their canonical upstreams:

  • I think this would also be a small, but highly impactful way to help modernize fedora packages. Including a metadata field like this would help bridge the gap from RPM’s “the tarball is the gold standard” design not that the vast majority of packages are maintained in a git repository somewhere online
  • This would enable tooling like CollectOSS to complement existing analysis with more social/people-oriented aspects of the community (contributors to issues/PRs, maintainer burnout/lottery factor, project velocity, etc), potentially enabling more proactive response to maintainer burnout or other potential threats (and maybe reducing potential sources of CVE/malware events in the first place)
  • Better visibility for fedora teams (like CommOps/Fedora Data) when measuring progress towards Strategy 2028 goals
  • expanded information in dnfinfo/packagekit to reduce friction for people[1] to find where to report bugs and submit fixes

The PURL proposal

In discussing this witn @adamwill, I learned about F45 Change Proposal: Adopt PURL Metadata (system-wide) - #10 by cssushiman

Upon first glance, I think this is a promising proposal to standardize an identifier format in Fedora to make it easier to link output from CVE scanners and SBOM data to Fedora packages. This change sounds like a helpful idea on its own, but I think what I’m proposing here goes slightly further.

Where PURL enables the link from SBOM → fedora package, my suggestion here would link Fedora package → upstream forge (github repo, gitlab repo, forgejo repo, etc). This would make the chain of traceability even more complete and both proposals are mutually beneficial.

Why not use the RPM Specfile’s URL field?

This is often one of the first things people ask when soliciting feedback for this proposal.

While the RPM specfile does already have a URL field, its use is not especially standardized and is often uses simply as a link to the homepage/website for the project (example: firefox, “some xfce packages”)

In order for this field to be useful and useable to the community, especially for more automated tasks like helping process the flood of CVEs that are hitting just about every project, the data needs to be available and follow very clearly defined, consistent rules.

Positive example: Hummingbird

One of the things that got me started on this was the recent announcement of Project Hummingbird.

While not part of the RPM specfile, their RPMs repo has a metadata folder containing a JSON file for each package. This json file contains a field (upstream_repo) that clearly identifies the canonical upstream repository for that package. This field is required to be populated when new packages are added, and all hummingbird packages have this field.

This data being available makes Hummingbird a very attractive first usecase for the work my team does ( analyzing the health of the upstream communities).

Edge cases

Life is definitely messy and perfection is not something I am aiming for here. Based on some existing conversations I have had surrounding hummingbirds ~400 or so RPM packages, I have been warned of various edge cases, including:

  • Upstreams that dont use a git repo (i.e. CVS, SVN)
  • Upstreams that dont use version control (i.e. their upstream is a tarball URL)
  • packages that have no “upstream” at all - they effectively are their own upstream. For example, fedora-release. (credit to adam’s comment)

I’d be open to discussion about edge cases like this (or others people know of). My initial thought is that I suspect these edge cases would be best to leave to those consuming the data (such as my team) since the way to handle each of these cases is likely to differ depending on what purpose you are using this upstream URL data for (i.e. security scanner vs community health metrics tooling). While I might log and skip non-git upstreams in our data analysis tooling (unless it becomes a significant number of repos), I suspect the same wont be true for a security scanner that promises 100% coverage.

Where I need help

This is mainly an initial idea that has come up in a lot of conversations in my little corner of the community that I believe can be useful to many, many other usecases as well.

I’m a little intimidated by the warnings on the documentation for the Fedora Change Proposal process, which is why I am posting this as a regular discussion post to refine the idea and gather support first.

I would be very interested to hear about:

  1. any feedback people have about this idea - any issues with the concept I outlined above?
  2. more usecases and reasons that not having this metadata is causing problems - I only know so much about what the whole Fedora ecosystem is doing
  3. help deciding what the best home for this metadata is in Fedora (RPM specfile? Sidecar JSON file? something else?)
  4. Help deciding how to proceed with this proposal. Is a Change Proposal the best venue for this?
  5. Help deciding how to implement this - Would the general process just look like me submitting one PR per package to include this metadata in fedora dist-git? Is that going to create an unnecessary burden on packagers, especially those who are maintaining several packages?

Thanks

Thanks to everyone who I’ve talked to so far about concepts relating to this proposal, including:

@mattdm
@cdolfi
@duffy
@adamwill
@kevin

  1. It seems like more people seemingly feel empowered to try Linux and make their first contributions today than even before the pandemic. ↩︎

1 Like
Including Upstream Branch information in Fedora Packages
2

Another edge case is repos that have no “upstream” at all - they effectively are their own upstream. For example, fedora-release.

3

This sounds like a long way of saying that you’d like packages to set the VCS tag that RPM already understands?

3 Likes
4

Well, “understands” is a strong word. It’s just a string that has no defined format and doesn’t do anything.

The question to ask: what would it change? As a package maintainer, the VCS tag isn’t terribly helpful for the majority of packages. I expect that such information isn’t going to be useful to motivate Fedora contributors more than they already are to participate in upstream projects. Scanning that and producing more data isn’t helpful either if it isn’t actionable. I certainly don’t want more useless reports from automated tooling than I already get now.

The PURL stuff is being done mostly because Red Hat Product Security promises that having the information would reduce the amount of false reporting we get from them. Otherwise it’s not only somewhat redundant, it’s also useless.

I am reluctant and wary to expand how much more data is required without proof we are going to do something that improves the quality of life of Fedora contributors.

AEO and ideas for discoverabilty of Fedora docs
5

I’m just an individual and can’t control what others decide to do as a result of having an upstream url as a reliable data source in Fedora packages. I think there are a lot of positive outcomes that can come from this too though (see below). This small link between a package and its upstream has uses that go far beyond sending maintainers unnecessary reports. Some of these positive uses, particularly the ones I am directly helping make happen (including via collaborations with Product Security) don’t involve us sending information to Fedora packagers at all.

I have also directly talked to people who have used AI to make first passes at gathering upstream urls for packages for their own usecases (and doing a surprisingly decent job at it). If the concern is that people with no regard for maintainers inboxes are going to get this data, the odds are they already have it, and may be currently relying on such AI-generated data for their automation.

Giving a picture of the health of the community of people behind open source projects (and doing so at scale) is what CollectOSS and 8Knot are designed for. Our tooling is run by OSPOs, package ecosystems, and researchers, all trying to understand different dimensions of the open source world, and in some cases where they can be most impactful with additional support/contribution.

The CHAOSS community, where CollectOSS is maintained, also has an AI alignment working group that is currently developing new CHAOSS metrics to help measure a community’s level of consent for AI tooling; measure the actual practiced uses of AI (to see if it lines up with what was consented to); and feedback loops between community, community leaders, and model builders to help make sure AI is used in ways that align with the interests of the community/its maintainers, rather than adding burden to their plates.

By creating a more reliable data source within Fedora where this information can be found, I think Fedora can get out ahead and support the communities and people who are doing this work. If package maintainers already know this information for their packages, why shouldn’t they share it to reduce needless repeat lookups of the information by everyone else who wants it? Especially when other distros have this already.

6

This sounds very similar to how the PURL proposal is standardizing the formatting of another RPM tag so that it becomes more useful. Could another PURL-like proposal to make the contents of the VCS tag more well-defined (and thus more useful to Fedora) be a better approach than trying to propose a new field of some kind?

7

Totally agree. But isn’t there a middle ground where we can adopt a standard without adding a requirement? That would allow the people who benefit from this metadata to take responsibility for it.

1 Like
8

Thanks for the idea. Here’s some feedback!

The thing about the PURL proposal is that it doesn’t require individual packaging changes. All it proposes is taking existing RPM virtual Provides generators and producing an additional Provides string based on the PURL format. Like for Python, we could make a small change to python-rpm-generators and cover ~4000 packages in one fell swoop. And existing tools that query rpm repositories can already use package Provides data, so this new metadata would be straightforward to work with.

For the VCS tag, that would require mass packaging changes to add the tag to each package piecemeal. There’s also the issue that the tag is not standardized, and I’m not sure how/if it’s included in the repository metadata — at least dnf is not able to query for it AFAICT. I’m not saying that we can’t do this, but I am also unsure on whether the effort is worth it.

Also, we do have other mappings of Fedora package → upstream repository or language package registry package that wouldn’t require additional packaging changes. Do none of these work for your usecase?

  • The proposed PURL Provides and the existing Provides generated on a per-language-ecosystem basis outlined in Changes/Adopt PURL Metadata - Fedora Project Wiki don’t map to upstream git forges, but they do map to packages in language package registries (e.g., PyPI, crates.io) that often provide links to upstream forges.
  • https://release-monitoring.org (Anitya) also maps upstream projects to package names in Fedora (and other distributions). This is already used for automatic “New upstream version of X” bugs for Fedora packages in Bugzilla. There is also a REST API available. Packages in Anitya can either be mapped to a package in a language package registry or to an upstream git repository.
  • Each RPM spec file has a Source0 that usually contains a link to an upstream source archive from the project’s git forge or a source distribution from a package registry like PyPI.
1 Like
9

It might be possible to use automation to help backfill the VCS tag to existing packages based on one of the above data sources, but again, I’m not sure it’s worth the churn.

There are pretty much three options — submit a bunch of PRs (which yes, does create extra work for packagers), automatically push changes en-masse as a provenpackager (if feasible), or rely on packagers to independently fix the packages they maintain (which takes longer, but is the only option for more complex/difficult to automate changes). See also https://docs.fedoraproject.org/en-US/fesco/Mass_package_changes/.

1 Like
11

Well, I do have a long-term goal to remove or replace the forge macros in Fedora :wink: . But that’s a separate discussion (see https://forge.fedoraproject.org/packaging/forge-srpm-macros/issues/14).

1 Like
12

Not quite true - the expected format is documented - it’s just not validated and treated as a free-form string field instead:

(Public) upstream source code VCS location. Format <vcs>:<address> with <vcs> being the VCS command used (e.g. git, svn, hg, …) and <address> being the location of the repository as used by the VCS tool to clone/checkout the repository (e.g. https://github.com/rpm-software-management/rpm.git).

from https://rpm.org/docs/6.0.x/manual/tags.html

13

Is there a reasonable way you would suggest approaching either improving the data in this field this field and/or moving to raise the validation bar moving forward?

14

Not sure - RPM itself doing some basic validation on the field would be nice, but that’s a more long-term thing. Maybe the CI that runs for package pull requests could have a pipeline step to check the VCS field for correct format, and flag it if it’s present but not valid?

1 Like
15

did some preliminary analysis on the RPM VCS field in fedora packages:

this is super quick so sorry about the overlapping text. green is “populated normally” and orange is “populated using macros”.

image
image790×700 35.6 KB

Edit: heres a graph with the dead packages excluded (same colors, the light gray is fetch failed)

image
image752×790 32.2 KB

it seems like trying to improve the existing VCS metadata is effectively the same path as making a new field in the sense that it would still likely require mass changes to be submitted to a majority of active fedora packages before it becomes substantially useful as a field

Does contributing to improving the adoption of the VCS field still sound like the best path forward if our team may end up needing to gather this data anyway (and wants to upstream it into fedora)?

1 Like
16

Are you aware of https://src.fedoraproject.org/lookaside/rpm-specs-latest.tar.xz? That’s a tarball of all rawhide spec files that should be easier to work with.

Also, I’m not able to read the labels on the pie graph. What’s the difference between the orange and green parts?

AEO and ideas for discoverabilty of Fedora docs
17

I was not aware of that, but it gives me an idea… ill make a new thread though since its maybe somewhat off this direct topic.

thanks for the info though!

18

Essentially the distinction is whether the RPM specfile is using a macro to populate the field. its more of a technical detail im interested in since im planning to actually use this data. If you just want to look at the graph in terms of adoption, green and orange are effectively “this has a value already set”

19

in 2022, someone already proposed a RepoURL RPM metadata field in Add several new optional URL tags by stanislav-brabec · Pull Request #2045 · rpm-software-management/rpm · GitHub

it was rejected as redundant with VCS and the docs were instead updated for VCS in Improve documentation for VCS tag by ffesti · Pull Request #2064 · rpm-software-management/rpm · GitHub

(thanks @prarit for finding these)

20

does this endpoint have a way to check whether/when the file was last updated? (like HTTP if-modified-since), or just documentation about it in general?

21

We hit this in hummingbird. We opted to point to the fedora location for the VCS field. That worked for us but it might not necessarily be the best thing for Fedora.