Revamp of MirrorManager

Hey, I thought I’d give an update and ask a question while at it.

As you may know I’ve been working on refreshing MirrorManager’s code. The main goal is to make it work properly on RHEL9, as RHEL7 is reaching the end of the line. But I’d also want to bring it to a state where it would be more easily maintained by other team members, using the same tooling and quality control, etc.

The web UI was mostly OK, what’s taking me much longer is the scripts (including some of the tooling and workarounds that are carried in ansible). I’m reaching a point where I can test it, and I’m currently having some memory issues in the crawler that I need to optimize. I’ll get there eventually.

One of the scripts crawl our mirror source to store the checksums of the repomd.xml files in the database, so that they can be compared with what’s on the mirrors. As a result, this script must have access to our shared NetApp storage. It’s my understanding that Red Hat wants specific IPs to allow access to the NetApp, even for read-only access, correct? If that’s true then we won’t be able to run the whole thing in Openshift, we’ll still need a VM with a fixed IP address to crawl the mirror source. Do you confirm?

3 Likes

One of the scripts crawl our mirror source to store the checksums of the repomd.xml files in the database, so that they can be compared with what’s on the mirrors. As a result, this script must have access to our shared NetApp storage. It’s my understanding that Red Hat wants specific IPs to allow access to the NetApp, even for read-only access, correct? If that’s true then we won’t be able to run the whole thing in Openshift, we’ll still need a VM with a fixed IP address to crawl the mirror source. Do you confirm?

I think that is not really true. If you look at how CentOS Stream mirror setup is configured you see that we use rsync and http for primary mirror crawling. That is, however, not as efficient as the current setup using fullfiletimelist mounted via NFS.

But the whole thing is more complicated. If you look at https://pagure.io/fedora-infra/ansible/blob/main/f/roles/mirrormanager/backend/templates/scan-primary-mirror.toml you can see that we use the new primary mirror scanner for Fedora EPEL, Fedora Archive and Fedora Secondary Arches. Also for CentOS Stream. CentOS Stream is especially interesting as it is not locally mounted.

This means we are not using the “new” Rust based scanner for “Fedora Linux” and “Fedora Alt”. “Fedora Alt” is special and I never changed it, but that should be easy to change.

“Fedora Linux” is more complicated. That is the only category where the directory in umdl or scan-primary-mirror (new rust based tool) is not the location of the fullfiletimelist. So it needs special code just to handle this category. I never changed that, but I probably should.

I would not put any effort in umdl, because it is only used by “Fedora Alt” and “Fedora Linux” and I always wanted to remove it completely from the code base.

So, finally to solve your problem, we should extend the rsync based scanner to be able to handle a fullfiletimelist and then we would no longer need NFS mounts but still have the the very efficient primary mirror scanning with just one single file and not by stat()ing all the files. Which is problematic for “Fedora Linux” and “Fedora Archive” (especially) as it just has too many files.

Another thing which might be problematic, however, some mirrors have only opened rsync for our crawler, so if the crawler requests are coming from different IPs that might be a real problem.

Oh yeah I only recently (post-FOSDEM) discovered that there was a rewrite of the primary mirror scanner in Rust as well. I thought only the mirrorlist handler had been rewritten. So, I’ve already put some work into the umdl code unfortunately.

I don’t think there are heavy performance issues with primary mirror crawling, is there? Was there another reason?

I don’t know Rust so I may be missing things, but it does not seem to handle the atomic “summary” files, do we still need this?

Cool.

That is interesting. There was already an attempt at a rewrite in the code (umdl2.py) that had never been completed, now there’s two more :smiling_face_with_tear:
Anyway, one less thing to do is always a good thing.

Cool. I’ve also seen mentions that the file size should be added to the fullfiletimelist, it’s easy to add because it’s “just” a shell script, but would that break your parser? Shouldn’t we also add the hashsums to that file when relevant, so that the scanner doesn’t even have to stat those files?

How much work do you think that would be?

That’s a very good point! Indeed quite a lot of mirrors that have rsync urls defined don’t respond to my development crawls from staging. Do you think they would be open to allowing an IP range?

One of the scripts crawl our mirror source to store the checksums of the repomd.xml files in the database, so that they can be compared with what’s on the mirrors. As a result, this script must have access to our shared NetApp storage. It’s my understanding that Red Hat wants specific IPs to allow access to the NetApp, even for read-only access, correct? If that’s true then we won’t be able to run the whole thing in Openshift, we’ll still need a VM with a fixed IP address to crawl the mirror source. Do you confirm?

I think that is not really true. If you look at how CentOS Stream mirror setup is configured you see that we use rsync and http for primary mirror crawling. That is, however, not as efficient as the current setup using fullfiletimelist mounted via NFS.

Yeah. While Red Hat storage team runs our netapp, it’s all ours and we
have pretty wide access to it to change things. We can export volumes on
our netapp to whatever network ranges we like. :wink:
So, it should be possibly to mount this in openshift (r/o) if we want.

But the whole thing is more complicated. If you look at https://pagure.io/fedora-infra/ansible/blob/main/f/roles/mirrormanager/backend/templates/scan-primary-mirror.toml you can see that we use the new primary mirror scanner for Fedora EPEL, Fedora Archive and Fedora Secondary Arches. Also for CentOS Stream. CentOS Stream is especially interesting as it is not locally mounted.

This means we are not using the “new” Rust based scanner for “Fedora Linux” and “Fedora Alt”. “Fedora Alt” is special and I never changed it, but that should be easy to change.

“Fedora Linux” is more complicated. That is the only category where the directory in umdl or scan-primary-mirror (new rust based tool) is not the location of the fullfiletimelist. So it needs special code just to handle this category. I never changed that, but I probably should.

I would not put any effort in umdl, because it is only used by “Fedora Alt” and “Fedora Linux” and I always wanted to remove it completely from the code base.

So, finally to solve your problem, we should extend the rsync based scanner to be able to handle a fullfiletimelist and then we would no longer need NFS mounts but still have the the very efficient primary mirror scanning with just one single file and not by stat()ing all the files. Which is problematic for “Fedora Linux” and “Fedora Archive” (especially) as it just has too many files.

Another thing which might be problematic, however, some mirrors have only opened rsync for our crawler, so if the crawler requests are coming from different IPs that might be a real problem.

We can set a egress ip in openshift… or it might just work as I think
thats just the default outgoing nat ip of our iad2 stuff. Either way we
could probibly work around whatever is the case.

The main reason to rewrite it was the mixing of code and configuration. The whole repomap.py business. That was always the biggest problem as changes to that file required a new release. Rust has proven to be very fast and efficient concerning database access and especially looping over the dictionaries and file lists. The previous experience with the cache generation and mirrorlist server was extremely positive that is why I did it.

No, that was never used. It was implemented many years ago but never used in production.

I am not aware of any discussions concerning changes to the file. But the parser in Python uses split() which works with any whitespace (according to the documentation) and the Rust code uses split('\t') as the file currently only uses tabs. So adding a field should not break anything.

Looking at the file I see that the hashsums are already included in the file, the problem was that the metalink wants md5, sha1, sha256 and sha512 and currently only sha1 are included in the file. That is why I ignored it. As it is only about hashing repomd.xml files and never felt important enough as those files are usually really small.

Sounds like a simple change. Not sure how quick-fedora-mirror handles changes to this file. I would expect it has no problems, but just something to keep in mind.

Hard to tell. But as Kevin said somewhere it might not be a problem as most services are using the same IP for outgoing connections. But usually it is difficult to communicate with mirror admins. A small percentage usually reacts fast, but most of the time you will not get an answer.

Ah, yes indeed. I see that you’ve moved it to a TOML file.
Thanks for your work on this, that’s one less thing on my plate :slight_smile:
I’m just worried that I may break it if I want to touch mirrormanager’s DB schema at some point.
For example, at the moment I’m finding out that storing the file metadata of non-repo files as a json blob in a column of the directory table is causing a bit of a memory issue, and was tempted to change that. It would need an adaptation in the primary mirror scanner. Not sure I’ll do it in the end, but you get the idea :slight_smile:

Nice! The comment about that was in the umdl code. Would it still be useful for your primary mirror scanner?

Agreed, but if we could just use the fullfiletimelist file to load data into the mirrormanager DB and not scan anything on the primary mirror, it would much simplify the NFS/rsync access issue. When you do rsync scanning, do you download the repomd.xml files locally to get the hashsum?

I’ve had the thought that btrfs send would be way more efficient than rsync for moving those large and frequently changing data sets between mirrors. I have no idea how (or if) that would actually work though.

There are some (potential) performance
advantages since btrfs already has the
metadata available that rsync must
(re)calculate, but in any case, not all
mirrors will be using (or can use) btrfs.
rsync is going to be the common tool
for a long long time.

I am sorry, but I actually do not understand what you are trying to do. But if database changes are necessary I am sure we can find a way to solve it.

This is a bit embarrassing (:wink:) but the RSYNC scanner uses HTTP to download the repomd.xml files. Implementing remote fullfiletimelist support means bascially it would be a single download which doing in HTTP is probably easier than in RSYNC.

We are talking about completely different things here. RSYNC scanner means we are only doing a file listing of all files remotely to get the metadata (time and size). Nothing is actually downloaded.

And as mentioned in the other reply, the primary mirror backend is not using btrfs and mirrors are potentially using something else. Sometimes an OS that does not have btrfs.

Yes! I think it would be pretty useful to get to that point. But the current format does not contain all the info we need. If I understand correctly the format is currently:
TIMESTAMP [tab] FILE_OR_DIR WORLD_READABLE [tab] SIZE [tab] PATH
where WORLD_READABLE is - when a directory is not world readable and * when a file is under a directory that is not world-readable (that’s what I gather by reading create-filelist).

There’s also a section called [Checksums SHA1] with the following format:
SHA1_CHECKSUM [tab] PATH

What else do we need? In the model we have MD5, SHA1, SHA256 and SHA512 sums for repomd.xml files (I think we are currently only checking SHA256, but we have them all). I suppose we could add a [Checksums] section for all of those sums, in theory it should not break compatibility with the parsers.

Do we need anything else? What do you think? Reasonable?

Do note that if we change this format, we need to change quick-mirror-fedora and… keep the format backward compatible for a long time since we have no ways to force people to update. ;(

I wonder if somehow we could move this to mdapi…

OK I’ve sent a PR to quick-fedora-mirror: PR#86: create-filelist: add Checksum sections for other hash algorithms - quick-fedora-mirror - Pagure.io. It should be backwards-compatible as it just adds sections.

Who are the users of this file besides mirrormanager?

Well, who knows? :slight_smile: But I know for sure quick-fedora-mirror and fedfind
use it.