Fedora 40 mirrors : errors leading from Western Europe to mirrors based in UA & RU?

Ask Fedora
, , ,
1

Hi to the Fedora community,

Following my update from Fedora 39 to Fedora 40 (I made a post about that a few days ago), I have noticed that Fedora has been reaching out to mirrors based further and further, until reaching out to Ukraine or Russia.

Being based in France, I understand that my OS should be accessing mirrors based in Western Europe, and that it may also depend on the VPN connection (I can use eg. FR, BE or NL).

Two days ago, I noticed though that Fedora update reached out to Poland and Ukraine, which was already unexpected. But today, it went as far as to two mirrors based in Russia, ie. Novotelecom(dot)ru and Sibirskie Seti Ltd - sdd3(dot)ru.

Looking at the network activity, it appears there were a number of errors with the http initial query to the mirrors, which seems to have led from BE to NL, then DE, CH, SE and later PL, UA and finally RU.

Here is for instance the error message received for the Belgian mirror Cu.be which should be working - and actually some activity took place with that mirror anyway.

GET /linux/updates/40/Everything/x86_64/repodata/8ecd4684ef1ee28b2bc4a6efa07f91a17589c41cd7974b1572f78e6d0992aa69-updateinfo.xml.zck HTTP/1.1
Host: fedora.cu.be
Range: bytes=0-6170
User-Agent: libdnf (Fedora Linux 40; workstation; Linux.x86_64)
Accept: */*

HTTP/1.1 301 Moved Permanently
Server: nginx/1.26.0
Date: Thu, 16 May 2024 11:20:05 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive
Location: https://fedora.cu.be/linux/updates/40/Everything/x86_64/repodata/8ecd4684ef1ee28b2bc4a6efa07f91a17589c41cd7974b1572f78e6d0992aa69-updateinfo.xml.zck
Permissions-Policy: interest-cohort=()

<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.26.0</center>
</body>
</html>

Similar or other types of error messages impacted other http queries, eg for Nluug.nl as well as several Western/Central European mirrors.

Oddly enough, the communication with that very remote RU mirror seems eventually mingled with communications to a French mirror.

Except from the fact that it is certainly not very efficient to reach out to a mirror based several thousands kilometers away, I am not comfortable to see Fedora update contacting servers based in a war zone. Cybersecurity/defense pros reading this forum could probably confirm that such kind of network activity may have consequences.

Has anyone else experienced such issues with Fedora 40 mirrors ? How can you avoid to be brought into that again ?

Many thanks.

2

Keep in mind that DNF should verify downloads with checksums and GPG signatures, so all of this is likely irrelevant.

Anyway, you can limit the metalink result to specific countries:

COUNTRY="BE,CH,DE,FR,NL,SE"
URL="$(grep -o -m 1 -e "^metalink=.*basearch" /etc/yum.repos.d/fedora.repo)"
sudo dnf config-manager --save --setopt "fedora.${URL}&country=${COUNTRY}"
URL="$(grep -o -m 1 -e "^metalink=.*basearch" /etc/yum.repos.d/fedora-updates.repo)"
sudo dnf config-manager --save --setopt "updates.${URL}&country=${COUNTRY}"

HTTP 301 is not an error, but a redirection status code.
Download managers normally follow redirects.

4 Likes
3

Hi, and thanks a lot for your message, much appreciated.

Many thanks also for the config file, I am going to add this today.

About the issue, I pasted in my post the first report which is a redirection indeed, and it was actually followed by more than 15 occurrences of “404 Not Found” for subsequent mirrors.

Could you please advise or explain what may have happened ?

Best regards

4

The metalink server returns the hash for the latest metadata, but mirrors asynchronously download metadata and packages from the main server, and some of them might not have finished synchronizing yet, specifically for fresh updates released just recently.

5

Thanks for the additional message - I marked the config change as solution to avoid that issue in the future.

More generally, and opening the question to anyone of course, still wondering please if there are other users who experienced that issue as well : actually 25 mirrors in total returned a “404 not found” http message, so it may have been noticed elsewhere. And if that event happened precisely at the time of a synchronization of several servers, that would be good to know, especially with regards to my previous post on Fedora 40 download.

Many thanks.

6

Timing with updates is often the cause of this issue. The metadata is loaded by dnf first and the included files follow as time, bandwidth, and repo mirror server configs permit.

Usually when such errors are seen just wait a short while (maybe an hour or two) then repeat the update and the errors are usually no longer there.

You must have been doing an update just after the files were released.

7

Hi, many thanks for your inputs.

Reading the below quote about the topic, I would like to re-clarify though what the actual issue is please:

Indeed, I know that open source mirrors can be unavailable sometimes because of file synchronization or other causes - and that’s of course fine. It happened to me several times in the past and that wasn’t an issue at all for me.

What was the actual issue however is the fact that after a redirection from a Belgian mirror - which used to work fine - a cascade of 25 errors with other European mirrors took that Fedora update all the way to Ukraine and even a few thousand kilometers away to Russia, which was not only unexpected but can be also very problematic for the impacted users whose internet connections are taken to a place connected with a war zone.

The inputs given above - what I marked as solution - help users to restrict Fedora reaching for updates only to a list of mirrors defined by the user.

The current incident still has two matters about which I would like to get support and information please :

1°) Choice and information of users / management of mirrors during Fedora updates: this incident brought to my knowledge - because I thought it was prudent to pay some extra attention to network activity after this potential ransomware alert during my Fedora 40 upgrade from Fedora 39 - that during an update (ie. dnf update), if a mirror fails or is unresponsive for whatever reason (eg. synchronization), Fedora reaches out to another mirror, and then another etc. which can lead that update very far.

It is therefore not the case that if the user tries to update and the mirror is not working at that moment, that the user just has to try again a bit later because 1°) you don’t get information that your mirror fails and 2°) Fedora automatically reaches out to further mirrors.

I checked the man pages for dnf and dnf.confbut there is not much information as to mirrors choice or management during updates.

Could anyone please explain what is the way mirrors are reached / selected / managed during Fedora updates ? That could be maybe something to add in a man page, with the solution to restrict mirrors’ list (not only with a URL but also with a country list) at users’ choice.

The bottom line of this matter could be seen as a user story such as:

" As a Fedora user, I would need to know how Fedora updates (dnf, yum, etc) handles access to mirrors / remote mirrors so that I may know what network connections should happen and are legitimate ones".

2°) About the two mirrors in .ru that have been reached by Fedora update - not in the Fedora mirror list ?

As mentioned in my initial post, Fedora update reached out to these two domains in .ru:

  • Novotelecom(dot)ru
  • Sibirskie Seti Ltd - sdd3(dot)ru

I am even more puzzled about these domains that they don’t seem to appear on Fedora 40 mirror list .

Please find here attached screenshots showing the three official mirrors based in Russia, and two screenshots from the Fedora update showing that it interacted with two .ru domains (not in the official mirror list), the communications with the first one being mingled with a French mirror.

Inputs from Fedora experts to hopefully clarify and close down on these questions would be greatly appreciated.

Many thanks.

Fedora-mirros-in-RU_Screenshot from 2024-05-20 12-07-47
Fedora-mirros-in-RU_Screenshot from 2024-05-20 12-07-471920Ă—1080 165 KB

a-1st_ru-domain_Fr-comms_Screenshot from 2024-05-20 12-25-52
a-1st_ru-domain_Fr-comms_Screenshot from 2024-05-20 12-25-521913Ă—546 152 KB

b-2nd-ru-domain_Screenshot from 2024-05-20 12-27-21
b-2nd-ru-domain_Screenshot from 2024-05-20 12-27-211913Ă—297 83.1 KB

8

You have enabled IP address resolution in Wireshark, which relies on rDNS and it is not required to match DNS, since rDNS is typically managed by the hosting provider separately from DNS and can be resolved to whatever the owners of the hosting want.

Note that the domain owners may not directly own the hosting server, moreover the domain can be resolved to multiple IPs hosted in different DCs or rely on CDN with actual hosting servers distributed across the region.

From a security perspective, it essentially makes no difference where the download comes from as long as the checksums and GPG signatures are correct, otherwise DNF will automatically switch to a different mirror.

1 Like
9

Hi, thank you very much for contributing to the topic again, much appreciated.

Actually, you’re right, I have enabled name resolution in Wireshark as it makes things a bit easier when having a first look. Then, I usually check the actual IP address with a reliable source - imho - such as ipinfo.io.

I have tried to find more information on dnf and how it interacts with (remote) mirrors but didn’t find these answers on the dnf documentation page. Some posts on other sites may get close to the question but these are rather old pages.

Basically, I would still be very happy if someone could please provide insights on:
a°) Choice and information of users / management of mirrors during Fedora updates and
b°) the two unexpected .ru domains that were accessed by Fedora update (one at least not being an official mirror).

About b°) I tried to find out what it could be and something similar happened today when I used a live Fedora 39 USB stick with a VPN connection to the region of Brussels, Belgium.

The official mirror that was initially identified was www.cu.be but the dnf process (a test installation of a package) then contacted somehow the same domain ending with sdd3(dot)ru - IP address: 89.189.177.241.

Adding three screenshots showing that

  • 26 Mb of data have been provided by sdd3(dot)ru, apparently acting as a Fedora mirror (but not on the official list as of today ?)
  • Wireshark name resolution links the domain sdd3(dot)ru with IP address 89.189.177.241
  • Ipinfo.io confirms that domain sdd3(dot)ru owns IP address 89.189.177.241.

Again, I am trying to get support on these questions please as this type of electronic communications (destination) can be very problematic (I am currently based in France), especially if the user is unsuspecting that such connection is initiated during a plain daily OS update or package installation.

That is why getting the bottom line on these two questions would be really appreciated - many thanks.

a - Screenshot from 2024-05-21 18-51-35
a - Screenshot from 2024-05-21 18-51-351476Ă—556 123 KB

b-Screenshot from 2024-05-21 19-10-15
b-Screenshot from 2024-05-21 19-10-151127Ă—694 130 KB

c-Screenshot from 2024-05-21 19-06-02
c-Screenshot from 2024-05-21 19-06-021808Ă—686 92.4 KB

10

Added mirror-manager

11

Resolving DNS and rDNS works like this:

> getent hosts mirror.linux-ia64.org
37.193.156.169  mirror.linux-ia64.org
89.189.177.241  mirror.linux-ia64.org

> getent hosts 37.193.156.169
37.193.156.169  l37-193-156-169.novotelecom.ru

> getent hosts 89.189.177.241
89.189.177.241  fantasy.sdd3.ru

That explains what you see in Wireshark.

1 Like
12

Thanks very much for your message, that’s good news on that question (much better this way!)

So basically, the page with the public list of Fedora 40 mirrors (and 39 also) is not updated yet. I also saw that this thread has been tagged - thanks:! - as mirror-manager so this may help with that topic.

Finally, insights on dnf / (remote) mirror choice & management would then be very helpful.

Best regards

13

About Fedora Software update / dnf, I still can’t find technical or product information outlining what the standard behavior is, in terms of accessing mirrors or how does the package manager manage mirror failures or temporary unavailability.

I am sorry but also had, for the moment, to un-select the proposed solution to restrict dnf to a list of countries, as there seem still to be, at network level, some evidence of inconsistencies as to which countries are contacted for Fedora software update.

The questions about the choice of mirror are somehow the opposites of those considered in that older post :

Many thanks.

14

What could go wrong with pulling packages from a .ru or .ua mirror server?

1 Like
15

The mirror selection algorithm for DNF looks like this:

  • Fetch a list of mirrors from the metalink server.
    • It can be optionally limited by the country codes.
  • Iterate through the list of mirrors from highest to lowest preference value.
    • The preference value is calculated by the metalink server.
  • Try downloading the metadata and packages from the selected mirror.
    • Resolve the mirror domain to a list of IPs and pick one randomly.
  • Switch to the next mirror in the list if there’s a problem with the current one.
    • It can be non-responsive, too slow, or return a wrong checksum.
16

Thank you for these inputs as to how dnf addresses mirrors.

Before getting to this matter in more details, I would need to come back a bit about the servers IP addresses & domain names.

Please note that you could indeed confirm the IP addresses were related to a Fedora mirror, as you knew beforehand the domain name of that specific mirror.

However, if someone doesn’t know this information upfront, then rDNS is used or required, for instance in a logging system or in ISPs’ network monitoring.

When using rDNS, the responses received back for these IP addresses actually point to domain names “fantasy.sdd3[.]ru” and “novotelecom[.]ru”. Which, everyone would agree I guess, gives a (rather) different information to third parties than if it were IP addresses that are unambiguously linked to a domain such as “mirror.linux-ia64.org”, does it not?

Thus and in addition to the place itself - as already discussed above - this makes the network connections initiated by Fedora update to such remote mirror not identifiable, unlike other mirrors such as www.cu.be, distrib-coffee.ipsl.jussieu.fr or mirror.nl.leaseweb.net as, for these other mirrors and probably for a number of other mirrors, domain names match IP addresses and vice-versa. Which clearly points at open source mirrors and not at other types of websites or domains.

About the information you shared about dnf, I am trying to link these to the behavior that was recorded.

For instance, if we consider the following process - done via http by default? - :

  • Try downloading the metadata and packages from the selected mirror.
    • Resolve the mirror domain to a list of IPs and pick one randomly.
  • Switch to the next mirror in the list if there’s a problem with the current one.
    • It can be non-responsive, too slow, or return a wrong checksum.

a°) Is there something like a fault-proof mechanism to avoid a cascade of events (such as domino’s falling) or at least to limit it to a number of errors, or will this process continue until it finds a mirror that is responsive or without errors?

b°) What is the actual scope of the random decision? For instance, if a mirror fails in Brussels or Paris, can this lead, step by step, to randomly pick any server, let’s say between Zurich or Seoul or again, are there any “circuit-breakers” to make decisions more predictable by users ?

Thank you very much and kind regards.

17

There is no regulation requiring rDNS to match DNS, and this is not even possible to enforce since one server can host multiple domains.

As far as I know, DNF will give up if none of the mirrors return a positive result, so the number of possible failures cannot exceed the size of the mirror list.

You can make it more predictable by restricting the mirror list to specific country codes as mentioned above, otherwise no limitations that I’m aware of are imposed on DNF by default.

18

You could also use BASEURL and point to a specific server at all times instead of using the mirror service.

2 Likes
19

Many thanks for the replies.

We would probably all agree that dnf offers a lot of features as a package manager. The event that triggered my post - ie. dnf reaching out unexpectedly for a Fedora update to problematic locations after 25 errors occurred with previous mirrors - identified though some areas of concerns, pertaining more specifically to mirror management, choice and information of users as to how the program behaves in that domain.

A quick comment about this mention:

There is no regulation requiring rDNS to match DNS, and this is not even possible to enforce since one server can host multiple domains.

Well, when it comes to user experience or information, I believe we can make choices that work well, are safe or bring value to the user without being necessarily imposed by a regulation… :wink: But, on the other hand, products, including in Open Source, are submitted to existing contexts or regulations (eg. situations where unsuspecting users may be exposed to various risks, such as in contexts where lists of countries may exist, sometimes reciprocally).

In summary, the above thread showed that :

i) the default behavior of dnf appears to successively try different mirrors to fetch an update, and will try to reach another mirror as long as the previously contacted mirrors fail. This process may lead to a lot of attempts (eg. 25+ mirrors) and, eventually, land very far from user’s site, which may, as already said, not be without implications for the users ;

ii) the possibility to restrict this default behavior to a number of predefined countries may address that issue, subject to further test (my own tests may have shown some inconsistencies) ; however, it looks this configuration option is not explicitly documented in dnf or dnf.conf man pages ;

iii) there is a possibility to use baseurl - apparently the same as <path/url> according to the dnf man page. This is however not the default behavior that is offered to users.

A few steps / suggestions could (Imho) prevent this kind of event:

a°) as the indefinite search for a mirror until one without error is found may lead to sub-optimal choices of mirrors, maybe setting a maximum number of tries - similarly to the parameter retries for packages in dnf.conf- could help. A suitable default value could be 5, tbd ;

b°) there is also a fastestmirror in dnf.conf. By defaut, it is set to False. Setting it by default to True - with information to users - could help, especially if that would lead Fedora update to be fetched from that fastest mirror only (similarly to the “baseurl” / “path/url” parameters. This may be the easiest solution ;

c°) as provided by @vgaetera above in this thread, the possibility to restrict updates to a predefined list of countries (eg. a list of servers based in the same jurisdiction or a region with common economic rules - for instance users or organizations choosing to fetch updates only in the European Union) can be a solution. As said above, I can’t fully test / validate it myself but I guess Fedora QA / Testing can hopefully help and include it in a man page for the benefit of Fedora users.

Have a good week-end everyone,

Alex

20

Your issue IMHO seems based on user perception of unsafe risks.

DNF, by default, always knows the checksum of the original file on the fedora site, and always compares the checksum of the downloaded rpm to the original. If they do not match then the file is considered corrupt and will be downloaded again from a different mirror. This process ensures that the file actually installed has not been modified from the original and that the user gets only valid, uncorrupted, images to use for updating.

By comparing in this way it prevents potential modification and corruption of what is being used for updates. As such it allows mirrors, wherever they may be located, to properly provide valid update files and is intended to prevent (possible) malicious modifications or download errors from corrupting the users system.

a. Limiting the number of mirrors tested may not be a bad idea, but also could result in failure when a particular region of the internet is not reachable by normally used paths.

b. using the fastesmirror choice is often a valid selection, though that also can have drawbacks once the database is built locally in cases where a particular mirror may be unreachable for a time.

c. No solution is perfect when working with the internet. Limiting the countries selected is mostly a guessing game based on domain names when domains may be hosted anywhere in the world and are not physically restricted to within a country or region boundaries.