Quality control on bad public mirror operators

I decided to share my experience as it could happen to any of us living any part of the world. There will be name-calling. I’ll try my best to be respectful, but if the tone is of concern, please take it down and I’ll write up a more softer generalised version of the post.

tl;dr: there’s a low quality package mirror that’s causing bad UX. There seems to be no quality control on public package mirrors.

At the time of writing, there’s only two public Fedora package mirrors in South Korea: namely KAIST and KRFOSS. The KAIST mirror has been around since the early days of Fedora. I know for a fact that it has been in good competent hands(the uni staff), hosted on more-or-less enterprise-grade hardware well suited for the purpose, with their uni network peered directly with the national backbone. I’ve experienced no big issues with them. If the mirror is down, it’s down. If it’s up, it’s up. There was no inbetween. Judging from several emails I’ve changed with the KAIST team over the years, I can tell they’re quite proud of what they’re running and maintain the mirror responsibly.

The competency of the operators of KRFOSS, on the other hand, has been questionable. First off, KRFOSS is a project. It’s a group of small mirror operators rather than a typical single independent mirror operator you’d expect: an organisation or individual in a single geologically location. KRFOSS services the Fedora mirror works by having one CDN front(Cloudflare) with the servers run by many actual hosts operated in various locations in South korea as source. How the traffic gets routed is not clear so even the operators themselves wouldn’t be able to locate the problem when HTTP 4xx and 5xx start popping up. Although not technically against any rules, but IMO, the use of CDN for hosting package mirror must be discouraged or even banned(if you’re not on a peering contract, don’t even think about hosting one). Using the script I wrote, I found that only a fraction of all public mirrors are hosted behind a some form of CDN including Cloudflare.

There’s a whole host of reasons why Fedora packages shouldn’t be cached with CDNs:

  1. Web caches cannot possibly keep up with the rate at which the build servers are churning out new packages. They simply cannot be treated as regular web contents
  2. There’s no way of knowing if the source is actually located where it says it is. In the case of KRFOSS, the source might as well be in China with no actual physical host in Korea. Can’t ping or traceroute the source. The only front reachable from outside are Cloudflare caches. We just have to take the operator’s word for it. Not a good idea

So I contacted the KRFOSS team a while back because I’d get occasional HTTP 404. This is no recent event. AFAIK, their mirror has been like this at least for the past 2 years. The response I got made it clear that they have no interest in operating the public mirror seriously.

The original email written in both Korean and English. Got a response in Korean, clearly machine translated. Not even bother to use LLM to get quality translation, just straight up old-fashioned machine translation. Whoever replied to my mail is definitely not a Korean national. Why would anyone who doesn’t speak the language volunteer to run a project like KRFOSS? This got me all paranoid and wondering if the real purpose of KRFOSS is a supply chain attack for another wave of cyberattack similar to the one in 2013(they host mirrors for a bunch of other distros and FOSS projects). They refer to themself as “X“. No real name, no organisation, no base of operation, no nothing. Just “X“ claiming to be the one who’s involved in the KRFOSS project. Not trustworthy at all. And why are they so obsessed with having multiple “sponsors” when they’re simply just using their CDN services? What are they trying to prove?

They admitted in the mail that they’re using some tier-2 mirror(mirrors.tuna.tsinghua.edu.cn) in China as source. So, they’re basically running the mirror with two major points of failure/bottleneck(CDN + tier 2 source) and it’s surprising that it’s working like 80% of the time.

(HTTP 522 is a Cloudflare-specific HTTP code returned when the origin times out)

I cc’d admin@fedoraproject.org. In my book, KRFOSS failed all my smell tests but no one at Fedora seem to care.

I get that Fedora is not RHEL. Users wouldn’t expect enterprise-grade quality control on the infrastructure, but the package mirror is still the most important infrastructure when it comes to Linux distros, no matter how big or small. There ought to be some quality controls around it. I’m sure KRFOSS is just one example and there are a few other public mirrors that are not up to the standard(or lack thereof). If RH cares about Fedora’s reputation, they really should do something about it.

Whenever I had issues with the mirror, I visited their website. From what I could gather, they don’t seem to be ashamed of having to host a public mirror on consumer-grade hardware from consumer-grade ISP network. To me, the whole project looks more like hobbyists’ playground than a real OS infrastructure. I’m not saying that enthusiasts should be allowed to host a public mirror. I’d do what I could to help KRFOSS guys out, had they been clear on their identities. I think public mirrors should be run by someone who at least has access to reliable hardware and IP transit. Unfortunately, KRFOSS guys are not the ones who could meet the criteria.


The montage of screenshots I compiled so far.

Translation: guide on bypassing daily transfer limit of ISP in various internet routers(consumer-grade network hardware) by sending a special DHCP option to the upstream DHCP server.

Background: there’s no true unlimited-data internet plan available to residential-premise customers in South Korea. Even 10G products exist, but all ISPs set a daily cap. Such guide would be useful when hosting a public mirror on ISP nets.

Translation: pulled Alpine and FreeBSD mirrors because there’s no storage. “Emergency maintence“(this isn’t just one-off incident. They get these “emergencies” quite frequently)

Taking a mirror down to update a firmware on a router(as in cheap home routers you can get for less than $100-200)…

Translation: taking a mirror down due to failed mobo

A mirror is down due to failed router.

[root@41e911af8a72 /]# dnf update
Updating and loading repositories:
 Fedora 43 openh264 (From Cisco) - x86_64                                                                                                                                   100% |   2.1 KiB/s |   5.8 KiB |  00m03s
 Fedora 43 - x86_64                                                                                                                                                         100% |   7.4 MiB/s |  36.4 MiB |  00m05s
>>> Status code: 404 for https://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/repomd.xml (IP: 104.18.2.112) -https://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/repom
>>> Status code: 404 for http://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/repomd.xml (IP: 104.18.2.112) -http://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/repomd.
>>> Status code: 404 for https://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/affb4c277d7a0a05ec39e29c3207de63fe207adf97143e88c61bccc0e2a0135b-primary.xml.zck (IP: 104.18.3.112) -https://mi
>>> Status code: 404 for https://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/73a6103e51d00abd1abd50eb54f98a07b398a86972555b532af6acdb98565c3d-comps-Everything.x86_64.xml.zst (IP: 104.18.3.1
>>> Status code: 404 for http://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/73a6103e51d00abd1abd50eb54f98a07b398a86972555b532af6acdb98565c3d-comps-Everything.x86_64.xml.zst (IP: 104.18.3.11
>>> Status code: 404 for http://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/affb4c277d7a0a05ec39e29c3207de63fe207adf97143e88c61bccc0e2a0135b-primary.xml.zck (IP: 104.18.3.112) -http://mirr
 Fedora 43 - x86_64 - Updates                                                                                                                                               100% |   3.6 MiB/s |  15.1 MiB |  00m04s
>>> Status code: 404 for https://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/repomd.xml (IP: 104.18.2.112) -https://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/repomd.xml   Status code: 404 for http://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/repomd.xml (IP: 104.18.2.112) -http://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/repomd.xml     Status code: 404 for https://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/319bdfe41801129e6a275520018afde70388e98579e369211392c61c263e1b7a-primary.xml.zck (IP: 104.18.3.112) -https://mirror
>>> Status code: 404 for http://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/319bdfe41801129e6a275520018afde70388e98579e369211392c61c263e1b7a-primary.xml.zck (IP: 104.18.3.112) -http://mirror.k
>>> Status code: 404 for https://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/296735a92b915a82ea639d5ee1c4a8af4a6e5d4194786ce9c83ab7e70cf09f80-comps-Everything.x86_64.xml.zst (IP: 104.18.3.112)
>>> Status code: 404 for http://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/296735a92b915a82ea639d5ee1c4a8af4a6e5d4194786ce9c83ab7e70cf09f80-comps-Everything.x86_64.xml.zst (IP: 104.18.3.112) -
>>> Status code: 404 for https://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/a6700f91cf6527f1d99d18493336af5a3a767ba141ed69ebb95b615560d262e0-updateinfo.xml.zck (IP: 104.18.3.112) -https://mir
>>> Status code: 404 for http://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/a6700f91cf6527f1d99d18493336af5a3a767ba141ed69ebb95b615560d262e0-updateinfo.xml.zck (IP: 104.18.3.112) -http://mirro 

repolist 404’d. KRFOSS team’s explanation: the source mirror is not syncing fast enough, so not our problem.

repolist 200’d but the actual meta DB file 404’d. The meta files are clearly not syncing atomically.

1 Like

I decided to share my experience as it could happen to any of us living any part of the world. There will be name-calling. I’ll try my best to be respectful, but if the tone is of concern, please take it down and I’ll write up a more softer generalised version of the post.

Well, name calling isn’t a good thing IMHO, but I’ll try and answer you
as best I can, no need to rewrite.

tl;dr: there’s a low quality package mirror that’s causing bad UX. There seems to be no quality control on public package mirrors.

At the time of writing, there’s only two public Fedora package mirrors in South Korea: namely KAIST and KRFOSS. The KAIST mirror has been around since the early days of Fedora. I know for a fact that it has been in good competent hands(the uni staff), hosted on more-or-less enterprise-grade hardware well suited for the purpose, with their uni network peered directly with the national backbone. I’ve experienced no big issues with them. If the mirror is down, it’s down. If it’s up, it’s up. There was no inbetween. Judging from several emails I’ve changed with the KAIST team over the years, I can tell they’re quite proud of what they’re running and maintain the mirror responsibly.

The competency of the operators of KRFOSS, on the other hand, has been questionable. First off, KRFOSS is a project. It’s a group of small mirror operators rather than a typical single independent mirror operator you’d expect: an organisation or individual in a single geologically location. KRFOSS services the Fedora mirror works by having one CDN front(Cloudflare) with the servers run by many actual hosts operated in various locations in South korea as source. How the traffic gets routed is not clear so even the operators themselves wouldn’t be able to locate the problem when HTTP 4xx and 5xx start popping up. Although not technically against any rules, but IMO, the use of CDN for hosting package mirror must be discouraged or even banned(if you’re not on a peering contract, don’t even think about hosting one). Using the script I wrote, I found that only a fraction of all public mirrors are hosted behind a some form of CDN including Cloudflare.

Cloudflare isn’t just a CDN, lots of sites use it for bot
protection/filtering, but not any content caching or the like.

There’s a whole host of reasons why Fedora packages shouldn’t be cached with CDNs:

  1. Web caches cannot possibly keep up with the rate at which the build servers are churning out new packages. They simply cannot be treated as regular web contents
  2. There’s no way of knowing if the source is actually located where it says it is. In the case of KRFOSS, the source might as well be in China with no actual physical host in Korea. Can’t ping or traceroute the source. The only front reachable from outside are Cloudflare caches. We just have to take the operator’s word for it. Not a good idea

Sure, there are issues with using CDNs.

So I contacted the KRFOSS team a while back because I’d get occasional HTTP 404. This is no recent event. AFAIK, their mirror has been like this at least for the past 2 years. The response I got made it clear that they have no interest in operating the public mirror seriously.

The original email written in both Korean and English. Got a response in Korean, clearly machine translated. Not even bother to use LLM to get quality translation, just straight up old-fashioned machine translation. Whoever replied to my mail is definitely not a Korean national. Why would anyone who doesn’t speak the language volunteer to run a project like KRFOSS? This got me all paranoid and wondering if the real purpose of KRFOSS is a supply chain attack for another wave of cyberattack similar to the one in 2013(they host mirrors for a bunch of other distros and FOSS projects). They refer to themself as “X“. No real name, no organisation, no base of operation, no nothing. Just “X“ claiming to be the one who’s involved in the KRFOSS project. Not trustworthy at all. And why are they so obsessed with having multiple “sponsors” when they’re simply just using their CDN services? What are they trying to prove?

They admitted in the mail that they’re using some tier-2 mirror(mirrors.tuna.tsinghua.edu.cn) in China as source. So, they’re basically running the mirror with two major points of failure/bottleneck(CDN + tier 2 source) and it’s surprising that it’s working like 80% of the time.

fedoraproject mirrors are encouraged to use larger mirrors near them,
all mirrors cannot pull from the master mirrors. Thats just not
sustainable, the master mirrors would be swamped. ;(

(HTTP 522 is a Cloudflare-specific HTTP code returned when the origin times out)

I cc’d admin@fedoraproject.org. In my book, KRFOSS failed all my smell tests but no one at Fedora seem to care.

ok, this I disagree with. I was on that thread and at the end of the
emails I saw they said they addressed your issues (at last as far as I
could tell, because I don’t read korean and had to use a translation
thing to read it). I didn’t see any replies from you after that saying
that the issue was not solved?

So, from my point of view, I wasn’t aware that it was still any kind of
issue…

I get that Fedora is not RHEL. Users wouldn’t expect enterprise-grade quality control on the infrastructure, but the package mirror is still the most important infrastructure when it comes to Linux distros, no matter how big or small. There ought to be some quality controls around it. I’m sure KRFOSS is just one example and there are a few other public mirrors that are not up to the standard(or lack thereof). If RH cares about Fedora’s reputation, they really should do something about it.

Well, fedora doesn’t run mirrors, their particular operators do. If they
are broken and continue to be the best we can do is talk to the
operators of them and if they cannot or will not fix issues we can
remove them.

there are definitely problems with mirrors from time to time, but it’s
IMHO rare and almost always just talking with the mirror operators gets
them to fix it. There have over the years been a few cases where there
was no response or the issues were not fixed and we disabled those
mirrors. Perhaps this is what needs to happen to this one, but it sure
wasn’t clear from the email thread (at least to me).

I’ll go ahead and disable them for now.

I never heard back from them since then. And I can’t possibly email them every time dnf chooses KRFOSS mirror and the meta DB files happen to be out of sync. It happened many times, and my script to check their health was not sophisticated enough. The script does not go further to check the case where the repo DB hash and the contents of meta do not match. Can’t really tell how long their mirror has been running like this.

I changed my yum.repos.d conf to use &country=us for a while now, but from time to time, I had to run dnf in containers or other VMs. That’s when I had issues with KRFOSS. I don’t do that often, but the fact that it happened so many times to the point I decided to write a long post about it on a public discussion forum says something about their downtime. In other words, this is no one-off incident and the problem with CDN has not been resolved since then. I wonder how many Fedora users in Korea(probably only a handful) experienced this issue and just turned their back to another distro, not knowing how to even tackle the issue.

I doubt that they have the necessary expertise to address the issue in the first place(use of consumer-grade hardware, using CDN to aggregate multiple mirrors to one single point of failure and so on: decisions that a competent engineers would never consider). I might sound condescending when I say that is okay as long as they continued to communicate with me to fix the problem. They did not. They just chose to sit this one out. That’s what’s been bugging me.

I don’t feel particularly good about this, though. Perhaps you should outline the reasons behind the decision and the actions they have to take to reinstate their status. I’d suggest:

  1. discourage the use of CDN and recommend exposing the actual hosts to the internet(they’re called “public“ mirrors for a reason) so that the mirrorlist can have more control over distributing the load as it’s been designed to do very well
  2. prove that they’re responsibly utilising the network, preferably proof that they have their own AS or IP transit contract with ISP/IX. Canonical has some rules regarding this. Hosting public mirrors from residential ISP net is definitely frown upon by Korean telcos. Korea doesn’t have internet infrastructure as good as the RIPE/ARIN region
  3. prove that they’re serious about keeping the uptime: use server hardware!

In the meantime, I’ll contact them again to see if they’re willing to join the conversation this time. IMO, they might as well remove the Fedora mirror as the traffic they were getting was probably insignificant.

There’s another unrelated issue of mirrorlist not preferring mirrors in Japan(although the mirrors in China might be geographically close, Japan is closer in global peering network). I think this has something to do with the dataset mirrorlist is making decisions off of, not the Rust code itself.

1 Like

I never heard back from them since then. And I can’t possibly email them every time dnf chooses KRFOSS mirror and the meta DB files happen to be out of sync. It happened many times, and my script to check their health was not sophisticated enough. The script does not go further to check the case where the repo DB hash and the contents of meta do not match. Can’t really tell how long their mirror has been running like this.

sure, but you could let us know before saying we don’t care?

…snip…

I don’t feel particularly good about this, though. Perhaps you should outline the reasons behind the decision and the actions they have to take to reinstate their status. I’d suggest:

  1. discourage the use of CDN and recommend exposing the actual hosts to the internet(they’re called “public“ mirrors for a reason) so that the mirrorlist can have more control over distributing the load as it’s been designed to do very well
  2. prove that they’re responsibly utilising the network, preferably proof that they have their own AS or IP transit contract with ISP/IX. Canonical has some rules regarding this. Hosting public mirrors from residential ISP net is definitely frown upon by Korean telcos. Korea doesn’t have internet infrastructure as good as the RIPE/ARIN region
  3. prove that they’re serious about keeping the uptime: use server hardware!

In the meantime, I’ll contact them again to see if they’re willing to join the conversation this time. IMO, they might as well remove the Fedora mirror as the traffic they were getting was probably insignificant.

I was going to email them politely, but I see you already contacted
them.

There’s another unrelated issue of mirrorlist not preferring mirrors in Japan(although the mirrors in China might be geographically close, Japan is closer in global peering network). I think this has something to do with the dataset mirrorlist is making decisions off of, not the Rust code itself.

Feel free to file a mirrormanager ticket on that.

1. Regarding the 404 errors and their duration

404 errors are issues that can occur at any time and in any environment. Mirrors must continuously synchronize to maintain up-to-date content. Other distributions also go through similar synchronization processes, and during this process, encountering 404 errors is entirely possible.

Therefore, making definitive claims about the quality of a mirror based solely on such occurrences is highly inappropriate. By that logic, any mirror that ever produces a 404 error would have to be considered unreliable, which is clearly unreasonable.

Let’s look at one of the example links mentioned in this discussion:
https://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/repomd.xml

At least at the time of writing this, the file exists and is accessible. It appears that dxdxdt is referring to a momentary 404 error that occurred during synchronization. However, they go further and claim something that was never stated by the ROKFOSS team:

“repolist 404’d. KRFOSS team’s explanation: the source mirror is not syncing fast enough, so not our problem.”

In reality, the ROKFOSS team did not ignore the issue. On the contrary, they acknowledged it and showed intent to resolve it (“We will update this issue soon”). Despite this, dxdxdt severely distorts the situation, portraying the ROKFOSS team as negligent and indifferent. This is clearly misleading and deserves criticism.

Therefore, this claim cannot be considered valid.

Additionally, part of dxdxdt’s argument is factually incorrect. They stated:

“AFAIK, their mirror has been like this at least for the past 2 years.”

However, the ROKFOSS PROJECT celebrated its first anniversary this past March. It is therefore baseless to claim that the mirror has been in this state for two years. This is an unfounded and false statement. You can verify the project’s first anniversary here:
https://news.krfoss.org/notice/ROKFOSS%201주년

To put this into perspective, it’s like looking at a one-year-old building and saying, “This building has had issues for two years.” Claiming problems existed before the project even existed is clearly inappropriate and raises serious concerns about the intent behind such statements.

2. Claims about nationality and supply chain concerns are political, inflammatory, and dangerous

The author states:

“The original email was written in both Korean and English. The response came only in Korean, clearly machine translated. Not even using an LLM, just old-fashioned machine translation. Whoever replied to my mail is definitely not a Korean national. Why would anyone who doesn’t speak the language volunteer to run a project like KRFOSS? This got me paranoid and wondering if the real purpose of KRFOSS is a supply chain attack similar to 2013.”

The only “evidence” presented is that the response “appeared to be machine translated.” This is not an objective fact, but merely the author’s subjective interpretation.

Moreover, the claim that the response was only in Korean is factually incorrect. The original email included both Korean and English. This was done as a courtesy, considering the possibility that the recipient might be Korean.

Despite this, the author makes unfounded and offensive claims such as:

  • “definitely not a Korean national”
  • “the source might as well be in China with no actual physical host in Korea”

These statements are baseless and insulting to the contributors.

This is particularly problematic as it introduces long-standing political tensions between Korea and China into a Fedora discussion, which is highly inappropriate and dangerous. This clearly violates the Fedora Code of Conduct.

Relevant violations include:

  • Saying insulting or derogatory comments and making personal attacks
  • Initiating controversy for its own sake
  • Public or private harassment

Furthermore, the author subtly connects unrelated points—such as the use of a Chinese mirror (mirrors.tuna.tsinghua.edu.cn)—to create the misleading impression that ROKFOSS is somehow a Chinese-backed project disguised as Korean. There is no evidence supporting this claim; it is entirely speculative.

Even more concerning is the reference to the 2013 supply chain attack, implying that ROKFOSS could be part of a similar cyberattack. This is a serious accusation with no substantiated basis and serves only to damage the project’s reputation.

Consider how this narrative flows: starting from a simple 404 issue, it gradually introduces political implications and security fears, shaping readers’ perceptions in a subtle yet manipulative way.

When people are influenced by such narratives, it leads to doubt, then suspicion, and eventually false conclusions. This is precisely how unnecessary conflicts arise.

The Fedora community should be a place for constructive and meaningful discussions—not for politically charged speculation. The post by dxdxdt disrupts this environment and clearly violates the Fedora Code of Conduct.

3. Regarding the use of CDNs

Using CDNs such as Cloudflare is common across many Linux distribution mirrors. In fact, some major mirrors rely on CDNs.

The issue is not the use of a CDN itself, but how content freshness is managed. Even if the origin server is up-to-date, stale cached data on a CDN could cause issues. However, if metadata is properly synchronized and cache is managed correctly, there is little practical difference compared to non-CDN mirrors.

From a user perspective, CDNs offer significant advantages in terms of download speed and accessibility. Their purpose is to improve performance—not degrade it.

As Kevin, likely a Fedora administrator, pointed out, Cloudflare is not just a CDN. It also provides protection against malicious traffic and attacks targeting mirrors.

Therefore, criticism should focus on improper cache management—not the mere use of a CDN. If freshness were not properly maintained, this issue would have been detected and addressed by Fedora systems already.

4. Regarding the attached screenshot and notices

Some content included in the screenshot is unrelated to this issue and unnecessarily drags unrelated contributors into the discussion.

The author also adds interpretations not present in the original text—for example, describing contributors’ equipment as “cheap home routers under $100–200.” This creates a misleading impression that the infrastructure is inherently low-quality.

This is another example of baseless and inflammatory framing.

Individual contributions are a cornerstone of the open-source ecosystem. Many widely used mirrors are operated by individuals. However, dxdxdt’s remarks demean these contributors and discourage participation, which is harmful to the community.

Driving contributors away is far more damaging than users leaving due to occasional mirror issues.

It is also unclear how many Fedora users exist in Korea. Regardless, suggesting that users will abandon Fedora over such issues is speculative and appears to be an attempt to pressure Fedora maintainers into action.

Conclusion

dxdxdt’s rude behavior deserves to be condemned, and their actions violate the Fedora Code of Conduct. Additionally, any actions taken based on such misleading statements should be reconsidered.

Sorry about that. You must be on corpo’s payroll. I’ll put it on the record by taking it back: you did care. I think I’ve done my part. Now I’ll stop embarrassing myself and see how this pans out.