I decided to share my experience as it could happen to any of us living any part of the world. There will be name-calling. I’ll try my best to be respectful, but if the tone is of concern, please take it down and I’ll write up a more softer generalised version of the post.
tl;dr: there’s a low quality package mirror that’s causing bad UX. There seems to be no quality control on public package mirrors.
At the time of writing, there’s only two public Fedora package mirrors in South Korea: namely KAIST and KRFOSS. The KAIST mirror has been around since the early days of Fedora. I know for a fact that it has been in good competent hands(the uni staff), hosted on more-or-less enterprise-grade hardware well suited for the purpose, with their uni network peered directly with the national backbone. I’ve experienced no big issues with them. If the mirror is down, it’s down. If it’s up, it’s up. There was no inbetween. Judging from several emails I’ve changed with the KAIST team over the years, I can tell they’re quite proud of what they’re running and maintain the mirror responsibly.
The competency of the operators of KRFOSS, on the other hand, has been questionable. First off, KRFOSS is a project. It’s a group of small mirror operators rather than a typical single independent mirror operator you’d expect: an organisation or individual in a single geologically location. KRFOSS services the Fedora mirror works by having one CDN front(Cloudflare) with the servers run by many actual hosts operated in various locations in South korea as source. How the traffic gets routed is not clear so even the operators themselves wouldn’t be able to locate the problem when HTTP 4xx and 5xx start popping up. Although not technically against any rules, but IMO, the use of CDN for hosting package mirror must be discouraged or even banned(if you’re not on a peering contract, don’t even think about hosting one). Using the script I wrote, I found that only a fraction of all public mirrors are hosted behind a some form of CDN including Cloudflare.
There’s a whole host of reasons why Fedora packages shouldn’t be cached with CDNs:
- Web caches cannot possibly keep up with the rate at which the build servers are churning out new packages. They simply cannot be treated as regular web contents
- There’s no way of knowing if the source is actually located where it says it is. In the case of KRFOSS, the source might as well be in China with no actual physical host in Korea. Can’t ping or traceroute the source. The only front reachable from outside are Cloudflare caches. We just have to take the operator’s word for it. Not a good idea
So I contacted the KRFOSS team a while back because I’d get occasional HTTP 404. This is no recent event. AFAIK, their mirror has been like this at least for the past 2 years. The response I got made it clear that they have no interest in operating the public mirror seriously.
The original email written in both Korean and English. Got a response in Korean, clearly machine translated. Not even bother to use LLM to get quality translation, just straight up old-fashioned machine translation. Whoever replied to my mail is definitely not a Korean national. Why would anyone who doesn’t speak the language volunteer to run a project like KRFOSS? This got me all paranoid and wondering if the real purpose of KRFOSS is a supply chain attack for another wave of cyberattack similar to the one in 2013(they host mirrors for a bunch of other distros and FOSS projects). They refer to themself as “X“. No real name, no organisation, no base of operation, no nothing. Just “X“ claiming to be the one who’s involved in the KRFOSS project. Not trustworthy at all. And why are they so obsessed with having multiple “sponsors” when they’re simply just using their CDN services? What are they trying to prove?
They admitted in the mail that they’re using some tier-2 mirror(mirrors.tuna.tsinghua.edu.cn) in China as source. So, they’re basically running the mirror with two major points of failure/bottleneck(CDN + tier 2 source) and it’s surprising that it’s working like 80% of the time.
(HTTP 522 is a Cloudflare-specific HTTP code returned when the origin times out)
I cc’d admin@fedoraproject.org. In my book, KRFOSS failed all my smell tests but no one at Fedora seem to care.
I get that Fedora is not RHEL. Users wouldn’t expect enterprise-grade quality control on the infrastructure, but the package mirror is still the most important infrastructure when it comes to Linux distros, no matter how big or small. There ought to be some quality controls around it. I’m sure KRFOSS is just one example and there are a few other public mirrors that are not up to the standard(or lack thereof). If RH cares about Fedora’s reputation, they really should do something about it.
Whenever I had issues with the mirror, I visited their website. From what I could gather, they don’t seem to be ashamed of having to host a public mirror on consumer-grade hardware from consumer-grade ISP network. To me, the whole project looks more like hobbyists’ playground than a real OS infrastructure. I’m not saying that enthusiasts should be allowed to host a public mirror. I’d do what I could to help KRFOSS guys out, had they been clear on their identities. I think public mirrors should be run by someone who at least has access to reliable hardware and IP transit. Unfortunately, KRFOSS guys are not the ones who could meet the criteria.
The montage of screenshots I compiled so far.
Translation: guide on bypassing daily transfer limit of ISP in various internet routers(consumer-grade network hardware) by sending a special DHCP option to the upstream DHCP server.
Background: there’s no true unlimited-data internet plan available to residential-premise customers in South Korea. Even 10G products exist, but all ISPs set a daily cap. Such guide would be useful when hosting a public mirror on ISP nets.
Translation: pulled Alpine and FreeBSD mirrors because there’s no storage. “Emergency maintence“(this isn’t just one-off incident. They get these “emergencies” quite frequently)
Taking a mirror down to update a firmware on a router(as in cheap home routers you can get for less than $100-200)…
Translation: taking a mirror down due to failed mobo
A mirror is down due to failed router.
[root@41e911af8a72 /]# dnf update
Updating and loading repositories:
Fedora 43 openh264 (From Cisco) - x86_64 100% | 2.1 KiB/s | 5.8 KiB | 00m03s
Fedora 43 - x86_64 100% | 7.4 MiB/s | 36.4 MiB | 00m05s
>>> Status code: 404 for https://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/repomd.xml (IP: 104.18.2.112) -https://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/repom
>>> Status code: 404 for http://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/repomd.xml (IP: 104.18.2.112) -http://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/repomd.
>>> Status code: 404 for https://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/affb4c277d7a0a05ec39e29c3207de63fe207adf97143e88c61bccc0e2a0135b-primary.xml.zck (IP: 104.18.3.112) -https://mi
>>> Status code: 404 for https://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/73a6103e51d00abd1abd50eb54f98a07b398a86972555b532af6acdb98565c3d-comps-Everything.x86_64.xml.zst (IP: 104.18.3.1
>>> Status code: 404 for http://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/73a6103e51d00abd1abd50eb54f98a07b398a86972555b532af6acdb98565c3d-comps-Everything.x86_64.xml.zst (IP: 104.18.3.11
>>> Status code: 404 for http://mirror.krfoss.org/fedora/releases/43/Everything/x86_64/os/repodata/affb4c277d7a0a05ec39e29c3207de63fe207adf97143e88c61bccc0e2a0135b-primary.xml.zck (IP: 104.18.3.112) -http://mirr
Fedora 43 - x86_64 - Updates 100% | 3.6 MiB/s | 15.1 MiB | 00m04s
>>> Status code: 404 for https://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/repomd.xml (IP: 104.18.2.112) -https://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/repomd.xml Status code: 404 for http://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/repomd.xml (IP: 104.18.2.112) -http://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/repomd.xml Status code: 404 for https://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/319bdfe41801129e6a275520018afde70388e98579e369211392c61c263e1b7a-primary.xml.zck (IP: 104.18.3.112) -https://mirror
>>> Status code: 404 for http://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/319bdfe41801129e6a275520018afde70388e98579e369211392c61c263e1b7a-primary.xml.zck (IP: 104.18.3.112) -http://mirror.k
>>> Status code: 404 for https://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/296735a92b915a82ea639d5ee1c4a8af4a6e5d4194786ce9c83ab7e70cf09f80-comps-Everything.x86_64.xml.zst (IP: 104.18.3.112)
>>> Status code: 404 for http://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/296735a92b915a82ea639d5ee1c4a8af4a6e5d4194786ce9c83ab7e70cf09f80-comps-Everything.x86_64.xml.zst (IP: 104.18.3.112) -
>>> Status code: 404 for https://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/a6700f91cf6527f1d99d18493336af5a3a767ba141ed69ebb95b615560d262e0-updateinfo.xml.zck (IP: 104.18.3.112) -https://mir
>>> Status code: 404 for http://mirror.krfoss.org/fedora/updates/43/Everything/x86_64/repodata/a6700f91cf6527f1d99d18493336af5a3a767ba141ed69ebb95b615560d262e0-updateinfo.xml.zck (IP: 104.18.3.112) -http://mirro
repolist 404’d. KRFOSS team’s explanation: the source mirror is not syncing fast enough, so not our problem.
repolist 200’d but the actual meta DB file 404’d. The meta files are clearly not syncing atomically.





