Dnf metadata refresh very slow

I create new toolbox containers and the first time I execute dnf, the metadata refresh takes uncomfortably long to finish: last 15 minutes its been mostly idling along and it’s just halfway through the second repo of the multitude of repos. Subsequent runs will be a bit faster, but it’s still always a nuisance. The download first boosts, then stalls for long time, then boosts again, then stalls again and so on and so forth.

I tried a bit of fiddling:

  • block some domains in the vain hope that it’s just some mirror that’s slow for me – no luck
  • limit the repo metalink URLs to country=XX, where XX have been several European countries – no luck
  • comment metalink and uncomment baseurl in the repo files with just one hand picked mirror where I confirmed fast download speeds with a 10GB file.

All without luck. If other users had this abysmal user experience, there would be widespread complaints. I suspect my ISP is doing nefarious things, as it also appears to be slowing down SSH connections, but metadata refresh should just be https connections, no? I have yet to experience slowdowns in http connections.

How would you go along proving/disproving the fault lies with the middle man?

1 Like

Hello @elpapsch,
There have been some registry issues I have noticed on the discussion forum here, about slow or failing repo connections. I don’t know if that is related to the problems you are experiencing, but I suspect they are related. I am currently without my toolbox containers as I am running F33SB rawhide, and toolbox is not working correctly on it right now. So I can’t test it from here without reverting to F32SB stable. What I can say is it took longer than usual for my toolbox container to pull from the registry today. (yes though I can’t use them I can still build them apparently). I think this also can be related to the infrastructure move Fedora did this summer, I would guess there are still some wrinkles to smooth out.

I wouldn’t know off the top of my head how to check this out, but I have my suspicions about my own ISP and it’s parent throttling bandwidth at times. Although they’re not supposed to be. Along those line though, with the added load on networks globally from the enforced work at home situation, bandwidth is likely getting gobbled up quickly.

1 Like

This looks like a flaw in the mirror selection protocol.
Perhaps checking speed and latency periodically can mitigate the problem.
Although this may require a complete protocol change.

For some ISPs, the less they know about your traffic, the better for you.
VPN is now almost one of the basic needs for internet access.

2 Likes

The mirrors were my first thought as well. Mirrors, type of connection to the Internet (Mbps? Wired or wireless?), and type of hardware would all be the first variables to check.

It doesn’t seem reasonable that an ISP would filter any traffic to or from Fedora unless you are in China or something.

It might not be content filtering, but traffic shaping or incorrect load balancing.
Net neutrality is a problem in many countries.

1 Like

Thanks @all for the input so far!

Mirrors: I modified the existing metalink URL by appending country=XX, i.e. https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-32&arch=x86_64&country=FR
Type of connection: It’s a broadband connection over TV cable with 400Mbit/s gross, while I typically get 50 - 70% of that.
Hardware: I have just one machine here. For a long time, this was a Lenovo X230, while recently it’s a X240. It’s the same Fedora installation, I just swapped the disk.

It could be less-nefarious traffic shaping or incorrect load balancing, as @vgaetera suggested.

1 Like

To be concrete, this is Vodafone we are talking about. I did not choose them. They bought my TV-cable based ISP last year and service seems to have deteriorated since then. That’s just my observation:

  • more outages occurring more often
  • SSH connections being slowed down or blocked. Case in point: 2 weeks ago I pressed Tab to have the shell complete a remote file path. I pressed Tab several times, suddenly the connection breaks, with “destination host unreachable” – yeah sure. Connection worked fine for another user working from elsewhere.
  • VPN being blocked “softly”. Before Vodafone bought the ISP, I could use Private Internet Access just fine. Months after the merger, I wanted to use PIA again. Networkmanager establishes the connection just fine. Once I generate some traffic, the bandwidth suddenly dwindles, until there is virtually no traffic going on. Same thing with Cisco VPN (vpnc), which my office uses. Other users working from elsewhere report no such problem.

I’ll try again with AirVPN today, which is another VPN which was working just fine months ago.

2 Likes

Check it out, I’d be interested in the results.

The neubot link is dead. I can find neubot.org via search, though there doesn’t seem to be any activity going on anymore.

The fast.com Netflix speed test shows me slower upload than the speed test which connects to a server in a ~60km away data center. “Internet Health Test” is another speed test with similar results. That’s too imprecise to be considered traffic shaping.

I tried AirVPN with the server in my same country. Bandwidth halfed, though internet connections are stable. DNF metadata refresh finally took only ~40 seconds instead of 40 minutes, which is great. :slight_smile:

So there are still some suspicions of bad ISP behavior, but I cannot pinpoint the culprit. The issue with PIA seems to be a problem of PIA. I listened for TCP RST packets using tcpdump, where in one attempt I got two RST packets from their server (but not in the other attempts, there it was just silently not working). I did not dig into the Cisco VPN issue.

Thanks @all for the suggestions!

In the end, finding a working VPN did the thing :slight_smile:

1 Like