Rpm-ostree upgrade: toooooooo slow


#1


I installed fedora sliverblue just now, and I tried rpm-ostree upgrade and the average download speed was 100KB/s. It wasted lots of time as there were too many things to download.:disappointed_relieved:


#2

So, right now, we only have 1 single mirror, and it is not exactly the best connectivity wise, especially from outside of the USA and Canada. I have been using a mirror setup by @miabbott by using this file:

$ cat /etc/ostree/remotes.d/fedora-workstation.conf 
[remote "fedora-workstation"]
url=https://faw.piratemirror.party/
gpg-verify=true
gpgkeypath=/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-28-primary

There is work going on to use the regular mirror network with ostree that should solve that issue.


#3

The first one will take a while if you’ve got an old build. The later ones don’t need to pull as much.


#4

Thanks for the shoutout @misc!

For those curious, the mirror is located in Amsterdam.

You can read about it here - https://faw.piratemirror.party/README.txt

I’m currently mirroring the stable F27 and F28 content; I’ll add in the F29 content when it is released and deprecate the F27 mirroring at that time.


#5

Is there an issue to track about mirroring all Silverblue content on official Fedora mirrors?


#6

Adding ostree repositories to the set of content that the mirrors mirror via rsync has not worked out well in the past - mirrors have been unhappy about having to rsync an ever-expanding gigantic tree of small files.

The hope is that rojig can largely solve this - clients will get the bulk of the data from packages that are already mirrored on the mirror network.

If that doesn’t work out for some reason, another option would be to mirror only static deltas. Updating via static deltas is much more efficient than downloading file-by-file, and in addition, a curated set of static deltas would be a much more reasonable thing for mirrors to rsync. I’m not sure how to split up the downloads in that way - http redirects? ostree feature to have a lookaside metalink for static deltas?

Finally, it might be possible to have an some mirrors mirror the content via ostree rather than rsync - this would be much more efficient and would also allow a mirror to only mirror to a certain depth, instead of mirroring all the data. This would probably need some work on mirrormanager and considerable coordination withe the mirror sites.

I don’t know of any tracking of this as a work item.


#7

Actually we are using amazon cloudfront for /objects/ and /deltas/ so I would normally expect this to be a lot faster. What version / release are you using? I assume other downloads are faster?


#8

So since the certificate did expire on the pirate mirror (we have all been there, I really can’t wait until lets encrypt automation will be done at the httpd server level with automated and default renewal), i switched back to the default mirror, and to answer @kevin , the speed is fine. So I guess the problem is solved.


#9

Oops…I knew I forgot to do something this weekend…


#10

Aaaand…we’re back!


#11

https://pagure.io/fedora-infrastructure/issue/5970


#12

Not solved yet…:disappointed_relieved:


#13

Upgrading FAW28 to Silverblue 29 seems jolly slow anyway (running in a VM over F28 but I assume that is not the problem).


#14

So you all are using the default settings and still seeing it be slow? Can you share exactly where you are seeing it be slow so we can try and isolate if its delta/objects download (cloudfront) or something else(possibly fedoraproject) or slow after it downloads?


#15

Location: China
A fresh installation in my GNOME boxes.
In fact, I am not familiar with rpm-ostree, so what kind of output shall I provide?


#16

Ah, so it had not even gotten to the point to give you an estimate?

@dustymabe is there any debugging we can do here? Perhaps we need to mirror more things?


#17

Yeah we definitely need to tackle ostree mirroring. Discussed in releng meeting a few weeks ago. We’ll try to prioritize this and work with patrick to implement.


#18

Sure, I was just wondering if there was more we could do now with the cloudfront setup to help.


#19

Can you use the normal fedora repos with rpm-ostree?


#20

Can you use the normal fedora repos with rpm-ostree?

They are used for things like rpm-ostree install, but the ostree content of Silverblue is hundreds of thousands of small files in a separate repo (i.e. not a yum/dnf repo).

My understanding is that the burden of hosting tens of thousands of larger RPM files for various Fedora releases is much more palatable and feasible for mirrors than what would be required to mirror the content in various ostree repos for each Fedora release.