Rpm-ostree update fails - out of memory

Ask Fedora
,
1

I think this never happened to me before. How to debug it?

$ rpm-ostree update
error: While pulling fedora/38/x86_64/silverblue: While fetching mirrorlist 'https://ostree.fedoraproject.org/mirrorlist': While fetching https://ostree.fedoraproject.org/mirrorlist: [27] Out of memory

I’m running Fedora 38 prerelease:

$ rpm-ostree status
State: idle
Deployments:
● fedora:fedora/38/x86_64/silverblue
                  Version: 38.20230322.n.0 (2023-03-22T08:08:28Z)
               BaseCommit: f100e8a6c6dd4274a4d7bbde24ed0c42c3998d8d71a529bac27270c8b9b112df
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
          LayeredPackages: langpacks-it

  fedora:fedora/38/x86_64/silverblue
                  Version: 38.20230320.n.0 (2023-03-20T08:07:36Z)
               BaseCommit: 6da331f0f1bdecc6dbea29a56619afbf761fde2690474043b905b50e6cb776c4
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
          LayeredPackages: langpacks-it

  fedora:fedora/37/x86_64/silverblue
                  Version: 37.20230315.0 (2023-03-15T00:44:37Z)
                   Commit: d0035ee2b86a33c58f07c5e424a96a998aeed98bb1444da5f1769f6ac7f1c14e
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
                   Pinned: yes
2

How much RAM is installed?
What is the output of free?

3

It doesn’t seem a problem of RAM:

$ free -m
               total        used        free      shared  buff/cache   available
Mem:            7786         995        4682         222        2108        6319
Swap:           7785           0        7785
[fede@fedora ~]$ rpm-ostree update
error: While pulling fedora/38/x86_64/silverblue: While fetching mirrorlist 'https://ostree.fedoraproject.org/mirrorlist': While fetching https://ostree.fedoraproject.org/mirrorlist: [27] Out of memory
4

Yeah, really weird that that would happen while fetching the mirrorlist! That file is only, like, 37 bytes.

5

I found this old discussion and indeed it’s a DNS server problem.

I changed the DNS server from the default provider to 1.1.1.1 and 8.8.8.8 and now I’m able to upgrade the system again.

6

It has been 0 days since it was DNS
It has been 0 days since it was DNS1080×721 75.6 KB

1 Like
7

Can you reproduce the error with any invalid DNS server or does it need to be the specific one?

At the very least the error message is bad, but at worst it could be a serious and possibly exploitable bug in handling an unexpected response.

8

Faulty local DNS is a common issue I’ve been dealing for years and it is not limited to this forum.
Often users are not knowledgeable enough to properly investigate the problem, thus addressing their negative experience to Linux in general and Fedora in particular.
This specifically happens when the source of the problem is poor compatibility with EDNS(0), or the difference in DNS query methods between Linux and Windows.
I wonder if dealing with DNS similar to NTP by preferring a global provider over a local one might lead to a better experience at large scale.

9

Just to be clear: the Internet connection works fine with this “faulty” DNS; only rpm-ostree is failing.

I always use global DNS servers such as 1.1.1.1, but I’ve just relocated and I was testing the connection with a new provider so I changed the router settings DNS to those of the provider.

$ grep nameserver /etc/resolv.conf 
nameserver 127.0.0.53

$ rpm-ostree update          
error: While pulling fedora/38/x86_64/silverblue: While fetching mirrorlist 'https://ostree.fedoraproject.org/mirrorlist': While fetching https://ostree.fedoraproject.org/mirrorlist: [27] Out of memory

Yes, the error message is bad.

10

I realize now that resolv.conf is not the place where DNS servers are saved.

Here’s what I see after changing the DNS servers in my router settings:

$ sudo cat /etc/NetworkManager/system-connections/FRITZ\!Box\ 7530\ FA.nmconnection | grep dns
dns=8.8.8.8;1.1.1.1;
dns-search=
dns-search=

$ grep nameserver /etc/resolv.conf 
nameserver 127.0.0.53
11

Keep in mind that one domain failing to resolve correctly means there may be others, perhaps not so critical as to be immediately identifiable, but still causing minor glitches or timeouts, which you may not even notice if the service falls back from its optimal operation method to sub-optimal, just to not fail entirely.

12

Now it’s failing despite the global DNS servers…
I must say that I had to set them in GNOME network settings, as setting them in my Fritz router settings was not enough maybe.

I’ve just realized that I should have typed resolvectl status to be sure which DNS servers have been used.

$ resolvectl status
Global
         Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (wlp1s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 8.8.8.8
       DNS Servers: 8.8.8.8 1.1.1.1 192.168.178.1
        DNS Domain: fritz.box
[fede@fedora ~]$ rpm-ostree update
error: While pulling fedora/38/x86_64/silverblue: While fetching mirrorlist 'https://ostree.fedoraproject.org/mirrorlist': While fetching https://ostree.fedoraproject.org/mirrorlist: [27] Out of memory

However, at the third try the update worked fine.

13

You can enable DoT to isolate the issue and prevent possible DNS hijacking.
It the problem persists, it is likely caused by IP connectivity and unrelated to DNS.
Perhaps some of the server IPs are unreachable for you for some reason.

14

There was something else about that particular error recently. Don’t remember the topic but is should be relatively easy to find.

15

Actually, there are a few curl related errors on the forum, although with different codes:

The current problem may as well indicate faulty RAM:

16

It’s happening again despite the good DNS servers. But the RAM is available:

$ rpm-ostree update
error: While pulling fedora/38/x86_64/silverblue: While fetching mirrorlist 'https://ostree.fedoraproject.org/mirrorlist': While fetching https://ostree.fedoraproject.org/mirrorlist: [27] Out of memory
[fede@fedora ~]$ free -m
               total        used        free      shared  buff/cache   available
Mem:            7785        1798        2370         328        3616        5339
Swap:           7784         118        7666
17

For sure I have DNS problems in my Fedora laptop.
Today my laptop is connected to Internet using my phone hotspot (DNS set by the provider). I can’t access a website on my Fedora laptop, but I can access it in my phone. They use the same connection settings.

Maybe the systemd DNS resolver. I vaguely remember it caused problems in the past.