Kernel corrupted by a failed update

Ask Fedora
,
1

Recently, I tried to update my Fedora 39 laptop with the command:


sudo dnf upgrade --refresh

My system had low memory, and anyway, I proceeded thinking that the update should abort if it sees insufficient memory.

However, the update started, packages got downloaded, and it showed the update could not be finished due to lack of storage. Then everything stopped working, and when I tried rebooting, the system could not boot due to a broken kernel.

Fortunately, I could boot through an alternate kernel from Grub, and I updated the kernel fully. However, I noticed that most drivers still need to be included, and I could not do many simple tasks like playing videos. I eventually did a clean install of Fedora 40.

Since DNF is handling the update and I am not using anything from a third party, why did such a scenario happen? DNF did not give any warning about low space. It should have been checked beforehand, and the update process should have been aborted. Low memory is a typical case; just trying to update in low storage should not result in a corrupt system.

2

This depends on the low storage being talked about. If this is ESP storage for the boot, then it could very well happen after the actual update of the system was completed (for the items in system storage, like the packages). The low memory, if a factor would likely show up as an out of memory error which would trigger the oomkiller and your dnf transaction would not complete. It appears you could have interrupted the final stages by what you say you did. If you try an update again, please try to get an alternate TTY open (<Ctrl><Alt><F2> for TTY2 ie). Then you can see if anything is going on and the GUI is just not being updated.

3

Welcome to Fedora @fedjaro

I guess it is too late to debug.

If you are already aware of low memory just fix this and do not trust to software who is made be a human being. Nobody is perfect to see all possibilities. And what about some changes made by the system (new version) where the actual dnf is not taking care of it yet?

As you made a clean install, the parameters of your partitions/spaces are different and giving you a clear answer to the question of why will be a guessing game now.

4

hi @fedjaro

You should try using ncdu to see the disk usage of your directories.

sudo ncdu /

5

An easier way, and what should have been done as soon as the update failed would have been to use the command df -h which would have given the usage and free space on each mounted file system.

At that point the decision on what to remove and identifying the cause of lack of space could easily have been performed.

This usually seems to happen when a user has limited space and/or has installed extra packages that use up space in /boot or fills up the root file system /

6

This has really be done before initiating an upgrade and or a new installation. Just this way we are able to find an eventual bug. It has to be reproducible. Just so the developers can catch/fix any bugs.

So next time please feel free to ask before you upgrade.

7

I can share the partitions. I always use the same partition table as follows:

  • 600 MiB /boot/efi partiton
  • 1 GiB ext4 /boot partiton
  • Rest / root btrfs partition about 220 GiB with a /home subvolume
8

This looks like a normal partitioning. The commands given above should have shown the problems had they been used before the reinstall.

9

So what is the recommendation here? suppose total size of downloadable packages through dnf update is 1GB. How much storage do i need approximately?

10

Here is df -h for a basic install of F40 on a 128GB SSD:

% df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc3       118G   49G   66G  43% /
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           1.6G  1.9M  1.6G   1% /run
tmpfs           3.9G   16K  3.9G   1% /tmp
/dev/sdc3       118G   49G   66G  43% /home
/dev/sdc2       974M  461M  446M  51% /boot
/dev/sdc1       599M   20M  580M   4% /boot/efi
tmpfs           788M  188K  788M   1% /run/user/1000

Note that btrfs shares space between root and home subvolumes, so if you add a lot of data to /home you won’t have enough space left in the root filesystem to do an upgrade.

1 Like