Grub2-install results in "not a correct XFS inode"

Ask Fedora
, ,
1

After upgrading to F39 I ran the post install step to upgrade grub2: grub2-install /dev/sda
The output shows no errors: Installation finished. No error reported.

However, after rebooting I get the following message in the console:
error: ../../grub-core/fs/xfs.c:541:not a correct XFS inode.

I have to press Space a lot of times until the grub menu appears, and after selecting a kernel I also have to press Space many more times before the system boots up.

I have searched a lot for this error, but unfortunately there are no solutions to fix this issue or I haven’t found them.

How can I fix this?

Additional info:

$ [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
BIOS
2

What version & spin of fedora are you using?

AFAIK the xfs file system is only installed by default with the server version (though I may be wrong)

What guide did you use that instructed you to do the grub2-install step?
Even if that was a correct step, you should be aware that for quite some time the boot loader has been much larger than the MBR on the drive and that an install of fedora in legacy/bios/MBR mode creates a ‘Bios Boot’ partition to hold the larger boot record.

3

I’m using Fedora Server and I have been upgrading the system for a few releases with dnf.

Even if that was a correct step, you should be aware that for quite some time the boot loader has been much larger than the MBR on the drive and that an install of fedora in legacy/bios/MBR mode creates a ‘Bios Boot’ partition to hold the larger boot record.

My current disk setup looks like this:

NAME                                          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda                                             8:0    0   32G  0 disk
├─sda1                                          8:1    0    2G  0 part  /boot
└─sda2                                          8:2    0   25G  0 part
  └─luks-e2d96d1c-0567-46b6-ab59-0695d3371e0b 253:0    0   25G  0 crypt /
sdb                                             8:16   0   64G  0 disk
└─data                                        253:1    0   64G  0 crypt /data
zram0                                         252:0    0  7.8G  0 disk  [SWAP]

Disk /dev/sda: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x6c32bb13

Device     Boot   Start      End  Sectors Size Id Type
/dev/sda1  *       2048  4196351  4194304   2G 83 Linux
/dev/sda2       4196352 56625151 52428800  25G 83 Linux

# dysk
┌─────────────────────────────────────────────────────┬────┬─────┬────┬─────────┬────┬────┬───────────┐
│                     filesystem                      │type│disk │used│   use   │free│size│mount point│
├─────────────────────────────────────────────────────┼────┼─────┼────┼─────────┼────┼────┼───────────┤
│/dev/mapper/data                                     │xfs │crypt│6.7G│10% ▌    │ 62G│ 69G│/data      │
│/dev/mapper/luks-e2d96d1c-0567-46b6-ab59-0695d3371e0b│xfs │crypt│ 16G│60% ███  │ 11G│ 27G│/          │
│/dev/sda1                                            │xfs │ HDD │303M│14% ▊    │1.8G│2.1G│/boot      │
└─────────────────────────────────────────────────────┴────┴─────┴────┴─────────┴────┴────┴───────────┘

This is on my test VM (Proxmox), but my bare-metal server in a DC looks the same, except that the partitions are bigger.

So what I am supposed to do? I do have a separate /boot partition. How would I create a BIOS boot partition (the one you were talking about) after ther fact?

4

Make sure that the GRUB configuration is correct. You can edit the GRUB configuration file/etc/default/grub. Open it and ensure that the GRUB_CMDLINE_LINUX parameter; it might include root=UUID=<UUID> or root=/dev/sdaX. Ensure it points to the correct root partition.

After making any changes to the GRUB configuration file, run the following command to update the GRUB configuration:
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

worse case scenarios :

sudo xfs_repair /dev/sdaX to attempt a repair, or sudo grub2-install /dev/sda and try reinstalling GRUB on the MBR (Master Boot Record) of your disk.

5

I’ve already done all of that before I even posted. But thanks a bunch for your reply.

6

Since I do not use the server version I cannot test (except in a VM) the problem you report.

It appears from the doc that you probably did the proper thing so that seems it may be a bug introduced by F39.

Please show us the output of sudo fdisk -l on the system that has the problem, even if you must boot to live media to enable that. I have no clue if the real system has the BIOS boot partition, but it seems your VM does not.

I just installed F38 server as BIOS boot on a VM, running under libvirt/QEMU/VMM, fully updated it, then ran lsblk -f which shows that there are 3 partitions created.

$ lsblk -f
NAME            FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sr0                                                                                              
zram0                                                                                            [SWAP]
vda                                                                                              
├─vda1                                                                                           
├─vda2          xfs                        8bad216e-7757-46d1-81c6-fe6eef742d7e    679.1M    29% /boot
└─vda3          LVM2_member LVM2 001       0B5Y02-lzIi-PF2g-XSiI-53uU-whnh-LnauGL                
  └─fedora-root xfs                        a6b7be73-234d-421b-b03a-fc494d307441       13G    13% /

I then upgraded it to F39 using the dnf system-upgrade process and after the upgrade completed and rebooted I then ran lsblk -f again.

$ lsblk -f
NAME            FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sr0                                                                                              
zram0                                                                                            [SWAP]
vda                                                                                              
├─vda1                                                                                           
├─vda2          xfs                        8bad216e-7757-46d1-81c6-fe6eef742d7e    622.8M    35% /boot
└─vda3          LVM2_member LVM2 001       0B5Y02-lzIi-PF2g-XSiI-53uU-whnh-LnauGL                
  └─fedora-root xfs                        a6b7be73-234d-421b-b03a-fc494d307441     12.9G    14% /

I then ran fdisk to see the full partition info

$ fdisk -l 
Disk /dev/vda: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 2BD76579-5B6E-424E-A7A7-6AF62783EFEA

Device       Start      End  Sectors Size Type
/dev/vda1     2048     4095     2048   1M BIOS boot
/dev/vda2     4096  2101247  2097152   1G Linux filesystem
/dev/vda3  2101248 41940991 39839744  19G Linux LVM


Disk /dev/mapper/fedora-root: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/zram0: 7.75 GiB, 8324644864 bytes, 2032384 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Note that the installer created 3 partitions on my 20GB virtual disk.
The default BIOS boot partition of 1M
/boot of 1GB
and the remainder as LVM for the root file system.

I even tested running grub2-install after completing the upgrade from F38 to F39 and it seemed to have no effect. I suspect it is not required at all with newer fedora versions, and do not know the age of that document that recommended it.

7

Yep, neither of them has the BIOS boot partition.

# fdisk -l
Disk /dev/sda: 32 GiB, 34359738368 bytes, 67108864 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x6c32bb13

Device     Boot   Start      End  Sectors Size Id Type
/dev/sda1  *       2048  4196351  4194304   2G 83 Linux
/dev/sda2       4196352 56625151 52428800  25G 83 Linux


Disk /dev/sdb: 64 GiB, 68719476736 bytes, 134217728 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/luks-e2d96d1c-0567-46b6-ab59-0695d3371e0b: 24.98 GiB, 26826768384 bytes, 52396032 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/zram0: 7.75 GiB, 8323596288 bytes, 2032128 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/data: 63.98 GiB, 68702699520 bytes, 134184960 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Hmmm, this document is updated every time there is a new Fedora release, which you can see at the beginning of the document where it states sudo dnf system-upgrade download --releasever=39 which means somebody had to change it so that it says 39.

Anyway, what is the process to fix this? I can increase my VM’s disk0 by 1 GB and add another 1MB partition with type BIOS boot. However, this partition will be after the root partition. I can’t just create one in the beginning of the disk.

The same is true for my bare metal box. I still have room left to add another partition. But yet again, it will be after the current partitions (which means /dev/sda3).

I can of course play around with my VM on Proxmox, since I can always rollback easily. So what are the steps after I created /dev/sda3 1 MB type: BIOS boot?

Edit: I should also be able to rsync /boot to somewhere else and remove the partition and create 2 partitions (bios boot and /boot) in the beginning of the disk instead of /dev/sda1. I still need to check what files I will have to update (besides /etc/fstab). Any pointers?

8

True, It was recently updated.

However, that does not mean the entire document was fully updated – nor can it be read to mean it is updated with every release.

I found this with a quick search
https://www.humblec.com/what-is-bios-boot-filesystem-type-in-fedora-and-why-it-is-required/

I have to admit I have not used a bios boot system for many years (at least 20) so I have no clues about how to fix an issue with no bios boot partition. The links I provided also imply that it only applicable for a GPT partitioned disk so it may not apply for your system.

9

This was introduced earlier, most probably in F38, since I did a successful grub2-install on F37 but ran into the same issue as the OP in F38 (see this thread).

I also ran xfs_repair & grub2-install again, with the latter only resulting in more than doubling the amount of errors reported. I have 1Mb of space before the first partition (holding /boot), with this layout created back on F31 (I think) & upgraded release-by-release using dnf system-upgrade; I ran grub2-install for the first time when upgrading to F37 & it worked there without any issues.

Like the OP, I use this VM to test upgrades before proceeding with the actual physical host. My host’s layout had 32k of space before the first partition & grub2-install in F37 was failing until I moved the start of the first partition (Dell diagnostics) from sector 63 to 2048, then worked flawlessly. I skipped grub2-install when upgrading the host to F38 & can keep doing that from now on, but would love to recover the VM; I unfortunately don’t have a snapshot to rollback to…

10

Thanks for the links. I created my partitions manually, but did not create a GPT label nor a BIOS partition, because this was never mentioned in the documentation when I installed Fedora server. I can’t recall which version it was, but it was something higher than 30. I moved to new HW back then and installed the OS. Since then I’ve only used dnf to upgrade the OS.

Either way, it seems that I am rather screwed right now. It’s not really possible to change the partition label without losing the partitions and data on that disk. Furthermore, if the links tell us that a GPT is required, what is to be done if one does not have a GPT label?

Somehow this situation is a problem. Well, it’s not an issue unless I run grub2-install. The only problem is that at one point I might have to run it, when the current loader in the MBR becomes incompatible with the data/layout in /boot.

So how am I supposed to solve the issue without reinstalling the entire OS?

11

As the partition table is of type dos aka mbr, then you do not need the bios-boot partition. grub2 would install itself in the space before the start of the first partition. In GPT formatted disk you do need the bios boot partition,

It is strange that the /boot file system is an xfs file system. I would expect it to be a ext4 file system.

Is the system bootable at the moment? If so you could leave it as is and be happy. Otherwise you can save the contents of the /boot partition using for example tar or cpio, recreate the boot partition as ext4 and restore the contents.

12

Which makes this even weirder. As you can see my first partition starts at 2048. And there are no errors during grub2-install.

Not really. Many, many moons ago, it was advertised that grub can handle xfs. A couple of years later, I thus created all my /boot partitions as xfs and never had any issues. Until now.

Yes, I rolled back the VM. But even with the error it boots up. It just takes many, many key presses (see my first post). So it cannot boot without interaction, which is rather useless.

I can certainly test this on my VM. However, something is off. Why wouldn’t XFS work anymore all of a sudden? Why are there no errors during grub2-install? I do have space enough before the first partition, so all should be good. But it is not.

Debugging is not possible without error messages, except the ones that happen before the grub menu (which are not logged anywhere except the console and which I have already put in my first post.)

Does anyone know grub2 devs? Shall I just open a ticket with the grub2 project? I am at a loss, especially since a usually not destructive post upgrade task basically screws up my system.

13

Open a ticket in the Fedora https://bugz.fedoraproject.org/grub2 . The Fedora version of grub2 is heavily modified from the upstream version.

14

I found a bug that seems to match this issue and left a comment.

https://bugzilla.redhat.com/show_bug.cgi?id=2254370

15

It’s been a while, but when I originally created this VM on F31, I just followed anaconda & accepted defaults whenever possible; I’m pretty sure the layout and FS types are what F31 Server Edition suggested…

16

That is still the suggested file system for /boot for the Server installation. More reasone to make sure it works.

17

That’s exactly it! The only difference is that the description claims downgrading GRUB to -109 fixes the issue, while mine happened with -108 & the last time it worked was with -94.

It appears -89 is available on F38, directly accessible via dnf downgrade– is it better to use that or try to manually download -94 from F37 repos?

18

Maybe you should add that info to the ticket.

It appears -89 is available on F38, directly accessible via dnf downgrade– is it better to use that or try to manually download -94 from F37 repos?

I’d try the downgrade approach but that’s just my opinion. I guess the grub devs can give a better answer.

19

Worked like a charm! Now I’m in the same boat as you: able to boot unattended, but potentially exposed at the next grub2-install.

20

This sounds like the details should be part of the bug report as a critical issue since upgrading grub may break the system again. Who knows how long that particular build may remain available?