Fstab LVM - Boot failure even though no systemctl status error

lamyer · April 25, 2023, 11:07pm

I have installed dualboot Ubuntu 23.04 with Fedora 38 on LVM on LUKS encrypted partition.

[root@fedora ~]# sgdisk --print /dev/nvme0n1
Disk /dev/nvme0n1: 1953525168 sectors, 931.5 GiB
Model: WD_BLACK SN770 1TB                      
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 8926997B-19D9-4350-B802-09E6920809AF
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1574911   768.0 MiB   8301  UbuntuBoot
   2         1574912         3147775   768.0 MiB   8300  FedoraBoot
   3         3147776         4720639   768.0 MiB   8301  UnassignedBoot
   4         4720640         4982783   128.0 MiB   EF00  EFI System Partition
   5         4982784      1953525134   929.1 GiB   8301  rootfs

Partition 1 is meant for /boot for Ubuntu while Partition 2 is meant for /boot for Fedora. Both of these share /boot/efi at Partition 4. Partition 5 is encrypted with LUKS and has LVM inside it:

[root@fedora ~]# lvs
  LV         VG       Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  Backup     vgubuntu -wi-a-----  30.00g                                                    
  Data       vgubuntu -wi-ao----  50.00g                                                    
  Documents  vgubuntu -wi-ao---- 100.00g                                                    
  FedoraHome vgubuntu -wi-ao----  50.00g                                                    
  FedoraRoot vgubuntu -wi-ao----  50.00g                                                    
  FedoraSwap vgubuntu -wi-ao----  32.00g                                                    
  Media      vgubuntu -wi-a----- 450.00g                                                    
  UbuntuHome vgubuntu -wi-a-----  50.00g                                                    
  UbuntuRoot vgubuntu -wi-a-----  50.00g                                                    
  UbuntuSwap vgubuntu -wi-a-----  32.00g

It has 10 logical volumes. Root, Home, Swap exclusively for Ubuntu and for Fedora respectively. And additionally I want to mount logical volumes: Backup, Data, Documents, Media.

I appended in the /etc/fstab and I did sudo systemctl daemon-reload after editing it as follows:

/dev/mapper/vgubuntu-Backup /mnt/Backup ext4 defaults 0 2
/dev/mapper/vgubuntu-Data /mnt/Data ext4 defaults 0 2
/dev/mapper/vgubuntu-Documents /mnt/Documents ext4 defaults 0 2
/dev/mapper/vgubuntu-Media /mnt/Media ext4 defaults 0 2

I got following errors while booting

This failure hangs the boot and shows message that root is locked. And then I have to forecefully poweroff as I can’t type anything.

Surprising thing is that root, home and swap logical volumes mount without any issue!

Then I modified the fstab with UUID from blkid as I thought it might be an issue with delay in loading mapper.

/dev/disk/by-uuid/a7a07d75-5664-4182-b403-07db1222d247 /mnt/Backup ext4 defaults 0 2
/dev/disk/by-uuid/09c74676-fc66-4765-b33c-c91febe40b81 /mnt/Data ext4 defaults 0 2
/dev/disk/by-uuid/e0a18f38-8a0d-46ed-bd93-eee7d69b477a /mnt/Documents ext4 defaults 0 2
/dev/disk/by-uuid/45e16cd6-0bcd-4743-b4ae-6aea102748bd /mnt/Media ext4 defaults 0 2

But again I am facing same issue while booting!

If I comment these fstab entries and boot it and again uncomment it and do mount -a, I get no error and all logical volumes are successfully mounted.

As per given in the above pictures I also checked the following (for each of the logical volume) but there is no error

$ systemctl status -l systemd-fsck@dev-mapper-vgubuntu\\x2dMedia.service
○ systemd-fsck@dev-mapper-vgubuntu\x2dMedia.service - File System Check on /dev/mapper/vgubuntu-Media
     Loaded: loaded (/usr/lib/systemd/system/systemd-fsck@.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead)
       Docs: man:systemd-fsck@.service(8)

Any suggestions?

vgaetera · April 26, 2023, 12:17am

Try to unmount the problematic volumes and force fsck them, then check the output:

sudo mount -a
sudo lvs
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo dracut -f --regenerate-all
sudo grep -e ^GRUB_CMDLINE_LINUX /etc/default/grub
sudo grep -r -e ^options /boot/loader

lamyer · April 26, 2023, 7:48am

I have a few question before I run the commands

Is there a possibility of data loss?
Does this affects dual boot with Ubuntu?
I am able to boot to Ubuntu with those logical volumes mounted without any issues. The problem is only with Fedora.

vgaetera · April 26, 2023, 8:45am

The idea is to update and check your GRUB configuration and rebuild initramfs when all the necessary volumes are mounted.

There’s always a probability of data loss, no matter what action you take.
It is a well-known fact that only the data which has independent copies is considered more or less safe, otherwise it is the data you should be ready to lose at any time for a variety of reasons.

Sorry, I don’t use dual boot, so I’m not sure what can affect it, but in any case, you should still be able to multi-boot with UEFI boot menu.

lamyer · April 26, 2023, 9:08am

The 4 problematic volume group are initially commented, so that I can boot into Fedora:

$ sudo -i
Place your right ring finger on the fingerprint reader
[root@fedora ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Apr 24 17:44:24 2023
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/vgubuntu-FedoraRoot /                       ext4    defaults,x-systemd.device-timeout=0 1 1
UUID=121d38ad-9451-4931-b49e-f42ac6809309 /boot                   ext4    defaults        1 2
UUID=AAE0-853F          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
/dev/mapper/vgubuntu-FedoraHome /home                   ext4    defaults,x-systemd.device-timeout=0 1 2
/dev/mapper/vgubuntu-FedoraSwap none                    swap    defaults,x-systemd.device-timeout=0 0 0
#/dev/disk/by-uuid/a7a07d75-5664-4182-b403-07db1222d247 /mnt/Backup ext4 defaults 0 2
#/dev/disk/by-uuid/09c74676-fc66-4765-b33c-c91febe40b81 /mnt/Data ext4 defaults 0 2
#/dev/disk/by-uuid/e0a18f38-8a0d-46ed-bd93-eee7d69b477a /mnt/Documents ext4 defaults 0 2
#/dev/disk/by-uuid/45e16cd6-0bcd-4743-b4ae-6aea102748bd /mnt/Media ext4 defaults 0 2

The problematic logical volume Data and Documents were manually mounted after booting.

[root@fedora ~]# sudo lvs
  LV         VG       Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  Backup     vgubuntu -wi-a-----  30.00g                                                    
  Data       vgubuntu -wi-ao----  50.00g                                                    
  Documents  vgubuntu -wi-ao---- 100.00g                                                    
  FedoraHome vgubuntu -wi-ao----  50.00g                                                    
  FedoraRoot vgubuntu -wi-ao----  50.00g                                                    
  FedoraSwap vgubuntu -wi-ao----  32.00g                                                    
  Media      vgubuntu -wi-a----- 450.00g                                                    
  UbuntuHome vgubuntu -wi-a-----  50.00g                                                    
  UbuntuRoot vgubuntu -wi-a-----  50.00g                                                    
  UbuntuSwap vgubuntu -wi-a-----  32.00g

Uncommenting the 4 problematic logical volumes from fstab:

[root@fedora ~]# sudo nano /etc/fstab
[root@fedora ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Apr 24 17:44:24 2023
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/vgubuntu-FedoraRoot /                       ext4    defaults,x-systemd.device-timeout=0 1 1
UUID=121d38ad-9451-4931-b49e-f42ac6809309 /boot                   ext4    defaults        1 2
UUID=AAE0-853F          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
/dev/mapper/vgubuntu-FedoraHome /home                   ext4    defaults,x-systemd.device-timeout=0 1 2
/dev/mapper/vgubuntu-FedoraSwap none                    swap    defaults,x-systemd.device-timeout=0 0 0
/dev/disk/by-uuid/a7a07d75-5664-4182-b403-07db1222d247 /mnt/Backup ext4 defaults 0 2
/dev/disk/by-uuid/09c74676-fc66-4765-b33c-c91febe40b81 /mnt/Data ext4 defaults 0 2
/dev/disk/by-uuid/e0a18f38-8a0d-46ed-bd93-eee7d69b477a /mnt/Documents ext4 defaults 0 2
/dev/disk/by-uuid/45e16cd6-0bcd-4743-b4ae-6aea102748bd /mnt/Media ext4 defaults 0 2

All logical volumes in FSTAB got mounted without any error:

[root@fedora ~]# sudo mount -a
mount: (hint) your fstab has been modified, but systemd still uses
       the old version; use 'systemctl daemon-reload' to reload.
[root@fedora ~]# sudo systemctl daemon-reload 
[root@fedora ~]# sudo mount -a
[root@fedora ~]# sudo lvs
  LV         VG       Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  Backup     vgubuntu -wi-ao----  30.00g                                                    
  Data       vgubuntu -wi-ao----  50.00g                                                    
  Documents  vgubuntu -wi-ao---- 100.00g                                                    
  FedoraHome vgubuntu -wi-ao----  50.00g                                                    
  FedoraRoot vgubuntu -wi-ao----  50.00g                                                    
  FedoraSwap vgubuntu -wi-ao----  32.00g                                                    
  Media      vgubuntu -wi-ao---- 450.00g                                                    
  UbuntuHome vgubuntu -wi-a-----  50.00g                                                    
  UbuntuRoot vgubuntu -wi-a-----  50.00g                                                    
  UbuntuSwap vgubuntu -wi-a-----  32.00g

Grub

[root@fedora ~]# sudo grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
File descriptor 3 (pipe:[262341]) leaked on vgs invocation. Parent PID 35216: grub2-probe
File descriptor 9 (pipe:[240152]) leaked on vgs invocation. Parent PID 35216: grub2-probe
File descriptor 3 (pipe:[262341]) leaked on vgs invocation. Parent PID 35216: grub2-probe
File descriptor 9 (pipe:[240152]) leaked on vgs invocation. Parent PID 35216: grub2-probe
File descriptor 3 (pipe:[262341]) leaked on vgs invocation. Parent PID 35250: grub2-probe
File descriptor 9 (pipe:[240152]) leaked on vgs invocation. Parent PID 35250: grub2-probe
File descriptor 3 (pipe:[262341]) leaked on vgs invocation. Parent PID 35250: grub2-probe
File descriptor 9 (pipe:[240152]) leaked on vgs invocation. Parent PID 35250: grub2-probe
Found Ubuntu 23.04 (23.04) on /dev/mapper/vgubuntu-UbuntuRoot
Adding boot menu entry for UEFI Firmware Settings ...
done
[root@fedora ~]# sudo dracut -f --regenerate-all
[root@fedora ~]# sudo grep -e ^GRUB_CMDLINE_LINUX /etc/default/grub
GRUB_CMDLINE_LINUX="resume=/dev/mapper/vgubuntu-FedoraSwap rd.luks.uuid=luks-4f9105ee-ae73-4e8c-b00f-a99035a95030 rd.lvm.lv=vgubuntu/FedoraRoot rd.lvm.lv=vgubuntu/FedoraSwap rd.lvm.lv=vgubuntu/UbuntuSwap rhgb quiet splash module_blacklist=hid_sensor_hub nvme.noacpi=1"

[root@fedora ~]# sudo grep -r -e ^options /boot/loader
/boot/loader/entries/9ff7856ce4f64250b8507daf68376f6a-6.2.9-300.fc38.x86_64.conf:options root=/dev/mapper/vgubuntu-FedoraRoot ro resume=/dev/mapper/vgubuntu-FedoraSwap rd.luks.uuid=luks-4f9105ee-ae73-4e8c-b00f-a99035a95030 rd.lvm.lv=vgubuntu/FedoraRoot rd.lvm.lv=vgubuntu/FedoraSwap rd.lvm.lv=vgubuntu/UbuntuSwap rhgb quiet splash module_blacklist=hid_sensor_hub nvme.noacpi=1 
/boot/loader/entries/9ff7856ce4f64250b8507daf68376f6a-0-rescue.conf:options root=/dev/mapper/vgubuntu-FedoraRoot ro resume=/dev/mapper/vgubuntu-FedoraSwap rd.luks.uuid=luks-4f9105ee-ae73-4e8c-b00f-a99035a95030 rd.lvm.lv=vgubuntu/FedoraRoot rd.lvm.lv=vgubuntu/FedoraSwap rd.lvm.lv=vgubuntu/UbuntuSwap rhgb quiet splash module_blacklist=hid_sensor_hub nvme.noacpi=1 
/boot/loader/entries/9ff7856ce4f64250b8507daf68376f6a-6.2.11-300.fc38.x86_64.conf:options root=/dev/mapper/vgubuntu-FedoraRoot ro resume=/dev/mapper/vgubuntu-FedoraSwap rd.luks.uuid=luks-4f9105ee-ae73-4e8c-b00f-a99035a95030 rd.lvm.lv=vgubuntu/FedoraRoot rd.lvm.lv=vgubuntu/FedoraSwap rd.lvm.lv=vgubuntu/UbuntuSwap rhgb quiet splash modul

_blacklist=hid_sensor_hub nvme.noacpi=1

I did what you said she then i still get the same error

As you can see it says that the root is locked and i can’t type anything :

vgaetera · April 26, 2023, 11:03am

I tried to reproduce your problem but couldn’t:

Installed Fedora 38 in a VM.
Used LUKS with LVM and some unallocated space.
Expanded the LUKS and PV.
Created a few LVs using free space.
Formatted the LVs to Ext4.
Added to fstab by copy-pasting your config and changing UUIDs.
Tested the config.
Rebooted.

Everything mounted properly with no errors.

lamyer · April 26, 2023, 11:32am

Both operaing System share EFI partition. Can that be a source of problem? I heard that separate boot partition should not interfere with other operating system in the dual boot configuration. Do you think Ubuntu can have some effect on my Fedora installation?

I am surprised because all logical volumes (except root, home and swap for Fedora) mount successfully on Ubuntu. While in Fedora only (root, home, swap for Fedora) gets mounted on reboot. Even though all logical volumes are in the same volume group and same partition of the same SSD. The ssd has no smart errors and it is new.

What do you suggest?

vgaetera · April 26, 2023, 11:46am

I recommend to set root password and proceed to recovery console:
How to debug Dracut problems :: Fedora Docs

Try collecting complete boot log to identify the cause of service failures.

lamyer · June 19, 2023, 10:17pm

The dualboot Ubuntu 23.04 with Fedora 38 had problem, as the fsck installed by Ubuntu 23.04 was more recent version than that by initial release of Fedora 38. So partitions and volumes created in Ubuntu 23.04 failed to mount when booting with Fedora due fsck enabled before every boot. Problem solved by creating volumes and partitions in Fedora.