Error: no such device after Fedora 30 upgrade on software RAID

rkudyba · June 13, 2019, 4:32pm

Unlike my other long discussion/thread, this is a different instance of a Fedora 29 → 30 upgrade failing. The server was just rebooting with a very quick Grub error that I could not capture. The drives are configured in a software RAID 10 set up, so I’m needing help how to get Grub repaired. When I mount

It boots now to a grub-rescue prompt, but I then booted to a Fedora 30 Live DVD and chose Troubleshooting. Where should I start?

fdisk -l
Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00089e1c

Device     Boot      Start        End    Sectors   Size Id Type
/dev/sda1             2048       6143       4096     2M 83 Linux
/dev/sda2  *          6144    1030143    1024000   500M 83 Linux
/dev/sda3          1030144 1639430143 1638400000 781.3G fd Linux raid autodetect
/dev/sda4       1639430144 3907028991 2267598848   1.1T  5 Extended
/dev/sda5       1639434240 3804579839 2165145600     1T fd Linux raid autodetect
/dev/sda6       3804581888 3906981887  102400000  48.8G 83 Linux


Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0009cb2d

Device     Boot      Start        End    Sectors   Size Id Type
/dev/sdb1             2048 1638402047 1638400000 781.3G fd Linux raid autodetect
/dev/sdb2       1638402048 3803547647 2165145600     1T fd Linux raid autodetect


Disk /dev/sdc: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0008ca54

Device     Boot      Start        End    Sectors   Size Id Type
/dev/sdc1             2048 1638402047 1638400000 781.3G fd Linux raid autodetect
/dev/sdc2       1638402048 3803547647 2165145600     1T fd Linux raid autodetect


Disk /dev/sdd: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000a8dd9

Device     Boot      Start        End    Sectors   Size Id Type
/dev/sdd1             2048 1638402047 1638400000 781.3G fd Linux raid autodetect
/dev/sdd2       1638402048 3803547647 2165145600     1T fd Linux raid autodetect
/dev/sdd3       3803547648 3873179647   69632000  33.2G 82 Linux swap / Solaris


Disk /dev/loop0: 1.6 GiB, 1686425600 bytes, 3293800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop1: 6.5 GiB, 6981419008 bytes, 13635584 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop2: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/live-rw: 6.5 GiB, 6981419008 bytes, 13635584 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/live-base: 6.5 GiB, 6981419008 bytes, 13635584 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/md127: 2 TiB, 2217105948672 bytes, 4330285056 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes


Disk /dev/md126: 1.5 TiB, 1677718454272 bytes, 3276793856 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes


 lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE   MOUNTPOINT
loop0         7:0    0   1.6G  1 loop   
loop1         7:1    0   6.5G  1 loop   
├─live-rw   253:0    0   6.5G  0 dm     /
└─live-base 253:1    0   6.5G  1 dm     
loop2         7:2    0    32G  0 loop   
└─live-rw   253:0    0   6.5G  0 dm     /
sda           8:0    0   1.8T  0 disk   
├─sda1        8:1    0     2M  0 part   
├─sda2        8:2    0   500M  0 part   
├─sda3        8:3    0 781.3G  0 part   
│ └─md126     9:126  0   1.5T  0 raid10 
├─sda4        8:4    0     1K  0 part   
├─sda5        8:5    0     1T  0 part   
│ └─md127     9:127  0     2T  0 raid10 
└─sda6        8:6    0  48.8G  0 part   
sdb           8:16   0   1.8T  0 disk   
├─sdb1        8:17   0 781.3G  0 part   
│ └─md126     9:126  0   1.5T  0 raid10 
└─sdb2        8:18   0     1T  0 part   
  └─md127     9:127  0     2T  0 raid10 
sdc           8:32   0   1.8T  0 disk   
├─sdc1        8:33   0 781.3G  0 part   
│ └─md126     9:126  0   1.5T  0 raid10 
└─sdc2        8:34   0     1T  0 part   
  └─md127     9:127  0     2T  0 raid10 
sdd           8:48   0   1.8T  0 disk   
├─sdd1        8:49   0 781.3G  0 part   
│ └─md126     9:126  0   1.5T  0 raid10 
├─sdd2        8:50   0     1T  0 part   
│ └─md127     9:127  0     2T  0 raid10 
└─sdd3        8:51   0  33.2G  0 part   [SWAP]
sr0          11:0    1   4.4G  0 rom    /run/initramfs/live

Edit some more:

cat /proc/mdstat
Personalities : [raid10] 
md126 : active (auto-read-only) raid10 sdb1[1] sda3[0] sdd1[3] sdc1[2]
      1638396928 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
      bitmap: 0/13 pages [0KB], 65536KB chunk

    md127 : active (auto-read-only) raid10 sdb2[1] sdc2[2] sda5[0] sdd2[3]
          2165142528 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
          bitmap: 0/17 pages [0KB], 65536KB chunk

Trying to us this how to but I can’t tell where the root file partition system is. Edit: found it, see below.

ls -l /dev/dm-0
brw-rw----. 1 root disk 253, 0 Jun 13  2019 /dev/dm-0
[root@localhost-live liveuser]# ls -l /dev/dm-1
brw-rw----. 1 root disk 253, 1 Jun 13  2019 /dev/dm-1
[root@localhost-live liveuser]# ls -l /dev/dm-2
ls: cannot access '/dev/dm-2': No such file or directory
[root@localhost-live liveuser]# cat /sys/class/block/dm-0/dm/name
live-rw
[root@localhost-live liveuser]# cat /sys/class/block/dm-1/dm/name
live-base
[root@localhost-live liveuser]# dmsetup table
live-base: 0 13635584 linear 7:1 0
live-rw: 0 13635584 snapshot 7:1 7:2 PO 8

That last one seems to be the DVD.

Update: /dev/sda6 contains the root partition, so I mounted it as well /dev/sda2 for boot. grub2-mkconfig –o /boot/grub2/grub.conf errors with:

### BEGIN /etc/grub.d/30_os-prober ###
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
mkdir: cannot create directory ‘/var/lib/os-prober/mount’: No such file or directory
### END /etc/grub.d/30_os-prober ###

/var is on a separate partition:

UUID=d7fb7755-f5c1-4e67-bc13-9e8d87634a40 /                       ext4    defaults        1 1
UUID=cdec73cb-1375-43c9-b20f-2a7e0ecf97cb /boot                   ext4    defaults        1 2
UUID=e6c3c501-3839-474b-bc0b-12428b7c1d7f /home                   ext4    defaults        1 2
UUID=e26a7ec3-7b9c-4600-96ed-7c973dd4796f /var                    ext4    defaults        1 2
UUID=b16f6528-af27-49e0-9565-41abaf512286 swap                    swap    defaults        0 0

rkudyba · June 13, 2019, 8:10pm

20190613_155913 20190613_155923

I’m seeing this now, older Fedora 28 Grub.

rkudyba · June 17, 2019, 9:06pm

Well I solved the Grub issue, does seem to be related to the known issue of having older Fedora installations. First I had to make sure to use the Fedora 30 DVD and boot using the Troubleshooting option. I had to mount the md0 RAID partition and mount it for /var. Then I ran grub2-install /dev/sda and grub2-mkconfig.

Oddly though NetworkManger for the 2nd port on the NIC is not resolving DNS. I manually set /etc/resolv.conf and restarted the service a few times but there might be something with the UUID. I’ve enabled debug and nothing obvious appears. What else can I check?

 systemctl status systemd-resolved
* systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/etc/systemd/system/../../../usr/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-06-17 17:03:13 EDT; 1s ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 3952 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   Memory: 2.4M
   CGroup: /system.slice/systemd-resolved.service
           `-3952 /usr/lib/systemd/systemd-resolved

Jun 17 17:03:11 ourdomain.edu systemd[1]: Starting Network Name Resolution...
Jun 17 17:03:13 ourdomain.edu systemd-resolved[3952]: Positive Trust Anchors:
Jun 17 17:03:13 ourdomain.edu systemd-resolved[3952]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Jun 17 17:03:13 ourdomain.edu systemd-resolved[3952]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jun 17 17:03:13 ourdomain.edu systemd-resolved[3952]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.>
Jun 17 17:03:13 ourdomain.edu systemd-resolved[3952]: Using system hostname 'ourdomain.edu'.
Jun 17 17:03:13 ourdomain.edu systemd[1]: Started Network Name Resolution.
[root@tartarus ~]# systemctl status NetworkManager  
* NetworkManager.service - Network Manager
   Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-06-17 16:40:39 EDT; 22min ago
     Docs: man:NetworkManager(8)
 Main PID: 3577 (NetworkManager)
    Tasks: 3 (limit: 4915)
   Memory: 5.7M
   CGroup: /system.slice/NetworkManager.service
           `-3577 /usr/sbin/NetworkManager --no-daemon

Jun 17 16:49:17 ourdomain.edu NetworkManager[3577]: <info>  [1560804557.6572] audit: op="checkpoint-create" arg="/org/freedesktop/NetworkManager/Checkpoint/2" pid=3508 uid=0 result="success"
Jun 17 16:49:17 ourdomain.edu NetworkManager[3577]: <trace> [1560804557.6729] auth: call[117]: CheckAuthorization(org.freedesktop.NetworkManager.settings.modify.system), subject=unix-process[pid=3508, uid=0, start=1236555] (succeeding for root)
Jun 17 16:49:17 ourdomain.edu NetworkManager[3577]: <trace> [1560804557.6730] auth: call[117]: completed: authorized=1, challenge=0 (simulated)
Jun 17 16:49:17 ourdomain.edu NetworkManager[3577]: <info>  [1560804557.6742] settings-connection[0x55f536bfa100,99422641-8c92-41a5-8a74-025808a6d11a]: write: failure to update connection: connection System em4 cannot change the UUID from 99422641-8c92-41a5-8a74-025808>
Jun 17 16:49:17 ourdomain.edu NetworkManager[3577]: <info>  [1560804557.6744] audit: op="connection-update" uuid="99422641-8c92-41a5-8a74-025808a6d11a" name="System em4" args="connection.interface-name,connection.timestamp,connection.uuid,connection.id,802-3-ethernet.m>
Jun 17 16:49:17 ourdomain.edu NetworkManager[3577]: <trace> [1560804557.6935] auth: call[118]: CheckAuthorization(org.freedesktop.NetworkManager.checkpoint-rollback), subject=unix-process[pid=3508, uid=0, start=1236555] (succeeding for root)
Jun 17 16:49:17 ourdomain.edu NetworkManager[3577]: <trace> [1560804557.6936] auth: call[118]: completed: authorized=1, challenge=0 (simulated)
Jun 17 16:49:17 ourdomain.edu NetworkManager[3577]: <info>  [1560804557.6936] checkpoint[0x55f536c731c0]: destroy /org/freedesktop/NetworkManager/Checkpoint/2
Jun 17 16:49:17 ourdomain.edu NetworkManager[3577]: <trace> [1560804557.6940] dbus-object[0x55f536c731c0]: unexport: "/org/freedesktop/NetworkManager/Checkpoint/2"
Jun 17 16:49:17 ourdomain.edu NetworkManager[3577]: <info>  [1560804557.6945] audit: op="checkpoint-destroy" arg="/org/freedesktop/NetworkManager/Checkpoint/2" pid=3508 uid=0 result="success"

em4 is the interface:

systemd-resolve --status
Global
       LLMNR setting: yes
MulticastDNS setting: yes
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes
  Current DNS Server: 150.108.4.11
         DNS Servers: 150.108.4.11
                      150.180.2.11
Fallback DNS Servers: 8.8.8.8
                      8.8.4.4
                      2001:4860:4860::8888
                      2001:4860:4860::8844
          DNS Domain: oudomain.edu
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 5 (em2)
      Current Scopes: none
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 4 (em1)
      Current Scopes: none
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 3 (em4)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 2 (em3)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
DefaultRoute setting: yes
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes
  Current DNS Server: 150.108.4.11
         DNS Servers: 150.108.4.11
                      150.180.2.11

rkudyba · June 18, 2019, 3:18pm

Figured out the NetworkManager issue. The default route was using the first NIC port, which is connected to a iSCSI VessRAID, and that was the default route.

ip route     
default via 10.0.10.1 dev em3 proto static metric 102 
default via 150.108.68.1 dev em4 proto static metric 103 
10.0.10.0/24 dev em3 proto kernel scope link src 10.0.10.3 metric 102 
x.y.z.0/24 dev em4 proto kernel scope link src x.y.z.35 metric 103

This command fixed the problem:
ip route add default via x.y.z.1

(Obfuscated our external IP address with x.y.z.)