Bootctl update fails with "Failed to update EFI: Failed to find ESP device"

Ask Fedora
, ,
1

Hi!

My CoreOS machine was off for a few weeks and upon reboot I was met with:

error: ../../grub-core/loader/arm64/linux.c:60:invalid magic number.
error: ../../grub-core/loader/arm64/linux.c:279:you need to load the kernel
first.

Press any key to continue...

running 37.20230401.3.0. I’m able to boot using the previous snapshot 37.20230322.3.0

I’m seeing coreos-bootupctl-update-aarch64.service is failing which I missed initially:

[core@majora ~]$ sudo systemctl status coreos-bootupctl-update-aarch64.service
× coreos-bootupctl-update-aarch64.service - Update aarch64 Bootloader
     Loaded: loaded (/usr/lib/systemd/system/coreos-bootupctl-update-aarch64.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-04-26 19:26:43 CDT; 19s ago
    Process: 1184 ExecStart=/usr/bin/bootupctl update (code=exited, status=1/FAILURE)
   Main PID: 1184 (code=exited, status=1/FAILURE)
        CPU: 11ms

Apr 26 19:26:42 majora systemd[1]: Starting coreos-bootupctl-update-aarch64.service - Update aarch64 Bootloader...
Apr 26 19:26:43 majora bootupctl[1184]: error: internal error: Failed to update EFI: Failed to find ESP device
Apr 26 19:26:43 majora systemd[1]: coreos-bootupctl-update-aarch64.service: Main process exited, code=exited, status=1/FAILURE
Apr 26 19:26:43 majora systemd[1]: coreos-bootupctl-update-aarch64.service: Failed with result 'exit-code'.
Apr 26 19:26:43 majora systemd[1]: Failed to start coreos-bootupctl-update-aarch64.service - Update aarch64 Bootloader.

Here is some additional information about the system:

 [core@majora ~]$ sudo bootupctl status
Component EFI
  Installed: grub2-efi-aa64-1:2.06-10.fc35.aarch64,shim-aa64-15.4-5.aarch64
  Update: Available: grub2-efi-aa64-1:2.06-88.fc37.aarch64,shim-aa64-15.6-2.aarch64
No components are adoptable.
CoreOS aleph image ID: fedora-coreos-35.20220424.3.0-metal.aarch64.raw
Boot method: EFI

[core@majora ~]$ rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Wed 2023-04-26 19:41:31 UTC)
Deployments:
  fedora:fedora/aarch64/coreos/stable
                  Version: 37.20230401.3.0 (2023-04-17T16:29:37Z)
               BaseCommit: e9287e4ec341ea061ce9763e5fea574c1d8e2e61b0a4bb2638960d6d24ae90a4
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
            SecAdvisories: 1 unknown severity, 1 low, 2 moderate
                     Diff: 36 upgraded, 2 added
          LayeredPackages: ...
           EnabledModules: cri-o:1.25

● fedora:fedora/aarch64/coreos/stable
                  Version: 37.20230322.3.0 (2023-04-03T20:34:30Z)
               BaseCommit: 7ca8df047f700f912dea94c3cb997d833cad064e0cd8490cfa3fe3eaf2c64f1c
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
          LayeredPackages: ...
           EnabledModules: cri-o:1.25
                   Pinned: yes

The machine is a Honeycomb LX2

I’m not sure if I’m just missing something or doing something incorrectly. Please let me know if I can try to provide any more information. Thank you in advance for your time!

cc @dustymabe

2

The coreos-bootupctl-update-aarch64.service was added in overlay/15fcos: upgrade bootloader on aarch64 at boot by bgilbert · Pull Request #2308 · coreos/fedora-coreos-config · GitHub as part of the fix for Old bootloader versions don't boot new aarch64 6.2+ kernels · Issue #1441 · coreos/fedora-coreos-tracker · GitHub. I’m suprised it’s failing for you. What do the following commands show on your system?

sudo ls -lh /dev/disk/by-partlabel/
sudo blkid
sudo lsblk
3

Sorry for my lack of knowledge here:

[core@majora ~]$ sudo ls -lh /dev/disk/by-partlabel/
total 0
lrwxrwxrwx 1 root root 10 Apr 26 19:26 boot-1 -> ../../sdb3
lrwxrwxrwx 1 root root 10 Apr 26 19:26 boot-2 -> ../../sdc3
lrwxrwxrwx 1 root root 10 Apr 26 19:26 esp-1 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Apr 26 19:26 esp-2 -> ../../sdc2
lrwxrwxrwx 1 root root 10 Apr 26 19:26 reserved-1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Apr 26 19:26 reserved-2 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Apr 26 19:26 root-1 -> ../../sdb4
lrwxrwxrwx 1 root root 10 Apr 26 19:26 root-2 -> ../../sdc4
[core@majora ~]$ sudo blkid
/dev/md127: LABEL="root" UUID="10276cdd-2aaf-40c3-8e5e-a5d46a0bb2ea" BLOCK_SIZE="512" TYPE="xfs"
/dev/sdb4: UUID="f775a7d2-6429-2741-03d5-14a0f5bf86bf" UUID_SUB="f9e1222e-8caa-c675-36ea-4bee209ca2ea" LABEL="any:md-root" TYPE="linux_raid_member" PARTLABEL="root-1" PARTUUID="92ae0212-48e2-448a-a21c-8c69f1ab0083"
/dev/sdb2: SEC_TYPE="msdos" LABEL_FATBOOT="esp-1" LABEL="esp-1" UUID="E72E-30AE" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="esp-1" PARTUUID="41c71c2c-73d4-4c27-bd43-9834bf6a40db"
/dev/sdb3: UUID="8b941b50-0f56-fcb9-a32a-99cf8910a68c" UUID_SUB="8a364cb7-a17b-5b58-21df-146d1da94ffa" LABEL="any:md-boot" TYPE="linux_raid_member" PARTLABEL="boot-1" PARTUUID="b7f195bd-a68c-4b9d-bbb6-7f27f9575794"
/dev/sdb1: UUID="9331-2667" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="reserved-1" PARTUUID="b7c25326-89cc-4180-9ad2-1e60818b631a"
/dev/sdc2: SEC_TYPE="msdos" LABEL_FATBOOT="esp-2" LABEL="esp-2" UUID="E734-C1E7" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="esp-2" PARTUUID="e3a4885a-8918-4d4e-8541-39718d8f38f1"
/dev/sdc3: UUID="8b941b50-0f56-fcb9-a32a-99cf8910a68c" UUID_SUB="e6097cf2-f3a5-8a9e-96e4-126324ff906b" LABEL="any:md-boot" TYPE="linux_raid_member" PARTLABEL="boot-2" PARTUUID="28b6d40a-7fc7-4bca-9c3a-fa721f5dbc3a"
/dev/sdc1: PARTLABEL="reserved-2" PARTUUID="8096454e-9801-4a2c-9b03-494b968276d3"
/dev/sdc4: UUID="f775a7d2-6429-2741-03d5-14a0f5bf86bf" UUID_SUB="6f019956-789e-7ebc-7a01-68af32f8cdac" LABEL="any:md-root" TYPE="linux_raid_member" PARTLABEL="root-2" PARTUUID="4abc3772-13bc-4ceb-808e-22682ddcaf1d"
/dev/md126: LABEL="boot" UUID="11e3e835-6704-422f-b04b-3a632a8bd78c" BLOCK_SIZE="1024" TYPE="ext4"
/dev/sda1: UUID="920479fe-ef22-4c1e-a664-3eab5dbb04a9" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="f32a03de-01"
[core@majora ~]$ sudo lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda         8:0    0 931.5G  0 disk
└─sda1      8:1    0 931.5G  0 part
sdb         8:16   0 223.6G  0 disk
├─sdb1      8:17   0     1M  0 part
├─sdb2      8:18   0   127M  0 part
├─sdb3      8:19   0   384M  0 part
│ └─md126   9:126  0 383.9M  0 raid1 /boot
└─sdb4      8:20   0 223.1G  0 part
  └─md127   9:127  0 222.9G  0 raid1 /var/lib/kubelet/pods/1a7720ca-e6e0-43fc-b9f1-b58e2217cb0a/volume-subpaths/sc-dashboard-provider/grafana/4
                                     /var/lib/kubelet/pods/1a7720ca-e6e0-43fc-b9f1-b58e2217cb0a/volume-subpaths/config/grafana/2
                                     /var/lib/kubelet/pods/1a7720ca-e6e0-43fc-b9f1-b58e2217cb0a/volume-subpaths/config/grafana/0
                                     /var/lib/kubelet/pods/e798cb03-480f-4e79-9a6e-84e85c422703/volume-subpaths/pvc-28276ae0-616f-4189-b6d2-30771ed717e5/prometheus/2
                                     /var/lib/kubelet/pods/1a7720ca-e6e0-43fc-b9f1-b58e2217cb0a/volume-subpaths/config/download-dashboards/0
                                     /var/lib/containers/storage/overlay
                                     /var
                                     /sysroot/ostree/deploy/fedora-coreos/var
                                     /usr
                                     /etc
                                     /
                                     /sysroot
sdc         8:32   0 223.6G  0 disk
├─sdc1      8:33   0     1M  0 part
├─sdc2      8:34   0   127M  0 part
├─sdc3      8:35   0   384M  0 part
│ └─md126   9:126  0 383.9M  0 raid1 /boot
└─sdc4      8:36   0 223.1G  0 part
  └─md127   9:127  0 222.9G  0 raid1 /var/lib/kubelet/pods/1a7720ca-e6e0-43fc-b9f1-b58e2217cb0a/volume-subpaths/sc-dashboard-provider/grafana/4
                                     /var/lib/kubelet/pods/1a7720ca-e6e0-43fc-b9f1-b58e2217cb0a/volume-subpaths/config/grafana/2
                                     /var/lib/kubelet/pods/1a7720ca-e6e0-43fc-b9f1-b58e2217cb0a/volume-subpaths/config/grafana/0
                                     /var/lib/kubelet/pods/e798cb03-480f-4e79-9a6e-84e85c422703/volume-subpaths/pvc-28276ae0-616f-4189-b6d2-30771ed717e5/prometheus/2
                                     /var/lib/kubelet/pods/1a7720ca-e6e0-43fc-b9f1-b58e2217cb0a/volume-subpaths/config/download-dashboards/0
                                     /var/lib/containers/storage/overlay
                                     /var
                                     /sysroot/ostree/deploy/fedora-coreos/var
                                     /usr
                                     /etc
                                     /
                                     /sysroot
4

Sorry for the delayed response here.

OK I see. It looks like you set up RAID on this system and it looks like bootupd doesn’t handle this appropriately. There is an existing open issue against bootupd at Support updating multiple EFIs in mirrored setups · Issue #132 · coreos/bootupd · GitHub. Can you also open an issue for this against GitHub - coreos/fedora-coreos-tracker: Issue tracker for Fedora CoreOS so we can document the incompatibility and track a fix and possibly document workarounds there.

5

This is now tracked in: