Kernel panic after upgrade to 37.20230315.0

I get the following message during boot after upgrade to version 37.20230315.0:

kernel panic not syncing timer doesn't work through interrupt-remapped io-apic

(Fortunately, there is boot to previous version.)

Any ideas?

1 Like

I was going to say that it worked for me, but then I checked rpm-ostree status… and, well…

~ ❱ rpm-ostree status
State: idle
Deployments:
  fedora:fedora/38/x86_64/silverblue
                  Version: 38.20230315.n.0 (2023-03-15T08:04:24Z)
               BaseCommit: 143466920d956602b58472b770680985a1ef243574d5a27a271efcbc746f93cb
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
            SecAdvisories: 1 low, 3 moderate
                     Diff: 68 upgraded, 5 removed, 53 added
          LayeredPackages: [...]

● fedora:fedora/38/x86_64/silverblue
                  Version: 38.20230313.n.0 (2023-03-13T08:07:32Z)
               BaseCommit: 34f81d41c76e8a8c54f86336f3fb874ce88ad8b4edf19d60484d3897ee26fce4
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
          LayeredPackages: [...]

So I guess I didn’t pay enough attention when booting this morning. In other words, it looks like I might’ve hit this here as well?

rpm-ostree status -v has the upgrade line for the kernel:

kernel 6.2.5-300.fc38 -> 6.2.6-300.fc38

People report success in bodhi, however… https://bodhi.fedoraproject.org/updates/FEDORA-2023-cb170d0440

I’m glad we have rollbacks (and apparently they even work automatically in some cases).

Be sure to pin your current working boot.

sudo ostree admin pin 0

If you get error: Cannot pin staged deployment, then run the same command, but with 1 instead:

sudo ostree admin pin 1

This tells ostree to keep the deployment around and to not garbage collect it, so you can always boot into it.

You’ll want to eventually unpin it at some point in the future. The pin numbers are all relative to what you see in rpm-ostree status, starting with 0. To unpin, add either -u or --unpin before the number.

What does rpm-ostree status say for you? Which version of Silverblue are you on? Is it the previous version working for you?

When searching for kernel panic not syncing timer doesn't work through interrupt-remapped io-apic, some people are saying (years ago, on a different distro… both Ubuntu and Arch) that they had a β€œbios problem”. And they fixed it with BIOS updates or worked around by setting ACPI off (which is probably not a good idea). But we shouldn’t have to do that, especially if it was working in the previous kernel release.


BTW: On my laptop’s boot today, I did notice blank screens and my keyboard not working when I expected it to… I guess that might be related. I should try rebooting to the latest a little later, probably without anything plugged into the laptop.


Edit: I managed to find some time to reboot. I’m in 38.20230315.n.0 and everything is fine here. I wasn’t able to reproduce the issue.

What are your system specs? Some of the blog posts I found when searching for the error you shared mention out of date BIOS/EFI for AMD-based mainboards. (It’s not all AMDs; my desktop was totally fine last night, for example.)

recently i also faced same kernel panic in silverblue

i think silverblue have some issues.

Thanks for the pinning command information.

~ $ rpm-ostree status -v
State: idle
AutomaticUpdates: disabled
Deployments:
  fedora:fedora/37/x86_64/silverblue (index: 0)
                  Version: 37.20230315.0 (2023-03-15T00:44:37Z)
               BaseCommit: d0035ee2b86a33c58f07c5e424a96a998aeed98bb1444da5f1769f6ac7f1c14e
                           β”œβ”€ repo-0 (2022-11-05T08:04:38Z)
                           β”œβ”€ repo-1 (2023-03-15T00:17:23Z)
                           └─ repo-2 (2023-03-15T00:20:06Z)
                   Commit: 9de0af737177932c97cc55bb818fa04067039a6d9d2d1c8dffd78e38d2fd05e4
                   Staged: no
                StateRoot: fedora
             GPGSignature: 1 signature
                           Signature made Wed 15 Mar 2023 11:46:15 using RSA key ID F55AD3FB5323552A
                           Good signature from "Fedora <fedora-37-primary@fedoraproject.org>"
                 Upgraded: amd-gpu-firmware 20230210-147.fc37 -> 20230310-148.fc37
                           conmon 2:2.1.6-3.fc37 -> 2:2.1.7-2.fc37
                           distribution-gpg-keys 1.84-1.fc37 -> 1.85-1.fc37
                           httpd 2.4.55-1.fc37 -> 2.4.56-1.fc37
                           httpd-core 2.4.55-1.fc37 -> 2.4.56-1.fc37
                           httpd-filesystem 2.4.55-1.fc37 -> 2.4.56-1.fc37
                           httpd-tools 2.4.55-1.fc37 -> 2.4.56-1.fc37
                           intel-gpu-firmware 20230210-147.fc37 -> 20230310-148.fc37
                           iwl100-firmware 39.31.5.1-147.fc37 -> 39.31.5.1-148.fc37
                           iwl1000-firmware 1:39.31.5.1-147.fc37 -> 1:39.31.5.1-148.fc37
                           iwl105-firmware 18.168.6.1-147.fc37 -> 18.168.6.1-148.fc37
                           iwl135-firmware 18.168.6.1-147.fc37 -> 18.168.6.1-148.fc37
                           iwl2000-firmware 18.168.6.1-147.fc37 -> 18.168.6.1-148.fc37
                           iwl2030-firmware 18.168.6.1-147.fc37 -> 18.168.6.1-148.fc37
                           iwl3160-firmware 1:25.30.13.0-147.fc37 -> 1:25.30.13.0-148.fc37
                           iwl3945-firmware 15.32.2.9-147.fc37 -> 15.32.2.9-148.fc37
                           iwl4965-firmware 228.61.2.24-147.fc37 -> 228.61.2.24-148.fc37
                           iwl5000-firmware 8.83.5.1_1-147.fc37 -> 8.83.5.1_1-148.fc37
                           iwl5150-firmware 8.24.2.2-147.fc37 -> 8.24.2.2-148.fc37
                           iwl6000-firmware 9.221.4.1-147.fc37 -> 9.221.4.1-148.fc37
                           iwl6000g2a-firmware 18.168.6.1-147.fc37 -> 18.168.6.1-148.fc37
                           iwl6000g2b-firmware 18.168.6.1-147.fc37 -> 18.168.6.1-148.fc37
                           iwl6050-firmware 41.28.5.1-147.fc37 -> 41.28.5.1-148.fc37
                           iwl7260-firmware 1:25.30.13.0-147.fc37 -> 1:25.30.13.0-148.fc37
                           iwlax2xx-firmware 20230210-147.fc37 -> 20230310-148.fc37
                           kernel 6.1.14-200.fc37 -> 6.1.18-200.fc37
                           kernel-core 6.1.14-200.fc37 -> 6.1.18-200.fc37
                           kernel-modules 6.1.14-200.fc37 -> 6.1.18-200.fc37
                           kernel-modules-extra 6.1.14-200.fc37 -> 6.1.18-200.fc37
                           libertas-usb8388-firmware 2:20230210-147.fc37 -> 2:20230310-148.fc37
                           libsmbclient 2:4.17.5-2.fc37 -> 2:4.17.6-0.fc37
                           libwbclient 2:4.17.5-2.fc37 -> 2:4.17.6-0.fc37
                           linux-firmware 20230210-147.fc37 -> 20230310-148.fc37
                           linux-firmware-whence 20230210-147.fc37 -> 20230310-148.fc37
                           mesa-dri-drivers 22.3.6-1.fc37 -> 22.3.7-1.fc37
                           mesa-filesystem 22.3.6-1.fc37 -> 22.3.7-1.fc37
                           mesa-libEGL 22.3.6-1.fc37 -> 22.3.7-1.fc37
                           mesa-libGL 22.3.6-1.fc37 -> 22.3.7-1.fc37
                           mesa-libgbm 22.3.6-1.fc37 -> 22.3.7-1.fc37
                           mesa-libglapi 22.3.6-1.fc37 -> 22.3.7-1.fc37
                           mesa-libxatracker 22.3.6-1.fc37 -> 22.3.7-1.fc37
                           mesa-va-drivers 22.3.6-1.fc37 -> 22.3.7-1.fc37
                           mesa-vulkan-drivers 22.3.6-1.fc37 -> 22.3.7-1.fc37
                           mod_lua 2.4.55-1.fc37 -> 2.4.56-1.fc37
                           nspr 4.35.0-4.fc37 -> 4.35.0-5.fc37
                           nss 3.88.1-1.fc37 -> 3.89.0-1.fc37
                           nss-softokn 3.88.1-1.fc37 -> 3.89.0-1.fc37
                           nss-softokn-freebl 3.88.1-1.fc37 -> 3.89.0-1.fc37
                           nss-sysinit 3.88.1-1.fc37 -> 3.89.0-1.fc37
                           nss-util 3.88.1-1.fc37 -> 3.89.0-1.fc37
                           nvidia-gpu-firmware 20230210-147.fc37 -> 20230310-148.fc37
                           openldap 2.6.3-1.fc37 -> 2.6.4-1.fc37
                           osinfo-db 20221130-1.fc37 -> 20230308-1.fc37
                           pipewire 0.3.66-1.fc37 -> 0.3.67-1.fc37
                           pipewire-alsa 0.3.66-1.fc37 -> 0.3.67-1.fc37
                           pipewire-gstreamer 0.3.66-1.fc37 -> 0.3.67-1.fc37
                           pipewire-jack-audio-connection-kit 0.3.66-1.fc37 -> 0.3.67-1.fc37
                           pipewire-libs 0.3.66-1.fc37 -> 0.3.67-1.fc37
                           pipewire-pulseaudio 0.3.66-1.fc37 -> 0.3.67-1.fc37
                           pipewire-utils 0.3.66-1.fc37 -> 0.3.67-1.fc37
                           qt-settings 37.0-1.fc37 -> 37.1-1.fc37
                           samba-client 2:4.17.5-2.fc37 -> 2:4.17.6-0.fc37
                           samba-client-libs 2:4.17.5-2.fc37 -> 2:4.17.6-0.fc37
                           samba-common 2:4.17.5-2.fc37 -> 2:4.17.6-0.fc37
                           samba-common-libs 2:4.17.5-2.fc37 -> 2:4.17.6-0.fc37
                           webp-pixbuf-loader 0.2.1-1.fc37 -> 0.2.2-1.fc37
                           which 2.21-35.fc37 -> 2.21-39.fc37
                           wireplumber 0.4.13-1.fc37 -> 0.4.14-1.fc37
                           wireplumber-libs 0.4.13-1.fc37 -> 0.4.14-1.fc37
                           xfsprogs 5.18.0-3.fc37 -> 6.1.0-1.fc37
      RemovedBasePackages: firefox firefox-langpacks 110.0-3.fc37
          LayeredPackages: epson-inkjet-printer-escpr langpacks-en

● fedora:fedora/37/x86_64/silverblue (index: 1)
                  Version: 37.20230310.0 (2023-03-10T01:11:58Z)
               BaseCommit: 0295325360e092d3ceac17666b85b32697fd0a053f4314002a13a17e124aed16
                           β”œβ”€ repo-0 (2022-11-05T08:04:38Z)
                           β”œβ”€ repo-1 (2023-03-10T00:41:44Z)
                           └─ repo-2 (2023-03-10T00:44:20Z)
                   Commit: 0c703e4a12f87d7b746e683857b0fa5b4bc4858ecc1f72342ed20ae98f5a6a9b
                           β”œβ”€ fedora (2022-11-05T08:04:38Z)
                           β”œβ”€ fedora-cisco-openh264 (2022-10-06T11:01:40Z)
                           β”œβ”€ fedora-modular (2022-11-05T07:58:03Z)
                           β”œβ”€ updates (2023-03-12T00:36:48Z)
                           β”œβ”€ updates-archive (2023-03-12T01:04:59Z)
                           └─ updates-modular (2023-02-22T11:28:37Z)
                StateRoot: fedora
             GPGSignature: 1 signature
                           Signature made Fri 10 Mar 2023 12:12:05 using RSA key ID F55AD3FB5323552A
                           Good signature from "Fedora <fedora-37-primary@fedoraproject.org>"
      RemovedBasePackages: firefox firefox-langpacks 110.0-3.fc37
          LayeredPackages: epson-inkjet-printer-escpr langpacks-en
~ $ sudo ostree admin pin 1
[sudo] password for baseline: 
Deployment 1 is now pinned
~ $

Is there a command that can be run to get the system specifications?

ASUS are only offering a beta BIOS for download; I’m assuming my BIOS is the current version.

You can see a lot of firmware with fwupdmgr get-devices β€” in the case of my desktop, it even lists firmware that it finds that isn’t supported. (It might or might not have the version that’s shown on the website for manually updating the firmware.)

If your hardware is supported (my desktop isn’t, but laptop is), then fwupdmgr update will help you update it.

$ fwupdmgr get-devices
WARNING: UEFI firmware can not be updated in legacy BIOS mode
See https://github.com/fwupd/fwupd/wiki/PluginFlag:legacy-bios for more information.
System manufacturer System Product Name
β”‚
β”œβ”€300 Series Chipset SATA Controller:
β”‚     Device ID:          7d29f2075dcafb4488b40c73f199cf46bb76bddb
β”‚     Current version:    02
β”‚     Vendor:             Advanced Micro Devices, Inc. [AMD] (PCI:0x1022)
β”‚     GUIDs:              17f97e61-e971-53d5-bb9e-6bcef9c6c23f ← PCI\VEN_1022&DEV_43B7
β”‚                         ef4a96a0-45d1-5cff-90d3-e6f387d130e7 ← PCI\VEN_1022&DEV_43B7&REV_02
β”‚                         8b139035-0f55-5b27-b1bc-c2c0064addd4 ← PCI\VEN_1022&DEV_43B7&SUBSYS_1B211062
β”‚                         2752f8e3-f453-5b2c-b26e-fbe10c23be07 ← PCI\VEN_1022&DEV_43B7&SUBSYS_1B211062&REV_02
β”‚                         1d3e715a-7904-528d-8c2f-ae4648e476e7 ← PCI\VEN_1022&DEV_1453
β”‚                         3c5c9a8d-93bc-592a-b193-1f8a742dca36 ← PCI\VEN_1022&DEV_1453&REV_00
β”‚                         ca12abed-f939-51d2-83cd-3537ffd291e3 ← PCI\VEN_1022&DEV_1453&SUBSYS_10438747
β”‚                         02100450-4f5b-5ada-9c7b-2806ccde0d3c ← PCI\VEN_1022&DEV_1453&SUBSYS_10438747&REV_00
β”‚     Device Flags:       β€’ Internal device
β”‚                         β€’ Cryptographic hash verification is available
β”‚   
β”œβ”€AMD Ryzen 5 1400 Quad-Core Processor:
β”‚     Device ID:          4bde70ba4e39b28f9eab1628f9dd6e6244c03027
β”‚     Current version:    0x08001138
β”‚     Vendor:             Advanced Micro Devices, Inc.
β”‚     GUIDs:              b9a2dd81-159e-5537-a7db-e7101d164d3f ← cpu
β”‚                         997104da-ec11-5e04-9d94-ac5501fbf609 ← CPUID\PRO_0&FAM_17
β”‚                         5fa201c0-e152-5003-a56b-93a9dd683fc7 ← CPUID\PRO_0&FAM_17&MOD_01
β”‚                         0eab26c9-612c-5655-957c-6a4bec51e214 ← CPUID\PRO_0&FAM_17&MOD_01&STP_1
β”‚     Device Flags:       β€’ Internal device
β”‚   
β”œβ”€HD161HJ:
β”‚     Device ID:          9afcd7db8643d511ee389321038ae4c56c7a45f3
β”‚     Summary:            ATA drive
β”‚     Current version:    GF100-07
β”‚     Vendor:             Samsung (ATA:0x144D, OUI:0000f0)
β”‚     Serial Number:      S0V3J9CQ715781
β”‚     GUIDs:              a6520ca8-2f92-52e8-a911-d7843604853f ← IDE\SAMSUNG_HD161HJ_________________________GF100-07
β”‚                         026faef4-2a1d-5f07-ada1-7284f8dbaf17 ← IDE\0SAMSUNG_HD161HJ_________________________
β”‚                         7cde58b9-1b94-5063-8360-b9dc69c66071 ← SAMSUNG HD161HJ
β”‚     Device Flags:       β€’ Internal device
β”‚                         β€’ Updatable
β”‚                         β€’ System requires external power source
β”‚                         β€’ Needs a reboot after installation
β”‚                         β€’ Device is usable for the duration of the update
β”‚   
β”œβ”€Hitachi HCS721010CLA332:
β”‚     Device ID:          26856ad5cbbaecc7579ecb42c8a409dbe14e936b
β”‚     Summary:            ATA drive
β”‚     Current version:    JP4OA3EA
β”‚     Vendor:             Western Digital (ATA:0x101C, OUI:000cca)
β”‚     Serial Number:      JP9960HZ1PBAEU
β”‚     GUIDs:              4d781e62-0c7c-5c97-911b-ee4f2498b1c4 ← IDE\Hitachi_HCS721010CLA332_________________JP4OA3EA
β”‚                         325d29fc-aa2a-52b3-8388-fc018d9be5aa ← IDE\0Hitachi_HCS721010CLA332_________________
β”‚                         a1719ebe-ec03-592c-8122-3c2deeea097a ← Hitachi HCS721010CLA332
β”‚     Device Flags:       β€’ Internal device
β”‚                         β€’ Updatable
β”‚                         β€’ System requires external power source
β”‚                         β€’ Needs a reboot after installation
β”‚                         β€’ Device is usable for the duration of the update
β”‚   
β”œβ”€TPM:
β”‚     Device ID:          c6a80ac3a22083423992a3cb15018989f37834d6
β”‚     Summary:            TPM 2.0 Device
β”‚     Current version:    3.7.0.3
β”‚     Vendor:             Advanced Micro Devices, Inc. (TPM:AMD)
β”‚     GUIDs:              ff71992e-52f7-5eea-94ef-883e56e034c6 ← system-tpm
β”‚                         9305de1c-1e12-5665-81c4-37f8e51219b8 ← TPM\VEN_AMD&DEV_0001
β”‚                         78a291ae-b499-5b0f-8f1d-74e1fefd0b1c ← TPM\VEN_AMD&MOD_AMD
β”‚                         65a3fced-b423-563f-8098-bf5c329fc063 ← TPM\VEN_AMD&DEV_0001&VER_2.0
β”‚                         5e704f0d-83cb-5364-8384-f46d725a23b8 ← TPM\VEN_AMD&MOD_AMD&VER_2.0
β”‚     Device Flags:       β€’ Internal device
β”‚   
└─Vega 10 XL/XT [Radeon RX Vega 56/64]:
      Device ID:          d05d536fa7bd6811a229e6b6e896497fa3cf1084
      Current version:    xxx-xxx-xxx
      Vendor:             Advanced Micro Devices, Inc. [AMD/ATI] (PCI:0x1002, PCI:0x1022)
      GUIDs:              887634ce-5760-54fe-8b49-89d905716eb5 ← PCI\VEN_1002&DEV_687F
                          3e5dccc2-9f56-5b7a-8778-30e98f64c3e2 ← PCI\VEN_1002&DEV_687F&REV_C1
                          787b837f-8ac5-507c-bd34-ff729ec533ec ← PCI\VEN_1002&DEV_687F&SUBSYS_14582308
                          78fcc526-29ac-5ea5-a401-5d386396d0b1 ← PCI\VEN_1002&DEV_687F&SUBSYS_14582308&REV_C1
                          83774004-d23a-5f43-a56b-a7693934dfb6 ← PCI\VEN_1022&DEV_1471
                          f6d469bc-28cc-5c0c-aaae-07ca05ae3027 ← PCI\VEN_1022&DEV_1471&REV_00
                          f1fbe3ed-78eb-5a91-acea-c7d7c5931eb5 ← PCI\VEN_1022&DEV_1471&SUBSYS_10221471
                          2dce2b4b-a226-5ed9-afc3-26517ab459c4 ← PCI\VEN_1022&DEV_1471&SUBSYS_10221471&REV_00
      Device Flags:       β€’ Internal device
                          β€’ Cryptographic hash verification is available

Just got the 37.20230322.0 update; no change, kernel panic persists:

ASUS still only offering beta version of the 6061 bios for the PRIME B350M-A motherboard.

Just got the 37.20230426 update: no change, kernel still panics.

Found out Asus PRIME B340M-A BIOS is 6042 dated 2022-04-28.

I’m having the same error message when trying to boot Fedora 37 (not Silverblue) with kernels 6.1.18 and 6.2.7. Kernel 6.1.15 works. This is with a Gigabyte A320M-HD2 motherboard, and happens even after updating the BIOS to the newest available version.

Same here. I originally skipped past this thread because I’m not running Silverblue, but I’ve got two systems on F37 Workstation that can’t boot either 6.1.18 or 6.2.7. I haven’t done any BIOS updates, but from what Nebbie Zebbie is saying, it doesn’t sound like it makes a difference anyway.

inxi output:

$ inxi -CMSG
System:
  Host: toolbox Kernel: 6.1.14-200.fc37.x86_64 arch: x86_64 bits: 64
    Desktop: GNOME Distro: Fedora release 37 (Thirty Seven)
Machine:
  Type: Desktop Mobo: ASUSTeK model: PRIME B350M-A v: Rev X.0x
    serial: <superuser required> BIOS: American Megatrends v: 6042
    date: 04/28/2022
Use of uninitialized value $value[0] in string ne at /usr/bin/inxi line 9227.
Use of uninitialized value $value[0] in string ne at /usr/bin/inxi line 9227.
CPU:
  Info: quad core (2-mt/2-st) model: AMD Ryzen 5 1400 bits: 64 type: MST AMCP
    cache: L2: 2 MiB
  Speed (MHz): avg: 1622 min/max: 1550/3200 cores: 1: 1550 2: 1375 3: 1550
    4: 1550 5: 1272 6: 2437
Graphics:
  Device-1: AMD Vega 10 XL/XT [Radeon RX 56/64] driver: amdgpu v: kernel
  Display: wayland server: X.Org v: 22.1.8 compositor: gnome-shell driver:
    dri: radeonsi gpu: amdgpu resolution: 1920x1080~60Hz
  API: OpenGL v: 4.6 Mesa 22.3.7 renderer: AMD Radeon RX Vega (vega10 LLVM
    15.0.7 DRM 3.49 6.1.14-200.fc37.x86_64)

Ah, that info there is helpful. I think Chris’ GPU speculation can be ruled out, as I have a GTX 1060; the consistent thing is Ryzen 1st gen CPUs (I have a Ryzen 5 1500X).

Kernel panics in the 37.20230401 update.

Kernel panics in the 37.20230402 update.

Just thought I would share my findings on this topic. I am running a Ryzen 1500x and GTX 1050 ti, I was experiencing kernel panics with the later 6.1.x and all of the 6.2.x kernels.

I managed to fix the issue by compiling a custom 6.2.9 kernel using the patch mentioned here

1 Like

Thanks for that update, Lenard.

It would seem the cause of the problem is identified.

I can see the patch has made it way into version 6.3 rc 5 of the kernel. Hopefully when the 6.3 kernel is released this will be resolved.

Thank you for that update, Lenard; very promising.

From uname -a I see I am on fedora 6.1.14-200.fc37.x86_64.

Maybe there is retroactive patch application?

And just for the record:

Kernel panics in the 37.20230409 update.