Fedora HDD stops working when starting VM

Hi together,

I hope someone can help me. Since the latest libvirt update it seems that my notebook stops working whenever I start a VM. It also does not matters which VM I start…
Rancher Desktop works perfectly fine (appimage) so it is not my notebook.
Once I close the VM, the iowait will be gone, but the filesytem is not write able anymore

root@fedora:~# dnf history info 45
Transaction ID : 45
Begin time     : Di 11 Jun 2024 09:58:10 CEST
Begin rpmdb    : a33061227f6ca4e15c65cf6cea04b0c11f8c8931775f5e0bee52547751efb056
End time       : Di 11 Jun 2024 09:58:27 CEST (17 seconds)
End rpmdb      : d70a75274332a1c72f928996146a8883fd0284be238535dee0b9f9e462fd860d
User           : XXXX
Return-Code    : Success
Releasever     : 40
Command Line   : update -y
Comment        : 
Packages Altered:
    Install  composefs-1.0.3-1.fc40.x86_64                                   @fedora
    Upgrade  appstream-data-40-8.fc40.noarch                                 @updates
    Upgraded appstream-data-40-7.fc40.noarch                                 @@System
    Upgrade  containers-common-5:0.59.1-1.fc40.noarch                        @updates
    Upgraded containers-common-5:0.58.0-2.fc40.noarch                        @@System
    Upgrade  containers-common-extra-5:0.59.1-1.fc40.noarch                  @updates
    Upgraded containers-common-extra-5:0.58.0-2.fc40.noarch                  @@System
    Upgrade  edk2-ovmf-20240524-3.fc40.noarch                                @updates
    Upgraded edk2-ovmf-20240524-1.fc40.noarch                                @@System
    Upgrade  flexiblas-3.4.4-1.fc40.x86_64                                   @updates
    Upgraded flexiblas-3.4.2-1.fc40.x86_64                                   @@System
    Upgrade  flexiblas-netlib-3.4.4-1.fc40.x86_64                            @updates
    Upgraded flexiblas-netlib-3.4.2-1.fc40.x86_64                            @@System
    Upgrade  flexiblas-openblas-openmp-3.4.4-1.fc40.x86_64                   @updates
    Upgraded flexiblas-openblas-openmp-3.4.2-1.fc40.x86_64                   @@System
    Upgrade  fontconfig-2.15.0-6.fc40.x86_64                                 @updates
    Upgraded fontconfig-2.15.0-4.fc40.x86_64                                 @@System
    Upgrade  iproute-6.7.0-2.fc40.x86_64                                     @updates
    Upgraded iproute-6.7.0-1.fc40.x86_64                                     @@System
    Upgrade  iproute-tc-6.7.0-2.fc40.x86_64                                  @updates
    Upgraded iproute-tc-6.7.0-1.fc40.x86_64                                  @@System
    Upgrade  libdrm-2.4.121-1.fc40.x86_64                                    @updates
    Upgraded libdrm-2.4.120-3.fc40.x86_64                                    @@System
    Upgrade  librsvg2-2.57.1-6.fc40.x86_64                                   @updates
    Upgraded librsvg2-2.57.1-4.fc40.x86_64                                   @@System
    Upgrade  libvirt-client-10.1.0-2.fc40.x86_64                             @updates
    Upgraded libvirt-client-10.1.0-1.fc40.x86_64                             @@System
    Upgrade  libvirt-daemon-10.1.0-2.fc40.x86_64                             @updates
    Upgraded libvirt-daemon-10.1.0-1.fc40.x86_64                             @@System
    Upgrade  libvirt-daemon-common-10.1.0-2.fc40.x86_64                      @updates
    Upgraded libvirt-daemon-common-10.1.0-1.fc40.x86_64                      @@System
    Upgrade  libvirt-daemon-config-network-10.1.0-2.fc40.x86_64              @updates
    Upgraded libvirt-daemon-config-network-10.1.0-1.fc40.x86_64              @@System
    Upgrade  libvirt-daemon-driver-interface-10.1.0-2.fc40.x86_64            @updates
    Upgraded libvirt-daemon-driver-interface-10.1.0-1.fc40.x86_64            @@System
    Upgrade  libvirt-daemon-driver-network-10.1.0-2.fc40.x86_64              @updates
    Upgraded libvirt-daemon-driver-network-10.1.0-1.fc40.x86_64              @@System
    Upgrade  libvirt-daemon-driver-nodedev-10.1.0-2.fc40.x86_64              @updates
    Upgraded libvirt-daemon-driver-nodedev-10.1.0-1.fc40.x86_64              @@System
    Upgrade  libvirt-daemon-driver-nwfilter-10.1.0-2.fc40.x86_64             @updates
    Upgraded libvirt-daemon-driver-nwfilter-10.1.0-1.fc40.x86_64             @@System
    Upgrade  libvirt-daemon-driver-qemu-10.1.0-2.fc40.x86_64                 @updates
    Upgraded libvirt-daemon-driver-qemu-10.1.0-1.fc40.x86_64                 @@System
    Upgrade  libvirt-daemon-driver-secret-10.1.0-2.fc40.x86_64               @updates
    Upgraded libvirt-daemon-driver-secret-10.1.0-1.fc40.x86_64               @@System
    Upgrade  libvirt-daemon-driver-storage-10.1.0-2.fc40.x86_64              @updates
    Upgraded libvirt-daemon-driver-storage-10.1.0-1.fc40.x86_64              @@System
    Upgrade  libvirt-daemon-driver-storage-core-10.1.0-2.fc40.x86_64         @updates
    Upgraded libvirt-daemon-driver-storage-core-10.1.0-1.fc40.x86_64         @@System
    Upgrade  libvirt-daemon-driver-storage-disk-10.1.0-2.fc40.x86_64         @updates
    Upgraded libvirt-daemon-driver-storage-disk-10.1.0-1.fc40.x86_64         @@System
    Upgrade  libvirt-daemon-driver-storage-gluster-10.1.0-2.fc40.x86_64      @updates
    Upgraded libvirt-daemon-driver-storage-gluster-10.1.0-1.fc40.x86_64      @@System
    Upgrade  libvirt-daemon-driver-storage-iscsi-10.1.0-2.fc40.x86_64        @updates
    Upgraded libvirt-daemon-driver-storage-iscsi-10.1.0-1.fc40.x86_64        @@System
    Upgrade  libvirt-daemon-driver-storage-iscsi-direct-10.1.0-2.fc40.x86_64 @updates
    Upgraded libvirt-daemon-driver-storage-iscsi-direct-10.1.0-1.fc40.x86_64 @@System
    Upgrade  libvirt-daemon-driver-storage-logical-10.1.0-2.fc40.x86_64      @updates
    Upgraded libvirt-daemon-driver-storage-logical-10.1.0-1.fc40.x86_64      @@System
    Upgrade  libvirt-daemon-driver-storage-mpath-10.1.0-2.fc40.x86_64        @updates
    Upgraded libvirt-daemon-driver-storage-mpath-10.1.0-1.fc40.x86_64        @@System
    Upgrade  libvirt-daemon-driver-storage-rbd-10.1.0-2.fc40.x86_64          @updates
    Upgraded libvirt-daemon-driver-storage-rbd-10.1.0-1.fc40.x86_64          @@System
    Upgrade  libvirt-daemon-driver-storage-scsi-10.1.0-2.fc40.x86_64         @updates
    Upgraded libvirt-daemon-driver-storage-scsi-10.1.0-1.fc40.x86_64         @@System
    Upgrade  libvirt-daemon-driver-storage-zfs-10.1.0-2.fc40.x86_64          @updates
    Upgraded libvirt-daemon-driver-storage-zfs-10.1.0-1.fc40.x86_64          @@System
    Upgrade  libvirt-daemon-kvm-10.1.0-2.fc40.x86_64                         @updates
    Upgraded libvirt-daemon-kvm-10.1.0-1.fc40.x86_64                         @@System
    Upgrade  libvirt-daemon-lock-10.1.0-2.fc40.x86_64                        @updates
    Upgraded libvirt-daemon-lock-10.1.0-1.fc40.x86_64                        @@System
    Upgrade  libvirt-daemon-log-10.1.0-2.fc40.x86_64                         @updates
    Upgraded libvirt-daemon-log-10.1.0-1.fc40.x86_64                         @@System
    Upgrade  libvirt-daemon-plugin-lockd-10.1.0-2.fc40.x86_64                @updates
    Upgraded libvirt-daemon-plugin-lockd-10.1.0-1.fc40.x86_64                @@System
    Upgrade  libvirt-daemon-proxy-10.1.0-2.fc40.x86_64                       @updates
    Upgraded libvirt-daemon-proxy-10.1.0-1.fc40.x86_64                       @@System
    Upgrade  libvirt-libs-10.1.0-2.fc40.x86_64                               @updates
    Upgraded libvirt-libs-10.1.0-1.fc40.x86_64                               @@System
    Upgrade  mdevctl-1.3.0-5.fc40.x86_64                                     @updates
    Upgraded mdevctl-1.3.0-4.fc40.x86_64                                     @@System
    Upgrade  mesa-dri-drivers-24.0.9-1.fc40.x86_64                           @updates
    Upgraded mesa-dri-drivers-24.0.8-1.fc40.x86_64                           @@System
    Upgrade  mesa-filesystem-24.0.9-1.fc40.x86_64                            @updates
    Upgraded mesa-filesystem-24.0.8-1.fc40.x86_64                            @@System
    Upgrade  mesa-libEGL-24.0.9-1.fc40.x86_64                                @updates
    Upgraded mesa-libEGL-24.0.8-1.fc40.x86_64                                @@System
    Upgrade  mesa-libGL-24.0.9-1.fc40.x86_64                                 @updates
    Upgraded mesa-libGL-24.0.8-1.fc40.x86_64                                 @@System
    Upgrade  mesa-libgbm-24.0.9-1.fc40.x86_64                                @updates
    Upgraded mesa-libgbm-24.0.8-1.fc40.x86_64                                @@System
    Upgrade  mesa-libglapi-24.0.9-1.fc40.x86_64                              @updates
    Upgraded mesa-libglapi-24.0.8-1.fc40.x86_64                              @@System
    Upgrade  mesa-va-drivers-24.0.9-1.fc40.x86_64                            @updates
    Upgraded mesa-va-drivers-24.0.8-1.fc40.x86_64                            @@System
    Upgrade  mesa-vulkan-drivers-24.0.9-1.fc40.x86_64                        @updates
    Upgraded mesa-vulkan-drivers-24.0.8-1.fc40.x86_64                        @@System
    Upgrade  open-vm-tools-12.4.0-1.fc40.x86_64                              @updates
    Upgraded open-vm-tools-12.3.5-3.fc40.x86_64                              @@System
    Upgrade  open-vm-tools-desktop-12.4.0-1.fc40.x86_64                      @updates
    Upgraded open-vm-tools-desktop-12.3.5-3.fc40.x86_64                      @@System
    Upgrade  podman-5:5.1.0-1.fc40.x86_64                                    @updates
    Upgraded podman-5:5.0.3-1.fc40.x86_64                                    @@System
    Upgrade  rsvg-pixbuf-loader-2.57.1-6.fc40.x86_64                         @updates
    Upgraded rsvg-pixbuf-loader-2.57.1-4.fc40.x86_64                         @@System
    Upgrade  vte-profile-0.76.2-2.fc40.x86_64                                @updates
    Upgraded vte-profile-0.76.2-1.fc40.x86_64                                @@System
    Upgrade  vte291-0.76.2-2.fc40.x86_64                                     @updates
    Upgraded vte291-0.76.2-1.fc40.x86_64                                     @@System
root@fedora:~#




Please post text that you copy and paste (using the </> button on the text entry toolbar).

When you post images it is impossible for others to search and find information about your problem and potential fixes.

The original text was already insert via the </> button.
If you can tell me, how to save text from pictures, I may do it. As the System will be read-only I can not even do screenshots or open anything. Therefor external Pictures are the only possibility…
Also they just visualize that iowait gets to the max, as the HDD becomes unusable / unavailable

It is ok to use images when text is not possible.

Is your hard drive failing? You can test using

smartctl
2 Likes

Also look at sudo dmesg and look for uncorrectable disk errors.
If you see them then the hdd is failing and needs replacing.
Backup any data you care about while you can.

I have no disk issues from what I can see and I also SMART is happy.
If it would be a disk sector issue, it should also just happen by a specific VM and not regardless of the VM, as I have 5 in total.
Also it would not explain, why Rancher Desktop with the Lima VM works without any issue. This leads me to a version problem, as it is an appimage and has it`s own libs

root@fedora-:~# dmesg | grep -i nvme
[    2.325231] nvme 0000:02:00.0: platform quirk: setting simple suspend
[    2.325290] nvme nvme0: pci function 0000:02:00.0
[    2.330172] nvme nvme0: missing or invalid SUBNQN field.
[    2.349524] nvme nvme0: 15/0/0 default/read/poll queues
[    2.353255]  nvme0n1: p1 p2 p3 p4 p5 p6
[   25.119609] EXT4-fs (nvme0n1p5): mounted filesystem 73c0b6d6-7945-4249-b692-2ae162953e3c r/w with ordered data mode. Quota mode: none.
[   26.560318] block nvme0n1: No UUID available providing old NGUID
root@fedora-:~# dmesg | grep -i error
[    1.714893] RAS: Correctable Errors collector initialized.
[   25.043134] EDAC igen6 MC1: HANDLING IBECC MEMORY ERROR
[   25.043136] EDAC igen6 MC0: HANDLING IBECC MEMORY ERROR
[   25.078687] hp_wmi: query 0x4 returned error 0x5
root@fedora-:~# dmesg | grep -i warning
[    0.000000] x86/split lock detection: #AC: crashing the kernel on kernel split_locks and warning on user-space split_locks
[   25.204455] ACPI Warning: \_SB.PC00.XHCI.RHUB.HS10._DSM: Argument #4 type mismatch - Found [Integer], ACPI requires [Package] (20230628/nsarguments-61)
[   46.684600] warning: `gnome-shell' uses wireless extensions which will stop working for Wi-Fi 7 hardware; use nl80211
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.9.6-200.fc40.x86_64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       TS4TMTE250S
Serial Number:                      H860000020
Firmware Version:                   A228VCJL
PCI Vendor/Subsystem ID:            0x1d79
IEEE OUI Identifier:                0x48357c
Total NVM Capacity:                 4.000.787.030.016 [4,00 TB]
Unallocated NVM Capacity:           0
Controller ID:                      0
NVMe Version:                       1.4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          4.000.787.030.016 [4,00 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            7c3548 521018df14
Local Time is:                      Mon Jul  1 09:21:20 2024 CEST
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005e):     Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x0f):         S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
Maximum Data Transfer Size:         64 Pages
Warning  Comp. Temp. Threshold:     85 Celsius
Critical Comp. Temp. Threshold:     90 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     9.00W       -        -    0  0  0  0        0       0
 1 +     4.60W       -        -    1  1  1  1   100000  100000
 2 +     3.80W       -        -    2  2  2  2   500000  500000
 3 -   0.0450W       -        -    3  3  3  3     2000    2000
 4 -   0.0040W       -        -    4  4  4  4     6000    8000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        44 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    4.791.255 [2,45 TB]
Data Units Written:                 8.548.125 [4,37 TB]
Host Read Commands:                 25.822.209
Host Write Commands:                87.383.557
Controller Busy Time:               250
Power Cycles:                       160
Power On Hours:                     422
Unsafe Shutdowns:                   26
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Thermal Temp. 1 Transition Count:   21
Thermal Temp. 1 Total Time:         4

Error Information (NVMe Log 0x01, 8 of 8 entries)
No Errors Logged

Self-test Log (NVMe Log 0x06)
Self-test status: No self-test in progress
Num  Test_Description  Status                       Power_on_Hours  Failing_LBA  NSID Seg SCT Code
 0   Extended          Completed without error                 422            -     -   -   -    -

This seems to say your RAM is not working reliably.
See In Band Error Correction Code (IBECC) — Zephyr Project Documentation explaining about what IBECC is.

Try removing the memory, cleaning the connectors carefully and reinserting the memory.
Also run from grub menu memtest86+, you can install this with dnf.

1 Like

Thanks, I totally missed that, as I did not focused on it.
That may also explain, why Rancher Desktop (sandbox) and Gnome Boxes (Sandbox) works as it is in a different RAM area and by launching the VMs, it may hit it.
At least I can just move on finding the error, once the RAM is replaced, of course if it then still exists.

Memfree86 instantly found the both errors, so I did not checked for longer, as the amount off errors does not matter as 1 is already to much :slight_smile:

1 Like

If your system is exposed to high humidity and/or salty ocean air you should investigate “contact enhancer”, which is now sold by auto parts stores due the many signal connectors in modern vehicles.

I could find the root cause of the issue/freeze. I had the Kernel Parameter iommu=on vfio-pci.ids=8086:a7a0" still, but i915 does not have full support for it (nearly none) but it will come with Intel XE driver.

Anyhow this lead to the freeze problem when starting VMs. very very rarely it also happened with Ranched Desktop (appimage). But now it seems to be gone.

Also I replaced the RAM DIMMs without any success, the same error will stay in dmesg as well as in memtest86+.
For that I have no solution, but at least my system is stable again with VMs :smiley:

1 Like

RAM SPD Issues mentions that some laptop systems with soldered RAM don’t provide the SMBus interface used to access SPD raw data. https://memtest.org/readme tells you how to disable SPD tests using the grub2 editor.