Unknown reason, random freeze

Ask Fedora
,
1

My machine (fedora server 44 installed) recently encounter random freezes while running…

I have recently updated from f42 to f44:

$ uname -ar
Linux fedodesk 7.1.8-200.fc44.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Aug 10 03:35:23 UTC 2026 x86_64 GNU/Linux

About recent changes to the system, its CPU was upgraded from R9-7950x to R9-9950x.

Also, I have a new btrfs setted up, and the problem occured after I start doing read/write to the btrfs partition (most of the time, but the issue is random so I am not really sure)…

The system become frozen (no tty, keyboard control does not work, ping timeout), I am not sure if it is really related to btrfs, cuz I made a lot of changes to that system recently (including hardware)

The last lines of journal from last boot says:

Aug 14 17:44:31 fedodesk systemd-journald[1043]: File /var/log/journal/4ae51c3969464444be3e6369efd4877e/user-1000.journal corrupted or uncleanly shut down, renaming and replacing.
Aug 14 17:44:34 fedodesk kernel: r8169 0000:4d:00.0 enp77s0: Link is Up - 2.5Gbps/Full - flow control off
Aug 14 17:44:35 fedodesk kernel: evm: overlay not supported
Aug 14 17:44:35 fedodesk kernel: bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
Aug 14 17:46:47 fedodesk kernel: ZFS: Unloaded module v2.4.3-1
Aug 14 17:46:47 fedodesk kernel: EXT4-fs (md127): re-mounted 38e70b35-5598-4a3f-bb9b-e7670c402941 ro.
Aug 14 17:46:47 fedodesk kernel: BTRFS info (device nvme2n1p1): scrub: started on devid 1
Aug 14 17:46:47 fedodesk kernel: BTRFS info (device nvme2n1p1): scrub: started on devid 2
Aug 14 17:47:02 fedodesk kernel: BTRFS info (device nvme2n1p1): scrub: finished on devid 2 with status: 0
Aug 14 17:47:03 fedodesk kernel: BTRFS info (device nvme2n1p1): scrub: finished on devid 1 with status: 0

My btrfs setup:

$ sudo btrfs filesystem show
Label: 'nvme-raid0'  uuid: b8828960-263c-48f1-a20a-2f875ef1f935
	Total devices 2 FS bytes used 130.15GiB
	devid    1 size 953.87GiB used 67.02GiB path /dev/nvme2n1p1
	devid    2 size 953.87GiB used 67.02GiB path /dev/nvme0n1p1

$ sudo btrfs device stats /mnt/nvme
[/dev/nvme2n1p1].write_io_errs    0
[/dev/nvme2n1p1].read_io_errs     0
[/dev/nvme2n1p1].flush_io_errs    0
[/dev/nvme2n1p1].corruption_errs  0
[/dev/nvme2n1p1].generation_errs  0
[/dev/nvme0n1p1].write_io_errs    0
[/dev/nvme0n1p1].read_io_errs     0
[/dev/nvme0n1p1].flush_io_errs    0
[/dev/nvme0n1p1].corruption_errs  0
[/dev/nvme0n1p1].generation_errs  0

My initial guesses is my DRAM or Curve optimizer settings (I am on 9950x CPU, 6200c32 total 32G RAM), so I just reset everything in BIOS to default, but that does not fix anything…

I have tried on kernel 6.19 and this seemed to be resolved… I will run more checks and test to confirm that, but I’ll provide the above for context…

Regards,
Miyuki

2

Updating on this issue…

So I decided to run the same set of stress tests on both of the kernels.
In four separated terminals…

sudo stress-ng --temp-path /mnt/raid1 --hdd 4 --hdd-bytes 1M --dentry 4 --metrics-brief --iostat 5
sudo stress-ng --temp-path /mnt/nvme --hdd 4 --hdd-bytes 1M --dentry 4 --metrics-brief --iostat 5
sudo stress-ng --cpu 32 --cpu-method matrixprod
sudo memtest 12G 20

…and monitor journalctl -xe -p 4 --dmesg -f at the same time.
That stress CPU up to 100% and IO together with memory, so anything non-stable can be caught…

In Kernel 7.1 something went wrong pretty quick:

Aug 14 21:58:44 fedodesk kernel: nvme nvme0: missing or invalid SUBNQN field.
Aug 14 21:58:46 fedodesk kernel: ryzen_smu: loading out-of-tree module taints kernel.
Aug 14 21:58:46 fedodesk kernel: md: async del_gendisk mode will be removed in future, please upgrade to mdadm-4.5+
Aug 14 21:58:46 fedodesk kernel: kauditd_printk_skb: 23 callbacks suppressed
Aug 14 21:58:46 fedodesk kernel: md127: echo current LBS to md/logical_block_size to prevent data loss issues from LBS changes.
                                         Note: After setting, array will not be assembled in old kernels (<= 6.18)
Aug 14 21:58:46 fedodesk kernel: Bluetooth: hci0: No dsm support to set reset delay
Aug 14 21:58:46 fedodesk kernel: snd_hda_intel 0000:5c:00.6: no codecs found!
Aug 14 21:58:46 fedodesk kernel:
Aug 14 21:58:46 fedodesk kernel: NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  610.57.04  Release Build  (dvs-builder@U22-I3-AF05-32-1)  Wed Jul 29 02:45:17 UTC 2026
Aug 14 21:58:51 fedodesk kernel: xhci_hcd 0000:57:00.0: xHC error in resume, USBSTS 0x401, Reinit
Aug 14 22:04:50 fedodesk kernel: ata3.00: exception Emask 0x10 SAct 0x0 SErr 0x48d0002 action 0xe frozen
Aug 14 22:04:50 fedodesk kernel: ata3.00: irq_stat 0x00000040, connection status changed
Aug 14 22:04:50 fedodesk kernel: ata3: SError: { RecovComm PHYRdyChg CommWake 10B8B LinkSeq DevExch }
Aug 14 22:04:50 fedodesk kernel: ata3.00: failed command: FLUSH CACHE EXT
Aug 14 22:04:50 fedodesk kernel: ata3.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 27
                                          res 40/00:d0:c0:0e:4b/00:00:74:00:00/40 Emask 0x10 (ATA bus error)
Aug 14 22:04:50 fedodesk kernel: ata3.00: status: { DRDY }
Aug 14 22:04:51 fedodesk kernel: ata3.00: retrying FLUSH 0xea Emask 0x10
Aug 14 22:04:54 fedodesk kernel: ata3.00: exception Emask 0x10 SAct 0x0 SErr 0x4050002 action 0xe frozen
Aug 14 22:04:54 fedodesk kernel: ata3.00: irq_stat 0x08000040, interface fatal error, connection status changed
Aug 14 22:04:54 fedodesk kernel: ata3: SError: { RecovComm PHYRdyChg CommWake DevExch }
Aug 14 22:04:54 fedodesk kernel: ata3.00: failed command: FLUSH CACHE EXT
Aug 14 22:04:54 fedodesk kernel: ata3.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 25
                                          res 40/00:c0:68:9d:4b/00:00:74:00:00/40 Emask 0x10 (ATA bus error)
Aug 14 22:04:54 fedodesk kernel: ata3.00: status: { DRDY }
Aug 14 22:04:55 fedodesk kernel: ata3.00: retrying FLUSH 0xea Emask 0x10
Aug 14 22:05:24 fedodesk kernel: ata3.00: exception Emask 0x10 SAct 0x0 SErr 0x4050002 action 0xe frozen
Aug 14 22:05:24 fedodesk kernel: ata3.00: irq_stat 0x08000040, interface fatal error, connection status changed
Aug 14 22:05:24 fedodesk kernel: ata3: SError: { RecovComm PHYRdyChg CommWake DevExch }
Aug 14 22:05:24 fedodesk kernel: ata3.00: failed command: FLUSH CACHE EXT
Aug 14 22:05:24 fedodesk kernel: ata3.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 23
                                          res 40/00:b0:f0:2c:51/00:00:74:00:00/40 Emask 0x10 (ATA bus error)
Aug 14 22:05:24 fedodesk kernel: ata3.00: status: { DRDY }
Aug 14 22:05:25 fedodesk kernel: ata3.00: retrying FLUSH 0xea Emask 0x10
Aug 14 22:05:52 fedodesk kernel: ata3: limiting SATA link speed to 3.0 Gbps
Aug 14 22:05:52 fedodesk kernel: ata3.00: exception Emask 0x50 SAct 0x0 SErr 0x48f0802 action 0xe frozen
Aug 14 22:05:52 fedodesk kernel: ata3.00: irq_stat 0x00000040, connection status changed
Aug 14 22:05:52 fedodesk kernel: ata3: SError: { RecovComm HostInt PHYRdyChg PHYInt CommWake 10B8B LinkSeq DevExch }
Aug 14 22:05:52 fedodesk kernel: ata3.00: failed command: FLUSH CACHE EXT
Aug 14 22:05:52 fedodesk kernel: ata3.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 18
                                          res 40/00:88:40:0c:56/00:00:74:00:00/40 Emask 0x50 (ATA bus error)
Aug 14 22:05:52 fedodesk kernel: ata3.00: status: { DRDY }
Aug 14 22:05:53 fedodesk kernel: ata3.00: retrying FLUSH 0xea Emask 0x50
Aug 14 22:07:26 fedodesk kernel: ata1.00: exception Emask 0x40 SAct 0x0 SErr 0x8f0802 action 0x6 frozen
Aug 14 22:07:26 fedodesk kernel: ata1: SError: { RecovComm HostInt PHYRdyChg PHYInt CommWake 10B8B LinkSeq }
Aug 14 22:07:26 fedodesk kernel: ata1.00: failed command: FLUSH CACHE EXT
Aug 14 22:07:26 fedodesk kernel: ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 23
                                          res 40/00:01:07:00:00/00:00:00:00:00/00 Emask 0x44 (timeout)
Aug 14 22:07:26 fedodesk kernel: ata1.00: status: { DRDY }
Aug 14 22:07:26 fedodesk kernel: ata1.00: retrying FLUSH 0xea Emask 0x44
Aug 14 22:10:17 fedodesk kernel: ata1.00: exception Emask 0x10 SAct 0x0 SErr 0x4050002 action 0xe frozen
Aug 14 22:10:17 fedodesk kernel: ata1.00: irq_stat 0x08000040, interface fatal error, connection status changed
Aug 14 22:10:17 fedodesk kernel: ata1: SError: { RecovComm PHYRdyChg CommWake DevExch }
Aug 14 22:10:17 fedodesk kernel: ata1.00: failed command: FLUSH CACHE EXT
Aug 14 22:10:17 fedodesk kernel: ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 30
                                          res 40/00:e8:20:e8:5d/00:00:74:00:00/40 Emask 0x10 (ATA bus error)
Aug 14 22:10:17 fedodesk kernel: ata1.00: status: { DRDY }
Aug 14 22:10:18 fedodesk kernel: ata1.00: retrying FLUSH 0xea Emask 0x10
Aug 14 22:11:08 fedodesk kernel: ata1.00: exception Emask 0x10 SAct 0x20040 SErr 0x4050002 action 0xe frozen
Aug 14 22:11:08 fedodesk kernel: ata1.00: irq_stat 0x08000040, interface fatal error, connection status changed
Aug 14 22:11:08 fedodesk kernel: ata1: SError: { RecovComm PHYRdyChg CommWake DevExch }
Aug 14 22:11:08 fedodesk kernel: ata1.00: failed command: WRITE FPDMA QUEUED
Aug 14 22:11:08 fedodesk kernel: ata1.00: cmd 61/00:30:00:86:18/02:00:09:00:00/40 tag 6 ncq dma 262144 out
                                          res 40/00:01:e0:4f:c2/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
Aug 14 22:11:08 fedodesk kernel: ata1.00: status: { DRDY }
Aug 14 22:11:08 fedodesk kernel: ata1.00: failed command: WRITE FPDMA QUEUED
Aug 14 22:11:08 fedodesk kernel: ata1.00: cmd 61/00:88:00:84:18/02:00:09:00:00/40 tag 17 ncq dma 262144 out
                                          res 40/00:01:e0:4f:c2/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
Aug 14 22:11:08 fedodesk kernel: ata1.00: status: { DRDY }
Aug 14 22:12:44 fedodesk kernel: ata1: limiting SATA link speed to 3.0 Gbps
Aug 14 22:12:44 fedodesk kernel: ata1.00: exception Emask 0x10 SAct 0x0 SErr 0x4050002 action 0xe frozen
Aug 14 22:12:44 fedodesk kernel: ata1.00: irq_stat 0x08000040, interface fatal error, connection status changed
Aug 14 22:12:44 fedodesk kernel: ata1: SError: { RecovComm PHYRdyChg CommWake DevExch }
Aug 14 22:12:44 fedodesk kernel: ata1.00: failed command: FLUSH CACHE EXT
Aug 14 22:12:44 fedodesk kernel: ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 23
                                          res 40/00:b0:b0:3d:58/00:00:74:00:00/40 Emask 0x10 (ATA bus error)
Aug 14 22:12:44 fedodesk kernel: ata1.00: status: { DRDY }
Aug 14 22:12:45 fedodesk kernel: ata1.00: retrying FLUSH 0xea Emask 0x10

The system does not freeze this time. I thought that might be a bad cable, so I get a replacement for that, however, same thing happened when re-testing.

However, under kernel 6.19, after stressing for 30+ mins, nothing went wrong (comparing to 6mins in 7.1)

$ sudo journalctl -xe -p 4 --dmesg -f
[sudo] password for yue:
Aug 14 22:58:44 fedodesk kernel: nvme nvme2: missing or invalid SUBNQN field.
Aug 14 22:58:44 fedodesk kernel: nvidia: loading out-of-tree module taints kernel.
Aug 14 22:58:44 fedodesk kernel:
Aug 14 22:58:44 fedodesk kernel: NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  610.57.04  Release Build  (dvs-builder@U22-I3-AF05-32-1)  Wed Jul 29 02:45:17 UTC 2026
Aug 14 22:58:47 fedodesk kernel: md: async del_gendisk mode will be removed in future, please upgrade to mdadm-4.5+
Aug 14 22:58:48 fedodesk kernel: Bluetooth: hci0: No dsm support to set reset delay
Aug 14 22:58:48 fedodesk kernel: snd_hda_intel 0000:5c:00.6: no codecs found!
Aug 14 22:58:48 fedodesk kernel: md127: echo current LBS to md/logical_block_size to prevent data loss issues from LBS changes.
                                         Note: After setting, array will not be assembled in old kernels (<= 6.18)
Aug 14 22:58:52 fedodesk kernel: xhci_hcd 0000:57:00.0: xHC error in resume, USBSTS 0x401, Reinit
Aug 14 22:58:53 fedodesk kernel: nvme nvme0: using unchecked data buffer

^C

$ time
shell  0.13s user 0.04s system 0% cpu 35:08.49 total
children  0.27s user 0.12s system 0% cpu 35:08.49 total

$ uptime
 23:37:19 up 38 min,  6 users,  load average: 51.64, 52.79, 47.54

I am really out of idea now, any help will be appreciated…

3

Might be related?

4

More update on this…
I added libata.force=1:3.0,3:3.0 to 7.1 grub params, it somehow stayed stable. And then I did some research and found that SATA_1 to SATA_4 on my motherboard uses the AMD controller instead of a third-party one, I moved the two disks to those ports and the disk problem seemed to be disappeared but the system still freezes from time to time…

dmesg -w / tty1/ journalctl -xe -b -1 All logged nothing sadly… Especially when I have no terminal access (even tty via physical screen) after the system freeze…