NetworkManager service crashes system

Ask Fedora
, , , , , , ,
1

I am using F39 and I think after the kernel update to 6.6.8, the NetworkManager service fails to load and forces a reboot of the system.

Here are some applicable logs:
uname -r
6.6.6-200.fc39.x86_64

/var/log/messages

Jan  1 23:48:39 my-system systemd[1]: Starting NetworkManager.service - Network Manager...
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9480] NetworkManager (version 1.44.2-1.fc39) is starting... (boot:273acdec-5eff-4cc8-8be0-ca5b54e2e691)
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9481] Read config: /etc/NetworkManager/NetworkManager.conf (lib: 20-connectivity-fedora.conf) (run: 15-carrier-timeout.conf)
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9522] manager[0x555800729ab0]: monitoring kernel firmware directory '/lib/firmware'.
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9538] hostname: hostname: using hostnamed
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9538] hostname: static hostname changed from (none) to "my-system"
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9540] dns-mgr: init: dns=systemd-resolved rc-manager=unmanaged (auto), plugin=systemd-resolved
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9549] rfkill1: found Wi-Fi radio killswitch (at /sys/devices/pci0000:00/0000:00:02.1/0000:04:00.0/0000:05:06.0/0000:09:00.0/ieee80211/phy0/rfkill1) (driver mt7921e)
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9550] manager[0x555800729ab0]: rfkill: Wi-Fi hardware radio set enabled
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9550] manager[0x555800729ab0]: rfkill: WWAN hardware radio set enabled
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9564] Loaded device plugin: NMAtmManager (/usr/lib64/NetworkManager/1.44.2-1.fc39/libnm-device-plugin-adsl.so)
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9585] Loaded device plugin: NMBluezManager (/usr/lib64/NetworkManager/1.44.2-1.fc39/libnm-device-plugin-bluetooth.so)
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9594] Loaded device plugin: NMWifiFactory (/usr/lib64/NetworkManager/1.44.2-1.fc39/libnm-device-plugin-wifi.so)
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9596] Loaded device plugin: NMWwanFactory (/usr/lib64/NetworkManager/1.44.2-1.fc39/libnm-device-plugin-wwan.so)
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9607] Loaded device plugin: NMTeamFactory (/usr/lib64/NetworkManager/1.44.2-1.fc39/libnm-device-plugin-team.so)
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9608] manager: rfkill: Wi-Fi enabled by radio killswitch; enabled by state file
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9608] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9608] manager: Networking is enabled by state file
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9610] settings: Loaded settings plugin: keyfile (internal)
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9621] dhcp: init: Using DHCP client 'internal'
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9622] manager: (lo): new Loopback device (/org/freedesktop/NetworkManager/Devices/1)
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9626] device (lo): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9628] device (lo): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9630] device (lo): Activation: starting connection 'lo' (6ae93b80-eecd-46ef-ad76-9dacfe49fe34)
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9635] manager: (enp10s0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
Jan  1 23:48:39 my-system NetworkManager[1881]: <info>  [1704174519.9636] device (enp10s0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Jan  1 23:48:39 my-system systemd[1]: Starting NetworkManager-dispatcher.service - Network Manager Script Dispatcher Service...
Jan  1 23:48:39 my-system systemd[1]: Started NetworkManager-dispatcher.service - Network Manager Script Dispatcher Service.
Jan  1 23:48:39 my-system audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan  1 23:48:39 my-system audisp-syslog[1449]: type=SERVICE_START msg=audit(1704174519.981:121): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' UID="root" AUID="unset"
Jan  1 23:48:40 my-system NetworkManager[1881]: <info>  [1704174520.1663] manager: (enp21s0u1u3): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Jan  1 23:48:40 my-system NetworkManager[1881]: <info>  [1704174520.1664] device (enp21s0u1u3): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Jan  1 23:48:40 my-system NetworkManager[1881]: <info>  [1704174520.1810] manager: (enp7s0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/4)
Jan  1 23:48:40 my-system NetworkManager[1881]: <info>  [1704174520.1810] device (enp7s0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Jan  1 23:48:50 my-system systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
Jan  1 23:48:50 my-system audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan  1 23:48:50 my-system audisp-syslog[1449]: type=SERVICE_STOP msg=audit(1704174530.461:125): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' UID="root" AUID="unset"
Jan  1 23:49:43 my-system kernel: watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [NetworkManager:1881]
Jan  1 23:49:43 my-system kernel: CPU: 15 PID: 1881 Comm: NetworkManager Not tainted 6.6.8-200.fc39.x86_64 #1

Output of sealert -l 1a30bb13-5115-4d01-bd86-43b034ad7f47

SELinux is preventing hostname from search access on the directory /var/lib/snapd/snap/core/16091/etc/NetworkManager/dispatcher.d/hook-network-manager.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that hostname should be allowed search access on the hook-network-manager directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'hostname' --raw | audit2allow -M my-hostname
# semodule -X 300 -i my-hostname.pp


Additional Information:
Source Context                system_u:system_r:cockpit_ws_t:s0
Target Context                system_u:object_r:sysctl_net_t:s0
Target Objects                /var/lib/snapd/snap/core/16091/etc/NetworkManager/
                              dispatcher.d/hook-network-manager [ dir ]
Source                        hostname
Source Path                   hostname
Port                          <Unknown>
Host                          my-system
Source RPM Packages           
Target RPM Packages           
SELinux Policy RPM            selinux-policy-targeted-39.3-1.fc39.noarch
Local Policy RPM              selinux-policy-targeted-39.3-1.fc39.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     my-system
Platform                      Linux my-system 6.6.6-200.fc39.x86_64 #1 SMP
                              PREEMPT_DYNAMIC Mon Dec 11 17:29:08 UTC 2023
                              x86_64
Alert Count                   22
First Seen                    2023-12-02 22:33:20 CST
Last Seen                     2024-01-02 00:58:21 CST
Local ID                      1a30bb13-5115-4d01-bd86-43b034ad7f47

Raw Audit Messages
type=AVC msg=audit(1704178701.532:103): avc:  denied  { search } for  pid=1958 comm="hostname" name="net" dev="proc" ino=236 scontext=system_u:system_r:cockpit_ws_t:s0 tcontext=system_u:object_r:sysctl_net_t:s0 tclass=dir permissive=0


Hash: hostname,cockpit_ws_t,sysctl_net_t,dir,search

Output of lspci

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14d8
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 14d9
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14db
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14db
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14db
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14dd
00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14dd
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 71)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e7
01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev 10)
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch (rev 10)
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX] (rev cc)
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio
03:00.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 USB
03:00.3 Serial bus controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 7444
04:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f4 (rev 01)
05:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
05:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
05:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
05:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
05:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
05:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
05:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
05:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
06:00.0 Non-Volatile memory controller: SK hynix Gold P31/BC711/PC711 NVMe Solid State Drive
07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
09:00.0 Network controller: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter
0a:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0b:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f4 (rev 01)
0c:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
0c:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
0c:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
0c:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
0c:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
0c:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
0c:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
0c:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43f5 (rev 01)
13:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43f7 (rev 01)
14:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43f6 (rev 01)
15:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43f7 (rev 01)
16:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43f6 (rev 01)
17:00.0 Non-Volatile memory controller: SK hynix Platinum P41/PC801 NVMe Solid State Drive
18:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raphael (rev c2)
18:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller
18:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP
18:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b6
18:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b7
18:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller
19:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b8
2

Would there be any additional information be helpful to resolving this?

3

With the selinux blocking things the very first thing I would suggest would be to do a full selinux relabel of the system.

This can be easily done by first doing sudo touch /.autorelabel then a reboot.
The relabel will be done during the startup so it may take quite some time to finish the reboot.
If SELinux is blocking the loading of networkmanager that should fix it for you.

The lspci output seems to indicate that it is not properly configuring the amd devices also, so once you have properly booted and have an internet connection then you probably should run sudo dnf upgrade --refresh to ensure that all packages are at the latest versions.

4

Turns out my motherboard, X670E Steel Legend, has a Realtek RTL8111 1G NIC which was causing the issue.
I disabled it in the BIOS and am using the second NIC, a Dragon RTL8125BG which seems to be working fine.
During my troubleshooting, I went as far as reimaging, but the installer would crash, so I knew it wasn’t a problem with my install.