I updated all my machines yesterday to get the sudo fix and now the r8169 code on all my physical machines is not working with the following
0000:02:00.0 enp2s0: rtl_txcfg_enptycond == 0 (loop: 666, delay: 100).
All 3 systems got a “NETDEV WATCHDOG: enpXsY (r8169): transmit queue 0 time out” at about 8:05 today.
Collect the output:
uname -a; lspci -n -n -k
Try to boot a previous kernel version as a workaround.
I’ve tried all 3 kernels on my server with no difference. I can’t get a grub menu on my notebook. Latest installed kernel is 5.10.10-200.fc33.x86_64. I’ve tried 5.10.9-201 and 5m10.8-200 but they all do this.
uname -a
Linux eros.ipa.ab-data.us 5.10.9-201.fc33.x86_64 #1 SMP Wed Jan 20 16:56:23 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
lspci -n -n -k
00:00.0 RAM memory [0500]: NVIDIA Corporation MCP61 Memory Controller [10de:03ea] (rev a1)
Subsystem: Elitegroup Computer Systems Device [1019:2604]
00:01.0 ISA bridge [0601]: NVIDIA Corporation MCP61 LPC Bridge [10de:03e0] (rev a2)
Subsystem: Elitegroup Computer Systems Device [1019:2604]
00:01.1 SMBus [0c05]: NVIDIA Corporation MCP61 SMBus [10de:03eb] (rev a2)
Subsystem: Elitegroup Computer Systems Device [1019:2604]
Kernel driver in use: nForce2_smbus
Kernel modules: i2c_nforce2
00:01.2 RAM memory [0500]: NVIDIA Corporation MCP61 Memory Controller [10de:03f5] (rev a2)
Subsystem: Elitegroup Computer Systems Device [1019:2604]
00:01.3 Co-processor [0b40]: NVIDIA Corporation MCP61 SMU [10de:03f4] (rev a2)
00:02.0 USB controller [0c03]: NVIDIA Corporation MCP61 USB 1.1 Controller [10de:03f1] (rev a3)
Subsystem: Elitegroup Computer Systems Device [1019:2604]
Kernel driver in use: ohci-pci
00:02.1 USB controller [0c03]: NVIDIA Corporation MCP61 USB 2.0 Controller [10de:03f2] (rev a3)
Subsystem: Elitegroup Computer Systems Device [1019:2604]
Kernel driver in use: ehci-pci
00:04.0 PCI bridge [0604]: NVIDIA Corporation MCP61 PCI bridge [10de:03f3] (rev a1)
00:05.0 Audio device [0403]: NVIDIA Corporation MCP61 High Definition Audio [10de:03f0] (rev a2)
Subsystem: Elitegroup Computer Systems Device [1019:a88d]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
00:06.0 IDE interface [0101]: NVIDIA Corporation MCP61 IDE [10de:03ec] (rev a2)
Subsystem: Elitegroup Computer Systems Device [1019:2604]
Kernel driver in use: pata_amd
Kernel modules: pata_amd, pata_acpi, ata_generic
00:08.0 IDE interface [0101]: NVIDIA Corporation MCP61 SATA Controller [10de:03f6] (rev a2)
Subsystem: Elitegroup Computer Systems Device [1019:2604]
Kernel driver in use: sata_nv
Kernel modules: sata_nv, pata_acpi, ata_generic
00:09.0 PCI bridge [0604]: NVIDIA Corporation MCP61 PCI Express bridge [10de:03e8] (rev a2)
Kernel driver in use: pcieport
00:0b.0 PCI bridge [0604]: NVIDIA Corporation MCP61 PCI Express bridge [10de:03e9] (rev a2)
Kernel driver in use: pcieport
00:0c.0 PCI bridge [0604]: NVIDIA Corporation MCP61 PCI Express bridge [10de:03e9] (rev a2)
Kernel driver in use: pcieport
00:0d.0 VGA compatible controller [0300]: NVIDIA Corporation C61 [GeForce 6100 nForce 405] [10de:03d1] (rev a2)
Subsystem: Elitegroup Computer Systems Device [1019:2604]
Kernel driver in use: nouveau
Kernel modules: nouveau
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration [1022:1100]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] Address Map [1022:1101]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] DRAM Controller [1022:1102]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] K8 [Athlon64/Opteron] Miscellaneous Control [1022:1103]
Kernel driver in use: k8temp
Kernel modules: k8temp
01:05.0 PCI bridge [0604]: Pericom Semiconductor PI7C8140A PCI-to-PCI Bridge [12d8:8140]
01:06.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller [10ec:8169] (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8169/8110 Family PCI Gigabit Ethernet NIC [10ec:8169]
Kernel driver in use: r8169
Kernel modules: r8169
02:0c.0 Multimedia controller [0480]: Philips Semiconductors SAA7130 Video Broadcast Decoder [1131:7130] (rev 01)
Subsystem: Philips Semiconductors SAA7130-based TV tuner card [1131:0000]
Kernel driver in use: saa7134
Kernel modules: saa7134
02:0d.0 Multimedia controller [0480]: Philips Semiconductors SAA7130 Video Broadcast Decoder [1131:7130] (rev 01)
Subsystem: Philips Semiconductors SAA7130-based TV tuner card [1131:0000]
Kernel driver in use: saa7134
Kernel modules: saa7134
02:0e.0 Multimedia controller [0480]: Philips Semiconductors SAA7130 Video Broadcast Decoder [1131:7130] (rev 01)
Subsystem: Philips Semiconductors SAA7130-based TV tuner card [1131:0000]
Kernel driver in use: saa7134
Kernel modules: saa7134
02:0f.0 Multimedia controller [0480]: Philips Semiconductors SAA7130 Video Broadcast Decoder [1131:7130] (rev 01)
Subsystem: Philips Semiconductors SAA7130-based TV tuner card [1131:0000]
Kernel driver in use: saa7134
Kernel modules: saa7134
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: ------------[ cut here ]------------
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: NETDEV WATCHDOG: enp1s6 (r8169): transmit queue 0 timed out
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: WARNING: CPU: 0 PID: 56 at net/sched/sch_generic.c:442 dev_watchdog+0x24d/0x260
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: Modules linked in: nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set nf_tables nfnetlink rfkill ip6table_filter ip6_tables iptable_filter powernow_k8 snd_hda_codec_realtek snd_hda_codec_generic edac_mce_amd ledtrig_audio kvm_amd snd_hda_intel snd_intel_dspcfg ccp soundwire_intel soundwire_generic_allocation saa7134 kvm snd_soc_core snd_compress snd_pcm_dmaengine tveeprom videobuf2_dma_sg xfs soundwire_cadence videobuf2_memops irqbypass videobuf2_v4l2 snd_hda_codec videobuf2_common snd_hda_core ac97_bus videodev snd_hwdep pcspkr snd_seq snd_seq_device mc k8temp snd_pcm snd_timer snd soundcore i2c_nforce2 nfsd nfs_acl auth_rpcgss lockd grace sunrpc nfs_ssc zram ip_tables dm_crypt
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: nouveau video mxm_wmi wmi i2c_algo_bit drm_kms_helper cec ttm drm r8169 serio_raw ata_generic pata_acpi sata_nv pata_amd fuse
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: CPU: 0 PID: 56 Comm: kworker/u4:1 Not tainted 5.10.9-201.fc33.x86_64 #1
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: Hardware name: PCCHIPS A13G/A13G, BIOS 6.00 PG 07/25/2007
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: Workqueue: kcryptd/253:0 kcryptd_crypt [dm_crypt]
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: RIP: 0010:dev_watchdog+0x24d/0x260
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: Code: b9 c9 fd ff eb a9 4c 89 f7 c6 05 90 c5 30 01 01 e8 f8 a8 fa ff 44 89 e9 4c 89 f6 48 c7 c7 40 47 48 a1 48 89 c2 e8 d0 c8 15 00 <0f> 0b eb 8a 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 66 66 66
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: RSP: 0018:ffffad2dc0003ec0 EFLAGS: 00010282
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: RAX: 000000000000003b RBX: ffff9b8a353cfc00 RCX: 0000000000000000
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: RDX: ffff9b8a6e0269e0 RSI: ffff9b8a6e018a80 RDI: ffff9b8a6e018a80
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: RBP: ffff9b8a0309c3dc R08: 0000000000000000 R09: ffffad2dc0003cf0
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: R10: ffffad2dc0003ce8 R11: ffffffffa1b44748 R12: ffff9b8a0309c480
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: R13: 0000000000000000 R14: ffff9b8a0309c000 R15: ffff9b8a353cfc80
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: FS: 0000000000000000(0000) GS:ffff9b8a6e000000(0000) knlGS:0000000000000000
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: CR2: 00007fda3d368168 CR3: 0000000003886000 CR4: 00000000000006f0
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: Call Trace:
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: <IRQ>
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: ? pfifo_fast_enqueue+0x150/0x150
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: call_timer_fn+0x29/0xf0
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: __run_timers.part.0+0x1af/0x210
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: ? ktime_get+0x38/0xa0
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: ? lapic_next_event+0x1d/0x20
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: ? clockevents_program_event+0x8d/0xf0
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: run_timer_softirq+0x26/0x50
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: __do_softirq+0xca/0x288
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: asm_call_irq_on_stack+0x12/0x20
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: </IRQ>
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: do_softirq_own_stack+0x37/0x40
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: irq_exit_rcu+0xc2/0x100
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: sysvec_apic_timer_interrupt+0x36/0x80
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: RIP: 0010:crypto_aes_decrypt+0xcd4/0xd40
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: Code: 46 8b 04 85 00 ed 09 a1 44 33 04 9d 00 f9 09 a1 0f b6 dc c1 e8 10 44 33 87 9c 00 00 00 44 33 04 9d 00 f1 09 a1 89 d3 c1 ea 18 <c1> eb 10 0f b6 c0 0f b6 db 44 33 04 9d 00 f5 09 a1 41 0f b6 db 4c
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: RSP: 0018:ffffad2dc00ebc20 EFLAGS: 00000212
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: RAX: 000000000000c06b RBX: 00000000070de1fa RCX: 00000000000000b2
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: RDX: 0000000000000007 RSI: ffff9b8a26c6c440 RDI: ffff9b8a350c2d58
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: RBP: 00000000000000cf R08: 000000005a458e6b R09: 000000009fecc6a3
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: R10: 00000000d0e4f963 R11: 0000000085996cad R12: 0000000000000010
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: R13: 0000000000000010 R14: ffff9b8a350c2c00 R15: ffff9b8a26c6c450
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: crypto_ecb_crypt+0x64/0x90
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: ? crypto_aes_encrypt+0xcf0/0xcf0
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: crypto_skcipher_decrypt+0x3b/0x60
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: xts_decrypt+0x97/0xd0
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: crypto_skcipher_decrypt+0x3b/0x60
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: crypt_convert+0xcab/0x1040 [dm_crypt]
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: ? newidle_balance+0x1c3/0x3a0
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: ? __switch_to+0x331/0x450
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: kcryptd_crypt_read_convert+0xc7/0x140 [dm_crypt]
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: process_one_work+0x1b6/0x350
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: worker_thread+0x53/0x3e0
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: ? process_one_work+0x350/0x350
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: kthread+0x11b/0x140
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: ? __kthread_bind_mask+0x60/0x60
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: ret_from_fork+0x22/0x30
Jan 28 09:20:25 eros.ipa.ab-data.us kernel: ---[ end trace f8dfba7e7d4cbcfc ]---
Jan 28 09:20:26 eros.ipa.ab-data.us abrt-dump-journal-oops[1872]: abrt-dump-journal-oops: Found oopses: 1
Jan 28 09:20:26 eros.ipa.ab-data.us abrt-dump-journal-oops[1872]: abrt-dump-journal-oops: Creating problem directories
Jan 28 09:20:27 eros.ipa.ab-data.us ssh[2147]: Starting up
Jan 28 09:20:27 eros.ipa.ab-data.us sudo[2148]: Starting up
Jan 28 09:20:27 eros.ipa.ab-data.us nss[2145]: Starting up
Jan 28 09:20:27 eros.ipa.ab-data.us pam[2146]: Starting up
Jan 28 09:20:27 eros.ipa.ab-data.us pac[2149]: Starting up
Jan 28 09:20:27 eros.ipa.ab-data.us abrt-dump-journal-oops[1872]: Reported 1 kernel oopses to Abrt
Jan 28 09:20:29 eros.ipa.ab-data.us systemd[1]: NetworkManager-dispatcher.service: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit NetworkManager-dispatcher.service has successfully entered the 'dead' state.
Jan 28 09:20:29 eros.ipa.ab-data.us audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jan 28 09:20:31 eros.ipa.ab-data.us audit[2151]: AVC avc: denied { write } for pid=2151 comm="abrt-action-sav" name="rpmdb.sqlite-wal" dev="dm-0" ino=1481001 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=0
Jan 28 09:20:31 eros.ipa.ab-data.us audit[2151]: AVC avc: denied { setattr } for pid=2151 comm="abrt-action-sav" name="rpmdb.sqlite-wal" dev="dm-0" ino=1481001 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=0
Jan 28 09:20:31 eros.ipa.ab-data.us audit[2151]: AVC avc: denied { write } for pid=2151 comm="abrt-action-sav" name="rpmdb.sqlite-shm" dev="dm-0" ino=1481002 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=0
Jan 28 09:20:31 eros.ipa.ab-data.us audit[2151]: AVC avc: denied { setattr } for pid=2151 comm="abrt-action-sav" name="rpmdb.sqlite-shm" dev="dm-0" ino=1481002 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=0
Jan 28 09:20:34 eros.ipa.ab-data.us audit[2151]: AVC avc: denied { write } for pid=2151 comm="abrt-action-sav" name="rpmdb.sqlite-wal" dev="dm-0" ino=1481001 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=0
Jan 28 09:20:34 eros.ipa.ab-data.us audit[2151]: AVC avc: denied { setattr } for pid=2151 comm="abrt-action-sav" name="rpmdb.sqlite-wal" dev="dm-0" ino=1481001 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=0
Jan 28 09:20:34 eros.ipa.ab-data.us audit[2151]: AVC avc: denied { write } for pid=2151 comm="abrt-action-sav" name="rpmdb.sqlite-shm" dev="dm-0" ino=1481002 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=0
Jan 28 09:20:34 eros.ipa.ab-data.us audit[2151]: AVC avc: denied { setattr } for pid=2151 comm="abrt-action-sav" name="rpmdb.sqlite-shm" dev="dm-0" ino=1481002 scontext=system_u:system_r:abrt_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=0
Jan 28 09:20:34 eros.ipa.ab-data.us systemd[1]: Created slice system-dbus\x2d:1.5\x2dorg.fedoraproject.Setroubleshootd.slice.
░░ Subject: A start job for unit system-dbus\x2d:1.5\x2dorg.fedoraproject.Setroubleshootd.slice has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit system-dbus\x2d:1.5\x2dorg.fedoraproject.Setroubleshootd.slice has finished successfully.
░░
░░ The job identifier is 920.
Jan 28 09:20:34 eros.ipa.ab-data.us systemd[1]: Started dbus-:1.5-org.fedoraproject.Setroubleshootd@0.service.
░░ Subject: A start job for unit dbus-:1.5-org.fedoraproject.Setroubleshootd@0.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit dbus-:1.5-org.fedoraproject.Setroubleshootd@0.service has finished successfully.
░░
░░ The job identifier is 919.
Jan 28 09:20:34 eros.ipa.ab-data.us audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dbus-:1.5-org.fedoraproject.Setroubleshootd@0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
It’s kind of weird that suddenly all the kernels stopped working. You should at least still have the previously-running kernel installed and that worked, right?
Interface names in the logs don’t match (enp2s0 vs enp1s6). Log snippets are from different machines?
This wouldn’t be helpful.
Initial message sounds like the issue happened only once. Is it reproducible?
And you updated which components from which old version to which new one?
I have 3 different machines that all have Realtek NICs. The device names are different on each. The old kernels behave the same. The kernel shared object was updated in last night’s dnf upgrade. I’m guessing that’s where the issue lies.
All of the log data is from a single host, eros. The device is ens1s6 throughout the log data.
Virtually identical error info on all 3 systems.
But obviously the machines have different versions of Realtek NICs. The rtl_txcfg_emptycond message can’t occur on the old RTL8169 PCI chip version.
Again the question: Is the tx timeout reproducible? And what means “The old kernels behave the same.”? The issue still occurs even if you go back to the kernel version before the update?
My notebookhas an 8168. The other 2 are identical 8169’s that I purchased 4 of together. The other 2 are in my firewall which is not running Fedora.
Within 30 seconds of the network starting on all 3 machines and all 3 kernels on each, thereby a 3x3 matrix of 9.
Do you want the dnf log?
I’m unclear exactly what you mean here.
I mean a new r8169.ko was installed.
The NICs are TRENDnet TEG-PCITXR.
So the notebook and eros (Athlon64 single core) both error out 2 minutes after a Nagios NRDP update. Both error on PID 0 with just a few registers different accounted for by different hardware.
The third machine (charon) errors out on PID 1501which is between lockd and nsfd in the process list.
Ok, I must be really tired. The card in Charon is ID’d as RTL8168f/8111f. The notebook reports RTL8168gu/8111gu. Eros reports RTL8169sb/8110sb. I must have gotten 1 card with a system then bought 3 more.
Right, but that’s part of the kernel package and gets put into the matching initrd, so if you pick the older kernel you will get the older driver. Unlike other packages, we retain multiple versions of the kernel on the system in parallel.
So, it’s obviously something other than the NIC driver. I’ll have to try to figure out which package caused this. I was hoping that since it happened on 3 diffenent hardware configs that others would be experiencing it too and someone else would be able to track it down.
Yeah, it’s really weird since it definitely seems like a kernel thing. How long has it been since you did updates? What other packages were updated? Is there any chance there was some other network change?
Hmmm — one thing you could do is boot the F33 Live image and make sure that still works.
I had done an update Mon., 1/25. I tend to keep on top of them. I’ll try the live image tomorrow.
Transaction ID : 561
Begin time : Wed 27 Jan 2021 07:26:47 PM EST
Begin rpmdb : 4048:c68964c124fc00c14f3cb0c68fc9ccc48b8ba1ff
End time : Wed 27 Jan 2021 07:30:44 PM EST (237 seconds)
End rpmdb : 4048:d96677d1fac3638e8af67d1cf7f8f690daadb421
User : Eric Ashley <eric>
Return-Code : Success
Releasever : 33
Command Line : upgrade
Comment :
Packages Altered:
Install kernel-5.10.10-200.fc33.x86_64 @updates
Install kernel-core-5.10.10-200.fc33.x86_64 @updates
Install kernel-devel-5.10.10-200.fc33.x86_64 @updates
Install kernel-modules-5.10.10-200.fc33.x86_64 @updates
Install kernel-modules-extra-5.10.10-200.fc33.x86_64 @updates
Upgrade SDL2-2.0.14-1.fc33.x86_64 @updates
Upgraded SDL2-2.0.12-4.fc33.x86_64 @@System
Upgrade bind-32:9.11.27-1.fc33.x86_64 @updates
Upgraded bind-32:9.11.26-2.fc33.x86_64 @@System
Upgrade bind-chroot-32:9.11.27-1.fc33.x86_64 @updates
Upgraded bind-chroot-32:9.11.26-2.fc33.x86_64 @@System
Upgrade bind-dnssec-doc-32:9.11.27-1.fc33.noarch @updates
Upgraded bind-dnssec-doc-32:9.11.26-2.fc33.noarch @@System
Upgrade bind-dnssec-utils-32:9.11.27-1.fc33.x86_64 @updates
Upgraded bind-dnssec-utils-32:9.11.26-2.fc33.x86_64 @@System
Upgrade bind-dyndb-ldap-11.3-6.fc33.x86_64 @updates
Upgraded bind-dyndb-ldap-11.3-5.fc33.x86_64 @@System
Upgrade bind-libs-32:9.11.27-1.fc33.x86_64 @updates
Upgraded bind-libs-32:9.11.26-2.fc33.x86_64 @@System
Upgrade bind-libs-lite-32:9.11.27-1.fc33.x86_64 @updates
Upgraded bind-libs-lite-32:9.11.26-2.fc33.x86_64 @@System
Upgrade bind-license-32:9.11.27-1.fc33.noarch @updates
Upgraded bind-license-32:9.11.26-2.fc33.noarch @@System
Upgrade bind-pkcs11-32:9.11.27-1.fc33.x86_64 @updates
Upgraded bind-pkcs11-32:9.11.26-2.fc33.x86_64 @@System
Upgrade bind-pkcs11-libs-32:9.11.27-1.fc33.x86_64 @updates
Upgraded bind-pkcs11-libs-32:9.11.26-2.fc33.x86_64 @@System
Upgrade bind-pkcs11-utils-32:9.11.27-1.fc33.x86_64 @updates
Upgraded bind-pkcs11-utils-32:9.11.26-2.fc33.x86_64 @@System
Upgrade bind-utils-32:9.11.27-1.fc33.x86_64 @updates
Upgraded bind-utils-32:9.11.26-2.fc33.x86_64 @@System
Upgrade crun-0.17-1.fc33.x86_64 @updates
Upgraded crun-0.16-3.fc33.x86_64 @@System
Upgrade fuse-overlayfs-1.4.0-1.fc33.x86_64 @updates
Upgraded fuse-overlayfs-1.3.0-1.fc33.x86_64 @@System
Upgrade jackson-annotations-2.11.4-1.fc33.noarch @updates
Upgraded jackson-annotations-2.11.3-1.fc33.noarch @@System
Upgrade jackson-core-2.11.4-1.fc33.noarch @updates
Upgraded jackson-core-2.11.3-1.fc33.noarch @@System
Upgrade jackson-databind-2.11.4-1.fc33.noarch @updates
Upgraded jackson-databind-2.11.3-1.fc33.noarch @@System
Upgrade libaom-2.0.1-3.fc33.x86_64 @updates
Upgraded libaom-2.0.1-2.fc33.x86_64 @@System
Upgrade libpskc-2.6.6-1.fc33.x86_64 @updates
Upgraded libpskc-2.6.5-1.fc33.x86_64 @@System
Upgrade mtr-2:0.94-1.fc33.x86_64 @updates
Upgraded mtr-2:0.92-7.fc33.x86_64 @@System
Upgrade nftables-1:0.9.3-7.fc33.x86_64 @updates
Upgraded nftables-1:0.9.3-6.fc33.x86_64 @@System
Upgrade pango-1.48.1-1.fc33.i686 @updates
Upgraded pango-1.48.0-1.fc33.i686 @@System
Upgrade pango-1.48.1-1.fc33.x86_64 @updates
Upgraded pango-1.48.0-1.fc33.x86_64 @@System
Upgrade pango-devel-1.48.1-1.fc33.x86_64 @updates
Upgraded pango-devel-1.48.0-1.fc33.x86_64 @@System
Upgrade perl-ExtUtils-Manifest-1:1.73-1.fc33.noarch @updates
Upgraded perl-ExtUtils-Manifest-1:1.72-457.fc33.noarch @@System
Upgrade python3-bind-32:9.11.27-1.fc33.noarch @updates
Upgraded python3-bind-32:9.11.26-2.fc33.noarch @@System
Upgrade python3-boto3-1.16.58-1.fc33.noarch @updates
Upgraded python3-boto3-1.16.53-1.fc33.noarch @@System
Upgrade python3-botocore-1.19.58-1.fc33.noarch @updates
Upgraded python3-botocore-1.19.53-1.fc33.noarch @@System
Upgrade python3-nftables-1:0.9.3-7.fc33.x86_64 @updates
Upgraded python3-nftables-1:0.9.3-6.fc33.x86_64 @@System
Upgrade sqlite-3.34.1-1.fc33.x86_64 @updates
Upgraded sqlite-3.34.0-1.fc33.x86_64 @@System
Upgrade sqlite-devel-3.34.1-1.fc33.x86_64 @updates
Upgraded sqlite-devel-3.34.0-1.fc33.x86_64 @@System
Upgrade sqlite-libs-3.34.1-1.fc33.i686 @updates
Upgraded sqlite-libs-3.34.0-1.fc33.i686 @@System
Upgrade sqlite-libs-3.34.1-1.fc33.x86_64 @updates
Upgraded sqlite-libs-3.34.0-1.fc33.x86_64 @@System
Upgrade sudo-1.9.5p2-1.fc33.x86_64 @updates
Upgraded sudo-1.9.5p1-1.fc33.x86_64 @@System
Upgrade sudo-python-plugin-1.9.5p2-1.fc33.x86_64 @updates
Upgraded sudo-python-plugin-1.9.5p1-1.fc33.x86_64 @@System
Upgrade urw-base35-bookman-fonts-20200910-2.fc33.noarch @updates
Upgraded urw-base35-bookman-fonts-20200910-1.fc33.noarch @@System
Upgrade urw-base35-c059-fonts-20200910-2.fc33.noarch @updates
Upgraded urw-base35-c059-fonts-20200910-1.fc33.noarch @@System
Upgrade urw-base35-d050000l-fonts-20200910-2.fc33.noarch @updates
Upgraded urw-base35-d050000l-fonts-20200910-1.fc33.noarch @@System
Upgrade urw-base35-fonts-20200910-2.fc33.noarch @updates
Upgraded urw-base35-fonts-20200910-1.fc33.noarch @@System
Upgrade urw-base35-fonts-common-20200910-2.fc33.noarch @updates
Upgraded urw-base35-fonts-common-20200910-1.fc33.noarch @@System
Upgrade urw-base35-fonts-legacy-20200910-2.fc33.noarch @updates
Upgraded urw-base35-fonts-legacy-20200910-1.fc33.noarch @@System
Upgrade urw-base35-gothic-fonts-20200910-2.fc33.noarch @updates
Upgraded urw-base35-gothic-fonts-20200910-1.fc33.noarch @@System
Upgrade urw-base35-nimbus-mono-ps-fonts-20200910-2.fc33.noarch @updates
Upgraded urw-base35-nimbus-mono-ps-fonts-20200910-1.fc33.noarch @@System
Upgrade urw-base35-nimbus-roman-fonts-20200910-2.fc33.noarch @updates
Upgraded urw-base35-nimbus-roman-fonts-20200910-1.fc33.noarch @@System
Upgrade urw-base35-nimbus-sans-fonts-20200910-2.fc33.noarch @updates
Upgraded urw-base35-nimbus-sans-fonts-20200910-1.fc33.noarch @@System
Upgrade urw-base35-p052-fonts-20200910-2.fc33.noarch @updates
Upgraded urw-base35-p052-fonts-20200910-1.fc33.noarch @@System
Upgrade urw-base35-standard-symbols-ps-fonts-20200910-2.fc33.noarch @updates
Upgraded urw-base35-standard-symbols-ps-fonts-20200910-1.fc33.noarch @@System
Upgrade urw-base35-z003-fonts-20200910-2.fc33.noarch @updates
Upgraded urw-base35-z003-fonts-20200910-1.fc33.noarch @@System
Upgrade virtualbox-guest-additions-6.1.18-1.fc33.x86_64 @updates
Upgraded virtualbox-guest-additions-6.1.16-1.fc33.x86_64 @@System
Upgrade xen-libs-4.14.1-2.fc33.x86_64 @updates
Upgraded xen-libs-4.14.1-1.fc33.x86_64 @@System
Upgrade xen-licenses-4.14.1-2.fc33.x86_64 @updates
Upgraded xen-licenses-4.14.1-1.fc33.x86_64 @@System
Removed kernel-5.9.16-200.fc33.x86_64 @@System
Removed kernel-core-5.9.16-200.fc33.x86_64 @@System
Removed kernel-devel-5.9.16-200.fc33.x86_64 @@System
Removed kernel-modules-5.9.16-200.fc33.x86_64 @@System
Removed kernel-modules-extra-5.9.16-200.fc33.x86_64 @@System
Scriptlet output:
1 /bin/stat: cannot statx '/var/named/chroot/dev/null': No such file or directory
2 /bin/stat: cannot statx '/var/named/chroot/dev/random': No such file or directory
3 /bin/stat: cannot statx '/var/named/chroot/dev/zero': No such file or directory
4 Warning: The unit file, source configuration file or drop-ins of vboxservice.service changed on disk. Run 'systemctl daemon-reload' to reload units.
5 Warning: The unit file, source configuration file or drop-ins of nftables.service changed on disk. Run 'systemctl daemon-reload' to reload units.
So the issue seems to originate in systemd-resolved. My FreeIPA instance is on a VM running under libvirt and since libvirt doesn’t like having an alias address, that means I had to put it into it’s own class C subnet. The primary DNS, is therefore on 172.26.23.x while the dns that resolves the world is on 172.26.22.x. Periodically I have to run resolvectl to reset the current dns server.
All my systems are fully functional with systemd-resolved disabled. I don’t have time to chase this bug at the moment. I’ll have another look once I’ve cleared some projects off my list.
On my notebook and charon (the VM host), systemd-resolved would segfault, then something cascades in the abrt handling. Eros runs bind, so maybe something in the bind update caused it’s systemd-resolved failure (no core dumps logged). Though I have bind installed on my notebook, it’s disabled.
I’ve not marked this solved since I don’t consider this to be the solution. It is a work-around solution, though. So let me know if I should mark it solved.