NFS (RPC) troubleshooting

tfmeier · October 1, 2024, 2:05pm

No the permission issue was my fault in the mount command. /mnt/nfs_share as opposed to (correct) /data/nfs_share

Once I changed that the command times out.

showmount - e 192.168.2.101 gives the correct output but it takes about 2mins on the client while it’s instantaneous on the server

tfmeier · October 1, 2024, 2:08pm

Client

root@dell-kvmhost:~# ip -6 address show; ip -6 neighbor show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fd32:645b:6ff8:fcad:489a:3996:7e3c:6b30/64 scope global dynamic noprefixroute
       valid_lft 1639sec preferred_lft 1639sec
    inet6 fe80::dbf9:fe5d:41c9:af6d/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
10: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UNKNOWN qlen 1000
    inet6 fe80::fc54:ff:fe3c:e19c/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
11: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UNKNOWN qlen 1000
    inet6 fe80::fc54:ff:fe23:fff3/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
12: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UNKNOWN qlen 1000
    inet6 fe80::fc54:ff:fe8a:38c3/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
fe80::a677:33ff:fecf:b08f dev br0 lladdr a4:77:33:cf:b0:8f STALE
fe80::dbf9:fe5d:41c9:af6d dev vnet2 FAILED
fe80::fa0f:41ff:fecf:48ca dev vnet1 FAILED
fe80::935d:6612:16a6:fb6 dev vnet2 FAILED
fe80::974e:ce5a:e9e:548f dev vnet1 FAILED
fe80::a677:33ff:fecf:b08f dev vnet0 FAILED
fe80::5c73:b486:1c80:4c65 dev vnet2 FAILED
fe80::6403:48f4:f18f:c2ee dev vnet2 FAILED
fe80::1cb7:99fc:3d1b:20b4 dev vnet2 FAILED
fd32:645b:6ff8:fcad:fa0f:41ff:fecf:48ca dev br0 lladdr f8:0f:41:cf:48:ca STALE
fe80::e81e:c841:b604:3f0a dev vnet2 FAILED
fe80::dbf9:fe5d:41c9:af6d dev vnet1 FAILED
fe80::935d:6612:16a6:fb6 dev vnet1 FAILED
fe80::fa0f:41ff:fecf:48ca dev br0 lladdr f8:0f:41:cf:48:ca STALE
fe80::84c7:cac9:5b8d:26ea dev vnet2 FAILED
fe80::974e:ce5a:e9e:548f dev br0 lladdr 1c:f2:9a:4c:a7:b7 STALE
fe80::5c73:b486:1c80:4c65 dev vnet1 FAILED
fe80::6403:48f4:f18f:c2ee dev vnet1 FAILED
fe80::1cb7:99fc:3d1b:20b4 dev vnet1 FAILED
fe80::fa0f:41ff:fecf:48ca dev vnet0 FAILED
fe80::974e:ce5a:e9e:548f dev vnet0 FAILED
fe80::e81e:c841:b604:3f0a dev vnet1 FAILED
fe80::935d:6612:16a6:fb6 dev br0 lladdr 00:b5:6d:02:84:10 STALE
fe80::e99d:ba6e:386e:8b97 dev br0 FAILED
fe80::84c7:cac9:5b8d:26ea dev vnet1 FAILED
fe80::a677:33ff:fecf:b08f dev vnet2 FAILED
fe80::5c73:b486:1c80:4c65 dev br0 lladdr 84:14:4d:a2:f2:7e STALE
fe80::dbf9:fe5d:41c9:af6d dev vnet0 FAILED
fe80::6403:48f4:f18f:c2ee dev br0 lladdr 52:54:00:8a:38:c3 router STALE
fe80::1cb7:99fc:3d1b:20b4 dev br0 lladdr da:d2:71:3b:c1:49 STALE
fe80::935d:6612:16a6:fb6 dev vnet0 FAILED
fe80::e81e:c841:b604:3f0a dev br0 lladdr 38:86:f7:03:ec:33 router STALE
fe80::5c73:b486:1c80:4c65 dev vnet0 FAILED
fe80::6403:48f4:f18f:c2ee dev vnet0 FAILED
fe80::1cb7:99fc:3d1b:20b4 dev vnet0 FAILED
fe80::a677:33ff:fecf:b08f dev vnet1 FAILED
fe80::84c7:cac9:5b8d:26ea dev br0 lladdr ac:67:84:e4:c1:1a STALE
fe80::e81e:c841:b604:3f0a dev vnet0 FAILED
fe80::fa0f:41ff:fecf:48ca dev vnet2 FAILED
fe80::974e:ce5a:e9e:548f dev vnet2 FAILED
fe80::84c7:cac9:5b8d:26ea dev vnet0 FAILED

Server

root@acer-bkp-server:~# ip -6 address show; ip -6 neighbor show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fd32:645b:6ff8:fcad:fa0f:41ff:fecf:48ca/64 scope global dynamic noprefixroute
       valid_lft 1743sec preferred_lft 1743sec
    inet6 fe80::fa0f:41ff:fecf:48ca/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
fe80::e81e:c841:b604:3f0a dev enp3s0 lladdr 38:86:f7:03:ec:33 router STALE
fd32:645b:6ff8:fcad:489a:3996:7e3c:6b30 dev enp3s0 lladdr 18:03:73:da:8c:f4 STALE
fe80::dbf9:fe5d:41c9:af6d dev enp3s0 lladdr 18:03:73:da:8c:f4 STALE

darin755 · October 1, 2024, 2:09pm

what does ping show between the hosts? Any missing packets?

tfmeier · October 1, 2024, 2:14pm

Ping shows no packet loss in both directions

darin755 · October 1, 2024, 2:23pm

Any spikes in latency?

tfmeier · October 1, 2024, 2:25pm

Both sides are between 0.2 and 0.5 ms

There must be something else fundamentally happening as showmount takes about 2 mins from the client

darin755 · October 1, 2024, 2:27pm

Have you tried turning off the Firewall on both sides? Anything useful in dmesg?

tfmeier · October 1, 2024, 2:32pm

Yep firewalls on both ends are temporarily off. So is SELinux.

What do I look for in dmesg?

darin755 · October 1, 2024, 2:34pm

Loom for anything that looks out of the ordinary such as errors or warnings. Also there should be an entry from the kernel when you attempted to mount NFS.

vgaetera · October 1, 2024, 2:35pm

# Client
ping fe80::fa0f:41ff:fecf:48ca%br0

# Server
ping fe80::dbf9:fe5d:41c9:af6d%enp3s0

tfmeier · October 1, 2024, 2:37pm

Ok, I’ll study that tomorrow… Massive output and it’s late here

tfmeier · October 1, 2024, 2:38pm

Both pings successful, <1 ms

vgaetera · October 1, 2024, 2:39pm

# Client
sudo mount [fe80::fa0f:41ff:fecf:48ca%br0]:/data/nfs_share /mnt/nfs_share

darin755 · October 1, 2024, 2:44pm

I would avoid IPv6 as that is just adding complexity. Also it is likely easier to type IPv4

vgaetera · October 1, 2024, 2:50pm

IPv6 LLA is not affected by IPv4 related issues which we want to rule out.

tfmeier · October 1, 2024, 2:52pm

Same outcome…

root@dell-kvmhost:~# sudo mount [fe80::fa0f:41ff:fecf:48ca%br0]:/data/nfs_share /mnt/nfs_share
mount.nfs: Connection timed out for [fe80::fa0f:41ff:fecf:48ca%br0]:/data/nfs_share on /mnt/nfs_share

vgaetera · October 1, 2024, 2:55pm

Check on the server:

systemctl status nfs-server.service
getenforce
sudo nft list ruleset

tfmeier · October 1, 2024, 10:37pm

Here’s the response. nft list ruleset doesn’t return anything which means the firewall is off?

root@dell-kvmhost:~# systemctl status nfs-server.service
● nfs-server.service - NFS server and services
     Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (exited) since Sun 2024-09-29 16:10:01 AEST; 2 days ago
       Docs: man:rpc.nfsd(8)
             man:exportfs(8)
   Main PID: 447807 (code=exited, status=0/SUCCESS)
        CPU: 25ms

Sep 29 16:10:00 dell-kvmhost systemd[1]: Starting nfs-server.service - NFS server and services...
Sep 29 16:10:01 dell-kvmhost systemd[1]: Finished nfs-server.service - NFS server and services.
root@dell-kvmhost:~# getenforce
Permissive
root@dell-kvmhost:~# sudo nft list ruleset
root@dell-kvmhost:~#
root@dell-kvmhost:~#

vgaetera · October 2, 2024, 12:58am

Your NFS service is disabled and the status output is incomplete.
Make sure to properly enable it and check again:

sudo systemctl enable nfs-server.service
sudo systemctl restart nfs-server.service
systemctl status nfs-server.service
hostnamectl status; rpm -q nfs-utils

tfmeier · October 2, 2024, 1:06pm

Ok have executed this; mount still times out

root@acer-bkp-server:~# sudo systemctl enable nfs-server.service
root@acer-bkp-server:~# sudo systemctl restart nfs-server.service
root@acer-bkp-server:~# sudo systemctl status nfs-server.service
● nfs-server.service - NFS server and services
     Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
             /run/systemd/generator/nfs-server.service.d
             └─order-with-mounts.conf
     Active: active (exited) since Wed 2024-10-02 22:57:35 AEST; 4s ago
       Docs: man:rpc.nfsd(8)
             man:exportfs(8)
    Process: 14772 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
    Process: 14773 ExecStart=/usr/sbin/rpc.nfsd (code=exited, status=0/SUCCESS)
    Process: 14785 ExecStart=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi (code=e>
   Main PID: 14785 (code=exited, status=0/SUCCESS)
        CPU: 86ms

Oct 02 22:57:34 acer-bkp-server systemd[1]: Starting nfs-server.service - NFS server and services...
Oct 02 22:57:35 acer-bkp-server systemd[1]: Finished nfs-server.service - NFS server and services.

root@acer-bkp-server:~# hostnamectl status; rpm -q nfs-utils
     Static hostname: acer-bkp-server
           Icon name: computer-desktop
             Chassis: desktop 🖥️
          Machine ID: 31fca6a444984e768fa495cc5846dfe3
             Boot ID: 2cf493de239a42b8aa6e431fd02a9e10
    Operating System: Fedora Linux 40 (Server Edition)
         CPE OS Name: cpe:/o:fedoraproject:fedora:40
      OS Support End: Tue 2025-05-13
OS Support Remaining: 7month 1w 2d
              Kernel: Linux 6.10.11-200.fc40.x86_64
        Architecture: x86-64
     Hardware Vendor: Acer
      Hardware Model: Aspire XC-603
    Firmware Version: P11-B4
       Firmware Date: Tue 2015-09-15
        Firmware Age: 9y 2w 4d
nfs-utils-2.7.1-0.fc40.x86_64
root@acer-bkp-server:~#

Topic		Replies	Views
Unable to mount NFS share Ask Fedora network , mount , nfs , networking , f40 , f41	1	121	March 8, 2025
Network issues in fedora 35 Project Discussion workstation-wg	5	1104	November 8, 2021
NFS client not work and noerror! Ask Fedora f35	3	1204	March 4, 2022
NFS problems after fresh install of Fedora 30 - problem solved Ask Fedora f30	8	2184	May 14, 2019
NFSv2 on Fedora 38 Ask Fedora f38 , nfs , server	3	357	January 29, 2024

NFS (RPC) troubleshooting

Related topics