I run various Fedora CoreOS VMs as lightweight NFS servers in Kubernetes cluster enviromments, already for quite a few years. And that has worked fine so far. After deployment of the Fedora CoreOS VM I have to remove the nfs-utils-coreos package and then install (layer) the full nfs-utils package.
I have installed a new such VM right now and encountered the following problem. The installation went fine. But the NFS server does no longer allow its clients to mount a subdirectory of an exported directory. Clients can now only mount directories that are explicitly listed in /etc/exports.
Previously, I for example exported /var/nfsshare on the NFS server VM and on the client I could mount /var/nfsshare/subdir - of course the directory had to exist. But I did not have to list it in /etc/exports.
I compared an older VM (where this still works) to my new VM and found that while both (!) now run FCOS 43.20251214.3 (I had recently updated the older VM), they have different versions of the nfs-utils package. The older VM (where it works) has nfs-utils-2.8.4-0.fc43.x86_64 while the new VM which I installed today has nfs-utils-2.8.7-1.fc43.x86_64 e.g. a newer version.
I can imagine that this problem has its root upstream but perhaps someone can explain it and where I might need to head to? Is this change intentional? Is there a new export option to get back the previous behaviour?
That’s strange, but sure, the commit messages are not very long, I can copy and paste those two here.
f36bd900a899088ca1925de079bd58d6205a1f3c
Fix access checks when mounting subdirectories in NFSv3
If a NFSv3 client asks to mount a subdirectory of one of the exported
directories, then apply the RPC credential together with any root
or all squash rules that would apply to the client in question.
42f01e6a78fed98f12437ac8b28cfb12b6bad056
mountd: Separate lookup of the exported directory and the mount path
When the caller asks to mount a path that does not terminate with an
exported directory, we want to split up the lookups so that we can
look up the exported directory using the mountd privileged credential,
and the remaining subdirectory lookups using the RPC caller’s
credential.
Meanwhile at least I found as a workaround that I can layer an explicit (older) version, e.g. this:
$ rpm-ostree install nfs-utils-1:2.8.4
(After first uninstalling the 2.8.7 version.)
Now I have the same version like on the older VM(s) and the subdirectory mounts work as before. At least this confirms that the problem comes from the newer nfs-utils 2.8.7 version.
I’m not sure it is a bug, from the description of the new mountd option that was added to the rpc.mountd man page, it sounds more like a security patch?
+.B \-c " or " \-\-apply-root-cred
+When mountd is asked to allow a NFSv3 mount to a subdirectory of the
+exported directory, then it will check if the user asking to mount has
+lookup rights to the directories below that exported directory. When
+performing the check, mountd will apply any root squash or all squash
+rules that were specified for that client.
+
+Performing lookup checks as the user requires that the mountd daemon
+be run as root or that it be given CAP_SETUID and CAP_SETGID privileges
+so that it can change its own effective user and effective group settings.
+When troubleshooting, please also note that LSM frameworks such as SELinux
+can sometimes prevent the daemon from changing the effective user/groups
+despite the capability settings.
+
+In earlier versions of mountd, the same checks were performed using the
+mountd daemon's root privileges, meaning that it could authorise access
+to directories that are not normally accessible to the user requesting
+to mount them. This option enables that legacy behaviour.
+
+.BR Note:
+If there is a need to provide access to specific subdirectories that
+are not normally accessible to a client, it is always possible to add
+export entries that explicitly grant such access. That ability does
+not depend on this option being enabled.
It looks like you are supposed to pass --apply-root-cred to mountd when starting it if you want the previous behavior with the newer version of the software.
The subdirectories in question were accessibly to anyone.
The actual mountd error messages that I got were:
authenticated mount request from 1.2.3.4:980 for /var/nfsshare/image-registry (/var/nfsshare)
Apr 07 13:16:41 nfs.clustername.domain.de rpc.mountd[2267]: can’t change groups for user 0-0. Operation not permitted
Apr 07 13:16:41 nfs.clustername.domain.de rpc.mountd[2267]: can’t open exported dir /var/nfsshare/image-registry: Operation not permitted
So it indeed looks like mountd is not permitted to change its groups.
I could either try to add the –apply-root-cred option to the systemd unit of mountd or I could check how to give it the CAP_SET[GU]ID privileges.
FWIW, in either case, I think you would need to create a /etc/systemd/system/nfs-mountd.service.d/override.conf to change those settings.
Good luck.
P.S. Since this is a Type=forking service, don’t forget that you have to clear the ExecStart setting with an explicit ExecStart= on a separate line before you add your custom ExecStart=<whatever> line. See the systemd.service man page if you are unfamiliar with that.
I succeeded with an override.conf adding –apply-root-cred to ExecStart.
I thought that the fix with CAP_SETUID/CAP_SETGID would be cleaner but either I did it wrong or it did not help. I added these two capabilites with AmbientCapabilities but it did not have any effect. Thinking over it it would be strange, anyway, because the rpc.mountd runs as root and thus would already have these privileges, right?
