I run a k8s cluster in my homelab using FCOS and smooth sailing for years. I noticed recently though that all my deployments were failed as they could not mount NFS (ones not needing that storage continued to run fine.) I ruled out hardware issues, NFS server issues, etc. before landing on it being due to the recently upgraded image.
The particular error encountered was akin to “Exit 32” (if I recall correctly) and “busy or already mounted or sharecache fail”
I verified that NFS itself was fine as I have other VMs using it, I mounted one on my desktop, all good. I rolled back one node to the previous image (44.20260419.3.1 (2026-05-10T20:44:27Z)) and all the containers started properly with NFS mounted up.
Does anyone know what might be the case here? The NFS server is Debian 13 Trixie with mainly defaults and no issues exporting to anything else.
Edit- for anyone experiencing the same I would advise rolling back rpm-ostree rollback –reboot (yes, this will reboot immediately upon reverting the image, remove the --reboot flag if you do not want that). Then pin the working image with ostree admin pin 0
Not sure what the issue could be exactly. That particular update path from 44.20260419.3.1 to 44.20260510.3.1 didn’t update the nfs packages. Maybe a kernel 7.0 issue?
Some more logs might help track down the issue. Also you can try with various in development builds if you want to see if the issue is fixed on a test system.
Let me see if I can take one and throw a dev build on it. And yes- that’s what struck me as odd as no libs or nfs-utils or anything had changed.
I don’t think it should matter but I also just run k0s on it (not full blooded k8s anymore as I just didn’t have the desire to keep mucking with it.) Though I wouldn’t think k0s would affect it. I’ll try to get to that soon if possible (busy couple weeks coming up sadly and was hoping someone would spot something I didn’t quickly )
Edit- I guess I should ask how I rebase onto this. I’m unsure of the process honestly.
Output: mount.nfs: /var/lib/k0s/kubelet/pods/ec66aae1-24b8-4ec5-9da8-3068d4c40c06/volumes/kubernetes.io~nfs/example-pv-testing is busy or already mounted or sharecache fail
kernel is
7.0.10-201.fc44.x86_64 vs 7.0.8-200.fc44.x86_64 on the stable version.
Somewhat of a solution- I had configured a mount on my NFS server in the ignition file. I debated doing it this way then just using “hostPath” for pods to access storage (debated it, never implemented.)
Seems that does something though what I’m not entirely sure. I unmounted it, killed the pod, and the new one now runs fine. To note- this NFS mount does not configure the same path as what I’m accessing in the pod itself (different subdir entirely) but it does do something odd.
The question then is what has changed between the earlier 44.x version and the new ones as to why this no longer works.
Not sure but we do have the tooling to help narrow down and figure it out.
If you really want to know I would iterate on our testing-devel stream to find which two versions of testing-devel the problem was introduced in (the regression was introduced in the set of packages that changed between the two). Then I would point an AI harness at the RPMs that changed to see if it could try to analyze the differences in software versions to see if it could find an upstream change that could have introduced the problem.
44.20260510.1.3 - STILL BROKEN
44.20260501.10.1 - WORKS
I threw this into AI and it barfed this out
--
Main Change: nfs-utils Security Update
Version 44.20260510.1.3 introduced an update to nfs-utils that tightened NFS export security by changing how subdirectories within exported trees are handled:
What Changed:
New behavior: The updated nfs-utils now rejects attempts to mount subdirectories of an exported tree unless those subdirectories are explicitly listed in /etc/exports or exported with wildcard options
Previous behavior (in 44.20260501.10.1): Clients could mount subdirectories without them being explicitly exported
Security rationale: This is a security-focused change to prevent implicit subdirectory access
Impact:
This change caused a regression where container workloads relying on mounting NFS subdirectory exports failed. Users reported:
Container deployments failing because they couldn’t mount NFS subdirectories
Workloads that didn’t require NFS storage continued working fine
Rolling back to earlier versions resolved the issue
Resolution in 44.20260510.1.3:
The later build (44.20260510.1.3) includes documentation and potentially fixes for this issue in its testing-devel changelog. Users had two main workarounds:
Update export configuration: Explicitly list subdirectories in /etc/exports instead of relying on parent directory exports
The nfs-utils-coreos package (used in Fedora CoreOS) enforces this restriction more strictly than the standard nfs-utils package, which allows users to layer the full package if they need the older semantics.
So given this I went in to the broken build and tried
Going to stand up a new test cluster on a different VLAN to test out NFS wildcard exporting and see if this works fine. I’ll update.
Edit- for the moment exporting it as is but with fsid=0 is working on that vlan. I need to read up more on that option in particular. It seems though that that is configured ONCE per nfs server and I have multiple paths (and mountpoints) that it goes across.
It would be more useful if you could tell us exactly which version of testing-devel broke things. i.e. start with 44.20260510.20.1 (probably is broken) and work backwards from there til you find one that works and report that here (i.e. last working and first bad).
I think the formatting broke it in the last post but I think these are the two closest images I could use off quay to isolate where it broke
44.20260510.1.3 - BROKEN
44.20260501.10.1 - WORKS
I honestly don’t know why it formats like that when I paste but those were the ones. I’ll doublecheck when I get a sec but I did work backwards to isolate it.
)