I run a k8s cluster in my homelab using FCOS and smooth sailing for years. I noticed recently though that all my deployments were failed as they could not mount NFS (ones not needing that storage continued to run fine.) I ruled out hardware issues, NFS server issues, etc. before landing on it being due to the recently upgraded image.
The particular error encountered was akin to “Exit 32” (if I recall correctly) and “busy or already mounted or sharecache fail”
I verified that NFS itself was fine as I have other VMs using it, I mounted one on my desktop, all good. I rolled back one node to the previous image (44.20260419.3.1 (2026-05-10T20:44:27Z)) and all the containers started properly with NFS mounted up.
Does anyone know what might be the case here? The NFS server is Debian 13 Trixie with mainly defaults and no issues exporting to anything else.
Edit- for anyone experiencing the same I would advise rolling back rpm-ostree rollback –reboot (yes, this will reboot immediately upon reverting the image, remove the --reboot flag if you do not want that). Then pin the working image with ostree admin pin 0
Not sure what the issue could be exactly. That particular update path from 44.20260419.3.1 to 44.20260510.3.1 didn’t update the nfs packages. Maybe a kernel 7.0 issue?
Some more logs might help track down the issue. Also you can try with various in development builds if you want to see if the issue is fixed on a test system.
Let me see if I can take one and throw a dev build on it. And yes- that’s what struck me as odd as no libs or nfs-utils or anything had changed.
I don’t think it should matter but I also just run k0s on it (not full blooded k8s anymore as I just didn’t have the desire to keep mucking with it.) Though I wouldn’t think k0s would affect it. I’ll try to get to that soon if possible (busy couple weeks coming up sadly and was hoping someone would spot something I didn’t quickly )
Edit- I guess I should ask how I rebase onto this. I’m unsure of the process honestly.
Output: mount.nfs: /var/lib/k0s/kubelet/pods/ec66aae1-24b8-4ec5-9da8-3068d4c40c06/volumes/kubernetes.io~nfs/example-pv-testing is busy or already mounted or sharecache fail
kernel is
7.0.10-201.fc44.x86_64 vs 7.0.8-200.fc44.x86_64 on the stable version.
Somewhat of a solution- I had configured a mount on my NFS server in the ignition file. I debated doing it this way then just using “hostPath” for pods to access storage (debated it, never implemented.)
Seems that does something though what I’m not entirely sure. I unmounted it, killed the pod, and the new one now runs fine. To note- this NFS mount does not configure the same path as what I’m accessing in the pod itself (different subdir entirely) but it does do something odd.
The question then is what has changed between the earlier 44.x version and the new ones as to why this no longer works.
Not sure but we do have the tooling to help narrow down and figure it out.
If you really want to know I would iterate on our testing-devel stream to find which two versions of testing-devel the problem was introduced in (the regression was introduced in the set of packages that changed between the two). Then I would point an AI harness at the RPMs that changed to see if it could try to analyze the differences in software versions to see if it could find an upstream change that could have introduced the problem.
)