Running out of file handles in pipewire?

Ask Fedora
, , , ,
1

It looks like this is not a common issue. It could be my hardware (Dell XPS i7-12700H laptop) or configuration. For the last few updates - kernel versions 6.3.6, 6.3.8 and 6.3.11 - I’ve been experiencing bad behavior. It starts with audio cutting out in the browser (YouTube pages e.g.) or repeating very quickly if I mouse over the KDE panel which is floating and auto-hidden. Video also stops or cuts out, or the audio cuts out. The same behavior occurs in both Firefox, my main browser, and Chrome. The audio issues are not permanent - I can sometimes get the audio restored by opening the audio widget and clicking on various radio buttons (normally selected as Analog Audio Output), or quitting and restarting the browser.

There are a lot of repeated journal entries indicating that the system has run out of file handles:

Jul 09 14:40:45 fedora pipewire[10163]: invalid memory type 12
Jul 09 14:40:45 fedora pipewire[10163]: mod.protocol-native: connection 0x562c12c860f0: can't DUP fd:895 Too many open files
Jul 09 14:40:45 fedora pipewire[10163]: mod.protocol-native: connection 0x562c12d33b10: can't DUP fd:1015 Too many open files
Jul 09 14:40:45 fedora pipewire[10163]: mod.protocol-native: connection 0x562c12c860f0: can't DUP fd:1015 Too many open files
Jul 09 14:40:45 fedora pipewire[10163]: pw.core: 0x562c1241bbb0: error -2 for resource 7: set_activation: No such file or directory
Jul 09 14:40:45 fedora pipewire[10163]: mod.client-node: 0x562c125a91f0: error seq:321 -2 (set_activation: No such file or directory)
...
Jul 09 15:04:45 fedora pipewire[10163]: pw.mem: 0x562c12428790: Failed to create memfd: Too many open files
Jul 09 15:04:46 fedora pipewire[10163]: pw.node: node 0x562c124cb9d0: write failed Bad file descriptor

That’s just a sample; all of those lines are repeated multiple times over multiple blocks of time, and most implicate or directly come from pipewire.

I’m also getting a popup telling me that the system is dangerously low on inotify handles: the system reserves around half a million.

Screenshot_20230709_144524

Output of lsof | awk '{print $1}' | uniq -c | sort -rn | head shows:

 297541 firefox
 155600 plasmashe
 100842 chrome
  50946 Isolated
  39326 chrome
  27251 akonadi_m
  24804 kwin_wayl
  20601 Isolated
  17091 Isolated
  14850 chrome

I currently have 7 Chrome windows and 8 Firefox windows open (with 77 tabs open in Firefox and 53 in Chrome: I often have many hundreds or worse open, so I doubt it’s an issue of number of tabs).

Is anyone else experiencing anything similar, or know what the issue might be? Any help in debugging or troubleshooting beyond journalctl and lsof would be welcome.

Again, recent builds, audio cutting out, videos skipping, and the system running low on handles.

Thanks.

1 Like
2

Inotify handles are not the same as file handles.
Suggest you increase the number of inotify handles as the pop up is warning you need to do that.

3

The issue is why I’m running out of inotify instance handles? I’m running a lighter than normal load since my most recent upgrade and reboot. This only started happening with the last 3 kernel upgrades (which may or may not be the issue; presumably not since the kernel has a significant level of testing). Upping the limit will just paper over the problem, not identify nor solve it.

1 Like
4

Yes I also just recently sometimes get this popup.

I use simpletabgroups with Firefox, so most tabs are disabled. Also Firefox itself disables Tabs, do you want to check that?

Maybe Chrome handles that worse, I dont use it. If you want different logins, bookmarks, whatever, use Firefox Profiles (about:profiles) and create a new ~/.local/share/applications entry launching firefox -p PROFILENAME

Really interesting, this must be a bug as half a million sounds crazy

5

Most of my Firefox tabs are in a quiescent or suspended state: I believe that’s the default now for recent versions of both Firefox and Chrome.

I got a new version of pipewire last night and so far I’m not seeing the pipewire issue crop up in the output from journalctl. However, it typically takes around 24 hours of use before I start to notice problems.

My inotify instance count, according to kde-inotify-survey is already hitting 60% so it looks like I’m not out of the woods yet.

If other people are seeing the problem then it’s unlikely to be something specific to my hardware or Fedora installation. I did boot into Windows 11 to see if there was anything untoward happening there, but other than using way more CPU resources than I think it ought to, and certainly more than Fedora, everything seemed fine.

6

Well. after 18+ hours of uptime since the last upgrade and reboot, including a new pipewire binary, the following is back in journalctl output; there are hundreds of these lines. It won’t be long before I’m dropping audio and having videos randomly stop when I mouse over the KDE panel (floating, auto-hidden, Wayland) if past experience is anything to go by:

Jul 11 18:35:50 fedora pipewire[2228]: pw.mem: 0x565471a3e790: Failed to create memfd: Too many open files
Jul 11 18:35:50 fedora pipewire[2228]: mod.protocol-native: connection 0x565471ece460: can't DUP fd:1010 Too many open files
Jul 11 18:35:50 fedora pipewire[2228]: mod.protocol-native: connection 0x565471ece460: can't DUP fd:1009 Too many open files
Jul 11 18:35:50 fedora pipewire[2228]: pw.mem: 0x565471a3e790: Failed to create memfd: Too many open files
Jul 11 18:35:50 fedora pipewire[2228]: mod.protocol-native: connection 0x565471ece460: can't DUP fd:1012 Too many open files
Jul 11 18:35:50 fedora pipewire[2228]: mod.protocol-native: connection 0x565471ece460: can't DUP fd:1011 Too many open files
Jul 11 18:35:50 fedora pipewire[2228]: pw.mem: 0x565471a3e790: Failed to create memfd: Too many open files
...

Firefox currently has over 500K open handles with 9 windows and around 100 tabs total.

I’m beginning to suspect Firefox, although it could also be some system service.

Any clues on how I can diagnose this further? There’s bound to be something in Firefox or in Linux that I can use to figure out what’s going on, or at least finger a culprit.

Danke.

7

Try restarting the browser and related services when the issue happens:

systemctl --user restart pipewire.service pipewire-pulse.service
8

lsof on the main Firefox process shows e.g.:

firefox 3758 user  DEL       REG                0,1             1194 /memfd:mozilla-ipc
firefox 3758 user  DEL       REG                0,1           114010 /memfd:gdk-wayland
firefox 3758 user  DEL       REG                0,1           111967 /memfd:gdk-wayland
firefox 3758 user  DEL       REG                0,1           111029 /memfd:gdk-wayland
firefox 3758 user  DEL       REG                0,1           112835 /memfd:gdk-wayland
firefox 3758 user  DEL       REG                0,1           111041 /memfd:gdk-wayland
firefox 3758 user  DEL       REG                0,1           112181 /memfd:mozilla-ipc
firefox 3758 user  DEL       REG                0,1            98637 /memfd:mozilla-ipc
firefox 3758 user  DEL       REG                0,1            98636 /memfd:mozilla-ipc
firefox 3758 user  DEL       REG                0,1             2441 /memfd:pulseaudio
firefox 3758 user  DEL       REG                0,1           107771 /memfd:mozilla-ipc

There are currently 1146 entries that match memfd, the vast majority of which are of this form:

firefox 3758 user  DEL       REG                0,1            98633 /memfd:mozilla-ipc

To me that looks like memory mapped files, or file sections that have been deleted or marked deleted but not closed? Could this be my issue? mmap’d files are not being closed?

9

Maybe not: another machine (Desktop, F38, KDE, Wayland, Firefox) is showing a similar number of DEL /memfd:mozilla-ipc entries in lsof -p and that isn’t showing a load of “out of file handles” messages in journalctl. I’ve not looked at this before but it does seem odd to me that a load of ostensibly deleted file handles are showing up in the output of lsof.

Currently this problem looks to be unique to this laptop (I have 2 other desktops with Fedora that I can check against)

10

I’ll try it, thanks. However, even though pipewire seems to be implicated in the journalctl output, I’m now veering towards it being something else that’s keeping file handles open, and pipewire is just a victim. I believe if I kill off Firefox then the situation will improve, but only temporarily based on recent experience with this problem.

11 
sudo mkdir -p /etc/systemd/user.conf.d
sudo tee /etc/systemd/user.conf.d/00-custom.conf << EOF > /dev/null
[Manager]
DefaultLimitNOFILE=$((2**19))
EOF
systemctl --user daemon-reload
systemctl --user show -p LimitNOFILE,LimitNOFILESoft \
    pipewire.service pipewire-pulse.service

systemd-system.conf

1 Like
12

Okay so this occurs after exceptionally long use. Thats why I havent recognized it yet

13

Restarting pipewire has made the problem go away - for now. Before restarting I looked in journalctl and in the last 1000 seconds there were over 30,000 entries from pipewire and little else:

Jul 12 22:33:33 fedora pipewire[10158]: mod.protocol-native: server 0x55d78d7aa030: failed to accept: Too many open files

When it happens again I’ll try the systemd work-around you provided although it looks like it’ll just defer the issue a bit further. I’m no closer to knowing what the actual problem is. It takes around 16 hours to repro with default limits and my current config/open apps etc.