I’ve recently started running into an issue with the systemd-oomd killer. I use tmux, and I usually have 3-4 sessions for different contexts (work, mail, project 1, project 2). I use gnome-terminal to run these.
Now, sometimes, let’s say in project 1, if I do something cpu/memory intensive, systemd-oomd will do its thing. But, systemd-oomd does not kill individual processes. I’m still trying to figure out exactly what it kills—scopes, slices? Anyway, what happens is that it kills the tmux server—the gnome-terminal application still lives, but the tmux-server is killed, which kills all my tmux sessions which kills all my terminal apps and multiple vim sessions. This is far from ideal.
So, I’ve been trying to figure out what to do here.
One solution, of course, is to disable systemd-oomd. I mean, sure, that’s an option, but I’d like to see if there are better ways of doing it before going down this “large hammer to a problem” type solution.
So, I thought: maybe I can limit each tmux-session to a different slice/scope so that only it is killed, leaving other sessions running without trouble? For this, one needs to run each tmux-session on a different tmux-server, because otherwise if the tmux-server is killed, all your sessions die too.
Thoughts? How would one go about doing this?
- is it possible to run tmux-server as a different unit and configure it such that systemd-oomd doesn’t, or avoids trying to kill it? I found this gist for example. Would this do? Any consequences to be aware of? I’d like to run tmux as a user service, of course—no need to run it as a system service here.
- if this cannot be done, I guess the next best thing is to run each tmux-session in a different tmux-server, thus isolating each so that if one is killed, the other lives on. Would this work? Is it enough to just use something like
tmux -L $RANDOM start-server \; new-session -s mysession
to start a new session in a new tmux-server? - Finally, how is the above different from running it using
systemd-run
like this:systemd-run --user --scope tmux -L $RANDOM start-server \; new-session -s mysession
? I see fromsystemd-cgls
that this method does not put my new tmux-server under the gnome-terminal slice. So I guess this is better isolation?
│ │ │ ├─app-org.gnome.Terminal.slice (#9910)
│ │ │ │ ├─vte-spawn-9d0fdcef-dd4e-4028-b34b-8f12db4c9158.scope (#56351)
│ │ │ │ │ ├─ 428416 bash
│ │ │ │ │ ├─ 428644 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 5551 start-server ; new-session -s default
│ │ │ │ │ ├─ 428679 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 5551 start-server ; new-session -s default
│ │ │ │ │ └─ 428681 /bin/bash
│ │ │ │ ├─vte-spawn-a57d7f0e-205b-4f06-bbbf-1e636de8347c.scope (#55727)
│ │ │ │ │ ├─ 426118 bash
│ │ │ │ │ ├─ 426260 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 5456 start-server ; new-session -s neuroml-paper
│ │ │ │ │ ├─ 426295 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 5456 start-server ; new-session -s neuroml-paper
│ │ │ │ │ └─ 426297 /bin/bash
│ │ │ │ ├─vte-spawn-47e29bf9-be49-423a-a8b1-0b8bad29d0fd.scope (#60400)
│ │ │ │ │ └─ 918388 bash
│ │ │ │ ├─vte-spawn-3d9f946b-b60f-4df2-80cf-551f193d002f.scope (#56447)
│ │ │ │ │ ├─ 428821 bash
│ │ │ │ │ ├─ 429007 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 22449 start-server ; new-session -s comms
│ │ │ │ │ ├─ 429042 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 22449 start-server ; new-session -s comms
│ │ │ │ │ ├─ 429044 /bin/bash
│ │ │ │ │ ├─ 663975 weechat
│ │ │ │ │ ├─ 664115 /bin/bash
│ │ │ │ │ └─ 664288 neomutt -F /home/asinha/Sync/99_private/work.neomuttrc
│ │ │ │ ├─gnome-terminal-server.service (#10390)
│ │ │ │ │ → user.xdg.inactive-since: -1
│ │ │ │ │ └─ 3062 /usr/libexec/gnome-terminal-server
│ │ │ │ └─vte-spawn-667770a6-b9e8-41f8-b380-f32d90f15ac3.scope (#55679)
│ │ │ │ ├─ 425627 bash
│ │ │ │ ├─ 425703 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 13610 start-server ; new-session -s neuroml
│ │ │ │ ├─ 425740 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 13610 start-server ; new-session -s neuroml
│ │ │ │ ├─ 425742 /bin/bash
│ │ │ │ └─ 801530 /bin/bash
..
│ │ │ ├─run-r333db369400d4b30b1e34d4a0e07bc14.scope (#61036)
│ │ │ │ ├─ 986371 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 7328 start-server ; new-session -s systemd-test
│ │ │ │ ├─ 986410 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 7328 start-server ; new-session -s systemd-test
│ │ │ │ ├─ 986413 /bin/bash
│ │ │ │ ├─ 988092 systemd-cgls
│ │ │ │ └─ 988093 less
The top bit shows me using option 2, all the tmux-servers are under the gnome-terminal slice. Does that mean, they’ll all still get killed if one of them ooms? At the bottom, we see me using option 3, where it’s not under the gnome-terminal slice. It has it’s own scope—which is fine, right?
Sorry if some of this doesn’t make sense. I’m still trying to wrap my head around slices and scopes and all this systemd jargon.
Edit
I tried method 3. I created two new tmux servers using systemd-run
, and then in one of them I went tail /dev/zero
. After a while, this one got killed, killing the tmux server but the other tmux server was unharmed. So this certainly works.
The one downside of having different tmux servers is that detaching/attaching becomes a lot more complex because one has to know what tmux servers are running. The workaround is to use human readable names for the sockets, I guess. One would go tmux -L $SOCKETNAME attach -T $SESSIONNAME
. In my case, the session name and socket name can be the same, since the idea is to use individual servers for each session.