How would one create new tmux servers, each isolated in a separate slice (so that if systemd-oomd kills one, the other tmux servers keep living)

I’ve recently started running into an issue with the systemd-oomd killer. I use tmux, and I usually have 3-4 sessions for different contexts (work, mail, project 1, project 2). I use gnome-terminal to run these.

Now, sometimes, let’s say in project 1, if I do something cpu/memory intensive, systemd-oomd will do its thing. But, systemd-oomd does not kill individual processes. I’m still trying to figure out exactly what it kills—scopes, slices? Anyway, what happens is that it kills the tmux server—the gnome-terminal application still lives, but the tmux-server is killed, which kills all my tmux sessions which kills all my terminal apps and multiple vim sessions. This is far from ideal.

So, I’ve been trying to figure out what to do here.

One solution, of course, is to disable systemd-oomd. I mean, sure, that’s an option, but I’d like to see if there are better ways of doing it before going down this “large hammer to a problem” type solution.

So, I thought: maybe I can limit each tmux-session to a different slice/scope so that only it is killed, leaving other sessions running without trouble? For this, one needs to run each tmux-session on a different tmux-server, because otherwise if the tmux-server is killed, all your sessions die too.

Thoughts? How would one go about doing this?

  • is it possible to run tmux-server as a different unit and configure it such that systemd-oomd doesn’t, or avoids trying to kill it? I found this gist for example. Would this do? Any consequences to be aware of? I’d like to run tmux as a user service, of course—no need to run it as a system service here.
  • if this cannot be done, I guess the next best thing is to run each tmux-session in a different tmux-server, thus isolating each so that if one is killed, the other lives on. Would this work? Is it enough to just use something like tmux -L $RANDOM start-server \; new-session -s mysession to start a new session in a new tmux-server?
  • Finally, how is the above different from running it using systemd-run like this: systemd-run --user --scope tmux -L $RANDOM start-server \; new-session -s mysession? I see from systemd-cgls that this method does not put my new tmux-server under the gnome-terminal slice. So I guess this is better isolation?
│   │ │ ├─app-org.gnome.Terminal.slice (#9910)
│   │ │ │ ├─vte-spawn-9d0fdcef-dd4e-4028-b34b-8f12db4c9158.scope (#56351)
│   │ │ │ │ ├─ 428416 bash
│   │ │ │ │ ├─ 428644 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 5551 start-server ; new-session -s default
│   │ │ │ │ ├─ 428679 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 5551 start-server ; new-session -s default
│   │ │ │ │ └─ 428681 /bin/bash
│   │ │ │ ├─vte-spawn-a57d7f0e-205b-4f06-bbbf-1e636de8347c.scope (#55727)
│   │ │ │ │ ├─ 426118 bash
│   │ │ │ │ ├─ 426260 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 5456 start-server ; new-session -s neuroml-paper
│   │ │ │ │ ├─ 426295 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 5456 start-server ; new-session -s neuroml-paper
│   │ │ │ │ └─ 426297 /bin/bash
│   │ │ │ ├─vte-spawn-47e29bf9-be49-423a-a8b1-0b8bad29d0fd.scope (#60400)
│   │ │ │ │ └─ 918388 bash
│   │ │ │ ├─vte-spawn-3d9f946b-b60f-4df2-80cf-551f193d002f.scope (#56447)
│   │ │ │ │ ├─ 428821 bash
│   │ │ │ │ ├─ 429007 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 22449 start-server ; new-session -s comms
│   │ │ │ │ ├─ 429042 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 22449 start-server ; new-session -s comms
│   │ │ │ │ ├─ 429044 /bin/bash
│   │ │ │ │ ├─ 663975 weechat
│   │ │ │ │ ├─ 664115 /bin/bash
│   │ │ │ │ └─ 664288 neomutt -F /home/asinha/Sync/99_private/work.neomuttrc
│   │ │ │ ├─gnome-terminal-server.service (#10390)
│   │ │ │ │ → user.xdg.inactive-since: -1
│   │ │ │ │ └─ 3062 /usr/libexec/gnome-terminal-server
│   │ │ │ └─vte-spawn-667770a6-b9e8-41f8-b380-f32d90f15ac3.scope (#55679)
│   │ │ │   ├─ 425627 bash
│   │ │ │   ├─ 425703 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 13610 start-server ; new-session -s neuroml
│   │ │ │   ├─ 425740 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 13610 start-server ; new-session -s neuroml
│   │ │ │   ├─ 425742 /bin/bash
│   │ │ │   └─ 801530 /bin/bash

..
│   │ │ ├─run-r333db369400d4b30b1e34d4a0e07bc14.scope (#61036)
│   │ │ │ ├─ 986371 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 7328 start-server ; new-session -s systemd-test
│   │ │ │ ├─ 986410 tmux -u -2 -f /usr/share/byobu/profiles/tmuxrc -L 7328 start-server ; new-session -s systemd-test
│   │ │ │ ├─ 986413 /bin/bash
│   │ │ │ ├─ 988092 systemd-cgls
│   │ │ │ └─ 988093 less

The top bit shows me using option 2, all the tmux-servers are under the gnome-terminal slice. Does that mean, they’ll all still get killed if one of them ooms? At the bottom, we see me using option 3, where it’s not under the gnome-terminal slice. It has it’s own scope—which is fine, right?

Sorry if some of this doesn’t make sense. I’m still trying to wrap my head around slices and scopes and all this systemd jargon.


Edit

I tried method 3. I created two new tmux servers using systemd-run, and then in one of them I went tail /dev/zero. After a while, this one got killed, killing the tmux server but the other tmux server was unharmed. So this certainly works.

The one downside of having different tmux servers is that detaching/attaching becomes a lot more complex because one has to know what tmux servers are running. The workaround is to use human readable names for the sockets, I guess. One would go tmux -L $SOCKETNAME attach -T $SESSIONNAME. In my case, the session name and socket name can be the same, since the idea is to use individual servers for each session.

Would systemd-nspawn provide the isolation desired?

1 Like

Edit:

The simplest way to do this is to modify the default-command variable tmux provides. So in the tmux config file (~/.tmux.config?):

# in tmux.conf
set-option -g default-command 'systemd-run --user --scope bash'

Older solution (round about, not needed)

I’m not sure tbh.

For the time being, I dug in a little more and realised I could run each tmux “window” in isolation. To create a new window in tmux one goes :new-window <command>. So, one can use:

:new-window  systemd-run --user --scope bash

And one can override the default bindings so that new windows are created like this:

# in .tmux.conf

bind-key c new-window systemd-run --user --scope bash

So, there’s only the one tmux server, but each window is isolated. Testing out the oom killer by running tail /dev/zero in one tmux window caused it to be killed, but did not affect other tmux windows or sessions, or the server.

So, i think this is perhaps the “cleanest” way to go about it.

3 Likes