Full system freeze on F37 after ssh connection timeout to a nspawn container launched via this ssh connection

Ask Fedora
, ,
1

Problem

F37 Workstation seemed to freeze entirely in an unrecoverable fashion after sshd server timed out a connection that launched a non-booted nspawn container with Fedora 37 in the container (running off of a file stored in /dev/shm) via that connection and had a python script with lots of CPU and memory utilization running in that SSH session while there was some VLC conversion happening launched from the local terminal. The SSH screen had a weird message before freezing that said something about a broken pipe and inability to allocate memory. Very strange. RAM is 16GB, the same script runs fine locally. /dev/shm container file size was almost the entire /dev/shm size, there was nothing else in /dev/shm/. The keyboard’s Num Lock toggle wouldn’t respond anymore level of freeze.

Cause

Not yet known.

Related Issues

Workarounds

Work around by removing connection time out in sshd_config for now.

2

I moved this out of Proposed Common Issues, because I don’t think this is likely to be a situation that many users find themselves.

Can you reproduce on F38 (or soon to be released F39)?

3

I haven’t tried reproducing it, no time. This is the first time Fedora crashed froze on me on a wide variety of scenarios and heavy everyday usage. This may crash some Red Hat servers so I think needs to be looked at by someone who is invested in this.

4

This WAS the first time I ran an nspawn container stored on /dev/shm/ vs hard drive. I am running now the same scenario - nspawn container stored on a local drive accessed via an ssh connection that does not time out anymore (although nothing much running locally) - and things are running smoothly.
Disclaimer: Ok, wasn’t the very first time but was the first time in this particular configuration and usage scenario.
EDIT: Doesn’t seem /dev/shm/ was necessarily the issue but it could’ve contributed to the freeze. Had another crash (but without the freeze) that said ‘broken pipe’ on the SSH client side. It seems like the nspawn container crashes specifically when it’s booted via an SSH connection. And that makes sense as integrity of the SSH connection would be paramount to the nspawn container function. Not sure why SSH drops (or maybe it is the nspawn container not jiving with SSH specifically) but this now seems a bit less severe than I thought at first.

6

Turns out Python script was leaking memory and eventually running out of memory. This, however, should never cause a full system freeze as it did in this case under any circumstances, especially over an SSH connection.