Cron jobs triggering systemd user session in F36?

Ask Fedora
,
1

So, this is weird.

I had one of those “events” today where my session spontaneously dropped me back to the login screen and lost track of all my processes — it happens, occasionally, but w/e. So when I logged back in and didn’t see any of the windows I had open, I knew there would be a lot of processes still running from my old session. I logged back out, switched to the console at Ctrl+Alt+F3, and logged in as root so I could do a killall -u ferd.

Which worked fine. Until a couple of minutes later, when there were new processes running as ferd. I killed them again, double-checked that they were all gone… and a couple of minutes later, they were back again!

Now, for Reasons™, I have a crontab set for my user account (with crontab -e), and one of the entries executes a script every 3 minutes to diddle some automatically-generated files that get written slightly wrong.

Apparently, every time that cron job fires off, it’s now causing the creation of a user session for my account, controlled by a /usr/lib/systemd/systemd --user process. And that process is starting all of the user units that I only intend to have running when I’m logged in. Those processes are staying running even after the cron job ends — in fact, a couple of them are daemons, so they never stop running because there’s nothing to terminate those units.

I don’t remember ever seeing this before upgrading to F36, though I can’t rule out the possibility.

I experimentally removed the crontab and the phantom sessions stopped, but they resumed as soon as I put it back in place. So now there’s no doubt it’s cron that’s triggering the systemd --user process to be started. The sessions don’t show up in loginctl at all, which says I’m completely logged out even though there are like a dozen processes running under my UID.

Anyone know how to get cron to stop launching user sessions? Or at least to launch them with a different target, so I can specify that it not start things like dropbox.service, pipewire.service, gvfs-daemon.service, and etc.?

2

Just a thing I found from the logs, but I have not enough systemd knowledge to explain it: cron goes via pam module crond, which includes system-auth, which contains a line
-session optional pam_systemd.so
If you comment out this line, the cron launched process does not add an item in the loginctl list. So: may be modifying the crond file to include a copy of system-auth without this line work-arounds the problem, but no idea about unwanted effects and whether this is the correct way to solve it.

1 Like
3

Ooh! Good call @hmmsjan , I didn’t even think of checking PAM, duh!

Hmm. It doesn’t, for me, even with that line present — does it show up for you? On my system, the crontab-initiated sessions aren’t shown in loginctl, but they still have plenty of processes running! In a sense that’s part of the problem, since if they were in loginctl I could at least terminate them with loginctl terminate-session / ... kill-session (and potentially hunt down what’s keeping them alive, or at least have more to go on).

In the meantime, I decided to work around the issue for the moment by adding a separate user account and shunting the frequent cron jobs over there, to at least keep them from triggering sessions for my main account.

That plan had its own challenges, plus the annoyance of having to juggle permissions so that the other account could update files under my /home/ferd/ account space, but in one particular aspect it proved enlightening.

What I see with the new account is that systemd --user is still launched for each run, and it still fires off user units. (Which remains odd to me, why would a cron job need to trigger units like pipewire.socket — I was wrong, it doesn’t actually start the service directly, just sets up the auto-activation socket — just to run a quick script on the filesystem? Seems like that should at least be optional.) However, when that happens for the new account, unlike with my account, the systemd --user daemon process shuts down the session the moment the cron job finishes.

So, the impression I get is that this is by design, and the real issue is that there’s something in my account’s user units that isn’t properly terminating when it’s supposed to. Which would also explain the long shutdown times I’ve been experiencing on the rare occasions I reboot the machine, since the same process must be blocking the system restart until it times out and meets the business end of systemd’s axe.

So, assuming I’m able to hunt down the source of those hang-on-close issues with my user sessions, the brief transient ones that get started with every cron job executed on my behalf will just be an oddity of the systemd New World Order™.

4

I’ve tested it only with a stupid “sleep 600” cron command, but what I indeed see is that a second user launches a second “systemd --user” instance when a cron job starts. In addition, a session is created.

Started session-20.scope - Session 20 of User user1.

But on my system, commenting out this pam call to pam_systemd prevents not only session creation, but also a second “systemd --user”. May be careful inspection of the logfiles shows the relation between cron, pam and systemd. On my system, commenting out that specific line in system-auth prevent not only a session, but also the complete setup of systemd-user, pipewire socket and so on.

1 Like
5

Hmm. Yeah, I see those log messages as well, I just don’t see any actual sessions.

The difference may be that unlike sleep 600, my crontab scripts complete within a second or two of being started, max — it could be that the session is already gone before I even get a chance to look for it. loginctl doesn’t appear to have any sort of history feature, so unless a session is currently running it’s not going to show up anymore.

My guess is that even when running under my user account, a session was created and terminated all in the space of 1-2 seconds (which is why I could never detect one)… it’s just that the actual work of migrating to exit.target post-session got delayed by whatever’s hanging up systemd during session termination.

6

OK, yes, As soon i kill the sleep the session is away and no longer visible in loginctl.
Tried to setup mpd (music player daemon) as user service. The sleep from cron launches mpd, so enabled user services start with the cron job, but it disappears if the sleep is ready. I would never have expected this cron-systemd interaction for simple cron jobs… Question stays why services do not stop if the cronjob is ready and why it does not help if you change the PAM file. I do not know whether a systemd.timer behaves the same way, that’s a new study…

1 Like
7

You may want to take a look at, loginctl enable-linger $USER?

Enable/disable user lingering for one or more users. If enabled for a specific user, a user manager is spawned for the user at boot and kept around after logouts. This allows users who are not logged in to run long-running services. Takes one or more user names or numeric UIDs as argument. If no argument is specified, enables/disables lingering for the user of the session of the caller.

If this only needs to run when you’re logged in I would look at making a systemd user timer.

I believe the observed behavior is intended for the user session, Re: [systemd-devel] How to quiet cron sessions logging with systemd-212?

1 Like
8

Thanks, now I finally understand why I always see user1000.slice starting up during boot. I have linger switched on without knowing. So it is intended behavior. I do not know how to get the original behavior of cron, just launch a program, without pam modifications.

9

…HUH. I do as well! And I have no idea why it would be set, as I’m not running MPD — or anything else — as a user service on this machine. (I vaguely remember linger being mentioned somewhere in the docs/configs as a way of accomplishing MPD-as-user-service, but I’m running it as a system service on a completely different machine. The mpd server isn’t even installed on this one.)

Only thing I can think of is that I might’ve set it to enable remote access, if that was necessary to reach desktop services?

Guess I’ll see what effect turning it off has; thanks for the pointer @grumpey!

10

I tried the following:
Make in /etc/pam.d a copy of system-auth text (symbolic link!) into system-auth-crond
Changed in the crond file system-auth into system-auth-crond.
Duplicated in system-auth-cron the line containing “pam_succeed_if” to before pam_systemd.
If I understood it correctly both pam_systemd.so and pam_unix.so are disabled now for cron. (If this works, the original system-auth could be modified in this way, which is wacky anyhow because it’s a symlink into authselect). The logfile now shows for session:

msg=‘op=PAM:session_open grantors=pam_loginuid,pam_keyinit,pam_limits acct=“user1” exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success’

which means pam_limits is the last one and both pam_systemd and pam_unix are skipped if it’s cron calling.

This should reduce the cron to what it was before and place it outside of systemd.
But for all users and all jobs, so side-effects are not excluded.
Disclaimer: system-auth depends on authselect!

According to the log it works and there is no session created. At least on my system…