I’m now trying to eliminate the hypothesis that this admin is created on-the-fly (with the located admin replaced by nimda).
The errors stop following
systemctl stop php-fpm
but of course, that breaks/stops Nextcloud.
Things get a little more complicated if I just stop Nextcould (by putting it into maintenance mode). I still get errors (at least for a while). I think because some of the background processes, launched by cron.php continue to run. I’ve gone through various trials to try and eliminate or confirm Nextcloud, cron.php, and related, but I’m not getting irrefutable evidence.
In the reverse order, things are clearer.
start php-fpm
resume Nextcloud (take out of maintenance mode
resume cron.php
No errors following 1 and 2, (even waiting 10 minutes).
Following 3, I get
May 17 21:10:40 vericalm audit[701712]: AVC avc: denied { search } for pid=701712 comm="admin" name="13419" dev="proc" ino=67936 scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:unconfined_r:gnome_atspi_t:s0-s0:c0.c1023 tclass=dir permissive=1
May 17 21:10:40 vericalm audit[701712]: AVC avc: denied { read } for pid=701712 comm="admin" name="comm" dev="proc" ino=67983 scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:unconfined_r:gnome_atspi_t:s0-s0:c0.c1023 tclass=file permissive=1
May 17 21:10:40 vericalm audit[701712]: AVC avc: denied { open } for pid=701712 comm="admin" path="/proc/13419/comm" dev="proc" ino=67983 scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:unconfined_r:gnome_atspi_t:s0-s0:c0.c1023 tclass=file permissive=1
May 17 21:10:40 vericalm audit[701712]: AVC avc: denied { getattr } for pid=701712 comm="admin" path="/proc/13419/comm" dev="proc" ino=67983 scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:unconfined_r:gnome_atspi_t:s0-s0:c0.c1023 tclass=file permissive=1
May 17 21:10:40 vericalm audit[701712]: AVC avc: denied { search } for pid=701712 comm="admin" name="13419" dev="proc" ino=67936 scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:unconfined_r:gnome_atspi_t:s0-s0:c0.c1023 tclass=dir permissive=1
May 17 21:10:42 vericalm systemd[1]: Starting setroubleshootd.service - SETroubleshoot daemon for processing new SELinux denial logs...
May 17 21:10:43 vericalm systemd[1]: Started setroubleshootd.service - SETroubleshoot daemon for processing new SELinux denial logs.
May 17 21:10:43 vericalm audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=setroubleshootd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 17 21:10:43 vericalm setroubleshootd[701785]: libsepol.context_from_record: invalid security context: "unconfined_u:unconfined_r:gnome_atspi_t:s0-s0:c0.c1023"
May 17 21:10:43 vericalm setroubleshootd[701785]: libsepol.context_from_record: could not create context structure
May 17 21:10:43 vericalm setroubleshootd[701785]: libsepol.context_from_string: could not create context structure
May 17 21:10:43 vericalm setroubleshootd[701785]: libsepol.sepol_context_to_sid: could not convert unconfined_u:unconfined_r:gnome_atspi_t:s0-s0:c0.c1023 to sid
May 17 21:10:43 vericalm setroubleshoot[701785]: Unable to process audit event: cannot access local variable 'syslog' where it is not associated with a value
May 17 21:10:43 vericalm setroubleshoot[701785]: Traceback (most recent call last):
May 17 21:10:43 vericalm setroubleshoot[701785]: File "/usr/lib/python3.13/site-packages/setroubleshoot/audit_data.py", line 1106, in compute_avcs
May 17 21:10:43 vericalm setroubleshoot[701785]: avcs.append(AVC(audit_event, record))
May 17 21:10:43 vericalm setroubleshoot[701785]: ~~~^^^^^^^^^^^^^^^^^^^^^
May 17 21:10:43 vericalm setroubleshoot[701785]: File "/usr/lib/python3.13/site-packages/setroubleshoot/audit_data.py", line 675, in __init__
May 17 21:10:43 vericalm setroubleshoot[701785]: self.derive_avc_info_from_audit_event(avc_record)
May 17 21:10:43 vericalm setroubleshoot[701785]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
May 17 21:10:43 vericalm setroubleshoot[701785]: File "/usr/lib/python3.13/site-packages/setroubleshoot/audit_data.py", line 1027, in derive_avc_info_from_audit_event
May 17 21:10:43 vericalm setroubleshoot[701785]: raise AVCError(_("%s \n**** Invalid AVC: bad target context ****\n") % self.avc_record)
May 17 21:10:43 vericalm setroubleshoot[701785]: setroubleshoot.audit_data.AVCError: node=vericalm type=AVC msg=audit(1747512640.952:105829): avc: denied { search } for pid=701712 comm="admin" name="13419" dev="proc" ino=67936 scontext=system_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:unconfined_r:gnome_atspi_t:s0-s0:c0.c1023 tclass=dir permissive=1
May 17 21:10:43 vericalm setroubleshoot[701785]:
May 17 21:10:43 vericalm setroubleshoot[701785]: **** Invalid AVC: bad target context ****
May 17 21:10:43 vericalm setroubleshoot[701785]: During handling of the above exception, another exception occurred:
...
So I fairly convinced the problem is induced by the background jobs launched from cron.php.
Also more clear is that the AVC denials come first, and the rest appear to be associated/consequential noise. Am I right?
Yes, I think so. There’s a genuine denial, and then there’s noise because of a bug in a Python script which tries to do some further analysis and logging of that denial.
As far as I can see from the Python source code, it would still be an issue in current versions.
(Edit: removed reference to a Bugzilla ticket, which was a different bug in the same Python script.)
I am now pretty sure the AVC denials are caused by the background processes launched by cron.php necessary for keeping Nextcloud working.
cron.php is no simple script, it launches separate processes depending on how Nextcloud is configured, when cron.php was last run, and so on. [I need to learn php, when I have time. It should not be too difficult given my proficiency in C++ and other languages.]
Let’s suppose that some Nextcloud third party App developer has tried to be too clever. For example, knowing that the process lauched in the background is generic (to cron.php) and wishing to convey more meaningful information to the user, they attempt to disguise the actual process name with the user, such as admin. Is this even possible?
In any case, why would any App wish to crawl through processes under /proc?
I notice there is something wrong with the AVC denial:
It is quite possible that cron.php is running admin (directly or indirectly) and then admin has done an exec system call to replace itself with coolwsd. In fact, that might explain whyadmin is poking around under /proc. If the admin command is forking and execing other processes, it might be attempting to look up some details about the spawned processes by reading from /proc.
That’s interesting that it is fairly easy, even in shell environment, for a process to masquerade its process name.
Why would a genuine process ever want to do that?
It’s not just the process name, there’s other things in the AVC denial that make me think it is so corrupt that it trips up setroubleshootd with invalid security context and subsequently further problems for setroubleshoot.
Let’s suppose some App running php as a non-priveled user apache inside Nextcloud has gone rogue (there’s a genuine bug — a coding error, or the coder is attempting to do something outside the accepted remit or scope of the app). This rogue app has effectively taken down a web site[1]. I would like to believe that layers of defences would have afforded some sort of protection against this. Hence my desire is get these defences bolstered by submitting bug reports against the relevant layers. But as this is not my field of expertise, I’d appreciate some help.
Obviously, I need to bug report the offending app. How can we help the developer? What is the app doing wrong that is causing the AVC denials? Is a process not allowed to inspect anything underneath /proc/<pid> where <pid> is its own process id?
It does not make sense to expect a sysadmin to adjust the security contexts of files under /proc because they are transitory. Is there something wrong with the SELinux defaults such that gnome_atspi_t is missing? Is there a bug with SELinux in this scenario at all?
I had presumed that Nextcloud is sufficiently mature that any App installed within its umbrella would be contained or shielded from causing system chaos. Is there a problem with the API presented to Nexcloud apps? Is there a problem with the vetting procedure for apps (like not marking an app as untested before it has been tested/vetted)?
Is there a problem with the web interface, specifically httpd and php-fpm, that can allow an apache user app to cause havoc?
Is there a problem with the setup on this machine that could be adjusted to mitigate this scenario? (I’ve already noted changing SELinux to enforcing, and to migrate from external Nextcloud installation to the official Fedora dnf installation). I don’t think my external facing layers of protection have been breached, but I’d welcome any suggestions for monitoring or bolstering.
Is there another layer in this list that I might have neglected to mention? Have I misconstrued anything?
This is almost a denial of service through floods of error messages leading to delayed responses, overflowing message queues etc. I’m having to reset user accounts because they have been locked out due to “too many requests”. ↩︎
According to the SELinux policy, no. (ls -alZ /proc shows that different processes have different SELinux labels, to enable fine-grained access control. In fact, even the default proc_t type shouldn’t be accessible by an arbitrary process.)
A system administrator can choose whether to relax that. However, earlier on you said that “My presumption is that all such processes are contained within Nextcloud and user apache. There should be nothing which could surprise SELinux.”
Based on that, is it right to conclude a Nextcloud process trawling through /proc breaches your expectations about containment?
Worth thinking about running Nextcloud as a container in Docker or Podman? (I believe Nextcloud provides a readymade Docker container to start from.) I find that makes it easier to reason about how the containment works.
It’s worth making a bug report in Bugzilla against Fedora’s setroubleshoot-server package, which contains the Python script causing that “UnboundLocalError” noise.
In turn there should be a bug report to the upstream (looks like a simple fix - the problem is this line of code, which attempts to use syslog before importing it).
The exec command isn’t masquerading the command name. It is just a quicker way to reallocate a process’ resources (mainly memory, but other things like the PID as well) to another process. The original process ceases to exist. Using the system call incorrectly can be a security problem, however, if the original process (e.g. admin) didn’t drop whatever privileges it had before calling exec.
FWIW, the following might also work around the problem, with the tradeoff of having slightly reduced security. It wouldn’t be quite as bad as running all of SELinux in permissive mode though.
sudo semanage permissive -a gnome_atspi_t
Edit: Nevermind, that wouldn’t prevent the logging of the access violation, which is apparently what you are needing to stop. Since the audit2allow tool seems to be having trouble parsing the logs, I guess you would have to craft the allow rule by hand. That should be doable.
If I was the developer of the offending code (the Collabora app) then I’d probably want to check/ensure that the process was created with the correct label(s).
As a user, are you telling me that the SELinux defaults for this system should include gnome_atspi_t such that new processes inherit this label? Could it be that the label gnome_atspi_t has been introduced since F30 was installed in 2019 and that the successive upgrades have failed to touch this? If so, what would be the command to correct this anomoly?
I’m not keen on a paste-over like a custom policy. That would add a complication for on-going maintenance, future upgrades, etc. Also, if there is something underlying that is wrong (like outdated defaults for SELinux), then I’d want that fixed rather risk recurrence if similar/worse issues in the future.
Neither Nextcloud or Collabora have an selinux policy. They only post that you relabel the files with the apache label type. This then defaults to using the apache policy. This policy is fine tuned for apache server, not every app that uses it.
When labels with the *_*_exec_t type run, they appear without the exec type label. So for instance in your case, gnome_atspi_t was run as gnome_atspi_exec_t first. All logs will display gnome_atspi_t as the running command label asking for permissions.
I can create a policy that allows only those four denials to succeed until or a nextcloud selinux policy has to be created tested and pushed upstream to get your ideal outcome.
Cool, but rather than being something that Nextcloud can fix, it’s a bug in setroubleshoot itself (actually, in terms of Fedora’s packaging schema, it’s in the setroubleshoot-server rpm.)
So in the first instance it’s a bug to raise in Fedora Bugzilla.
There should also be a corresponding issue raised in the Gitlab repo upstream (setroubleshoot / setroubleshoot · GitLab), though I’m not sure if the Fedora package maintainer will raise that or whether they prefer you to raise it.