F40 Change Proposal: Systemd Security Hardening (System-Wide)

Project Discussion Change Proposals
,
1

Enable systemd service hardening features for default system services

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Wiki
Announced

:link: Summary

Improve security by enabling some of the high level systemd security hardening settings that isolate and sandbox default system services.

:link: Owner

:link: Detailed Description

systemd provides a number of settings that can harden security for services. We are selecting a few high level ones to enable by default on a service by service basis as suitable for that particular service.

  • PrivateTmp=yes
  • ProtectSystem=yes/full/strict
  • ProtectHome=yes/read-only
  • ProtectClock=yes
  • ProtectHostname=yes
  • ProtectControlGroups=yes
  • ProtectHostname=yes
  • ProtectKernelLogs=yes
  • ProtectKernelModules=yes
  • ProtectKernelTunables=yes
  • ProtectProc=invisible
  • PrivateDevices=yes
  • PrivateNetwork=yes
  • NoNewPrivileges=yes
  • User=

If we want to go further, we could also consider:

  • CapabilityBoundingSet=
  • DevicePolicy=closed
  • KeyringMode=private
  • LockPersonality=yes
  • MemoryDenyWriteExecute=yes
  • PrivateUsers=yes
  • RemoveIPC=yes
  • RestrictAddressFamilies=
  • RestrictNamespaces=yes
  • RestrictRealtime=yes
  • RestrictSUIDSGID=yes
  • SystemCallFilter=
  • SystemCallArchitectures=native

We will aim to cover all the default system services as well as some of the high profile services such as Nginx or PostgreSQL. All of these settings need to be configured on a per service basis instead of using a global override to facilitate fine tuning the settings based on service requirements and limit the impact for users on upgrades. Certain services have a very targeted scope. For instance, a service that only needs to read or write from only one directory could leverage more fine grained settings to restrict access even further. We will enable as many of these as feasible for the services but not every knob is going to be applicable to every service. For example, PrivateNetwork=yes can only be used for services that does not need network connectivity by default. We have to choose between DynamicUser=yes or User if either is feasible for the service to use. As a base starting point, from Fedora 39 workstation, we have the following system services installed by default which should considered within the scope of the change (excluding systemd associated ones which already have a number of these security settings enabled). We may also consider doing this for some of the high profile services including say Nginx and PostgreSQL permitting time considerations and other contributors if any joining this effort. We will prioritize critical or long running services.

  • abrtd.service
  • abrt-journal-core.service
  • abrt-oops.service
  • abrt-pstoreoops.service
  • abrt-vmcore.service
  • abrt-xorg.service
  • accounts-daemon.service
  • alsa-restore.service
  • alsa-state.service
  • anaconda-direct.service
  • anaconda-fips.service
  • anaconda-nm-config.service
  • anaconda-nm-disable-autocons.service
  • anaconda-noshell.service
  • anaconda-pre.service
  • anaconda.service
  • anaconda-sshd.service
  • arp-ethers.service
  • auditd.service
  • auth-rpcgss-module.service
  • avahi-daemon.service
  • blivet.service
  • blk-availability.service
  • bluetooth.service
  • bolt.service
  • brltty.service
  • canberra-system-bootup.service
  • canberra-system-shutdown-reboot.service
  • canberra-system-shutdown.service
  • chronyd-restricted.service
  • chronyd.service
  • chrony-wait.service
  • colord.service
  • console-getty.service
  • cups-browsed.service
  • cups.service
  • dbus-broker.service
  • dbus-daemon.service
  • dbus-org.freedesktop.hostname1.service
  • dbus-org.freedesktop.import1.service
  • dbus-org.freedesktop.locale1.service
  • dbus-org.freedesktop.login1.service
  • dbus-org.freedesktop.machine1.service
  • dbus-org.freedesktop.portable1.service
  • dbus-org.freedesktop.timedate1.service
  • debug-shell.service (opens a user shell that must be able to do arbitrary stuff)
  • dm-event.service
  • dnf-makecache.service
  • dnf-system-upgrade-cleanup.service
  • dnf-system-upgrade.service
  • dnsmasq.service
  • dracut-cmdline.service
  • dracut-initqueue.service
  • dracut-mount.service
  • dracut-pre-mount.service
  • dracut-pre-pivot.service
  • dracut-pre-trigger.service
  • dracut-pre-udev.service
  • dracut-shutdown-onfailure.service
  • dracut-shutdown.service
  • emergency.service (opens a user shell that must be able to do arbitrary stuff)
  • fedora-third-party-refresh.service
  • firewalld.service
  • flatpak-add-fedora-repos.service
  • flatpak-system-helper.service
  • fprintd.service
  • fsidd.service
  • fstrim.service
  • fwupd-offline-update.service
  • fwupd-refresh.service
  • fwupd.service
  • gdm.service
  • geoclue.service
  • grub-boot-indeterminate.service
  • gssproxy.service
  • htcacheclean.service
  • httpd.service
  • hypervfcopyd.service
  • hypervkvpd.service
  • hypervvssd.service
  • iio-sensor-proxy.service
  • import-state.service
  • initrd-cleanup.service
  • initrd-parse-etc.service
  • initrd-switch-root.service
  • initrd-udevadm-cleanup-db.service
  • instperf.service
  • ipp-usb.service
  • iscsid.service
  • iscsi-init.service
  • iscsi-onboot.service
  • iscsi.service
  • iscsi-shutdown.service
  • iscsi-starter.service
  • iscsiuio.service
  • kdump.service
  • kmod-static-nodes.service
  • ldconfig.service
  • libvirtd.service
  • libvirt-guests.service
  • livesys-late.service
  • livesys.service
  • loadmodules.service
  • logrotate.service
  • low-memory-monitor.service
  • lvm2-lvmdbusd.service
  • lvm2-lvmpolld.service
  • lvm2-monitor.service
  • man-db-cache-update.service
  • man-db-restart-cache-update.service
  • mcelog.service
  • mdcheck_continue.service
  • mdcheck_start.service
  • mdmonitor-oneshot.service
  • mdmonitor.service
  • ModemManager.service
  • ndctl-monitor.service
  • netavark-dhcp-proxy.service
  • NetworkManager-dispatcher.service
  • NetworkManager.service
  • NetworkManager-wait-online.service
  • nfs-blkmap.service
  • nfsdcld.service
  • nfs-idmapd.service
  • nfs-mountd.service
  • nfs-server.service
  • nfs-utils.service
  • nftables.service
  • nis-domainname.service
  • nm-priv-helper.service
  • numad.service
  • nvmefc-boot-connections.service
  • nvmf-autoconnect.service
  • ostree-boot-complete.service
  • ostree-finalize-staged-hold.service
  • ostree-finalize-staged.service
  • ostree-prepare-root.service
  • ostree-remount.service
  • packagekit-offline-update.service
  • packagekit.service
  • pam_namespace.service
  • pcscd.service
  • plocate-updatedb.service
  • plymouth-halt.service
  • plymouth-kexec.service
  • plymouth-poweroff.service
  • plymouth-quit.service
  • plymouth-quit-wait.service
  • plymouth-read-write.service
  • plymouth-reboot.service
  • plymouth-start.service
  • plymouth-switch-root-initramfs.service
  • plymouth-switch-root.service
  • podman-auto-update.service
  • podman-clean-transient.service
  • podman-restart.service
  • podman.service
  • polkit.service
  • power-profiles-daemon.service
  • psacct.service
  • qemu-guest-agent.service
  • qemu-pr-helper.service
  • quotaon.service
  • raid-check.service
  • rc-local.service (this can do arbitrary stuff)
  • realmd.service
  • rescue.service
  • rpcbind.service
  • rpc-gssd.service
  • rpc-statd-notify.service
  • rpc-statd.service
  • rpmdb-migrate.service
  • rpmdb-rebuild.service
  • rtkit-daemon.service
  • saslauthd.service
  • selinux-autorelabel-mark.service
  • selinux-autorelabel.service
  • selinux-check-proper-disable.service
  • speech-dispatcherd.service
  • spice-vdagentd.service
  • spice-webdavd.service
  • sshd.service
  • ssh-host-keys-migration.service
  • sssd-autofs.service
  • sssd-kcm.service
  • sssd-nss.service
  • sssd-pac.service
  • sssd-pam.service
  • sssd.service
  • sssd-ssh.service
  • sssd-sudo.service
  • switcheroo-control.service
  • system-update-cleanup.service
  • tcsd.service
  • thermald.service
  • udisks2.service
  • unbound-anchor.service
  • upower.service
  • uresourced.service
  • usbmuxd.service
  • vboxclient.service
  • vboxservice.service
  • vgauthd.service
  • virtinterfaced.service
  • virtlockd.service
  • virtlogd.service
  • virtnetworkd.service
  • virtnodedevd.service
  • virtnwfilterd.service
  • virtproxyd.service
  • virtqemud.service
  • virtsecretd.service
  • virtstoraged.service
  • vmtoolsd.service
  • wpa_supplicant.service
  • zfs-fuse-scrub.service
  • zfs-fuse.service
  • zvbid.service

We will also coordinate with upstream following Patch status :: Fedora Docs and encourage package maintainers to upstream these changes. Systemd will ignore any of these settings it does not understand on older versions. Hence this should be safe for upstream to merge on any services.

:link: Feedback

:link: Benefit to Fedora

Fedora services will get a significant security boost by default by avoiding or mitigating any unknown security vulnerabilities in default system services.

:link: Scope

Packaging guidelines will have to be modified to add recommendations to use more of the systemd security features by default. In particular, we should add a security settings section in Packaging:Systemd - Fedora Project Wiki. Current the guidance only recommends a couple of settings for long running services. Sample text:

Systemd services included in Fedora are recommended to use as many of the following security settings as applicable while maintaining the default functionality of the service.

  • PrivateTmp=yes
  • ProtectSystem=yes/full/strict
  • ProtectHome=yes
  • PrivateDevices=yes
  • ProtectKernelTunables=yes
  • ProtectKernelModules=yes
  • ProtectKernelLogs=yes
  • ProtectControlGroups=yes
  • NoNewPrivileges=yes
  • PrivateNetwork=yes

The full list of sandboxing features are available in systemd.exec. Note that if you are submitting changes to upstream as recommended, systemd will warn and ignore any of these features it doesn’t support. So it should be safe for upstream to enable as many of these features as applicable and not worry about distribution support for ones using older versions of systemd.

  • Trademark approval: N/A

:link: Upgrade/compatibility impact

Packages will automatically get additional security features enabled by default transparently. In limited circumstances, they may need to override the defaults. Refer to user experience section for details.

:link: How To Test

You can use tools like systemd-analyze security and systemctl cat to verify that specific security features are enabled by default. Default services with the default features should have no adverse impact and users shouldn’t have to do anything beyond using the software as intended and report any regressions. High profile services not installed by default that gain these security features would benefit from more targeting testing to spot any unintended consequences especially for niche or advanced functionality. If advanced non-default functionality requires overrides default settings, we can document those in the release notes to provide guidance.

:link: User Experience

This should be largely transparent change for users. The goal is to have the services work as expected with the default functionality but to potentially require tweaking the settings if the configuration is changed by users after installation. For instance, if we add ProtectHome=yes to Apache httpd.service and the user wishes to serve files out of their home directory, they will need to override the systemd setting to ProtectHome=read-only to allow for the service to read from the user home directory in addition to changing the service specific configuration files to enable this feature.

:link: Dependencies

None. We are merely enabling some of systemd security features by default for default system services and potentially some high profile services.

:link: Contingency Plan

  • Contingency mechanism: These settings can be enabled/disabled at a per service level. No wholesale reverts is necessary. If we don’t finish the work for all the services, we can follow up in future releases.
  • Contingency deadline: N/A
  • Blocks release? No

:link: Documentation

:link: Release Notes

systemd security hardening features are enabled for default system services and following high profile services.

  • PostgreSQL
  • Apache Httpd
  • Nginx
  • MariaDB

…

If you wish to turn off any particular settings, you can follow the standard systemd method of overriding the config. For example,

$ cat /etc/systemd/system/httpd.service.d/override.conf

[Service]

ProtectHome=no

$ sudo systemctl daemon-reload

$ sudo systemctl restart httpd.service

$ systemctl status httpd.service

â—Ź httpd.service - The Apache HTTP Server)

11 Likes
2

IME that is not OK. If a project ships unit files that result in warnings being generated, it will result in bug reports from users directly, and/or support tickets raised with the distribution vendor, neither of which is a desirable situation as it creates a support burden for both upstream and the vendor to respond to this. As an upstream maintainer, I’d expect any patches to either not include settings incompatible with our minimum required systemd version, or to provide build system magic to conditionally build the default unit file.

3

Fair point. I was thinking more from the perspective on whether these features will break the service outright but from a support perspective, having warnings can still be a problem. I have added your feedback to the feedback section and update the guidance on upstream changes.

1 Like
4

The investigation about whether each one of those security features work for a service will have to happen on a per-service basis. Each one is a special case so we can’t really enable any of this by default for every service.

The service security changes should also be discussed and merged upstream first to make sure nothing gets broken. They would then land “automatically” in Fedora as part of a regular update.

Is there any particular reason this needs to be a change request?

Have you already made an initial investigation about which security options are missing for which service?

1 Like
5

Hi Timothée

I have noted the per service nature of the change in the proposal. Fedora will include the very latest version of systemd and can enable many more of the settings and have it be functional while taking into account the default configuration. Upstream will have other considerations including the minimum version of systemd that they want to support and therefore may be able to take advantage of only a smaller subset of these features even if they are willing to use them. It’s been well over a decade since I packaged systemd for Fedora originally and most services at the moment don’t take advantage of much of the features. Fedora can provide a more secure configuration by default for its users and act as a coordination point in increasing adoption of these features as it has done with systemd itself.

6

Just to add my £0.02, I think it’s also worth considering the ways that we already harden packages that upstream may not be aware of. From subtle ways (our long string of GCC args) to major ways (SELinux), this is a tradeoff we’ve already begun to make.

Annoyingly, there’s no real equivalent to setroubleshoot for these hardenings. We therefore need to consider how we can best make these easy to toggle on and off for troubleshooting. Perhaps we always include them as drop-in units, even if the base systemd unit is already entirely Fedora-provided. That would also make it more obvious to an administrator that something has changed when inspecting the service, as systemctl status will list drop-ins.

Whilst it’s likely that most of these changes will be transparent to users (particularly if they shadow existing SELinux policies), I’d expect a few of them to really hit hard. I can foresee a world of cgi-bin scripts that don’t survive this migration without administrators intervening. If we go ahead with this initiative (and I think we should) then we need to shout this from the rooftops. Considering that F40 will become EL10, we should helpfully be able to get some support from Red Hat on that.

I think it would also be a good idea to update the packager docs to reflect these new recommendations as a list of sensible defaults. For the packages mentioned in this proposal, we’d also need to decide if making the changes is maintainer’s responsibility or the change owners.

Despite sounding pessimistic, I’m really onboard with this change. I can think of a number of recently vulnerabilities this would have gone a long way to mitigating, and if there’s anything I can do to help I’d be happy to do so.

7

Hi Daniel

Thanks for the detailed feedback.

I agree additional tooling support can be useful. I am however not sold on the idea of using drop-ins since those are intended for changes by users/administrators and IMO that’s not the mechanism we can use. I am proposing to do this work on a purely voluntary basis and I have no affiliation with Red Hat but to the extend this will have an impact on RHEL 10, someone else should likely do some internal coordination for this. Perhaps that would be the FPL or change wrangler.

As part of the change proposal I have already includes the suggestion to update the packaging guidelines and also included a starting draft for discussion. Since we can’t reliably predict the exact impact especially for upgrades, we do need to highlight these changes prominently and I have included some sample text for the release notes as well. Whether change owners should be responsible or the package maintainers, my working assumption is that it would be collaborative effort with the package maintainers making the final call but this is part of FESCo’s decision to make as they review this proposal.

I would be happy to get additional help. There is a lot of ground to cover and since the exact settings will vary based on the service, I can certainly use the participation for creating the PRs, testing the changes, working on the docs etc. If you (and others) would like to participate, please feel free to add yourself as a owner or indicate that to me here or offline and can setup some ways for us to coordinate.

8

Drop-ins can be installed in /usr/lib, and that is already used in Fedora, sometimes even by the same package that owns the main unit file. I like this idea from a support perspective; being able to have a user mask away all downstream hardening would be a useful diagnostic. It doesn’t seem like systemd has great tooling for that, but mkdir and touch isn’t too hard.

I have one small concern though. Drop-ins always take precedence over unit files, no matter where those files are installed. That means an if an admin copies a service to /etc (e.g. with systemctl edit --full), the vendor hardening would apply even if explicitly disabled. This would be visible in systemctl status and especiallysystemctl cat but I worry that it would be confusing.

To ensure that systemctl edit (without --full) works, the name of the drop-in would need to sort before override.conf, maybe something like 00-fedora-hardening.conf.

9

Can every setting that is proposed to be used be reset to its default in a systemd override file?

10

I concur. The gold standard here should be that the hardening is fully integrated into the upstream provided .service file and maintained by upstream too, not as a Fedora patch, nor as an extra Fedora drop-in unit config.

More generally, having tried systemd hardening upstream for one of the libvirt daemons ([libvirt PATCH] logging: lockdown the systemd service configuration - Devel - Libvirt List Archives), I think we should be wary of doing any hardening as downstream only patches, for non-trivial daemons.

The impact of some of the hardening settings is non-obvious, as there can be surprising ways applications interact with the system that are not apparent unless you have intimate knowledge of the code (which most Fedora maintainers do not have), not to mention users doing unexpected configurations.

A parallel would be the development of SELinux policy downstream in Fedora, which got us a long standing reputation for breaking applications when the policy was too restrictive to allow the app to work correctly. I’ve experienced similar with Seccomp, where seemingly obvious things to deny end up breaking valid functionality, even when I’m familiar with the application code. Systemd hardening is highly likely to share these risks. The best way to mitigate risk of breakage is for the upstream maintainers to own this problem, as they’re most knowledegable about what their code expects to be able to do.

11

Do you have a few precise, validated examples for this claim?

With a thorough review, I’m sure that we’ll find some services that can be hardened a bit but I don’t think the issue is as widespread as presented here and it’s likely that most services that are not hardened right now either can not be (they need admin permissions anyway) or need a lot of work, better done upstream.

12

Yes. If we use drop-ins as Chris has suggested, then you would also have to make sure the lexical ordering makes it such that the overrides gets read last. If we use names he has indicated as an example, then it would just work.

13

Hi Timothée

Sure. Here is a concrete example. Nginx in Fedora uses only PrivateTmp

https://src.fedoraproject.org/rpms/nginx/blob/rawhide/f/nginx.service#_19

We could at the minimum consider the following:

[Service]
LockPersonality=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
ProtectClock=true
ProtectControlGroups=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectProc=invisible
ProtectSystem=full
RestrictRealtime=true
RestrictSUIDSGID=true
SystemCallArchitectures=native

You could with a few tweaks run it as a non root user but even otherwise, all of this is still useful.

14

This talks mainly about hardening of systemd system units, but can something similar be done for systemd user units?

For example, if I run systemd-analyze --user securiy pretty much everything is marked as UNSAFE, except for pipewire and wireplumber units which are a bit lower.
It does make sense if that is expected since those units likely run with the same privileges as the user, but still…

15

Hi Mateus

Yes, it can work for user units as well and many of them likely could use even more fine grained settings compared to system units. However, as indicated in the change proposal, I scoped it to system units only for this release because of time constraints. If this is approved and we cover only the default system units (and some select high profile services), we could potentially learn from the feedback post release from users, upstream developers etc and then adjust our strategy before we cover more the next release. I think there might be value in approaching this incrementally even if we have others joining the effort.

16

That is not what I was getting at if you add a setting can how do I set to it default value?

Can I use the PropName= with no value? Or do I have to set to its default?
This is likely covered in the systemd docs, but I not sure where to look.

17

There is no generic mechanism to reset a directive to default. PropName= only works for an accumulating list (like ExecStart) or a string where you’re just setting it to ''. When used with other types, it’s ignored as invalid.

Most of these settings are (at least in part) booleans, but ProtectProc is not. It’s an enum that can only be reset to default with ProtectProc=default.

18

OK, so from this example, it does look like we could potentially enable a few of those by default for all services. I would expect the following to work:

LockPersonality=true
MemoryDenyWriteExecute=true
ProtectClock=true
ProtectControlGroups=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectProc=ptraceable
RestrictRealtime=true
SystemCallArchitectures=native

Unfortunately we don’t have a way to set such defaults nor do we have a way to exclude services from it so that would be a per-service change. Here we would have to exclude chronyd from ProtectClock for example.

I don’t think nginx is a good example however as it’s not installed by default (we should likely focus on hardening all installed by default services in all of our variants) and it’s likely doing all sorts of weird things depending on the configuration.

1 Like
19

Hi Timothée

The change proposal includes changing settings at a per service level and calls that out several times. Although it is not part of the change proposal, if you want to do invert this, you could very well set ProtectClock as true by default and then explicitly disable this for services like Chronyd as an example. A default override that is applicable for all services is supported within systemd.

Regardless of whatever odd things Nginx may be doing based on user configuration, I think it is very reasonable to prevent it from changing the clock, hostname or mucking with kernel modules by default. In any case, Nginx was just a high profile example. Other system services typically have a much more targeted scope compared to a web server. The same settings I provided as an example are largely applicable to nearly every default system service with a few exceptions. Chrony needs to manage the clock, NetworkManager needs to manage devices etc. Many of the system services can use other settings such as ProtectHome.

20

I expect this is liable to either break, or degrade performance of, any application linked to libpcre2, as its regex JIT attempts to use writable + executable memory IIRC.

I expect this could impact libvirt, docker, podman

Liable to impact libvirt’s ability to enable IP forwarding.

IIUC polkitd needs to be able to access /proc/$PID but does not necessarily have ptrace() capability

Also, very many of the tunables listed imply NoNewPrivileges=yes if the service has dropped CAP_SYS_ADMIN so their impact is potentially larger than their name suggests.