CoreOS Hardening

I have not seen specific hardening advice, focusing on Fedora CoreOS. Is Fedora CoreOS already hardened enough by default ? (there is a lonely note recommending to mask the docker.service so it is not accidentally socket-activated, but other than that …)

There are plenty of hardening guides for Linux servers, of varying degrees of detail and obsolescence.

Some of the recommendations are very old and quite general, applying to any kind of software. Pretty much any guide will tell you to regularly apply security updates (at this point in time this should be simply put in the in the “duh!” category). Then it’s another classic: reduce the attack surface by keeping only the essential services and components running/installed.

CoreOS already covers these two as part of its design, and there is much more, locked accounts with no passwords, with a focus on key-based ssh access and so on.

On top of that comes the discussion about containers. Again, many guides never fail to mention that people should not be lulled into a sense of false security just because they run things in containers and not directly on the host system. At that point in the discussion usually there is a natural segue into the often given advice to use “rootless” containers – so to try to run containers as non-root – moment when maybe podman gets talked about.

To stir the pot, add some performance considerations into the mix and you get the ingredients of a spirited debate that can go on for a very long time, without much end in sight.

Yes, it is a big and complex subject and there are no silver bullets. There should be no expectations that it can be fully and decidedly solved.

So without going into too much detail, any particular advice applying to CoreOS hardening, with a focus on standalone usage ?

Thanks.

This is a complicated topic. As the joke goes… the strongest way to harden a system — Fedora CoreOS or otherwise — is to put it in a lead box, lock the lead box, throw away the key, drop the lead box into the bottom of the ocean, and then hurl the entire Earth into the sun. Everything short of that is compromise in some way or another.

In other words, it’s not really that it’s a problem that can’t be fully solved, but that it’s a problem that doesn’t have any single solution. It’s best to think about threats and risks. Where are you running the system? What is it doing? Where are the exposure points? Who needs to have access? What kind of access? What things must not be accessed, and by whom? If that fails, what are the consequences?

We do have a lot of security features in play — many of them advantages that other operating systems don’t have. In addition to rootless containers, the default SELinux policy offers another layer of protection.

Hello @fedhex ,
Are you talking about using FCOS as a desktop single user system? If that is the case, I would strongly recommend using Silverblue or Kinoite as the base starting point as they too employ the security features @mattdm has mentioned in his response, but they are geared to the single system/single user ootb, whereas FCOS is more hands on at the installation phase and a fairly steep learning curve for the average user. It is (FCOS) meant as a server of containerized workloads, I think. As well, they (Silverblue and Kinoite) are created via a declarative set of files, they just don’t use the same build tools as FCOS at this time. Daemonless container management is generally more secure than using the traditional method with a daemon such as Docker, which can now too run containers without a daemon. The real security rub with containers would come down to what you chose to expose to it of the host system, and how. Podman switched from runc to crun, which is more secure I think.

1 Like

@mattdm, thank you for the response.

True, the solution space is huge. Indeed, and what you are describing, any solution (or path towards a solution) should be framed with a specific threat model. That starts with a foundation: what exactly is attacked; in this case, how would a CoreOS install look like …

The crucial thing is the balance between security and convenience; as you said, you can harden a system so well that the convenience becomes 0, or a joke (the ping times accessing that box in the middle of the sun must be atrocious … yes, thank you, I’ll be working remotely). There must be some convenience built into the model, otherwise it’ll be too hard to use. And the optimization is to make at least any compromise a convenience. (conveniences such as ssh key access that are also a big security improvement are rare)

I can see some things that CoreOS does trying to strike a balance between the two. What I was dreaming of is to get a good picture of where was a security decision, where was a convenience decision.
Maybe I can take the pain in trading off some clear convenience for more security, if is really worth it.
But I’ll be thinking twice++ of reducing security for extra convenience: unless I have some super-deep understanding of what is going on, I would rather trust CoreOS and its designers to do the right thing instead.
So, deciding what to leave as it is and what to customize and how much.

An example of the small, early decisions to be made – but they could end up later on steer significantly the direction of the whole architecture.
Take for example the default regular user account. It’s clear a lot of care has been used in setting it up. No default password login, you can simply add a ssh key and you’re good to go. Security! On the other hand, core is in %sudo in sudoers (or something to that effect), so no authentication with sudo. Convenience!

Is it worth setting a password on the default account and have sudo authenticate you ? (or, more extreme, set up an entirely new user… then make sure the account does still not allow password logins…) A lot of system administration will become more inconvenient and arbitrary automation, especially through ssh, will suffer (or at least it will require setting up a fixed list of NOPASSWD scripts … )

Then let’s pin some variables, an attempt to narrow down a mythical “minimal common” case. Single-node install. These days, some form of cloud install. Multiple containers, some having ports exposed to public access; others, not publicly exposed; both might have some sort of privately accessible communication channel that you can tunnel to. In the first category, some kind of server, most likely a web server. In the second, stuff like a database (now that I describe all this, starts to sound like a frontend and a backend…).

Given this fairly common layout, there is another major fork in the road: rootful or rootless containers. I am also focusing just on podman here, since I understood docker has some wrinkles with rootless.
There is smooth convenience running rootful: add the unit in the ignition. Done.
For rootless, there is more friction: user-level systemd units are more complicated to create, assuming that you will do all the container related setup from inside one of them. Or there is an idiomatic way to start with system units and switch to a certain user/group inside (User=, Group=); maybe I have not got deep enough down the systemd docs rabbit hole to find out.
Does the convenience biased towards rootful amount to an admission that rootful is OK enough and not the “OMG, run-don-t-walk-away-from-it” that many container articles seem to imply ? (again, in the context of podman)

And despite the advances in the convenience of running rootless, there are still limitations that sometimes pop up, usually late during the design or, worse, testing process. If you hit a wall at that point, going back and re-arranging everything to run rootful turns the rootless attempt into wasted effort.

I got a lesson in this kind of outcome when trying to change some default CoreOS settings: many hardening guides recommend moving the sshd from listening on the default port to something else. The advice is controversial, the critics’ point being that you basically decrease convenience for negligible increase in security through obscurity. Just in case, as an experiment, I tried it. It turned out that is not possible due to SELinux settings (it can be done, but in very ugly, inelegant ways), so I gave up and stuck with the default.

This is the kind of advice that would be useful: is this just a limitation of the current way of applying ignition settings or it is considered not worth supporting such an use case and the sshd on the default is the way to go ?

I guess I’ll stop here, this is already getting really long.

1 Like

Thanks for the advice @jakfrost !
I am aware of Silverblue, etc. but I am talking about servers here, not desktop.

Excellent then, FCOS is indeed the beast you want.

FWIW, system administration and arbitrary automation are discouraged on Fedora CoreOS. You can do them, of course, but we recommend reconfiguring machines by updating the Ignition config and reprovisioning.

I’d say no, rootless is preferred even though it’s less convenient. We haven’t yet put a lot of work into optimizing the container workflow (odd, I know) since users often bring their own container orchestration.

As sometimes happens, it’s a combination of features colliding with each other. It’d be good to fix this (and we’d welcome help doing so) but unfortunately there are nontrivial unsolved problems there.

One specific note on hardening: it’s good practice to ensure that untrusted services on the machine cannot read the Ignition config. On virtualization platforms this isn’t generally a problem, but some cloud providers serve the Ignition config from a metadata service that’s accessible from containers unless you firewall it or delete the config.

(Ideally you wouldn’t put secrets in the Ignition config, and would keep them in a separate secret store instead. Setting that up in a secure way isn’t necessarily trivial, though, and we don’t have any guides for it.)

1 Like

Not yet, but there is a feature request for it in the Podman GitHub project: