@mattdm, thank you for the response.
True, the solution space is huge. Indeed, and what you are describing, any solution (or path towards a solution) should be framed with a specific threat model. That starts with a foundation: what exactly is attacked; in this case, how would a CoreOS install look like …
The crucial thing is the balance between security and convenience; as you said, you can harden a system so well that the convenience becomes 0, or a joke (the ping times accessing that box in the middle of the sun must be atrocious … yes, thank you, I’ll be working remotely). There must be some convenience built into the model, otherwise it’ll be too hard to use. And the optimization is to make at least any compromise a convenience. (conveniences such as ssh key access that are also a big security improvement are rare)
I can see some things that CoreOS does trying to strike a balance between the two. What I was dreaming of is to get a good picture of where was a security decision, where was a convenience decision.
Maybe I can take the pain in trading off some clear convenience for more security, if is really worth it.
But I’ll be thinking twice++ of reducing security for extra convenience: unless I have some super-deep understanding of what is going on, I would rather trust CoreOS and its designers to do the right thing instead.
So, deciding what to leave as it is and what to customize and how much.
An example of the small, early decisions to be made – but they could end up later on steer significantly the direction of the whole architecture.
Take for example the default regular user account. It’s clear a lot of care has been used in setting it up. No default password login, you can simply add a ssh key and you’re good to go. Security! On the other hand, core is in %sudo in sudoers (or something to that effect), so no authentication with sudo. Convenience!
Is it worth setting a password on the default account and have sudo authenticate you ? (or, more extreme, set up an entirely new user… then make sure the account does still not allow password logins…) A lot of system administration will become more inconvenient and arbitrary automation, especially through ssh, will suffer (or at least it will require setting up a fixed list of NOPASSWD scripts … )
Then let’s pin some variables, an attempt to narrow down a mythical “minimal common” case. Single-node install. These days, some form of cloud install. Multiple containers, some having ports exposed to public access; others, not publicly exposed; both might have some sort of privately accessible communication channel that you can tunnel to. In the first category, some kind of server, most likely a web server. In the second, stuff like a database (now that I describe all this, starts to sound like a frontend and a backend…).
Given this fairly common layout, there is another major fork in the road: rootful or rootless containers. I am also focusing just on podman here, since I understood docker has some wrinkles with rootless.
There is smooth convenience running rootful: add the unit in the ignition. Done.
For rootless, there is more friction: user-level systemd units are more complicated to create, assuming that you will do all the container related setup from inside one of them. Or there is an idiomatic way to start with system units and switch to a certain user/group inside (User=, Group=); maybe I have not got deep enough down the systemd docs rabbit hole to find out.
Does the convenience biased towards rootful amount to an admission that rootful is OK enough and not the “OMG, run-don-t-walk-away-from-it” that many container articles seem to imply ? (again, in the context of podman)
And despite the advances in the convenience of running rootless, there are still limitations that sometimes pop up, usually late during the design or, worse, testing process. If you hit a wall at that point, going back and re-arranging everything to run rootful turns the rootless attempt into wasted effort.
I got a lesson in this kind of outcome when trying to change some default CoreOS settings: many hardening guides recommend moving the sshd from listening on the default port to something else. The advice is controversial, the critics’ point being that you basically decrease convenience for negligible increase in security through obscurity. Just in case, as an experiment, I tried it. It turned out that is not possible due to SELinux settings (it can be done, but in very ugly, inelegant ways), so I gave up and stuck with the default.
This is the kind of advice that would be useful: is this just a limitation of the current way of applying ignition settings or it is considered not worth supporting such an use case and the sshd on the default is the way to go ?
I guess I’ll stop here, this is already getting really long.