F42 Change Proposal: Remove fips-mode-setup (system-wide)

Project Discussion Change Proposals
,
1

Remove fips-mode-setup

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Wiki
Announced

:link: Summary

Remove the fips-mode-setup command provided by the crypto-policies package. The functionality it currently performs will be partially removed, partially replaced by automation.

:link: Owner

:link: Detailed Description

fips-mode-setup in the crypto-policies package is currently used to switch Fedora systems into FIPS mode. FIPS mode is designed to change cryptographic components in Fedora (the kernel’s crypto API, OpenSSL, NSS, GnuTLS, and libgcrypt) to comply with the United States Federal Information Processing Standards requirements for cryptographic modules, better known as FIPS 140.

We propose removing this utility. Instead, systems should be placed into FIPS mode using one of the following methods:

  • Pass fips=1 on the kernel command line of the Anaconda installer (i.e., install in FIPS mode)
  • Use the osbuild-composer image build functionality with the

[customizations]\n fips=true

customization to build an image in FIPS mode

This will remove all tooling to switching in and out of FIPS mode after installation. Switching after installation is a bad idea anyway, see below for the rationale. Switching out of FIPS mode without reinstalling does not make sense from a compliance point of view.

:link: Context information on FIPS

For compliance with these standards, the cryptographic modules must be submitted to the Cryptographic Module Validation Program run by the US National Institute of Standards and Technology. Such certifications are lengthy efforts involving a company accredited to perform review and testing before submission, the so-called Cryptographic and Security Testing Laboratories (CSTLs). It takes upwards of a year to receive a certification, and costs a mid five-figure sum per CPU architecture to do this. Due to lack of funding and the fast pace of Fedora, Fedora’s modules are not submitted. As a consequence, FIPS mode on Fedora is useful for testing only. Several distributions downstream of Fedora (RHEL, AlmaLinux, Oracle Linux, Amazon Linux), perform their own FIPS validations.

Occasionally, users think that enabling FIPS mode improves security. While running in FIPS mode does enable some additional checks and disables some older algorithms, we see FIPS mode as a compliance tool, not as a security tool. In some situations, enabling FIPS mode actually decreases security: for example, LUKS disk encryption normally uses the modern GPU-resistant Argon2 key derivation function, but because it is not FIPS-approved, it falls back to the older and generally considered weaker PBKDF2 in FIPS mode. FIPS mode also does not allow the use of Curve25519 for key exchange (used by default in TLS 1.3), or the ChaCha20-Poly1305 authenticated encryption algorithm (again, used in TLS 1.3) both of which are considered modern and good. We do not recommend running FIPS mode to “improve security”. A well-configured crypto policy is a much better tool to limit the use of legacy algorithms without the side effect of disabling newer non-US-specified algorithms.

:link: Why remove fips-mode-setup?

To correctly switch a system to FIPS mode, multiple knobs need to be adjusted:

  • The system-wide cryptographic policy needs to be FIPS or based on it
  • The fips dracut initramfs module needs to be enabled to run an integrity test (a FIPS 140 requirement) on the kernel image, followed by re-generating the initramfs
  • The kernel command line needs to contain fips=1 to switch the kernel to FIPS mode (which also enables a slower FIPS-compliant DRBG)
  • If /boot is on a separate partition, a boot=UUID=<uuid-of-boot-partition> flag needs to be added to the kernel command line for the fips dracut initramfs module to be able to find the expected message authentication code for the kernel binary.

This process is somewhat brittle due to the number of different changes that need to be made. Additionally, fips-mode-setup --disable, which only exists for testing purposes because the compliance requirements do not support switching systems out of FIPS mode, does not undo adding the fips module to the initramfs to avoid a risk that the system will no longer boot afterwards. fips-mode-setup --check subsequently reports an inconsistent configuration, which is frequently confusing users.

This user confusion is one of the reasons why we propose to remove this tool.

Instead, we will turn the fips=1 kernel command line flag into the single source of truth for whether FIPS mode is enabled. We have already removed or will remove some of the separate knobs and instead automate them to follow the single source of truth:

  • The system-wide cryptographic policy will automatically follow the fips=1 kernel command line flag to remove the need for the manual switch previously performed by fips-mode-setup. This is already implemented by a dracut module, which will add two bind-mounts before pivoting to the root filesystem if the system is in FIPS mode, but the crypto-policy is not.
  • For systems that are not started using dracut (e.g., containers with systemd), the crypto-policies package ships fips-crypto-policy-overlay.service, which runs early during boot and also applies these bind mounts. See 2325499 – systemd presets request - fips-crypto-policy-overlay.service. This change is independent of this change proposal, and we will pursue this regardless of whether the community agrees that this change is a good idea, since it does make sense as a separate incremental improvement.
  • Containers started using podman on hosts in FIPS mode did already create these bind mounts, but we are also modernizing that, see containers/common#2130 and containers/common#2174.
  • The crypto-policies package has been adjusted to detect and handle this situation during upgrades and when update-crypto-policies --set is called manually. This also improves the behavior of update-crypto-policies --set in containers in FIPS mode, which was a lot more brittle before.
  • We will enable the dracut fips module on all systems. This is required for unified kernel images anyway. The fips dracut module does nothing if the system is not in FIPS mode (i.e., fips=1 is not on the kernel command line).

The other big reason for removing fips-mode-setup is that it allows switching to FIPS mode after installation, which means that certain algorithmic choices have already been made. For example, cryptsetup will use PBKDF2 as key derivation function in FIPS setup, but Argon2 otherwise. A system with LUKS disk encryption initially set up outside of FIPS mode but switched into FIPS mode later using fips-mode-setup would end up in one of two states: either, non-compliant due to the use of Argon2, or unbootable in case the cryptographic library refuses the attempt to use Argon2. OpenSSH is also affected during installation or first system boot with its hostkey algorithm choices. Additionally, even keys that would be usable in FIPS mode must be considered tainted on systems switched to FIPS mode after installation, because key generation behaves different in FIPS mode (e.g., all RSA keys would have to be thrown away and re-generated).

For these reasons, we have come to the conclusion that the only sane way to create a system in FIPS mode, is to install it in FIPS mode. fips-mode-setup is therefore actively harmful.

:link: Feedback

Users that do not care about compliance but only run Fedora in FIPS mode for testing may not want to reinstall from scratch just for that purpose. For these cases, and when the above limitations are well understood, the fips=1 kernel command line flag can be modified directly, e.g. using

grubby --update-kernel=ALL --args=“fips=1 boot=UUID=$(blkid --output value --match-tag UUID “$(findmnt --first --noheadings -o SOURCE /boot)”)”

The boot=UUID=<partition-uuid> part of this command is only required if /boot is a separate partition, because the fips dracut initramfs module will otherwise not be able to locate the message authentication code to verify the integrity of the kernel image.

Users that do not care about compliance that want to switch systems out of FIPS mode can equally use grubby to remove the fips=1 kernel command line flag.

:link: Benefit to Fedora

  • Upstream testing on systems in FIPS mode will use consistent algorithm choices everywhere.
  • Misconfigurations such as the use of the Argon2 key derivation function for LUKS disk encryption in FIPS mode will be reduced.
  • Misconfigurations such as the use of a non-FIPS system-wide cryptographic policy in FIPS mode will automatically be avoided
  • Maintenance for fips-mode-setup can be stopped, freeing up resources for other changes

:link: Scope

  • Proposal owners: Most of the changes mentioned above are incremental improvements that made sense on their own and have already landed. Removal of the fips-mode-setup tool is the remaining step that could not be done without a change proposal and TBD.
  • Other developers: No further changes are expected from other developers. From a user point of view, FIPS mode will continue to work as it did before. The only change is on how to enable FIPS mode. The anaconda installer that used to invoke fips-mode-setup has already been changed to call /usr/libexec/fips-setup-helper provided by the crypto-policies package instead.
  • Release engineering: #12472
  • Policies and guidelines: not needed for this Change
  • Trademark approval: not needed for this Change
  • Alignment with the Fedora Strategy: This simplifies the ecosystem connections because it reduces the number of downstream changes.

:link: Upgrade/compatibility impact

Systems in FIPS mode that were configured to not use the FIPS system-wide cryptographic policy will see their policy automatically be changed to FIPS mode. Running in FIPS mode with a non-FIPS crypto-policy is considered a misconfiguration.

Scripts that used fips-mode-setup will stop working after the upgrade.

:link: Early Testing (Optional)

Do you require ‘QA Blueprint’ support? No

:link: How To Test

Testers that will try the three methods outlined above to switch to FIPS mode are welcome. From downstream testing, we expect no issues.

Using dnf reinstall crypto-policies on a system that has the two bind-mounts for the FIPS crypto-policy (i.e., that’s in FIPS mode but its crypto-policy was not manually switched to FIPS) should pass without issue, and the files in /etc/crypto-policies/back-ends should correspond to those in /usr/share/crypto-policies/back-ends/FIPS. We also expect no issues here from downstream testing.

Testers that run Fedora containers with systemd on hosts in FIPS mode (i.e., /proc/sys/crypto/fips_enabled contains 1) both with podman and with other container runtimes are very welcome.

:link: User Experience

Users will notice that fips-mode-setup no longer exists. Users that want to switch systems to FIPS mode need to follow different steps.

At the same time, switching to FIPS mode now comes with fewer pitfalls and fewer possibilities for misconfigurations and inconsistencies.

:link: Dependencies

The following packages have already adapted to this potential change and will continue to work:

  • anaconda
  • dracut
  • podman
  • crypto-policies (in particular update-crypto-policies and the package %pre and %post scriptlets)

:link: Contingency Plan

  • Contingency mechanism: We will keep fips-mode-setup and try again next release
  • Contingency deadline: beta freeze (or even later, if required)
  • Blocks release? No

:link: Documentation

:link: Release Notes

The fips-mode-setup command that could be used to switch systems to FIPS mode after installation has been removed.

Switching to FIPS mode after installation has subtle bugs and is no longer recommended. To switch systems to FIPS mode,

  • add fips=1 to the kernel command line of the installer
  • use osbuild-composer with the following customization to build a FIPS-enabled image:

[customizations] fips = true

To disable FIPS mode, re-install your system without enabling FIPS mode. Disabling FIPS mode without reinstalling does not make sense from a compliance point of view.

Last edited by @amoloney 2024-12-03T17:55:51Z

Last edited by @amoloney 2024-12-03T17:55:51Z

1 Like
2

How do you feel about the proposal as written?

  • Strongly in favor
  • In favor, with reservations
  • Neutral
  • Opposed, but could be convinced
  • Strongly opposed
0 voters

If you are in favor but have reservations, or are opposed but something could change your mind, please explain in a reply.

We want everyone to be heard, but many posts repeating the same thing actually makes that harder. If you have something new to say, please say it. If, instead, you find someone has already covered what you’d like to express, please simply give that post a :heart: instead of reiterating. You can even do this by email, by replying with the heart emoji or just “+1”. This will make long topics easier to follow.

Please note that this is an advisory “straw poll” meant to gauge sentiment. It isn’t a vote or a scientific survey. See About the Change Proposals category for more about the Change Process and moderation policy.

3

The fips-mode-setup --check and fips-mode-setup --is-enabled options still seem potentially useful, at least for some transition period.

4

Checking whether FIPS mode is enabled can be done with cat /proc/sys/crypto/fips_enabled instead. fips-mode-setup --check is especially brittle because of the many knobs it checks, and its presence is one of the reasons why we propose removing it, because it often leads to confusion on user’s ends.

5

I’m torn on this one.

Personal me understands completely where you guys are coming from with this change proposal, and on the surface it seems to make complete sense. Make things simpler, reduce confusion, reduce upkeep. Ever since I joined the community I have acquired an ever increasing appreciation for the amount of effort that goes into making Fedora, so naturally I empathize with the desire to simplify and streamline.

Professional me had a mild heart attack. I work for a defense contractor and administer multiple blended Windows/RHEL environments (and many other infra platforms underneath and in between). I STIG. HARD. Compliance is a huge part of my day to day life. I also have to balance compliance directives against reality:

Many things don’t work on “fully compliant” systems.

Government enclaves have a process for dealing with areas where they need to be non-compliant as a matter of reality. It shows up as a ding on their compliance paperwork, but the administrators and operators document the rationale and move on with life. Far more common, however, is the need for administrators to partially or temporarily deviate from compliance, execute a series of critical tasks, and then restore a system to its compliant state.

For example, we all know and love SELinux. It would not be uncommon for various teams to have Ansible playbooks wrapped with a “Make SELinux Permissive” task and an “Always” task to put the system back in Enforcing mode for certain situational workloads. It’s just “in the way” while that task is being performed, and most people would rather not permanently relax the compliance posture. Another example: Windows systems getting software updates or new software installs may have their antivirus temporarily disabled so that it doesn’t interfere.

It is common practice for various compliance controls to be temporarily disabled so that some process can complete. Systems are typically re-scanned for compliance afterwards.

I have seen software, notably Hashicorp Vault, that would not install while the system was in FIPS mode. It’s been a while, but I think it had something to do with either the RPM or some component being signed in a weak manner. This may not even be the case anymore, but it was a very real problem for my team at the time. Anyways, a fairly direct approach might be to alter the automation for installing and updating Vault (because it has to be done carefully) to peel the system out of FIPS mode, reboot it, update and reconfigure the software, put the system back in FIPS mode, reboot again, and validate the FIPS configuration.

The fips-mode-setup allows the administrator to ‘pop the hood’ for this particular compliance measure while engaging in maintenance activities. Imagine trying to check your oil if someone came by and decided to just weld the hood shut. Could it be done? Probably, but we’d have to go through some tedious, complicated, and potentially costly and/or disruptive measures to get there.

I agree with your statement that popping in and out of FIPS mode doesn’t make sense from a compliance perspective, and so would every Information System Security Manager (ISSM) and Authorizing Official (AO) out there. They don’t have to deal with the reality of administering and maintaining said systems, let alone entire populations of them.

I would be a lot more comfortable with this proposal if it still provided a way for people to deviate from compliance, execute maintenance tasks, and put everything back into compliance afterwards.

6

I get that there are scenarios where switching to FIPS mode after installation is problematic, but there are also likely scenarios where switching to FIPS mode after installation is fine & valid. Removing ‘fips-mode-setup’ solves the former problems, but breaks the the latter.

“Time of installation” is not as simple a concept as one might think, in non-bare metal worlds. For example, VM disk images can often be thought of as having a two phase installation process, first you build the general disk image template offline, and then at time of VM deployment the template is specialized for a particular VM instance, but this change is implicitly suggesting that FIPS has to be enabled at the time the disk image is built, not deployed. This would potentially imply that anyone shipping VM disk image templates needs to build & ship duplicates of each with FIPS enabled.

In the case of VM disk images, the generic templates would NOT be expected to have any OpenSSH host key present. In terms of LUKS, on first boot the disk image would undergo a re-encrypt process to either add LUKS to the plain rootfs, or at least to replace the primary key, giving a chance to set the keyslot alg choice to PDKDF2 instead of Argon2.

Rather than purging fips-mode-setup entirely, has thought been given to making it validate known problem areas. ie have it check if OpenSSH chose a non-compliant key alg, or if LUKS chose a non-compliant key alg. Refuse to switch to FIPS mode if the config is unsound, but otherwise proceed. This wouldn’t be foolproof, but it would stop the common foot-guns, and retain functionality for users whom know what they’re doing.

It feels like config validation would facilitate Paul’s scenario above where a FIPS enabled system has to be temporarily switched out of FIPS mode to perform some action, and then re-instated.

7

@aggraxis I understand where you’re coming from. I believe there are options for you even after fips-mode-setup has been removed:

  1. You can use a custom subpolicy of the FIPS system-wide cryptographic policy to re-enable certain features; this is already common practice e.g. with the FIPS:AD-SUPPORT policy module and other similar modules.
  2. You can still enable and disable FIPS mode by modifying the fips=1 kernel command line argument. This is explained in the first paragraphs of the Feedback section of the proposal at Changes/RemoveFipsModeSetup - Fedora Project Wiki.
  3. You can “virtually” disable FIPS mode by bind-mounting a file containing 0 over /proc/sys/crypto/fips_enabled.
1 Like
8

@berrange We are aware of the two-stage build of, for example, VM images. We are even recommending the use of osbuild-composer and bootc in the proposal. You do not need to build two sets of images, one for FIPS and one without, but it is an option to simplify deployment of compliant systems (and in some cases even required, because you can, for example, not specify modified kernel command line arguments when booting a VM on AWS as far as I know).

If you have an image that does not contain OpenSSH host keys and will set up LUKS on the first boot, it is absolutely enough to just boot that with fips=1 for the first boot, since the problematic algorithm choices have not been made. However, if your first boot steps contained fips-mode-setup --enable followed by OpenSSH key generation and LUKS key setup without a reboot in between, it would have used the wrong algorithms until now, but will use the correct algorithms after the change.

Rather than purging fips-mode-setup entirely, has thought been given to making it validate known problem areas. ie have it check if OpenSSH chose a non-compliant key alg, or if LUKS chose a non-compliant key alg. Refuse to switch to FIPS mode if the config is unsound, but otherwise proceed. This wouldn’t be foolproof, but it would stop the common foot-guns, and retain functionality for users whom know what they’re doing.

Thought has been given to this, but it is

  • more effort to write custom handlers for these cases
  • an area of silent failure if we miss adding a handler for a set of keys or algorithm choices

The last part is especially problematic, since there is no central record of what keys have been generated on a system and where they are stored. We do have a check for the LUKS key slot added in RHEL 9.5, but we do not consider this a maintainable solution for the future.

It feels like config validation would facilitate Paul’s scenario above where a FIPS enabled system has to be temporarily switched out of FIPS mode to perform some action, and then re-instated.

I believe I have provided suitable alternatives for Paul’s scenario. On the other hand, I do not think that we should make operations that potentially break compliance easier. We do not want users to accidentally violate compliance requirements.

9

Modifying the kernel command line parameter to fips=1 and rebooting will be sufficient for our typical deployments. I have no other reservations about the change proposal as written.

10

Note that another option to set the fips=1 kernel argument early during boot, before any keys are generated, is to use ignition, which runs in the initramfs contrary to cloud-init which runs later. This could be adopted for cloud images to avoid having to create two disk images.

1 Like
11

@siosm I don’t think that will be enough, since the kernel also switches behavior depending on whether fips=1 is on the command line, so the Ignition run would not be in FIPS mode, i.e., it couldn’t generate compliant keys. One of the changes in the kernel is a different random number generator, and keys generated with the kernel’s default RNG cannot be considered FIPS-compliant.

Ignition could potentially do this correctly if it added the kernel command line flag as the first operation and rebooted again. Is that what is happening?

12

Yes :slight_smile:

13

Well, it’s not exactly ignition doing it but related to it (the option comes via the ignition config): os/overlay.d/05rhcos/usr/lib/dracut/modules.d/40rhcos-fips/rhcos-fips.sh at c248ce256df8573f5499c4f9bdb2fd29eb4bdcef · openshift/os · GitHub

We’ve been looking at fully integrating it in ignition: Add `fips: true` · Issue #1323 · coreos/ignition · GitHub

14

@siosm This is actually great news! This is a nice and simple way to enable FIPS mode that doesn’t require editing the kernel command line manually. Thank you!

2 Likes
15

This change proposal has now been submitted to FESCo with ticket #3303 for voting.

To find out more, please visit our Changes Policy documentation.