Social engineering attack -> no best practices in crypto on website: how to contact website team to solve that? (no immediately exploitable vulnerability, but practical in future in some circumstances)

Project Discussion
,
1

The user @dobymick made us aware that there is an issue with the verification of signatures in the way we provide ISO files on our servers.

It is a best practice to ensure end to end verification to our users, which means, the users should be able to verify that the ISO files they get from the mirrors are equal to the files we originally provided to the mirrors.

The current process is prone to a manipulation that many users are likely to not identify:

ISOs and CHECKSUM files are currently provided both by the mirrors. This means, the mirrors could change both. I do not fear about the trustworthiness of the organizations that currently provide Fedora or the https/x509v3 architecture, but the possibility to change files makes every mirror to a single point of failure → this leads to a big attack surface, and the more widespread Fedora gets, the more attractive our mirrors become for attacks.

The issue: we offer users on the website the possibility to verify the CHECKSUM file with our gpg file, but the integrated signature only verifies the HEADER.

If the attacker would manipulate an ISO, and then just add the manipulated ISO checksum above the header, the verification will be successful as the malicious checksum is outside the header (thus ignored by the current type of gpg verification), whereas many (if not most users) would in the subsequent step consider the malicious’ ISOs hash in the verification process. As some of our CHECKSUM files have many files contained, a user might not see at first glance that an additional one is contained and that their ISO actually does not fit an hash that is inside the header but outside (the warning signs are easily overlooked in all that text, and at the top: it says manipulated ISO = OK). I guess many users don’t know about headers and such at all.

Therefore, the suggestion is to provide only CHECKSUM files without a contained signature in future, and provide detached signatures for them on the mirrors.

This social engineering attack also illustrates that we suggest a very complex process for verification on our website, which in this context cannot be considered a best practice. This might be improved as well.

A slight addition: it might be added that the average user might not be attracted by the little “verify” button and likely doesn’t know much what this is about anyway (people avoid to click on things that do not make sense to them at first glance).

This is NOT an immediate attack vector, but it creates a large attack surface and makes mirrors more attractive for attackers as long as average users (hyperbolic speaking) ain’t cryptographers and as long as Fedora is in widespread use: maybe there could be some improvement discussed, to avoid making mirrors attractive for attacks, to protect users if they get Fedora without being experts, and to lead the community with good practices: people tend to adopt what they see.

Off the cuff, I do not know what team is currently responsible for that. Maybe someone from commops/mindshare/marketing can help and let me know? May you can let me know what tag to add, or where I shall open a ticket

I just reproduced and documented …

… a full example of Doby Mick’s social engineering attack: assuming I have captured one or more mirrors that users use to get Fedora ISOs:

I focus on KDE because it is widespread, and because the CHECKSUM file contains many ISO files and thus causes a lot of text that distracts users.

  1. I captured a mirror, let’s assume I found any administrator access in any mirror (let’s assume the admin’s password was t00r - my favorite!)
  2. I created a malicious ISO of Fedora KDE: In reality, I took our ISO and did echo "thisIsBackdoored" >> Fedora-hackedKDE-Live-x86_64-40-1.14.iso
  3. I created the sha256sum of this backdoored Fedora KDE, and I have added this to the CHECKSUM file that Fedora provides through the mirrors, but I added it to the top ABOVE the header. Here is the malicious CHECKSUM file that is now on our corrupted mirror:
# Fedora-KDE-Live-x86_64-40-1.14.iso: 2645645312 bytes
SHA256 (Fedora-hackedKDE-Live-x86_64-40-1.14.iso) = f0c25a9da8f3ee430aa233c664f6e5e2abe13d245e3975d30513ff2503b19017

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

# Fedora-Budgie-Live-x86_64-40-1.14.iso: 2146633728 bytes
SHA256 (Fedora-Budgie-Live-x86_64-40-1.14.iso) = 18a2876d031fd58d00d9a37d3a823caa03cf7059309bed1de79eb3a46dc4bc21
# Fedora-Cinnamon-Live-x86_64-40-1.14.iso: 2558238720 bytes
SHA256 (Fedora-Cinnamon-Live-x86_64-40-1.14.iso) = 081ac2402d3e2e704d36ee6aacd67ba631d1939e0c35cf517e996dbb70117735
# Fedora-KDE-Live-x86_64-40-1.14.iso: 2645645312 bytes
SHA256 (Fedora-KDE-Live-x86_64-40-1.14.iso) = 8b9da20cfa947b16c8f0c5187e9b4389e13821e31b062688a48cf1b3028c335c
# Fedora-LXDE-Live-x86_64-40-1.14.iso: 1741191168 bytes
SHA256 (Fedora-LXDE-Live-x86_64-40-1.14.iso) = 3be326068be6bd7e98d7e7e89f4427648fe83484ecd36e1e77304af1369d1920
# Fedora-LXQt-Live-x86_64-40-1.14.iso: 1845794816 bytes
SHA256 (Fedora-LXQt-Live-x86_64-40-1.14.iso) = ef0fbed423acc7484b6fea14a062c41a3026ca93d71edf28debe2d883f9ce61c
# Fedora-MATE_Compiz-Live-x86_64-40-1.14.iso: 2454198272 bytes
SHA256 (Fedora-MATE_Compiz-Live-x86_64-40-1.14.iso) = 3242eb53dd2d0bad66dff04f7e54848aa750910171a110e2f8677aa23b8e84e0
# Fedora-SoaS-Live-x86_64-40-1.14.iso: 1440874496 bytes
SHA256 (Fedora-SoaS-Live-x86_64-40-1.14.iso) = 4287b380b4be2f2c421178b9dfdcaf8c64ed58e343baa7d1d482c4f3481975fb
# Fedora-Sway-Live-x86_64-40-1.14.iso: 1637679104 bytes
SHA256 (Fedora-Sway-Live-x86_64-40-1.14.iso) = 6974428b65153156e133fe0165cdf5cb4420b987fe27c53218480414b685e602
# Fedora-Xfce-Live-x86_64-40-1.14.iso: 1889988608 bytes
SHA256 (Fedora-Xfce-Live-x86_64-40-1.14.iso) = 519db49a21587c007b8135cfc8fba470951937d2f7edc01a8f4b96abc772370c
# Fedora-i3-Live-x86_64-40-1.14.iso: 1617053696 bytes
SHA256 (Fedora-i3-Live-x86_64-40-1.14.iso) = 1f55028b79c6633178a0106fdb3d34ccb489f1dc039c5e96a829cea28624d7a8
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEEV35rvhXhT7oRF0KBydwfqFbecwFAmYisCQACgkQBydwfqFb
ecwIsQ//R7hHyRG1csTnL5WtYMM4Rq3fequ7CFDIvL5iywNXEPed7vL7Z9DG9qU1
eiNIaWbElYy58tBG6JQVf2ucqehbYqgVI/rXqQHbs+XpHTu6TtRB+BHdKEbWmQNf
QCzdqyzSVhIz24A3XM7Ya4bDGA3LdR5VUCACGqx8VI5+wYJFySSQixCENEusbCtG
a7G0eqhkMzwVfOsCTvX1EmZ6akQHcWZV9bvOKLqq+i5uXwzEzHlY7Jskt86QtSdR
6i/h30BKDmKw8UTrtftQ0Ayt/YNQw1EezeC6wHx8bRH0Zi1S2M33KQou5VXAfa2N
H7ePUWs89Rpq4mjrjavGsIiZf5e7QuNiIPJOr4k6yzxXCDTejKIBWtNHiFdaEYMy
eW+/CknTgoVmU1ry5SZ7nuaECY6kUYnqFXrPWfwGTEcvnaC2JUVqMH9Mgpd/kZIF
JamLBiF66GuanyFThxuD1MMJX6hUwcNzANMzMTXanVqeTz/gs+40r3adUqbpZz4f
kMUUuOW07IhLGp190Skyeg/AkYEJFdJo5eutCD7Lb0OKRmpjGtPddxpxjxDVhfVc
a6qcFM+FhZ7sRe+l0GnX0S8hNm1QijcpqmgX2zMbYMsulwcVIUjquOM3NJipqx8o
xcXip44A7xpt6NlJTcD5ku/Qp7MrTHfpKMknwUoQDtr9oi0Ajb0=
=O9rW
-----END PGP SIGNATURE-----
  1. As a average user, I now follow the elaboration of Fedora on their website (go to the KDE download page and click the verify button: then you get the current HowTo.
  2. Given the HowTo, I already downloaded the CHECKSUM file and the ISO from the mirror (when I tested, I always got the same mirror for both - this can be expected to apply if people download both at the roughly same time from the same place).
  3. I do curl -O https://fedoraproject.org/fedora.gpg
  4. I do gpgv --keyring ./fedora.gpg Fedora-Spins-40-1.14-x86_64-CHECKSUM with the manipulated CHECKSUM file, but the result is TRUE / SUCCESSFUL because only the header is considered and everything above ignored:
gpgv: Signature made Fri Apr 19 19:55:48 2024 CEST
gpgv:                using RSA key 115DF9AEF857853EE8445D0A0727707EA15B79CC
gpgv: Good signature from "Fedora (40) <fedora-40-primary@fedoraproject.org>"
  1. I now think the file is fine, so I proceed with sha256sum -c Fedora-Spins-40-1.14-x86_64-CHECKSUM, and I get the following output:
Fedora-hackedKDE-Live-x86_64-40-1.14.iso: OK
sha256sum: Fedora-Budgie-Live-x86_64-40-1.14.iso: No such file or directory
Fedora-Budgie-Live-x86_64-40-1.14.iso: FAILED open or read
sha256sum: Fedora-Cinnamon-Live-x86_64-40-1.14.iso: No such file or directory
Fedora-Cinnamon-Live-x86_64-40-1.14.iso: FAILED open or read
Fedora-KDE-Live-x86_64-40-1.14.iso: OK
sha256sum: Fedora-LXDE-Live-x86_64-40-1.14.iso: No such file or directory
Fedora-LXDE-Live-x86_64-40-1.14.iso: FAILED open or read
sha256sum: Fedora-LXQt-Live-x86_64-40-1.14.iso: No such file or directory
Fedora-LXQt-Live-x86_64-40-1.14.iso: FAILED open or read
sha256sum: Fedora-MATE_Compiz-Live-x86_64-40-1.14.iso: No such file or directory
Fedora-MATE_Compiz-Live-x86_64-40-1.14.iso: FAILED open or read
sha256sum: Fedora-SoaS-Live-x86_64-40-1.14.iso: No such file or directory
Fedora-SoaS-Live-x86_64-40-1.14.iso: FAILED open or read
sha256sum: Fedora-Sway-Live-x86_64-40-1.14.iso: No such file or directory
Fedora-Sway-Live-x86_64-40-1.14.iso: FAILED open or read
sha256sum: Fedora-Xfce-Live-x86_64-40-1.14.iso: No such file or directory
Fedora-Xfce-Live-x86_64-40-1.14.iso: FAILED open or read
sha256sum: Fedora-i3-Live-x86_64-40-1.14.iso: No such file or directory
Fedora-i3-Live-x86_64-40-1.14.iso: FAILED open or read
sha256sum: WARNING: 17 lines are improperly formatted
sha256sum: WARNING: 9 listed files could not be read

→ consider the first line: Fedora-hackedKDE-Live-x86_64-40-1.14.iso: OK
→ I would like to say that the WARNINGS about formatting and such are caused by the social engineering attack, but unfortunately, our original CHECKSUM file causes the same issue :frowning:
→ Therefore, if a user does not know about the implications of the different commands we suggest and their impact on each other (and what they actually do and not do), and trusts these outputs, they might get a malicious ISO and even think it was successfully verified
→ Additional note: The normal KDE is “ok” as well because I have it in the same directory as well.

The suggestion would be to replace this method already by F41, or at least by F42 release: We might discuss a best practice that is less error prone and easier to be conducted for average users - one example is the one suggested above. @dobymick maybe wants to add further points.

In any case, thanks to @dobymick for making us aware, and for providing the idea of using detached signatures!

4 Likes
2

Added mindshare

3

Added marketing-team

4

An alternative workflow needs to be simple to end users or machinery too. If we steer people toward using gpg to verify ISO files (a good idea!), we need their gpg’s to Just Work with respect to these signatures. i.e., we should preset the fedora keys/certificates somehow.

5

Would it be any better/simpler to have the user run sha256sum (or similar) on the ISO and have them search for the checksum on our website much the way we do with the GPG fingerprints (Fedora keeps you safe | The Fedora Project)?

6

Maybe, but that’d require internet access, and (in the worst case) searching through a list of hexadecimal blobs for a match.

7

We shortly discussed that idea in the other topic. Indeed, I liked the idea to also have just the sha256sums on our website: so, when users click the “verify” button, they might also get the sha256sums. sha256 is cryptographically secure, and the hash can be trusted, and if the user can trust getfedora.org, they can trust that hash as well if it is from our website.

However, this is just an addition for which I would be +1 in any case, but we still should provide the signatures at the mirrors because users get fedora not just from our website, but also other pages link to the mirrors, whereas the mirrors can be accessed directly. We cannot control nor determine if and how often users end up at the mirrors without “passing” our website (there are some linux magazines or so that might link to mirrors or so - we cannot control that).

So I would like to do both, and maybe make the “verify” button more visible and present it in a way that encourages users to check out what is contained.

If they are on the website, it should be fine to emphasize the sha256sums in the HowTo (which opens when clicking the “verify” button; obviously, in that case, users “pass” us), and add a button within the HowTo box to get the gpg file or so, or make it a separate button - whatever.

We have to keep in mind that the signature files in the mirrors also add a psychological dimension to average users, as many have no real awareness of potential impacts: it leads to the question what is this and why is it there.

Also, we effectively follow that way the practice that seems to have become common. So I would like a combination.

1 Like
8

It all sounds reasonable to me. :slightly_smiling_face: I’d say just open some issues (or MRs) at fedora / Fedora Websites and Apps / Fedora Websites / fedora-websites-3.0 · GitLab once you’ve worked out a better system and the Websites and Apps team will review the ideas and implement them if they seem reasonable. Thanks.

3 Likes
9

Thanks for the link! That’s what I was looking for :classic_smiley:

10

Please add instructions on how a Windows user can check the checksum.
I’m thinking of people that are going to dual boot with Fedora.
There is a built in windows tool that will calculate the sha256.

1 Like
11

Good point! The same for Mac.

Interesting. I didn’t know that :open_mouth:

12

Thank you Chris for taking my concerns seriously, especially the social engineering attack I described in my part 2 post. I agree entirely with your comments about the challenges between security and ease of use.

I would like to write some documentation about using GnuPG to verify Fedora Linux ISO files. I am very knowledgeable of GnuPG and the OpenPGP standard as defined in RFC 4880.

I believe the ideal solution would be to produce a detached signature over the checksum file like we discussed. I will write more later, to describe a verification procedure that is both secure and easy to use.

If you want to work on this and contribute, I am happy to check out for you what team is managing this, if you want to join them. Shall I check out for you? If you contribute to the maintenance, you might achieve to provide more in this respect, although you will have to be able to make compromises to not cause a denial of service to average users, whom we also serve here (In short, we prefer them to have a SELinux that protects the most sensitive areas of their system but not much more, rather than they have no SELinux at all :classic_smiley: ).

Thank you I would really like to work on this and to join the team managing signing the releases. Thank you for your help.

2 Likes
13

100%.

Let me try to roughly sum up the current suggestion we elaborated here:

  • Add the detached signature to the existing ISO and CHECKSUM on the servers for all those who end up on the mirrors without passing us (and to raise some user awareness about these signature files),
  • and change the content of the “verify” button to contain directly the sha256sums, a short explanation to use it (1 command), but also provide elaboration for Windows/Mac (I guess on average-user-linux-systems we can expect sha256sum to be available),
  • and additionally, the “verify” box should still contain at the bottom (not the emphasis for those who are on our website) either a simple explanation or a link about how to use the signature files as well, just that users also know why these files are there (if something is there, something should be said about it, that avoids uncertainty for users who might think if they did something wrong if there are files they don’t use if they pass our website).

Does that as rough summary for a solution make sense? Other thoughts/alternatives?

If you want, you can open the ticket at the website team, Gregory mentioned the link above. I admit, I would be happy if I could pass that to someone else as I would need to focus something else atm :classic_smiley:

If you do, please mention the discussion in the topic here and also let us know the link to the ticket here, in order to ensure that both are connected and can consider each other: then readers from both sides can read posts in both, and might add relevant thoughts & perspectives.

I suggest to work with the holistic attack scenario I elaborated above when you explain the case: I assume not everyone reading the ticket has a crypto & gnupg background, and then a strong & reproducible context makes it easier to understand the impact of the issue, and when and where changes are necessary. It also ensures the same context in both topics/tickets.

Feel free to let them know in the ticket that you are interested in joining :classic_smiley: I am not sure who does the signatures, but surely they will know who does.

14

Note: Edited this message entirely because I was incorrect in my first draft.

In Fedora CoreOS, a detached signature is computed over the Fedora CoreOS release files. A checksum file is provided, but it is not protected by any sort of GPG signature.

15

Hey folks. I am not sure where to reply to this topic since there’s
several threads and lots of discussion, so I’ll just chime in here at
the end of this topic. :wink:

First, thanks for bringing this up! Things in open source improve
because people notice/propose changes and advocate for them and
convince/get changes in.

I think detached signatures are worth considering, but they may be
tooling, proccess and documentation needed. definitely worth a wider
discussion.

Finally, I personally wish we could get away from using/needing to have
our users use gpg. I’ve used it since the 90’s and it was a wonderous
thing back then, but these days it’s just not something we should be
asking our users to deal with. I think this post from Moxie Marlinspike
sums up my feelings aound it now:

We have been looking at a new build system/setup ( konflux ) and if we
do decide to move to that, perhaps we could also retool our verfication
setup and use something like sigstore ( https://www.sigstore.dev/ )

Anyhow, shorter term we could definitely look into what it would take to
do detached signatures.

2 Likes
16

I have the exact same sentiment.

I am not familiar with konflux or sigstore. I will take a deeper look later today. We could also use minisign as a simpler drop in replacement for GnuPG. Unfortunately, minisign for Windows does not currently have an embedded X.509 signature.

I am not familiar with Fedora Project developers / maintainers. I assume you work with the release signing team?

17

In many cases I agree that it is useless to use gpg. However, having an end to end verification makes sense to limit attack vectors and make attack vectors less attractive, and increase (from attackers’ perspectives) risk of manipulations getting identified, and thus also make mirrors less interesting for attacks.

This is more a statistical means: if 1 of 100 users who download a file from a mirror verify the integrity, there is a good chance that 1 of the 10 in 1000 makes us aware of the issue (or the mirror provider). Thus, if Fedora is so popular at some time that 100.000 users download from that mirror (which means it is attractive for attackers to capture a mirror), we already have 1000 people that identified the issue, and it is likely that 1 of them makes us aware. In any case, any attacker has to expect that (higher risk for a shorter/smaller return on investment).

I already made the point in the other topic that imho the major protection is at the moment the tls/x509v3 around https. But providing a means that allows users to verify reliably end to end verification should be contained, for the given reasons. So far, gpg seems to be the best compromise for that so far (I don’t like gpg too - for several reasons): therefore, …

… I would be happy to consider alternatives on the mid/long term :classic_smiley:

If I find some time in the next days I might look into this. Don’t know that yet. Thanks for making aware :wink:

18

I am not familiar with konflux or sigstore. I will take a deeper look later today. We could also use minisign as a simpler drop in replacement for GnuPG. Unfortunately, minisign for Windows does not currently have an embedded X.509 signature.

yeah, always worth looking at options, even if they aren’t fully
acceptable.

I am not familiar with Fedora Project developers / maintainers. I assume you work with the release signing team?

Yeah, my job at Red Hat is to help maintain the Fedora Infrastructure,
but I have been involved in release engineering in my ‘spare time’ for
longer than I have worked on infrastructure. :slight_smile:

2 Likes
19

I don’t know how to use GPG. I have tried many times and have never been able to. Trying to use GPG makes me feel stupid.

I do know enough to get a SHA256 sum, especially in KDE where it’s in the right click menu of a file.

Currently, I click the checkmark button on the download page and it tells me to use GPG to verify it against keys that are fetched from fedoraproject.org.

I think it’s easier to skip all that and just post the checksums directly on the download page as text.

1 Like
20

Understandable. Most cryptographic tools require at least a basic background in cryptography. It does not help that GnuPG is massively complex; the manual is several dozen pages.

The point of “all that” is to provide end to end verification. We can post the checksums directly on the download page as an easier means of integrity checking, but GnuPG (or its replacement: minisign, signify, etc.) is here to stay.