A lot of [SSL certificate problem: certificate has expired] when running "dnf update"

[marco@t420-tovis ~]$>ls -l /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
-r--r--r--. 1 root root 221037 20 mar 23.21 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

…nothing…

[marco@t420-tovis ~]$>rpm -Va \*curl\*
[marco@t420-tovis ~]$>

Done (with chattr instead of chmod); let’s see…

[marco@t420-tovis ~]$>sudo chattr +i  /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
[sudo] password for marco: 
[marco@t420-tovis ~]$>
[marco@t420-tovis ~]$>lsattr  /etc/pki/ca-trust/extracted/pem/
----i---------e------- /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
--------------e------- /etc/pki/ca-trust/extracted/pem/objsign-ca-bundle.pem
--------------e------- /etc/pki/ca-trust/extracted/pem/README
--------------e------- /etc/pki/ca-trust/extracted/pem/email-ca-bundle.pem
--------------e------- /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.sav
[marco@t420-tovis ~]$>

1 Like

OK, and what about the ca-certificates package which includes these files:

$ rpm -qf /etc/pki/ca-trust/
ca-certificates-2021.2.52-3.fc36.noarch

$ rpm -Va \*ca-certificates\*

We’re just trying to see if these files are different from what the Fedora package provides given that you’ve noted that they’ve been changed/corrupted. If they are, we need to see what is modifying them.

Here you go:

[marco@t420-tovis ~]$>rpm -qf /etc/pki/ca-trust/
ca-certificates-2021.2.52-1.0.fc35.noarch
[marco@t420-tovis ~]$>
[marco@t420-tovis ~]$>rpm -Va \*ca-certificates\*
[marco@t420-tovis ~]$>
1 Like

In that case these files are as they should be, at least at this point in time but I guess that is expected if you’ve fixed your issue recently. It’ll be good to run these checks again when you think they’ve been changed so we can try to figure out what’s changing them.

Other checks:

  • that this package is from the Fedora repos and that no third party repo is providing it and thus overwriting files on an update:
sudo dnf list \*ca-certificates\*
Installed Packages
ca-certificates.noarch                                             2021.2.52-3.fc36                                              @fedora

It should only return a package from Fedora for you too

  • that there isn’t another package (not ca-certificates) that is also providing these files. On my F36 where I’m not seeing these issues, these are the only packages that touch the files in the folder:
$ sudo dnf whatprovides '/etc/pki/ca-trust/*'
ca-certificates-2021.2.52-3.fc36.noarch : The Mozilla CA root certificate bundle
Repo        : @System
Matched from:
Filename    : /etc/pki/ca-trust/README
Filename    : /etc/pki/ca-trust/ca-legacy.conf
Filename    : /etc/pki/ca-trust/extracted
Filename    : /etc/pki/ca-trust/extracted/README
Filename    : /etc/pki/ca-trust/extracted/edk2/README
Filename    : /etc/pki/ca-trust/extracted/edk2/cacerts.bin
Filename    : /etc/pki/ca-trust/extracted/java
Filename    : /etc/pki/ca-trust/extracted/java/README
Filename    : /etc/pki/ca-trust/extracted/java/cacerts
Filename    : /etc/pki/ca-trust/extracted/openssl
Filename    : /etc/pki/ca-trust/extracted/openssl/README
Filename    : /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
Filename    : /etc/pki/ca-trust/extracted/pem
Filename    : /etc/pki/ca-trust/extracted/pem/README
Filename    : /etc/pki/ca-trust/extracted/pem/email-ca-bundle.pem
Filename    : /etc/pki/ca-trust/extracted/pem/objsign-ca-bundle.pem
Filename    : /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
Filename    : /etc/pki/ca-trust/source
Filename    : /etc/pki/ca-trust/source/README
Filename    : /etc/pki/ca-trust/source/anchors
Filename    : /etc/pki/ca-trust/source/blacklist
Filename    : /etc/pki/ca-trust/source/blocklist
Filename    : /etc/pki/ca-trust/source/ca-bundle.legacy.crt

ca-certificates-2021.2.52-3.fc36.noarch : The Mozilla CA root certificate bundle
Repo        : fedora
Matched from:
Filename    : /etc/pki/ca-trust/README
Filename    : /etc/pki/ca-trust/ca-legacy.conf
Filename    : /etc/pki/ca-trust/extracted
Filename    : /etc/pki/ca-trust/extracted/README
Filename    : /etc/pki/ca-trust/extracted/edk2/README
Filename    : /etc/pki/ca-trust/extracted/edk2/cacerts.bin
Filename    : /etc/pki/ca-trust/extracted/java
Filename    : /etc/pki/ca-trust/extracted/java/README
Filename    : /etc/pki/ca-trust/extracted/java/cacerts
Filename    : /etc/pki/ca-trust/extracted/openssl
Filename    : /etc/pki/ca-trust/extracted/openssl/README
Filename    : /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
Filename    : /etc/pki/ca-trust/extracted/pem
Filename    : /etc/pki/ca-trust/extracted/pem/README
Filename    : /etc/pki/ca-trust/extracted/pem/email-ca-bundle.pem
Filename    : /etc/pki/ca-trust/extracted/pem/objsign-ca-bundle.pem
Filename    : /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
Filename    : /etc/pki/ca-trust/source
Filename    : /etc/pki/ca-trust/source/README
Filename    : /etc/pki/ca-trust/source/anchors
Filename    : /etc/pki/ca-trust/source/blacklist
Filename    : /etc/pki/ca-trust/source/blocklist
Filename    : /etc/pki/ca-trust/source/ca-bundle.legacy.crt

freeipa-client-common-4.9.8-3.fc36.noarch : Common files used by IPA client
Repo        : fedora
Matched from:
Filename    : /etc/pki/ca-trust/source/ipa.p11-kit

tog-pegasus-2:2.14.1-65.fc36.x86_64 : OpenPegasus WBEM Services for Linux
Repo        : fedora
Matched from:
Filename    : /etc/pki/ca-trust/source/anchors/localhost-pegasus.pem

OK; so I guess the “immutable” flag must be removed (see @kpfleming posts).
These are the checks results:

[marco@t420-tovis ~]$>sudo dnf list \*ca-certificates\*
Last metadata expiration check: 1:05:34 ago on ven 25 mar 2022, 09:57:28.
Installed Packages
ca-certificates.noarch           2021.2.52-1.0.fc35                    @updates
[marco@t420-tovis ~]$>sudo dnf whatprovides '/etc/pki/ca-trust/*'
[sudo] password for marco: 
Last metadata expiration check: 1:02:34 ago on ven 25 mar 2022, 09:57:28.
ca-certificates-2021.2.50-3.fc35.noarch : The Mozilla CA root certificate bundle
Repo        : fedora
Matched from:
Filename    : /etc/pki/ca-trust/README
Filename    : /etc/pki/ca-trust/ca-legacy.conf
Filename    : /etc/pki/ca-trust/extracted
Filename    : /etc/pki/ca-trust/extracted/README
Filename    : /etc/pki/ca-trust/extracted/edk2/README
Filename    : /etc/pki/ca-trust/extracted/edk2/cacerts.bin
Filename    : /etc/pki/ca-trust/extracted/java
Filename    : /etc/pki/ca-trust/extracted/java/README
Filename    : /etc/pki/ca-trust/extracted/java/cacerts
Filename    : /etc/pki/ca-trust/extracted/openssl
Filename    : /etc/pki/ca-trust/extracted/openssl/README
Filename    : /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
Filename    : /etc/pki/ca-trust/extracted/pem
Filename    : /etc/pki/ca-trust/extracted/pem/README
Filename    : /etc/pki/ca-trust/extracted/pem/email-ca-bundle.pem
Filename    : /etc/pki/ca-trust/extracted/pem/objsign-ca-bundle.pem
Filename    : /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
Filename    : /etc/pki/ca-trust/source
Filename    : /etc/pki/ca-trust/source/README
Filename    : /etc/pki/ca-trust/source/anchors
Filename    : /etc/pki/ca-trust/source/blacklist
Filename    : /etc/pki/ca-trust/source/blocklist
Filename    : /etc/pki/ca-trust/source/ca-bundle.legacy.crt

ca-certificates-2021.2.52-1.0.fc35.noarch : The Mozilla CA root certificate bundle
Repo        : @System
Matched from:
Filename    : /etc/pki/ca-trust/README
Filename    : /etc/pki/ca-trust/ca-legacy.conf
Filename    : /etc/pki/ca-trust/extracted
Filename    : /etc/pki/ca-trust/extracted/README
Filename    : /etc/pki/ca-trust/extracted/edk2/README
Filename    : /etc/pki/ca-trust/extracted/edk2/cacerts.bin
Filename    : /etc/pki/ca-trust/extracted/java
Filename    : /etc/pki/ca-trust/extracted/java/README
Filename    : /etc/pki/ca-trust/extracted/java/cacerts
Filename    : /etc/pki/ca-trust/extracted/openssl
Filename    : /etc/pki/ca-trust/extracted/openssl/README
Filename    : /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
Filename    : /etc/pki/ca-trust/extracted/pem
Filename    : /etc/pki/ca-trust/extracted/pem/README
Filename    : /etc/pki/ca-trust/extracted/pem/email-ca-bundle.pem
Filename    : /etc/pki/ca-trust/extracted/pem/objsign-ca-bundle.pem
Filename    : /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
Filename    : /etc/pki/ca-trust/source
Filename    : /etc/pki/ca-trust/source/README
Filename    : /etc/pki/ca-trust/source/anchors
Filename    : /etc/pki/ca-trust/source/blacklist
Filename    : /etc/pki/ca-trust/source/blocklist
Filename    : /etc/pki/ca-trust/source/ca-bundle.legacy.crt

ca-certificates-2021.2.52-1.0.fc35.noarch : The Mozilla CA root certificate bundle
Repo        : updates
Matched from:
Filename    : /etc/pki/ca-trust/README
Filename    : /etc/pki/ca-trust/ca-legacy.conf
Filename    : /etc/pki/ca-trust/extracted
Filename    : /etc/pki/ca-trust/extracted/README
Filename    : /etc/pki/ca-trust/extracted/edk2/README
Filename    : /etc/pki/ca-trust/extracted/edk2/cacerts.bin
Filename    : /etc/pki/ca-trust/extracted/java
Filename    : /etc/pki/ca-trust/extracted/java/README
Filename    : /etc/pki/ca-trust/extracted/java/cacerts
Filename    : /etc/pki/ca-trust/extracted/openssl
Filename    : /etc/pki/ca-trust/extracted/openssl/README
Filename    : /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
Filename    : /etc/pki/ca-trust/extracted/pem
Filename    : /etc/pki/ca-trust/extracted/pem/README
Filename    : /etc/pki/ca-trust/extracted/pem/email-ca-bundle.pem
Filename    : /etc/pki/ca-trust/extracted/pem/objsign-ca-bundle.pem
Filename    : /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
Filename    : /etc/pki/ca-trust/source
Filename    : /etc/pki/ca-trust/source/README
Filename    : /etc/pki/ca-trust/source/anchors
Filename    : /etc/pki/ca-trust/source/blacklist
Filename    : /etc/pki/ca-trust/source/blocklist
Filename    : /etc/pki/ca-trust/source/ca-bundle.legacy.crt

freeipa-client-common-4.9.7-2.fc35.noarch : Common files used by IPA client
Repo        : fedora
Matched from:
Filename    : /etc/pki/ca-trust/source/ipa.p11-kit

freeipa-client-common-4.9.8-1.fc35.noarch : Common files used by IPA client
Repo        : updates
Matched from:
Filename    : /etc/pki/ca-trust/source/ipa.p11-kit

tog-pegasus-2:2.14.1-61.fc35.x86_64 : OpenPegasus WBEM Services for Linux
Repo        : fedora
Matched from:
Filename    : /etc/pki/ca-trust/source/anchors/localhost-pegasus.pem
2 Likes

This all looks good, so everything is as it should be. If/when you run into issues again, we should be able to track down what causes it.

1 Like

Hi everybody. Here we go again…
Something happened (during a “dnf update”, I assume) that corrupted this file:

-r–r–r–. 1 root root 218254 20 ago 22.36 tls-ca-bundle.pem

causing the next “dnf update” run into the curl errors:
Errors during downloading metadata for repository ‘tor’:

Replacing the “tls-ca-bundle.pem” with a different one (taken from my second laptop, running Fedora35 as well) solved the problem.

Thanks.

Marco

What version of ca-certficates do you have?

ca-certificates-2022.2.54-1.2.fc36.noarch

is what I have and everything runs just fine. That’s the same version as F35 also. Did you reboot your system after the update and see if that helps?

what’s the version of ca-certificate on this machine?

Well, that’s a thing…

On the machine having problems the version/release is 2022.2.54/1.2.fc35
On the other one is 2022.2.54/1.0.fc35

Thanks

It sounds like you might have some duplicate packages installed. This can happen if a system update is interrupted before it finishes and can leave your system in a weird state. You might want to run package-cleanup with the clean-dupes option to remove duplicate older versions of packages.

Thank you Scott.
Done, no duplicates it says:

[marco@t420 ~]$>sudo package-cleanup --cleandupes
Last metadata expiration check: 0:00:47 ago on lun 22 ago 2022, 17:01:42.
Error: No duplicated packages found for removal.
[marco@t420 ~]$>

1 Like

So, I don’t see any bugs about this, and the only karma that the update received for the latest version (2022.2.54-1.2) was also positive. On my F36, this version is also running just fine without any issues. This indicates that this is somehow limited to your system (at least for the moment until we get more users noting that they’re running into the same issue).

Can you recall exactly when this began? Was it immediately after an update? What other packages were updated?

The package update will install the new files for the updated version. As I note, this works fine for us. So we need to see if the version of the file that was installed on your system by the update is:

  • the correct file
  • modified by something (which you note as “corrupted”)

Also, what does rpm -V ca-certificates say?

2 Likes

Well, I opened this discussion on March, 16th, but I was struggling with this issue since weeks before. Of course it must have been a consequence of an update, but to say exactly when is not possible.
Since then, from time to time, not every time, running a dnf update throws all those curl errors, meaning that something happened after the previous update. What I’m currently doing is running updates each and every day, in order to narrow the time frame to be investigated when the problem will occur next time.

It says nothing at all.

Thanks for taking care.

Marco

1 Like

That implies that the files from the package are as they should be—they have not been modified by anything else. Could you please run this periodically to keep checking that nothing is modifying the files in any way?

Yes, sure!

2 Likes

What do you do if you don’t have an older version of Fedora with a good pem file?
As I’m unable to update the system at all.

@whitequill ,
Did the solution here solve your issue? If not maybe your issue is different.

1 Like

No. Because I don’t have a previous version of Fedora available.
clean-dups is not installed and package-cleanup is not installed and nothing new can be installed through dnf because of the certificate errors.

I have ca-certificate-2022.2.54-1.2 installed.

Could you open a new issue please? We haven’t quite reached the root of the issue here so even though you’re both seeing the same error, it’s hard to say if it’s caused by the same underlying issue—sure “using in old ca-certificate package” seems to work around it, here but since so many of us are not experiencing this issue, I don’t think the issue is with the ca-certificate package. So I think we need to debug the issue more to figure out what is causing it.

In the meantime, you can manually download the package from the build system and install it using dnf to see if this workaround also works for you:

https://koji.fedoraproject.org/koji/packageinfo?packageID=6260

Before you do that, though, could you please open a new topic and provide us with some info?

  • output of sudo fpaste --sysinfo --printonly
  • rpm -q ca-certificates
  • rpm -V ca-certificates

and anything else that may be relevant, e.g.:

  • are you able to browse the internet in general?
  • what ISP are you using?
  • have you made any changes to the network config? Firewall, DNS, and so on?
1 Like

You’ve got a DNS problem; not related to certificates. Probably best to move that post to a new topic.