Talk: F38 to F39 dnf system-upgrade can fail on Raspberry Pi

This is a discussion topic for the following Common Issue:

You can discuss the problem and its solutions here, but please note that debugging and technical feedback should primarily go to the issue trackers (e.g. Bugzilla) linked in the Common Issue, because that’s the place that developers watch, not here.

If there are any updates/changes/amendments for the Common Issue description, which you believe should be performed, please post it here.

Please see the Common Issue for solution/workarounds:

The workaround as currently written provides the “how” but not the “why”. That is I have not explained why systemd-timesyncd works and chrony does not. Nor is there any discussion of the trade-offs between systemd-timesyncd and chrony and why one might revert the change after a successful upgrade. Is this needed? Discussion in bugzilla does go into why the problem is cropping up with this upgrade and not with prior upgrades.

I suspect that is related to the timing of when the network is available and when ntp is used to update the clock. Systemd starts the network very early (and also probably systemd-timesyncd) and chrony starts later. It may be that chrony does not even get started during the system-upgrade reboot stage. I would need to see logs to know and I have never seen a log that would be available at that stage of the upgrade.

Yes, chrony is not started after the system-upgrade reboot when the upgrade is happening. systemd-timesyncd also stores a timestamp locally in the file system on system shutdown and reads that time stamp during boot and can use it to set the system clock very early in the boot process. This is useful for devices like the RPI4 that do not have real time clocks. The trade off is that systemd-timesyncd does not function as an ntp server whereas chrony does. So for use cases where chrony is used as an ntp service, it will need to be restored after the upgrade.

The dnf system upgrade plug in does have a log which can be accessed after upgrade regardless of success or failure via sudo dnf system-upgrade log --number=-1.

https://bugzilla.redhat.com/show_bug.cgi?id=2242759#c5
https://bugzilla.redhat.com/show_bug.cgi?id=2242759#c6

That’s true, the “why” could be elaborated in the Cause section. If you want to improve the wording (and cover also other points that you mentioned), just add the excerpt here and I’ll apply it to the common issue description (because I don’t think you can edit it directly anymore). But I wouldn’t go into very deep technical details, there’s the bug report for people who are interested in that. Just a general summary. Thanks.

Thanks Kamil. I really appreciate your help. After reviewing the common issue, I only recommend adding the paragraph below at the end of the workaround section.

Systemd-timesyncd is a suitable network time protocol (NTP) client, but not as a full featured replacement for all of the capabilities of chrony. If the Raspberry Pi machine is used as an NTP server then chrony will need to be re-enabled after the successful upgrade to Fedora 39.

many thanks!

Done, thanks.

@kparal - based on excellent information in https://bugzilla.redhat.com/show_bug.cgi?id=2242759#c41, it is clear that some of the information I included in this common issue is not correct. Specifically information about the date/time early in the boot cycle on RPi or other devices without a real time clock. The comment from Zbigniew includes 1 option (https://bugzilla.redhat.com/show_bug.cgi?id=2242759#c41) that is much easier. Zbigniew has also pushed an updated build for systemd in F38 which may make this whole issue moot. I will try to test this shortly but am short on time today.

Hi Brad, I updated the description. I kept the previous workaround in place as well, just in case the newer one doesn’t work for everyone. If you think the description should be tweaked somehow, please let me know, thanks :slight_smile:

hi Kamil

Looks good to me. Well done.

In https://bugzilla.redhat.com/show_bug.cgi?id=2242759#c41 Zbigniew notes that “Systemd has a built-in mechanism to set the clock during early boot.
It’ll use either a built-in TIME_EPOCH (which is usually the build time of systemd itself),
or the mtime of /usr/lib/clock-epoch.” This explains the time stamps in the dnf log file.

There might be a few users who would appreciate this type of technical detail, but it is likely too much for most.

I ran into this issue while attempting to upgrade my Raptor Systems Blackbird POWER9 PPC64le from Fedora 38 to 39.
The GUI upgrade failed silently. The ‘unsupported’ dnf method failed silently until I hit ESC to see that every package was failing a cert check because every cert lived in the future.
The entirety of the fix was:

sudo systemctl enable --now systemd-timesyncd

…then run the system-upgrade again.

And the PPC system is now running 39.