F40 Change Proposal: Drop Delta RPMs (System Wide)

Wiki Link https://fedoraproject.org/wiki/Changes/Drop_Delta_RPMs

Announce Link https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/CE4I6P6WWQIJTR4KGW5BANFJ7MX7GP44/

This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

:link: Summary

Stop producing Delta RPMs during the compose process, and disable deltarpm support in the default configuration of DNF / DNF5.

:link: Owner

:link: Detailed Description

Delta RPMs are an optimization that reduces the amount of data that needs to be downloaded for package updates, by only downloading the parts of the package that have changed between the locally installed version and the new version, and reassembling a complete RPM package for the new version locally.

Due to the way they are generated during the compose process, it is not possible to produce Delta RPMs for all packages that are involved in an upgrade (since only one previous version is considered for delta generation). This can lead to upgrades that involve hundreds of packages, but only a tiny fraction of them (or none at all) also have appropriate Delta RPMs available in the repository.

Additionally, reconstruction of the new version of the RPM from the currently installed version and the Delta RPM can fail, causing an additional download of the complete RPM for the new version, wasting bandwidth.

On top of these issues, the presence of Delta RPMs in package repositories also inflates the size of the repository metadata, which needs to be downloaded by all users, whether the actual upgrade involves Delta RPMs or not. This most affects users that upgrade via GUI utilities like GNOME Software - because the PackageKit libdnf backend has no support for Delta RPMs at all, or users of OSTree based variants / spins of Fedora - where Delta RPMs are not used either.

According to early feedback from Release Engineering, it looks like it will not be feasible to address the shortcomings of Delta RPMs as they are currently implemented, and since they often no longer result in a net reduction in download sizes for upgrades, this Change proposes both to disable the generation of DeltaRPMs during the compose process, and to disable the deltarpm support in dnf / dnf5 by default.

:link: Some statistics

I have collected data for system upgrades with dnf across three different Fedora installs (Fedora 38 Workstation, Fedora 37 KDE, and Fedora 38 KDE, respectively, all with the “updates-testing” repository enabled) for two months, with varying intervals between upgrades (between one day and one month) in an attempt to capture different situations. To summarize the results:

  • Delta RPMs were available for only 25 out of 42 upgrades
  • Delta RPMs saved 7.5 MB / 8% of downloads on average
  • Delta RPMs saved 22.5 MB of download on average (if there were no failures)
  • Delta RPMs wasted 52.7 MB of download on average (if there were failures)

As an anecdote, most of the savings are attributable to one package (firefox).

For reference, the current sizes of repository metadata for Fedora 38 (as of 2023-09-07):

  • fedora: 86.8 MB
  • updates: 33.2 MB
  • updates-testing: 13.3 MB

Note that refreshing repository metadata only causes deltas to be downloaded, not a full re-download (due to using zchunk for repodata).

:link: Feedback

N/A

:link: Benefit to Fedora

Benefits for Fedora infrastucture:

  • simplification of the compose process for “updates” and “updates-testing” repositories (generation of Delta RPMs will be skipped)
  • reduction of storage requirements in Fedora infrastructure and on repository mirrors (both due to smaller metadata and dropped Delta RPMs)
  • reduction in bandwith use for repository metadata updates

Benefits for Fedora users:

  • more reliable upgrades (i.e. failures to re-assemble RPMs from Delta RPMs are eliminated)
  • reduction in bandwith use for repository metadata updates
  • reduction in CPU usage for package upgrades (local re-assembly of RPMs from on-disk data and Delta RPMs is eliminated)

:link: Scope

  • Proposal owners:
    • modify compose configuration to skip generation of Delta RPMs for Fedora 40+
    • modify default configuration for dnf / dnf5 to disable deltarpm support
  • Other developers:
    • coordination with dnf and dnf5 developers and / or package maintainers
  • Release engineering: releng#11665
    • merge compose changes to skip generation of Delta RPMs for Fedora 40+
  • Policies and guidelines: N/A (not needed for this Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with Community Initiatives: N/A (no currently active initiatives)

:link: Upgrade/compatibility impact

Installations that get upgraded to Fedora 40 should not be negatively affected by this Change. The configuration change for dnf / dnf5 will only be automatically applied if there were no modifications to /etc/dnf/dnf.conf. However, since the repositories will no longer contain Delta RPMs, this configuration change is not strictly required.

Alternatively, the default dnf / dnf5 configuration could be changed at the code level instead of changes to configuration files.

:link: How To Test[edit]

Users should be able to verify that support for Delta RPMs is disabled in dnf on Fedora 40, for example by checking for deltarpm = 0 in the output of dnf config-manager --dump | grep deltarpm.

:link: User Experience

Users who upgrade their systems with GNOME Software (or other similar tools) should not be affected by this change, other than faster and smaller repository metadata downloads.

Users who upgrade via dnf / dnf5 should see a similar improvement. Additionally, there will be no more wasted downloads due to failed reassembly of RPMs from Delta RPMs.

:link: Dependencies

  • Release Engineering changes (modifications of the compose process)
  • dnf / dnf5 package changes (disabling deltarpm support, either via configuration changes, or changing the default from true to false in the code itself)

:link: Contingency Plan

  • Contingency mechanism:

If the compose configuration cannot be modified to skip generation of Delta RPMs in time for Fedora 40, changing the default configuration in dnf / dnf5 will still cause some of the benefits outlined above (except for those caused by the smaller repository metadata sizes).

If the default dnf configuration cannot be changed in time for Fedora 40, the benefits outlined above should still apply in almost all circumstances.

If neither changes can be implemented in time, the change can be postponed to the next Fedora release without causing problems.

  • Contingency deadline: Fedora 40 Beta Freeze

  • Blocks release? No

:link: Documentation

N/A

:link: Release Notes

As of Fedora 40, package repositories no longer contain Delta RPMs, and package managers (dnf, dnf5) have disabled support for deltarpms by default. They are not useful in many common circumstances (i.e. upgrades via GNOME Software or on OSTree based variants), and often no longer even cause less data to be downloaded for upgrades.

3 Likes

As the one who did most of the original work of getting deltarpms into Fedora, I wholeheartedly support this change. I’m sorry to see them go, but it’s time.

10 Likes

Well, delta RPMs were great at the time they worked (before changes in how Fedora updates were published); but its true that they were essentially not working for a long time. So, they are effectively removed already, and I’d really prefer if the change was about fixing the issues and make it work again. But, I understand that there is nobody willing to fix it, so it can be forgotten :frowning:

And far larger download sizes are coming with containerized applications (e.g. Flatpaks) anyway…

2 Likes

Thank you, @decathorpe, for driving this forward. (And thank you @jdieter for creating this in the first place.)

2 Likes

OMFG yes please. Quite a while ago, I compiled another set of extensive stats demonstrating why deltarpms just aren’t worth it anymore.

Quoting the denouement:

(Those numbers were totals across 3-ish months of ~weekly dnf upgrade runs, plus any necessary dnf install runs during the same period.)

Edit: I take it back, the script only counted log messages that mentioned deltaRPMs, so dnf install wouldn’t have been included. Since a dnf install couldn’t possibly take advantage of .drpms, it would’ve polluted the stats to include those transactions. Fortunately 3-years-ago me was smarter than today-me, and realized that.

3 Likes

I think it’s sad to make life for people on (expensive) metered or slow connections (yes, that is still a reality for millions) more difficult or leave them behind. It would be more inclusive to fix delta rpm and enable bandwidth savings for those who need it.

1 Like

I am actually not sure whether Delta RPMs even help you on metered connections. As mentioned in the Change Proposal, the benefits of using Delta RPMs is, on average, just a few MB per upgrade. But they also have a constant cost, in the form of repository metadata.

The difference is that repository metadata includes information about all Delta RPMs in the repository, not only the ones for updates that affect the local system, so this is a constant factor that increases download size even if you get zero benefit from it for some updates.

To make things worse, repository metadata is downloaded using zchunk to only download changes, but the Delta RPMs in the repository change every day, so even that doesn’t help.

I’m not sure how big the savings would be if there were no Delta RPMs in the repositories, but since metadata is usually refreshed at least once per day automatically, it will not be insignificant - especially when you’re on a metered connection, where - I assume - you do not upgrade daily.

1 Like

It also didn’t count occurrences of upgrades where there were no Delta RPMs involved at all, if I’m reading things correctly? So the actual averages might be even worse.

Let me just clarify something again: yes, delta rpms in its current state are almost useless and provide not much benefits, but it is because our repositories do NOT generate them properly since a few years.

So, if somebody were going to fix the problems we currently have, DeltaRPMs provided a HUGE benefit for metered connections (more than 50% savings in almost all cases, and frequently more than 70% savings was not uncommon at those golden days).

But they are broken for a long time, and there was nobody all these years who would step up to fix it. So, DeltaRPMs can go, not because it is useless, but because it is broken in its current state.

1 Like

Failed Delta RPMs increased 297.1 MB of updates to 355.1 MB (16.3% wasted)

Will there be an alternative for people who want/need to save bandwidth?

But sometimes I got up to 5-7% saves. (Not that it’s needed, Gig optic, but newertheless.)

I think CoreOS, Silverblue, and other Atomic Desktop options with delta upgrades are the best path forward. A different approach, but lots of benefits.

1 Like