F40 Change Proposal: Drop Delta RPMs (System Wide)

Project Discussion Change Proposals
,
1

Wiki Link https://fedoraproject.org/wiki/Changes/Drop_Delta_RPMs

Announce Link https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/CE4I6P6WWQIJTR4KGW5BANFJ7MX7GP44/

This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

:link: Summary

Stop producing Delta RPMs during the compose process, and disable deltarpm support in the default configuration of DNF / DNF5.

:link: Owner

:link: Detailed Description

Delta RPMs are an optimization that reduces the amount of data that needs to be downloaded for package updates, by only downloading the parts of the package that have changed between the locally installed version and the new version, and reassembling a complete RPM package for the new version locally.

Due to the way they are generated during the compose process, it is not possible to produce Delta RPMs for all packages that are involved in an upgrade (since only one previous version is considered for delta generation). This can lead to upgrades that involve hundreds of packages, but only a tiny fraction of them (or none at all) also have appropriate Delta RPMs available in the repository.

Additionally, reconstruction of the new version of the RPM from the currently installed version and the Delta RPM can fail, causing an additional download of the complete RPM for the new version, wasting bandwidth.

On top of these issues, the presence of Delta RPMs in package repositories also inflates the size of the repository metadata, which needs to be downloaded by all users, whether the actual upgrade involves Delta RPMs or not. This most affects users that upgrade via GUI utilities like GNOME Software - because the PackageKit libdnf backend has no support for Delta RPMs at all, or users of OSTree based variants / spins of Fedora - where Delta RPMs are not used either.

According to early feedback from Release Engineering, it looks like it will not be feasible to address the shortcomings of Delta RPMs as they are currently implemented, and since they often no longer result in a net reduction in download sizes for upgrades, this Change proposes both to disable the generation of DeltaRPMs during the compose process, and to disable the deltarpm support in dnf / dnf5 by default.

:link: Some statistics

I have collected data for system upgrades with dnf across three different Fedora installs (Fedora 38 Workstation, Fedora 37 KDE, and Fedora 38 KDE, respectively, all with the “updates-testing” repository enabled) for two months, with varying intervals between upgrades (between one day and one month) in an attempt to capture different situations. To summarize the results:

  • Delta RPMs were available for only 25 out of 42 upgrades
  • Delta RPMs saved 7.5 MB / 8% of downloads on average
  • Delta RPMs saved 22.5 MB of download on average (if there were no failures)
  • Delta RPMs wasted 52.7 MB of download on average (if there were failures)

As an anecdote, most of the savings are attributable to one package (firefox).

For reference, the current sizes of repository metadata for Fedora 38 (as of 2023-09-07):

  • fedora: 86.8 MB
  • updates: 33.2 MB
  • updates-testing: 13.3 MB

Note that refreshing repository metadata only causes deltas to be downloaded, not a full re-download (due to using zchunk for repodata).

:link: Feedback

N/A

:link: Benefit to Fedora

Benefits for Fedora infrastucture:

  • simplification of the compose process for “updates” and “updates-testing” repositories (generation of Delta RPMs will be skipped)
  • reduction of storage requirements in Fedora infrastructure and on repository mirrors (both due to smaller metadata and dropped Delta RPMs)
  • reduction in bandwith use for repository metadata updates

Benefits for Fedora users:

  • more reliable upgrades (i.e. failures to re-assemble RPMs from Delta RPMs are eliminated)
  • reduction in bandwith use for repository metadata updates
  • reduction in CPU usage for package upgrades (local re-assembly of RPMs from on-disk data and Delta RPMs is eliminated)

:link: Scope

  • Proposal owners:
    • modify compose configuration to skip generation of Delta RPMs for Fedora 40+
    • modify default configuration for dnf / dnf5 to disable deltarpm support
  • Other developers:
    • coordination with dnf and dnf5 developers and / or package maintainers
  • Release engineering: releng#11665
    • merge compose changes to skip generation of Delta RPMs for Fedora 40+
  • Policies and guidelines: N/A (not needed for this Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with Community Initiatives: N/A (no currently active initiatives)

:link: Upgrade/compatibility impact

Installations that get upgraded to Fedora 40 should not be negatively affected by this Change. The configuration change for dnf / dnf5 will only be automatically applied if there were no modifications to /etc/dnf/dnf.conf. However, since the repositories will no longer contain Delta RPMs, this configuration change is not strictly required.

Alternatively, the default dnf / dnf5 configuration could be changed at the code level instead of changes to configuration files.

:link: How To Test[edit]

Users should be able to verify that support for Delta RPMs is disabled in dnf on Fedora 40, for example by checking for deltarpm = 0 in the output of dnf config-manager --dump | grep deltarpm.

:link: User Experience

Users who upgrade their systems with GNOME Software (or other similar tools) should not be affected by this change, other than faster and smaller repository metadata downloads.

Users who upgrade via dnf / dnf5 should see a similar improvement. Additionally, there will be no more wasted downloads due to failed reassembly of RPMs from Delta RPMs.

:link: Dependencies

  • Release Engineering changes (modifications of the compose process)
  • dnf / dnf5 package changes (disabling deltarpm support, either via configuration changes, or changing the default from true to false in the code itself)

:link: Contingency Plan

  • Contingency mechanism:

If the compose configuration cannot be modified to skip generation of Delta RPMs in time for Fedora 40, changing the default configuration in dnf / dnf5 will still cause some of the benefits outlined above (except for those caused by the smaller repository metadata sizes).

If the default dnf configuration cannot be changed in time for Fedora 40, the benefits outlined above should still apply in almost all circumstances.

If neither changes can be implemented in time, the change can be postponed to the next Fedora release without causing problems.

  • Contingency deadline: Fedora 40 Beta Freeze

  • Blocks release? No

:link: Documentation

N/A

:link: Release Notes

As of Fedora 40, package repositories no longer contain Delta RPMs, and package managers (dnf, dnf5) have disabled support for deltarpms by default. They are not useful in many common circumstances (i.e. upgrades via GNOME Software or on OSTree based variants), and often no longer even cause less data to be downloaded for upgrades.

5 Likes
2

As the one who did most of the original work of getting deltarpms into Fedora, I wholeheartedly support this change. I’m sorry to see them go, but it’s time.

11 Likes
3

Well, delta RPMs were great at the time they worked (before changes in how Fedora updates were published); but its true that they were essentially not working for a long time. So, they are effectively removed already, and I’d really prefer if the change was about fixing the issues and make it work again. But, I understand that there is nobody willing to fix it, so it can be forgotten :frowning:

And far larger download sizes are coming with containerized applications (e.g. Flatpaks) anyway…

3 Likes
4

Thank you, @decathorpe, for driving this forward. (And thank you @jdieter for creating this in the first place.)

2 Likes
5

OMFG yes please. Quite a while ago, I compiled another set of extensive stats demonstrating why deltarpms just aren’t worth it anymore.

Quoting the denouement:

(Those numbers were totals across 3-ish months of ~weekly dnf upgrade runs, plus any necessary dnf install runs during the same period.)

Edit: I take it back, the script only counted log messages that mentioned deltaRPMs, so dnf install wouldn’t have been included. Since a dnf install couldn’t possibly take advantage of .drpms, it would’ve polluted the stats to include those transactions. Fortunately 3-years-ago me was smarter than today-me, and realized that.

3 Likes
6

I think it’s sad to make life for people on (expensive) metered or slow connections (yes, that is still a reality for millions) more difficult or leave them behind. It would be more inclusive to fix delta rpm and enable bandwidth savings for those who need it.

1 Like
7

I am actually not sure whether Delta RPMs even help you on metered connections. As mentioned in the Change Proposal, the benefits of using Delta RPMs is, on average, just a few MB per upgrade. But they also have a constant cost, in the form of repository metadata.

The difference is that repository metadata includes information about all Delta RPMs in the repository, not only the ones for updates that affect the local system, so this is a constant factor that increases download size even if you get zero benefit from it for some updates.

To make things worse, repository metadata is downloaded using zchunk to only download changes, but the Delta RPMs in the repository change every day, so even that doesn’t help.

I’m not sure how big the savings would be if there were no Delta RPMs in the repositories, but since metadata is usually refreshed at least once per day automatically, it will not be insignificant - especially when you’re on a metered connection, where - I assume - you do not upgrade daily.

1 Like
8

It also didn’t count occurrences of upgrades where there were no Delta RPMs involved at all, if I’m reading things correctly? So the actual averages might be even worse.

9

Let me just clarify something again: yes, delta rpms in its current state are almost useless and provide not much benefits, but it is because our repositories do NOT generate them properly since a few years.

So, if somebody were going to fix the problems we currently have, DeltaRPMs provided a HUGE benefit for metered connections (more than 50% savings in almost all cases, and frequently more than 70% savings was not uncommon at those golden days).

But they are broken for a long time, and there was nobody all these years who would step up to fix it. So, DeltaRPMs can go, not because it is useless, but because it is broken in its current state.

2 Likes
10 

Failed Delta RPMs increased 297.1 MB of updates to 355.1 MB (16.3% wasted)

Will there be an alternative for people who want/need to save bandwidth?

11

But sometimes I got up to 5-7% saves. (Not that it’s needed, Gig optic, but newertheless.)

12

I think CoreOS, Silverblue, and other Atomic Desktop options with delta upgrades are the best path forward. A different approach, but lots of benefits.

1 Like
13

I’m not sure I buy that, entirely. I believe they used to represent a significant reduction in package sizes, but I don’t believe that it’s merely issues with how they’re generated that have eliminated that savings.

If that were the case, then even though there may be relatively few DeltaRPMs available in the repos (due to the missing version combinations not being generated), for the ones that are available the size reduction should be that significant 50-70% you’re quoting. But my data didn’t remotely bear that out.

I addressed that somewhat in my original discussion:

If there’s even one line of code changed, or one timestamp different, any compiled binaries (executables and libraries) in an RPM will be completely different from the previous build, which means they all get excluded from the deltaRPMs.

In fact, the experience that led me to originally collect those statistics and post that thread, three years ago, was seeing a bunch of texlive package deltaRPMs actually show significant savings — and realizing what an outlier that was. Texlive packages, containing almost exclusively data files, are the ideal conditions for deltaRPMs, since most of their contents will be unchanged from build to build. But they’re both the exception that proves the rule, and so tiny as to be meaningless either way.

14

I’m not sure I buy that, entirely. I believe they used to represent a significant reduction in package sizes, but I don’t believe that it’s merely issues with how they’re generated that have eliminated that savings.

Well, you can refer to these:
https://pagure.io/fedora-infrastructure/issue/7008

Apparently, it was about 5 years ago where they stopped to work correctly, which seems to be with F25 release. So yeah, 3 years ago it didn’t work either.

And I’m not the only one talking about big savings, this is a quote from @robatino:

… They used to save around 75% on average. …

If there’s even one line of code changed, or one timestamp different, any compiled binaries (executables and libraries) in an RPM will be completely different from the previous build, which means they all get excluded from the deltaRPMs.

I highly doubt that’s the case. AFAIK, it was not a simple: pack only changed files; and it created a diff for a changed file too. It was not “all or nothing”. Although, I might be wrong, but if it was the case it should be useless from day zero.

15

OK, I’m sure that you are wrong:

Courgette, deltarpm and bsdiff (the tool) are all based on the bsdiff algorithm, which is very efficient.

It explicitly talks about binary diff algorithms, as it is used in delta rpms too. And there is also this output for Firefox:

I doubt changes from Firefox 3.5.4 to 3.5.6 (from 15M down to 434K) was merely in non-source files, and it certainly was rebuilt, as is the case with Fedora packages.

But I also acknowledge that it had its own share of problems, as Jonathan has outlined in a number of posts. Nevertheless, it was a huge win as I perfectly remember.

1 Like
16

To clarify on how drpms work:

They must be generated at the same time you make the repository data.
So, you have to know and have access to all the rpms you want to make
drpms from at the exact same time you compose the repository.
Our compose tool works by gathering the exact packages that are supposed
to be in the repo and building things from those. It only has a small
hack that lets it look at the last compose for drpms. This is why
there’s so few of them.

In order to fix that, drpms would need to be decoupled from the repodata
creation, dnf would need to learn to look for them in another place,
bunches of them would need to be synced out, etc.

The actual way they work on the end user machine is that dnf sees that
you have version X.Y installed of a package, that you are upgrading to
A.B and there is a drpm available. That drpm contains only the binary
diff between the two versions. You download that and then it uses the
installed X.Y package you have + drpm to make a A.B rpm. Then, that
version is used to upgrade you. (just like you used a A.B rpm directly,
it’s exactly the same rpm). If you don’t upgrade really often (like,
once a day), the odds are good that there’s no drpm at all available for
you. And yeah, some packages are better than others for binary diff.
(or never can have usable drpms, like kernel packages).

Basically drpms are a trade off. You are trading download bits for cpu
and time assembling them.

Anyhow, if someone wants a project, you could work on untangling drpms
from createrepo_c, but it’s by no means a quick and easy project, it
would be a ton of work. ;(

3 Likes
17

And really that’s the core issue: For anyone except those users on metered connections, that’s an actively TERRIBLE tradeoff. Unless you’re heavily restricted as to how many bytes you can pull down, just downloading the entire package will always be less effort (CPU cycles, memory requirements, filesystem accesses, time, complexity) than painstakingly reassembling it locally from diffs.

I don’t deny the potential utility of drpms for users with bandwidth restrictions… but they’re really the only ones for whom deltas make any sense.

1 Like
18

That post is from 2009. That probably falls into the “old days before compilers started using application security features like address randomization by default.” period I mentioned.

The firefox RPM is also not the best example. It contains relatively few / relatively small compiled binary files, at least today. The single largest binary component is the 125MB /usr/lib64/firefox/libxul.so file. The actual firefox binary itself (/usr/lib64/firefox/firefox-bin) is under 1MB in size.

Most of the package’s contents are .xpi files, which are zip archives that will lend themselves to binary diffing.

Still, I downloaded both the current firefox-119.0-2.fc38.x86_64.rpm package from F38 updates, and the firefox-111.0.1-1.fc38.x86_64.rpm that F38 shipped with. That’s not the best example, since there’s a lot more distance between those than you’d want, but it’s what we have.

$ makedeltarpm firefox-111.0.1-1.fc38.x86_64.rpm firefox-119.0-2.fc38.x86_64.rpm firefox-delta.drpm 
$ ls -l firefox*.*rpm
-rw-r--r--. 1 root root 71M Nov  6 08:43 firefox-111.0.1-1.fc38.x86_64.rpm
-rw-r--r--. 1 root root 65M Nov  6 08:44 firefox-119.0-2.fc38.x86_64.rpm
-rw-r--r--. 1 ferd ferd 47M Nov  6 08:45 firefox-delta.drpm

Not great. And that’s with “only” 6 months’ worth of package changes. (Again, I acknowledge that if the two builds were closer, the delta would be much smaller.)

I don’t have access to the courgette algorithm directly, but I do have bsdiff installed. So, let’s see how it does with individual binary files in the archive(s):

$ bsdiff firefox-111/firefox-111.0.1-1.fc38.x86_64/usr/lib64/firefox/firefox-bin firefox-119/firefox-119.0-2.fc38.x86_64/usr/lib64/firefox/firefox-bin firefox-bin.bsdiff
$ ls -l firefox-111/firefox-111.0.1-1.fc38.x86_64/usr/lib64/firefox/firefox-bin firefox-119/firefox-119.0-2.fc38.x86_64/usr/lib64/firefox/firefox-bin firefox-bin.bsdiff
-rwxr-xr-x. 1 ferd ferd 739k Mar 21  2023 firefox-111/firefox-111.0.1-1.fc38.x86_64/usr/lib64/firefox/firefox-bin
-rwxr-xr-x. 1 ferd ferd 483k Oct 26 20:00 firefox-119/firefox-119.0-2.fc38.x86_64/usr/lib64/firefox/firefox-bin
-rw-r--r--. 1 ferd ferd 233k Nov  6 08:51 firefox-bin.bsdiff

$ bsdiff firefox-111/firefox-111.0.1-1.fc38.x86_64/usr/lib64/firefox/firefox-bin firefox-119/firefox-119.0-2.fc38.x86_64/usr/lib64/firefox/firefox-bin firefox-bin.bsdiff
$ bsdiff firefox-111/firefox-111.0.1-1.fc38.x86_64/usr/lib64/firefox/firefox-bin firefox-119/firefox-119.0-2.fc38.x86_64/usr/lib64/firefox/firefox-bin firefox-bin.bsdiff
-rwxr-xr-x. 1 ferd ferd 153M Mar 21  2023 firefox-111/firefox-111.0.1-1.fc38.x86_64/usr/lib64/firefox/libxul.so
-rwxr-xr-x. 1 ferd ferd 124M Oct 26 20:00 firefox-119/firefox-119.0-2.fc38.x86_64/usr/lib64/firefox/libxul.so
-rw-r--r--. 1 ferd ferd  52M Nov  6 08:57 libxul.bsdiff

So, yes, I’ll concede that i was somewhat pessimistic about the ability of deltas to work on compiled binaries. The bsdiff of firefox-bin is especially impressive: The diff size is 233k, which is less than the amount the binary has grown between the two versions! (The Firefox 119 firefox-bin is 256k larger than the one from Firefox 111.) I’m not sure how that even works.

I’m an idiot who can’t read. The Firefox 119 firefox-bin file is the smaller one. So the Firefox 119 binary is 256k smaller than the Firefox 111 version, at only 483k. But the diff version of that binary from Firefox 111 is 233k — nearly half of its total size.

But what I was not wrong about is the poor tradeoff in resources:

$ time bsdiff firefox-111/firefox-111.0.1-1.fc38.x86_64/usr/lib64/firefox/libxul.so firefox-119/firefox-119.0-2.fc38.x86_64/usr/lib64/firefox/libxul.so libxul.bsdiff
bsdiff firefox-111/firefox-111.0.1-1.fc38.x86_64/usr/lib64/firefox/libxul.so   259.62s user 1.40s system 99% cpu 4:22.52 total
$ time bspatch firefox-111/firefox-111.0.1-1.fc38.x86_64/usr/lib64/firefox/libxul.so libxul-119.so libxul.bsdiff
bspatch firefox-111/firefox-111.0.1-1.fc38.x86_64/usr/lib64/firefox/libxul.so  8.62s user 0.33s system 97% cpu 9.135 total

Calculating that libxul.bsdiff file takes over FOUR MINUTES on my system! And that time is largely dependent on the size of the input files — having a “closer” binary, say from the immediately previous build, might produce a smaller diff, but it would still take a long time to generate that diff from two 125MB input files.

Applying the diff it generates is, thankfully, a much quicker affair, taking “only” 9 seconds to re-create the Firefox 119 libxul.so from the Firefox 111 version and the .bsdiff.

Courgette presumably leaves out some of bsdiff’s most time-consuming features, considering the makedeltarpm command to generate a diff of those two packages “only” takes 1m5s total, on the same system. OTOH, its tradeoffs go the other direction: applying that .drpm to reconstruct the Firefox 119 RPM from the Firefox 111 RPM (even with access to the actual RPM itself, not using the installed on-disk files) takes over 1.5 minutes:

$ time applydeltarpm -r firefox-111.0.1-1.fc38.x86_64.rpm firefox-delta.drpm firefox-119-reconstructed.rpm
applydeltarpm -r firefox-111.0.1-1.fc38.x86_64.rpm firefox-delta.drpm   95.64s user 0.31s system 99% cpu 1:36.41 total

Though, here again,with closer versions the apply would be a quicker process. The make, probably somewhat quicker, but not necessarily substantially quicker.

And 1 minute to generate a delta is a long time. Fortunately, generating additional deltas wouldn’t have to cause the exponential growth in resource consumption that one might expect. deltarpm comes with the combinedeltarpm command, which can build new deltas out of old ones (relatively quickly). So, given a set of old deltas, a new package build, and the previous package build, deltas for every version represented could be created by running makedeltarpm prev.rpm new.rpm prev..new.drpm, then running combinedeltarpm prev-1..prev.drpm prev..new.drpm prev-1..new.drpm, and so on for each existing *..prev.drpm in turn.

It would still create exponential growth in drpm files themselves, though. Say a package has deltas for three versions, A, B, and C. You’d have A..B.drpm, A..C.drpm, and B..C.drpm. Performing a new build D would be followed by the lengthy makedeltarpm C.rpm D.rpm C..D.rpm process, followed by:

combinedeltarpm A..C.drpm C..D.drpm A..D.drpm
combinedeltarpm B..C.drpm C..D.drpm B..D.drpm

That gets you 7 total .drpm files for four versions. Add a fifth version, and after generating D..E.drpm you have to combinedeltarpm all three existing *..D.drpm files, producing four new .drpms, for a total of 11. Next version adds 5 new deltas, for a total of 16. And so on.

19

Just to be even more fair, I dug through Bodhi and pulled out the two most recent Firefox builds for F40, firefox-119.0-2.fc40.rpm and firefox-119.0-3.fc40.rpm.

Running makedeltarpm on those two packages, which are (as you’d expect) nearly identical in size at 66MB, takes 16 seconds, and produces a .drpm that’s 14MB. Applying that .drpm to reconstruct firefox-119.0-3.fc40.rpm from firefox-119.0-2.fc40.rpm takes 1m37s.

1 Like
20

Good news here — container deltas are under active development. This should benefit both Flatpaks and the new CoreOS / rpm-ostree distribution method. So, we should have something that works better (with lower cost on both sides) which still also gives decent bandwidth reduction.

2 Likes