With the unavoidable upgrade to createrepo_c >= v1.0 it will be hard, if not impossible, to retain exactly the same settings for repositories metadata.
I’ve drafted a change proposal for Rawhide/F41, but I’d like to start the discussion early, since it is really meant for Rawhide and then land into F41.
Summarizing: with createrepo_c < 1.0 we’re currently using different settings for Rawhide repository metadata and stable releases repository metadata.
Rawhide
gzip compression for all metadata
no updateinfo.xml file is generated (no updates repository exists)
sqlite database of repodata is generated and compressed with gzip as well
comps repodata is made available both as uncompressed xml and gzipped
zchunk is active
DRPMs disabled
F40
gzip compression for primary metadata
updateinfo.xml (for updates repository) is compressed with XZ
sqlite database of repodata is generated and compressed with BZ2
comps repodata is made available both as uncompressed xml and gzipped
zchunk is active
DRPMs disabled (approved change for F40)
F<40
gzip compression for primary metadata
updateinfo.xml (for updates repository) is compressed with XZ
sqlite database of repodata is generated and compressed with BZ2
comps repodata is made available both as uncompressed xml and gzipped
zchunk is active
DRPMs enabled
With createrepo_c > 1.0 moving to zstd as the default compression type, we want to have consistent settings for all new releases and to specify those settings manually, so that possible future changes of defaults don’t cause unexpected breakages. So we propose the following settings:
Rawhide/F>=41
use --general-compress-type set to zstd to compress all metadata to zstd
updateinfo.xml (for updates repository) compressed with zstd as well
disable generating the additional sqlite database, as it was only useful for yum
comps repodata will be available only zstd compressed
zchunk is active
DRPMs disabled
F<=40
nothing should change by using the --compatibility flag of createrepo_c >= 1.0
Note that updating bodhi-composer to use createrepo_c >= 1.0 will also introduce an unavoidable change into EPEL8 updates repositories: comps repodata will be made available only in compressed format (which is set to XZ in EPEL8). Other EPEL releases (EPEL7 and EPEL9) can use the --compatibility flag to maintain actual settings. This is due to the fact createrepo_c >= 1.0 only generate compressed comps repodata, while using the --compatibility flag does generate the uncompressed file, but also overwrite all other settings. So using the flag on EPEL8 would mean to change the general metadata compression from XZ currently used to gzip.
With the unavoidable upgrade to createrepo_c >= v1.0 it will be hard, if not impossible, to retain exactly the same settings for repositories metadata.
I’ve drafted a change proposal for Rawhide/F41, but I’d like to start the discussion early, since it is really meant for Rawhide and then land into F41.
Yes. Early is good.
I think we can’t really land this in fedora land until rhel7 goes EOL
(for epel7 builds). So, it might be worth noting that in the change…?
(Unless a way to pass compat flags to createrepo_c in koji appears
before then)
Summarizing: with createrepo_c < 1.0 we’re currently using different settings for Rawhide repository metadata and stable releases repository metadata.
Thanks for the summary!
A few random questions I had:
Does zchunk care about the compression? I don’t think it does, but
would be good to confirm.
I assume dnf / dnf5 is fine with all of these changes right?
As I understand it, using --compatibility flag should make rhel7 happy about generated repodata. So we can use createrepo_c >= 1.0 also for epel7 composes.
Again, here it is how I understand it: zchunk files are just provided aside to other chosen compressed format and it has its own compression method (which is based on zstd, AFAIK). So that should not be impacted by changing other compression method.
I’m not completely sure about dnf / dnf5, but I guess they’re ok since we’re switching to use new createrepo_c defaults. I suppose they just use whatever they find in repository and adapt themselves.
As I understand it, using --compatibility flag should make rhel7 happy about generated repodata. So we can use createrepo_c >= 1.0 also for epel7 composes.
Yeah, I guess you are right. I was thinking of the koji buildroot case
where we have currently no way to pass --compatibility.
Again, here it is how I understand it: zchunk files are just provided aside to other chosen compressed format and it has its own compression method (which is based on zstd, AFAIK). So that should not be impacted by changing other compression method.
I’m not completely sure about dnf / dnf5, but I guess they’re ok since we’re switching to use new createrepo_c defaults. I suppose they just use whatever they find in repository and adapt themselves.