[TRIAGED] [rawhide] anyone aware of why cargo has been memleaking?

Ask Fedora
1

For random reasons most if not all of the rust packages on Terra Rawhide are facing this exact issue[1][2][3] since the last few weeks:

rpmbuild │ + /usr/bin/env CARGO_HOME=.cargo RUSTC_BOOTSTRAP=1 'RUSTFLAGS=-Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Cstrip=none -Cforce-frame-pointers=yes -Clink-arg=-specs=/usr/lib/rpm/redhat/redhat-package-notes --cap-lints=warn -C link-arg=-fuse-ld=mold' /usr/bin/cargo build -j4 -Z avoid-dev-deps --profile rpm
rpmbuild │ warning: both `/github/home/rpmbuild/BUILD/ruffle-nightly-20250523-build/ruffle-nightly-2025-05-23/.cargo/config` and `/github/home/rpmbuild/BUILD/ruffle-nightly-20250523-build/ruffle-nightly-2025-05-23/.cargo/config.toml` exist. Using `/github/home/rpmbuild/BUILD/ruffle-nightly-20250523-build/ruffle-nightly-2025-05-23/.cargo/config`
rpmbuild │     Updating crates.io index
rpmbuild │     Updating git repository `https://github.com/ruffle-rs/rust-flash-lso`
rpmbuild │     Updating git repository `https://github.com/ruffle-rs/nellymoser`
rpmbuild │     Updating git repository `https://github.com/ruffle-rs/regras3`
rpmbuild │     Updating git repository `https://github.com/ruffle-rs/h263-rs`
rpmbuild │     Updating git repository `https://github.com/ruffle-rs/nihav-vp6`
rpmbuild │     Updating git repository `https://github.com/kyren/gc-arena.git`
rpmbuild │  Downloading crates ...
rpmbuild │ error: failed to download `unicode-segmentation v1.12.0`
rpmbuild │ 
rpmbuild │ Caused by:
rpmbuild │   Out of memory
rpmbuild │ error: Bad exit status from /var/tmp/rpm-tmp.SldqOd (%build)

This only happens on Rawhide. Since Fedora 42 and Rawhide have the same cargo version, we can assume this is an issue not with cargo itself.

The command is the expansion of %cargo_license_summary_online, our own version of %cargo_license_summary:

$ rpm -E '%cargo_license_summary'

(
set -euo pipefail
/usr/bin/env CARGO_HOME=.cargo RUSTC_BOOTSTRAP=1 RUSTFLAGS='-Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Cstrip=none -Cforce-frame-pointers=yes --cap-lints=warn' /usr/bin/cargo tree -Z avoid-dev-deps --workspace --offline --edges no-build,no-dev,no-proc-macro --no-dedupe --target all --prefix none --format "# {l}" | sed -e "s: / :/:g" -e "s:/: OR :g" | sort -u
)

$ rpm -E '%cargo_license_summary_online'

(
set -euo pipefail
/usr/bin/env CARGO_HOME=.cargo RUSTC_BOOTSTRAP=1 RUSTFLAGS='-Copt-level=3 -Cdebuginfo=2 -Ccodegen-units=1 -Cstrip=none -Cforce-frame-pointers=yes --cap-lints=warn' /usr/bin/cargo tree -Z avoid-dev-deps --workspace --edges no-build,no-dev,no-proc-macro --no-dedupe --target all --prefix none --format "# {l}" | sed -e "s: / :/:g" -e "s:/: OR :g" | sort -u
)

As you can see the only difference is the lack of the --offline flag.

Is Fedora upstream also facing the same issue? Is there an ongoing fix / bug tracker?

  1. bump(nightly): legcord-nightly zed-nightly · terrapkg/packages@0100e50 · GitHub ↩︎

  2. bump: rust-mise · terrapkg/packages@782e3df · GitHub ↩︎

  3. bump: ruffle-nightly chezmoi electron · terrapkg/packages@a8df0e6 · GitHub ↩︎

2

The fedora package builders (mock/koji) doesn’t have network access during build so doesn’t download anything.

3

so it has something to do with broken networking on rawhide?

4

please follow the steps outlined here [1]

[1] Rust Packaging Guidelines :: Fedora Docs

5

This discussion is related to why cargo on rawhide is broken, not about how things should be packaged in Fedora.

6

The reason I’m posting this here is that I don’t know if Fedora is aware of the fact that cargo has been straight up memleaking since the last few weeks. As aforementioned, I believe the root cause of the issue lies within Fedora Rawhide, not in rust or our builders. I’m looking for a possible ETA for the fix, and I don’t want the issue to affect anything outside of rawhide (e.g. 42).

7

what is the spec fie you’re using?

8

Examples of the spec files:

From the last example, it is clear that the issue is not specific to cargo tree, as %cargo_build also failed (with internet access).

9

Seems like it has been failing since 2025-05-05. The last successful build was on 2025-05-04 02:55 UTC. The first failing build was on 2025-05-05 09:08.

10

We have seen zero OOM issues in rawhide builds for “official” Fedora packages, so this is the reason why we’ve not been “aware”. Our builds don’t have network access, so it’s possible that we just don’t hit the code paths that cause this issue.

The only change that I could point to that might be a possible cause is that on ~May 3rd, curl was updated to 8.14-rc1. No other relevant system-level libraries were updated during that timeframe that I can see.

Can you try reproducing this issue with upstream cargo? i.e. install a stable Rust toolchain via rustup and then run the command that OOMs with that toolchain to see if it fails with the “official” toolchain too?

1 Like
11

Good idea, just tried that out in podman with our builders, the rustup version of cargo works fine.

12

Between 2025-05-03 00:59 and 2025-05-04 01:06, registry.fedoraproject.org/fedora-minimal@sha256:3abc736dccc79916b85bed8d5769ba377520791a38b14910645e33ba390a82e0 was pulled.
On the next day (2025-05-05 01:04) 23e48b985d9da980a03ffd208a65d1651ae303cad42fff4168a396ff3d210de2 was pulled.

I went ahead and diffed their rpm -qa '*', the only difference is:

libcurl-8.13.0-1.fc43.x86_64
curl-8.13.0-1.fc43.x86_64

becomes

libcurl-8.14.0~rc1-1.fc43.x86_64
curl-8.14.0~rc1-1.fc43.x86_64

I think you are right

13

Thank you for confirming - can you please file a bug against the curl package in Fedora then?

14

already on it!

1 Like
15
1 Like
16

Thanks to everyone in the discussion!

1 Like