F42 Change Proposal: Unify /usr/bin and /usr/sbin (System-Wide)

Hi @zbyszek,
I’m not sure if this is the right discussion thread since the change is now accepted for Fedora 42 but I couldn’t find a newer one.
On Fedora 42 at this moment, /usr/sbin is linked to /bin but the $PATH still contains /usr/sbin. The summary says, that /usr/sbin should be removed from $PATH. When is this change coming? I was looking for it in the shadowutils package but I cannot see it there.

The problem can be spotted by running which ruby which returns /usr/sbin/ruby. Ruby is actually located in /usr/bin/ruby.

Oh, hmmm. This probably should be updated to say F42. Confused me for a moment there :slight_smile:

You are probably confused by something that incorrectly extends $PATH. There are multiple candidates: .bashrc, .bash_profile, /etc/profile, the desktop environment, possibly some other things. I’m pretty sure that systemd provides a correct initial value of $PATH.

To test going through the levels:

$ sudo systemd-run -t env | grep PATH
PATH=/usr/local/bin:/usr/bin
$ systemd-run --user -t env | grep PATH
PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin
$ env | grep PATH          
PATH=/home/zbyszek/bin:/home/zbyszek/.local/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin

So in my case, the system manager sets the path correctly, but the environment in the user instance is already wrong. This is most likely not because the manager sets it wrong, but because something injects the value into the user manager environment block.

When you see a wrong value, please specify where you are seeing it, and ideally open a bug against the appropriate component.

EDIT: two comments:

  • I restarted the container I was testing things in, and now the user session doesn’t have the duplicated $PATH. So it seems something changed, but I’m not sure in which component. This needs further testing in more clean environments…
  • To get the simplified $PATH, the system needs to be fully merged. On upgraded systems, at this point, most likely /usr/sbin is still a directory.
1 Like

I first observed this issue in our CI that uses Testing Farm where a fresh system image is used for every run. When I install a Fedora Rawhide VM without a desktop environment, I see the same behavior. I do not see this problem on updated systems because the folders are not merged. I ran the commands in a Testing Farm machine and in a fresh Fedora 42 VM (so the /usr/sbin/ is a symlink in my case) and got a similar result as you.

Testing Farm:

# systemd-run -t env | grep PATH
PATH=/usr/local/bin:/usr/bin
# systemd-run --user -t env | grep PATH
PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin
# env | grep PATH
PATH=/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

Freshly installed Fedora 42 VM:

# systemd-run -t env | grep PATH
PATH=/usr/local/bin:/usr/bin
# systemd-run --user -t env | grep PATH
PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin
# env | grep PATH
PATH=/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

The reason why I brought up shadow-utils is that I and my colleague determined that is who might be setting $PATH initially. I had a look at https://src.fedoraproject.org/rpms/shadow-utils/blob/7b5d11c4b7d0d91c4d879b18facaefd7975544a5/f/shadow-utils.login.defs#_88 which documents that the default path for all users contains /usr/sbin. The actual default path is then contained in the shadow-utils source and I didn’t see any patches in the dist-git repo for that. I’m not quite sure how PATH is actually set, it was just something that we found and thought could be the issue… Is that completely irrelevant?

From the files that you suggested, .bashrc and .bash_profile do not modify the $PATH on these systems, I checked. However, /etc/profile that belongs to the setup package could add some sbin locations:

pathmunge () {
    case ":${PATH}:" in
        *:"$1":*)
            ;;
        *)
            if [ "$2" = "after" ] ; then
                PATH=$PATH:$1
            else
                PATH=$1:$PATH
            fi
    esac
}

# Path manipulation
if [ "$EUID" = "0" ]; then
    pathmunge /usr/sbin
    pathmunge /usr/local/sbin
else
    pathmunge /usr/local/sbin after
    pathmunge /usr/sbin after
fi

This would suggest that for root, /usr/local/sbin:/usr/sbin should be added at the start of the $PATH if the folders are not already there. For other users, /usr/local/sbin:/usr/sbin should be added at the end if the folders are not already there.

I don’t think that this file modifies the $PATH in this case since the $PATH from our reproducers ends with /usr/local/sbin:/usr/bin:/usr/sbin and not /usr/local/sbin:/usr/sbin. I think the $PATH looks suspiciously similar to what I saw in shadow-utils, I am just not sure if that is the case. What do you think?

Ooops, this was indeed a problem in systemd. In fact, it was caused by a change I introduced myself :frowning:

https://src.fedoraproject.org/rpms/systemd/c/cc9bce7893d956c5e0a5041e0890150b21c0d6d7?branch=rawhide
should fix the issue. I won’t rebuild just for this, but we’ll rebuild soon and then this should be resolved.

Thank you for the report and the debugging!

Closing change proposals for F40. The ship has sailed. Of course we can still discuss if needed, but please open new topics for that.

The merge script fails when upgrading from F41.

Here’s a way to force it from a text TTY: Ctrl+Alt+F5

  • Check the status message to identify the conflicting packages:
sudo dnf reinstall filesystem -y | cat
  • Temporarily remove the conflicting packages until the issue goes away:
sudo rpm -e --nodeps gdm
sudo rpm -e --nodeps dnsmasq
  • Restore the removed packages and restart the affected services:
sudo dnf install gdm dnsmasq
sudo systemctl restart gdm.service
  • Verify the results:
> echo ${PATH}
/usr/local/bin:/usr/bin

> readlink /usr/local/sbin
bin

> readlink /usr/sbin
bin

What do you mean by fails? If the script reports that sbin could not be merged, this is not an error, just informative output.

This is OK if you know what you’re doing, but is a fairly dangerous step, easy to get wrong.

If the script cannot merge the directories, we will not be able achieve the declared goals and consolidate the PATH when upgrading to F42 unless we take manual action.

If this behavior is expected, then it is probably worth documenting in the OP to clarify that the merge will not happen automatically on upgrade.

Based on my years of experience upgrading Fedora, it is best to minimize the gap between an upgraded installation and a new one rather than rely on backward compatibility.

1 Like

Perhaps not in F42, but soon enough, upgrades
will fail in all sorts of unusual ways as there will
be an assumption[0] that /usr/bin and /usr/sbin
merging has already successfully occured. Adjust
the upgrade scripts to do what is needed, adjust
the related rpms, or abort the upgrade proposal
and revert. There is no other proper way.

[0] Which will demonstrate that assume means
to make an [ass] of [u] and [me].

1 Like

Packages that still have files in /usr/sbin prevent the full merge from being finalized. This is usually because the package FTBFS and wasn’t rebuilt successfully in the mass rebuild or later. This is a bug on its own, since we want packages to be buildable. Please complain to the maintainers of those packages (there should be a bugzilla bug already open, and it’s best to ping the maintainers there) or ideally help to fix the FTBFS is the maintainers have trouble doing that. I cannot handle all the cases myself ;(

I submitted an update for criu and a PR for dnsmasq today. But there’s still many other packages to be rebuilt…

1 Like

Thanks for confirming this is a failed proposal as
it is not complete and should be marked as such.
Maybe for F43 when you are actually complete?
As a PP, you have, under your capabilities, the
ability to fix this to be actually complete.

This proposal has been accepted and implemented for Fedora 42. Reverting it at this point would be a lot of work. If there are packages that needs to be updated, it would be better to file individual bugs for those packages and contribute the changes if possible. Thanks

1 Like

Is there a list of packages that are still being worked on? I looked at the tracker bug for this change, but the list of blocking bugs looks incomplete.

On my pretty standard Fedora 42 Workstation system (upgraded from Fedora 40 → 41 → 42), there are still some packages installed that prevent the migration:

  • dmraid / dmraid-events
  • iscsi-initiator-utils
  • vpnc

And two non-link files which are unowned by any package:

  • /usr/sbin/iconvconvert
  • /usr/sbin/ldconfig

How are those ever supposed to get cleaned up? If no package owns them, no package update can remove them …

I don’t think there’s a list. I use find /usr/sbin/ -type f | xargs rpm -qf | pkgname | sort -u to get a list of packages affecting a system.

  • dmraid / dmraid-events
  • iscsi-initiator-utils
  • vpnc

They need to be rebuilt. Those all failed in the mass rebuild. This is something the maintainers must fix.

And two non-link files which are unowned by any package:

  • /usr/sbin/iconvconvert
  • /usr/sbin/ldconfig

I don’t know anything about iconvconvert.

ldconfig should be a symlink to /usr/bin/ldconfig. I suspect that you might have upgraded early before all the kinks in the scriptlets were ironed out. AFAIK, now the scriptlets work correctly. If it’s just you, i.e. if it was only an issue in the early versions of the packages, I would prefer if you fix it yourself. But if it’s a more widespread issue, the scriptlets can be adjusted to handle it.

1 Like

This an experience report, having just upgraded to Fedora 42 from a system that is fairly “boring,” but has been upgraded over quite a few Fedora releases.

$ find /usr/sbin/ -type f | xargs rpm -qf | pkgname | sort -u

bpftool
dnsmasq
file /usr/sbin/capsh is not owned by any package
file /usr/sbin/getcap is not owned by any package
file /usr/sbin/getpcaps is not owned by any package
file /usr/sbin/iconvconfig is not owned by any package
file /usr/sbin/lockdev is not owned by any package
file /usr/sbin/sasldblistusers2 is not owned by any package
file /usr/sbin/saslpasswd2 is not owned by any package
glibc
iscsi-initiator-utils
iscsi-initiator-utils-iscsiuio
libcap
sheepdog
syslinux-extlinux
vpnc

The files /usr/sbin/{capsh,getcap,getpcaps} all had mtime 2024-07-17, and had corresponding, newer equivalents in /usr/bin provided by libcap. The file /usr/sbin/lockdev also had mtime 2024-07-17, and had a newer equivalent in /usr/bin provided by lockdev. The file /usr/sbin/iconvconfig had mtime 2025-03-12, and had a newer equivalent in /usr/bin provided by glibc. The files /usr/sbin/{sasldblistusers2,saslpasswd2} had mtime 2024-07-16, and had newer equivalents in /usr/bin provided by cyrus-sasl-lib. I don’t have any insight into how these unowned files came to exist in /usr/sbin, but I concluded that manually removing each of them was the right thing to do.

I was still surprised to see packages like glibc and libcap apparently blocking the merge.

1 Like

I don’t think glibc and libcap are a problem. I’m seeing successful upgrades with those files installed. Maybe glibc is reported because it has a virtual Provides:/usr/sbin/ldconfig ?

Some packages were fixed. I filed some more pull requests and bodhi updates that should most of the problems listed by your:

https://bodhi.fedoraproject.org/updates/FEDORA-2025-db618ec20b

https://bodhi.fedoraproject.org/updates/FEDORA-2025-1c01f56ace

3 Likes