Status of marking the base Python environment as externally managed (PEP 668)

Ask Fedora
1

What’s the status of this change? Changes/PythonMarkExternallyManaged - Fedora Project Wiki

It is marked as targeting Fedora 38, but it does not seem to have been implemented (fd EXTERNALLY-MANAGED / finds nothing), and I could not find if it was actually implemented then reverted for some reason and whether it is planned to go in a future Fedora release.

2

That proposal was never submitted. There seems to be a summary of the issues that held it up in the linked thread on Python’s Discourse:

Basically, the intent was that pip would install to /usr/local and only /usr would be protected, but the PEP doesn’t allow that.

3

And worse the PEP does not allow a user to install into ~/ either.

You must use a venv or add a scary --i-want-to-mess-up-the-system option to pip.

4

As I understand it, that behavior is wanted because installing into ~/.local/lib/pythonX.Y/site-packages/ shadows system packages, which can result in effectively breaking them.

I had hoped the PEP would be implemented, since it’s IMHO a great reliability improvement, but thanks for explaining that it was intentional not to do it.

5

(Note that Debian and thus Ubuntu did implement PEP 668, for what it’s worth.)

6

Fedora added the -s to the shebang lines to avoid seeing the packages in ~/.local.

But that is not seen as a solution that was acceptable.

7

Did it? I’m on Fedora 38 and I have

~ $ head -1 /usr/bin/meson
#!/usr/bin/python3
~ $ head -1 /usr/bin/dnf
#!/usr/bin/python3

I unpacked the meson RPM from meson-1.2.3-1.fc39 | Build Info | koji to see if it had changed in Fedora 39 but the shebang line is the same.

8

I also read this in PEP 668:

System Python

Shouldn’t distro software just run with the distro site-packages directory alone on sys.path and ignore the local system administrator’s site-packages as well as the user-specific one? This is a worthwhile idea, and various versions of it have been circulating for a while under the name of “system Python” or “platform Python” (with a separate “user Python” for end users writing Python or installing Python software separate from the system). However, it’s much more involved of a change. First, it would be a backwards-incompatible change. As mentioned in the Motivation section, there are valid use cases for running distro-installed Python applications like Sphinx or Ansible with locally-installed Python libraries available on their sys.path. A wholesale switch to ignoring local packages would break these use cases, and a distro would have to make a case-by-case analysis of whether an application ought to see locally-installed libraries or not.

Furthermore, Fedora attempted this change and reverted it, finding, ironically, that their implementation of the change broke their package manager.

9

(CC @churchyard)

10

Some packages do use -s other do not use the -s, it was never consistent.

For example these do use the -s:

rpmdiff #! /usr/bin/python3 -sP
rpmlint #! /usr/bin/python3 -sP
rust2rpm #! /usr/bin/python3 -sP
rust2rpm.~ #! /usr/bin/python3 -sP
11

This change was never submitted. I have tried to implement a proof of concept and I failed.

If you are interested in details, here’s my talk on the topic from PyConPl’22.

Regarding the -s flag, there are two types of Fedora packages that don’t use it in Python shebangs.

  1. Packages that explicitly opt-out from the flag because they want to support user-pip-installed Python extensions/modules/plugins… This includes e.g. python-notebook, ansible, pytest, or tox.
  2. Package using non-Python installation methods (i.e. they install Python files by autottools or cmake, or cp) where we don’t automatically manipulate the shebang flags; hence, they don’t get the flag unless they opted in (which they usually don’t). This seems to include meson and dnf, but also many others.

Sometimes, the packages from the second group should belong to the first group (for example dnf supports pip-installed (dependencies for) plugins, meosn might as well) rather than be fixed to have the flag.

Nevertheless, the -s flag does not solve all the problems.

1 Like