Is it possible to unravel the Python 'magic' packaging clusterfsck?

Project Discussion
1

Python has (at least) two different implementations of a magic tool for divining file [mime]types based on filename and contents.

The file command’s libmagic has a set of Python bindings available, which are also published on PyPi as file-magic.

Then there’s the popular project GitHub - ahupp/python-magic: A python wrapper for libmagic, (published to PyPi under the same python-magic name), which as the repo link notes also uses libmagic under the hood.

For the longest time, the Fedora file source package produced a subpackage for libmagic’s official Python bindings, named python3-magic.

Back in 2020, @eclipseo pointed out that this name was a problem, as it blocked packaging of ahupp’s python-magic, and asked the file maintainers to rename their subpackage python3-file-magic in accordance with the corresponding PyPi package name.

That request was technically acquiesced to, but with the caveat that python3-file-magic would still have to have a Provides: for python3-magic to avoid breaking packages that depended on it under that name.

Ultimately, not only does python3-file-magic have a Provides: python3-magic, but it also both Obsoletes: and Conflicts: with that package. Despite the package being renamed, the file maintainers kind of salted the earth as they were vacating the old name.

Not that I’m blaming them. The real problem is, the two packages do conflict on a fundamental level.

  • python3-file-magic wants to install a Python module named magic.py, the bindings for libmagic.
  • python3-magic wants to install a package named magic.

If you were to try to install both packages together into the same Python instance, you’d end up with this unworkable site-packages structure:

/usr/lib/python3.13/site-packages/
├── magic.py
├── magic
│   ├── __init__.py

Worse still, both packages’ primary interface to the functionality of libmagic is a class named magic.Magic … with completely different, incompatible APIs.

The net result: While python-magic is packaged for Fedora (the current Fedora 41 release is python3-magic-0.4.27-9.fc41), it’s impossible to install unless you first remove python3-file-magic. (And vice-versa, technically.)

Removing python3-file-magic may be difficult to do, though, especially if you’re a Fedora packager — one of the packages with a direct requirement on it is rpmlint. All in all, on my system in order to remove python3-file-magic I’d have to be willing to do without not only rpmlint, but fedora-review, fedpkg, and python3-rpkg. Not a workable scenario.

But I therefore also have to be willing to do without python3-magic.

Oh, and as one more “fun” twist, remember those conflicting magic.Magic APIs I mentioned earlier?

There are other packages in the Fedora repo, like python3-eyed3, that contain code to interface with python3-magic. (Though without the declared dependency, because that would make them uninstallable alongside rpmlint, fedpkg, fedora-review, etc…)

If eyeD3 is run on a Fedora install without python3-magic, but with python3-file-magic, the eyeD3 mimetype plugin will successfully import magic.Magic… and immediately start throwing tracebacks, when it tries to initialize a python-file-magic object using python-magic arguments it doesn’t support.

Like I said: clusterfsck.

1 Like
2

From Ask Fedora to Project Discussion

3

Added package-maintainers

4

#TIL that Python can handle that situation (meaning, it doesn’t immediately uninstall itself in protest over being treated that way), but it handles it by completely ignoring the magic.py module when there’s a package directory of the same name. If you do this:

$ python3 -m venv test_venv
$ . ./test_venv/bin/activate
$ python3 -m pip install file-magic
$ python3 -m pip install python-magic

Then an import magic in the venv’s Python interpreter will load magic/__init__.py instead of magic.py. There’s no way to access magic.py from file-magic, in that scenario, unless you first python3 -m pip uninstall python-magic. (Or use importlib to load the module directly from the magic.py file. I suppose that would work.)