Is it possible to unravel the Python 'magic' packaging clusterfsck?

Python has (at least) two different implementations of a magic tool for divining file [mime]types based on filename and contents.

The file command’s libmagic has a set of Python bindings available, which are also published on PyPi as file-magic.

Then there’s the popular project GitHub - ahupp/python-magic: A python wrapper for libmagic, (published to PyPi under the same python-magic name), which as the repo link notes also uses libmagic under the hood.

For the longest time, the Fedora file source package produced a subpackage for libmagic’s official Python bindings, named python3-magic.

Back in 2020, @eclipseo pointed out that this name was a problem, as it blocked packaging of ahupp’s python-magic, and asked the file maintainers to rename their subpackage python3-file-magic in accordance with the corresponding PyPi package name.

That request was technically acquiesced to, but with the caveat that python3-file-magic would still have to have a Provides: for python3-magic to avoid breaking packages that depended on it under that name.

Ultimately, not only does python3-file-magic have a Provides: python3-magic, but it also both Obsoletes: and Conflicts: with that package. Despite the package being renamed, the file maintainers kind of salted the earth as they were vacating the old name.

Not that I’m blaming them. The real problem is, the two packages do conflict on a fundamental level.

  • python3-file-magic wants to install a Python module named magic.py, the bindings for libmagic.
  • python3-magic wants to install a package named magic.

If you were to try to install both packages together into the same Python instance, you’d end up with this unworkable site-packages structure:

/usr/lib/python3.13/site-packages/
├── magic.py
├── magic
│   ├── __init__.py

Worse still, both packages’ primary interface to the functionality of libmagic is a class named magic.Magic … with completely different, incompatible APIs.

The net result: While python-magic is packaged for Fedora (the current Fedora 41 release is python3-magic-0.4.27-9.fc41), it’s impossible to install unless you first remove python3-file-magic. (And vice-versa, technically.)

Removing python3-file-magic may be difficult to do, though, especially if you’re a Fedora packager — one of the packages with a direct requirement on it is rpmlint. All in all, on my system in order to remove python3-file-magic I’d have to be willing to do without not only rpmlint, but fedora-review, fedpkg, and python3-rpkg. Not a workable scenario.

But I therefore also have to be willing to do without python3-magic.

Oh, and as one more “fun” twist, remember those conflicting magic.Magic APIs I mentioned earlier?

There are other packages in the Fedora repo, like python3-eyed3, that contain code to interface with python3-magic. (Though without the declared dependency, because that would make them uninstallable alongside rpmlint, fedpkg, fedora-review, etc…)

If eyeD3 is run on a Fedora install without python3-magic, but with python3-file-magic, the eyeD3 mimetype plugin will successfully import magic.Magic… and immediately start throwing tracebacks, when it tries to initialize a python-file-magic object using python-magic arguments it doesn’t support.

Like I said: clusterfsck.

1 Like

From Ask Fedora to Project Discussion

Added package-maintainers

#TIL that Python can handle that situation (meaning, it doesn’t immediately uninstall itself in protest over being treated that way), but it handles it by completely ignoring the magic.py module when there’s a package directory of the same name. If you do this:

$ python3 -m venv test_venv
$ . ./test_venv/bin/activate
$ python3 -m pip install file-magic
$ python3 -m pip install python-magic

Then an import magic in the venv’s Python interpreter will load magic/__init__.py instead of magic.py. There’s no way to access magic.py from file-magic, in that scenario, unless you first python3 -m pip uninstall python-magic. (Or use importlib to load the module directly from the magic.py file. I suppose that would work.)

I just ran head-first into this same mess: logconv.py: python3-magic conflits with python3-file-magic · Issue #6544 · 389ds/389-ds-base · GitHub

I agree the current situation is not remotely optimal. It’s silly.

We should probably declare that only one of these things is allowed to be magic for Fedora’s system Python, and rename the other one (or just drop it). Anything that really, really wants to use the other one would have to be patched for the rename.

Right now there is an annoyingly 50/50 split between things using each:

[adamw@toolbx 389-ds-base (main *)]$ sudo dnf repoquery --whatrequires python3-file-magic
autokey-common-0:0.96.0-10.fc42.noarch
cobbler-0:3.3.7-3.fc42.noarch
diffoscope-0:285-1.fc42.x86_64
python3-ginga+recommended-0:5.1.0-3.fc42.noarch
python3-pass-import+decrypt-0:3.5-2.fc42.noarch
rpmlint-0:2.6.1-2.fc42.noarch
[adamw@toolbx 389-ds-base (main *)]$ sudo dnf repoquery --whatrequires python3-magic
alot-0:0.11-2.fc42.noarch
gnome-feeds-0:2.2.0-2.fc42.noarch
python3-Mastodon-0:1.8.1-8.fc42.noarch
retrace-server-0:1.24.2-9.fc42.noarch
s3cmd-0:2.4.0-5.fc42.noarch

Which means you can’t install any of those simultaneously, which is pretty silly. In addition to the rpmlint dep, sos recommends python3-file-magic, and sos is in both the ‘standard’ and ‘workstation-product’ groups, so it’s in an awful lot of Fedora installs. That makes me think it should be the winner, and python3-magic should be forced to rename.

diffoscope gets bonus points for requiring python3-file-magic and recommending python3-magic. good luck with that.

@churchyard do you have any thoughts here?

I don’t know how much the API differs, but if we are to rename the module and patch every dependent to use the downstream-only name, would it be easier to patch the dependents to use the other package instead?

This pretty much says that it should be “trivial” to port everything over to python3-magic just by changing the metadata.

I doubt it. Even if they were very similar (and I’m not sure they are; python-magic’s magic.Magic constructor takes a bunch of arguments, file-magic’s takes none) – but even if they were, a rename only requires patching the import statements, not any of the calls.

…The compatibility layer could be some help, but it’s kind of in the wrong package. file-magic is the one that we’re “stuck with” (due to existing uses). Converting file-magic users to use python-magic via its compatibility API is kind of going in the wrong direction.

Holy hell. I never even noticed that. Way to escalate from clusterfsck to clusterfisting, diffoscope maintainers! :+1:

That was my initial thinking as well. Also, python3-file-magic is the “official” bindings (the ones built from the libmagic source tree), so my knee-jerk reaction is to say that they get control of the name purely based on that.

But then I thought about it some more. The thing is that if we were to rename python-magic… what does it get renamed to? That’s the tricky question.

Whereas python-file-magic has two very good, very obvious options for renaming its module: file_magic.py (to match the PyPi package name), and libmagic.py (obvious). magic.py is actually a pretty BAD name for that module, even though it’s the one the libmagic maintainers give it.

There’s a part of me that sort of wants to try and engage the maintainers of these packages in this, and sort out a rename upstream (somewhere) so that this gets solved for everyone, permanently. Rather than us having to carry downstream patches to who knows how many packages forever.

In a sense we almost have a “duty of care” to encourage the upstreams to arrive at some sort of compromise. The situation feels untenable, even if venvs allow both packages to pretend the other doesn’t exist for the most part. Seems like the two camps should want to work this out. But I haven’t seen any sign that they’ve ever discussed the issue directly. (I haven’t looked extensively, tho, and since you can’t prove a negative I can’t say that they definitely HAVEN’T, either. I just haven’t seen it.)

And so there’s a saner, less idealistic part of me that dreads the thought of that. This is a long-standing issue that’s clearly been noticed by (presumably) both camps, whether or not they’ve discussed it between them.

Discussions like this one are often contentious and rarely fun for anyone involved. Even if my approach of gentle obsequiousness did eventually carry the day, and lead to a happy ending in that particular case. (And believe me, I’m as surprised as anyone. More, actually, because I understand just how completely out of character that was for me.)

1 Like