Virsh attach-disk: unable to execute QEMU command 'device_add'

Ask Fedora
1

Fedora 36 Workstation here. This appears to come up sometimes (there’s even this closed #1012085 from 2013 about this), and I first suspected SELinux to be the reason for this:

$ virsh attach-disk --domain sles12 --source /mnt/nfs/data/tmp/sles12.iso --target vdb
error: Failed to attach disk
error: internal error: unable to execute QEMU command 'device_add': Could not open '/mnt/nfs/data/tmp/sles12.iso': Permission denied

Yes, virt_use_nfs is enabled, but it still would not work. But even with disabling SELinux it wouldn’t budge. Weird. The .iso should really be accessible by all users on this machine, including the qemu user:

$ ls -ld /{,mnt{,/nfs{,/data{,/tmp{,/sles12.iso}}}}}
dr-xr-xr-x.  1 root root        158 Aug 12 22:51 /
drwxr-xr-x.  1 root root         52 Aug  9 15:57 /mnt
drwxr-xr-x.  1 root root         18 Oct 24  2021 /mnt/nfs
drwxr-xr-x. 13 root root       4096 Jul 25  2021 /mnt/nfs/data
drwxrwxrwt.  9 root root       4096 Sep 20 17:46 /mnt/nfs/data/tmp
-r--r--r--.  2 root root 3836739584 Sep 14  2018 /mnt/nfs/data/tmp/sles12.iso

$ sudo -u qemu file /mnt/nfs/data/tmp/sles12.iso
/mnt/nfs/data/tmp/sles12.iso: ISO 9660 CD-ROM filesystem data (DOS/MBR boot sector) 'SLE-12-SP3-Server-DVD-x86_640473' (bootable)

When trying to attach the disk, the following happens in syslog:

$ getenforce 
Permissive

$ journalctl
[....]
Sep 20 18:00:29 polkitd[1571]: Registered Authentication Agent for unix-process:489289:11927827 (system bus name :1.50320 [/usr/bin/pkttyagent --process 489289 --notify-fd 4 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_GB.UTF-8)
Sep 20 18:00:29 virtqemud[489296]: libvirt version: 8.1.0, package: 2.fc36 (Fedora Project, 2022-03-13-01:12:58, )
Sep 20 18:00:29 virtqemud[489296]: XML error: Invalid value for attribute 'value' in element 'allowReboot': 'default'.
Sep 20 18:00:29 polkitd[1571]: Unregistered Authentication Agent for unix-process:489289:11927827 (system bus name :1.50320, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_GB.UTF-8) (disconnected from bus)
Sep 20 18:00:29 audit[1442]: AVC avc:  denied  { read } for  pid=1442 comm="systemd-machine" scontext=system_u:system_r:systemd_machined_t:s0 tcontext=unconfined_u:unconfined_r:svirt_t:s0:c17,c959 tclass=file permissive=1
Sep 20 18:00:29 audit[1442]: AVC avc:  denied  { search } for  pid=1442 comm="systemd-machine" name="481421" dev="proc" ino=1646351 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=unconfined_u:unconfined_r:svirt_t:s0:c17,c959 tclass=dir permissive=1
Sep 20 18:00:29 audit[1442]: AVC avc:  denied  { open } for  pid=1442 comm="systemd-machine" path="/proc/481421/cgroup" dev="proc" ino=1677031 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=unconfined_u:unconfined_r:svirt_t:s0:c17,c959 tclass=file permissive=1
Sep 20 18:00:29 audit[1442]: AVC avc:  denied  { getattr } for  pid=1442 comm="systemd-machine" path="/proc/481421/cgroup" dev="proc" ino=1677031 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=unconfined_u:unconfined_r:svirt_t:s0:c17,c959 tclass=file permissive=1
Sep 20 18:00:29 audit[1442]: AVC avc:  denied  { ioctl } for  pid=1442 comm="systemd-machine" path="/proc/481421/cgroup" dev="proc" ino=1677031 ioctlcmd=0x5401 scontext=system_u:system_r:systemd_machined_t:s0 tcontext=unconfined_u:unconfined_r:svirt_t:s0:c17,c959 tclass=file permissive=1
Sep 20 18:00:29 virtqemud[489296]: error from service: GDBus.Error:org.freedesktop.machine1.NoMachineForPID: PID 481421 does not belong to any known machine
Sep 20 18:00:29 virtqemud[489296]: internal error: unable to execute QEMU command 'device_add': Could not open '/mnt/nfs/data/tmp/sles12.iso': Permission denied
Sep 20 18:00:29 virtqemud[489296]: cannot lookup default selinux label for /mnt/nfs/data/tmp/sles12.iso

$ ps -fp 489296,481421
UID          PID    PPID  C STIME TTY          TIME CMD
christi+  489296    2786  0 18:00 ?        00:00:00 /usr/sbin/virtqemud --timeout=120
christi+  481421    2786  1 17:41 ?        00:00:29 /usr/bin/qemu-system-x86_64 -name guest=sles12,debug-threads=on -S [...]

I’ve never seen virtqemud before, but maybe that’s a new thing now. The daemon is running as the same user (myself) trying to attach that disk, and yes, I do have access to that .iso as can be seen above. Also, this is a read-only NFS mount, so I cannot use magic SELinux commands, but this doesn’t seem to be related to SELinux anyway here.

Any ideas why this isn’t working?

2

What are the ownership, permissions, and yes, even the selinux content of ‘/mnt/nfs/data/tmp/sles12.iso’?
ls -laZ /mnt/nfs/data/tmp/sles12.iso will show that.
Yes, even the selinux permissions of the running process virtqemud may be involved even with selinux in permissive mode. Ownership and permissions of the daemon accessing the file, the path to the file, and the file itself are a factor whenever the error message is about permissions.

3

Found a workaround, this even works with SELinux enabled again (as it should be):

$ sudo losetup -r -f /mnt/nfs/data/tmp/sles12.iso
$ sudo setfacl -m g:libvirt:rw /dev/loop0
$ sudo chcon -t svirt_image_t /dev/loop0

$ virsh attach-disk --domain sles12 --source /dev/loop0 --target vdb --mode readonly

But it’d be nice to get this done w/o this trick :slight_smile:

4

Indeed, forgot about the SELinux information in that ls output:

$ ls -Zld /{,mnt{,/nfs{,/data{,/tmp{,/sles12.iso}}}}}
dr-xr-xr-x.  1 root root system_u:object_r:root_t:s0           158 Aug 12 22:51 /
drwxr-xr-x.  1 root root system_u:object_r:mnt_t:s0             52 Aug  9 15:57 /mnt
drwxr-xr-x.  1 root root unconfined_u:object_r:mnt_t:s0         18 Oct 24  2021 /mnt/nfs
drwxr-xr-x. 13 root root system_u:object_r:nfs_t:s0           4096 Jul 25  2021 /mnt/nfs/data
drwxrwxrwt.  9 root root system_u:object_r:nfs_t:s0           4096 Sep 20 17:46 /mnt/nfs/data/tmp
-r--r--r--.  2 root root system_u:object_r:nfs_t:s0     3836739584 Sep 14  2018 /mnt/nfs/data/tmp/sles12.iso

So, in this particular case I do have access to the NFS server, and the files. But the server is not configured with SELinux, and I cannot change the lables on a readonly mount. And, as it doesn’t work with SELinux disabled, it might not even be related to SELinux this time.

5

Looks to me like the daemon requires acl access or selinux access (or both) even though the nfs file system is readable and selinux was permissive.

Your workaround set both acl access and selinux context for the file access.

6

Please note, the workaround even works with SELinux set to “Permissive” and no SELinux labels needed to be set (only permissions, of course) – as it should be.

sudo losetup -D
sudo rmmod loop
sudo losetup -r -f /mnt/nfs/data/tmp/sles12.iso
sudo setfacl -m g:libvirt:rw /dev/loop0

No extra SELinux label was set here:

$ ls -lZ /dev/loop0 
brw-rw----+ 1 root disk system_u:object_r:fixed_disk_device_t:s0 7, 0 Sep 21 12:11 /dev/loop0
7

FWIW, it turns out the ISO images can be attached to the VM, but one really has to use --mode readonly for the target if the source is readonly too (be it via its Unix permissions, or because it’s located on a readonly mount):

$ test -w /mnt/nfs/data/tmp/sles12.iso; echo $?
1

$ virsh attach-disk --domain sles12 --source /mnt/nfs/data/tmp/sles12.iso --target vdb
error: Failed to attach disk
error: internal error: unable to execute QEMU command 'device_add': Could not open '/mnt/nfs/data/tmp/sles12.iso': Permission denied

$ virsh attach-disk --domain sles12 --source /mnt/nfs/data/tmp/sles12.iso --target vdb --mode readonly
Disk attached successfully

$ test -w /mnt/nfs/data/tmp/debian.iso; echo $?
0

$ virsh attach-disk --domain sles12 --source /mnt/nfs/data/tmp/debian.iso --target vdc 
Disk attached successfully

$ ls -lZ /mnt/nfs/data/tmp/{debian,sles12}.iso
-rw-rw-rw-. 1 root root system_u:object_r:nfs_t:s0  351272960 Jan 12  2020 /mnt/nfs/data/tmp/debian.iso
-r--r--r--. 2 root root system_u:object_r:nfs_t:s0 3836739584 Sep 14  2018 /mnt/nfs/data/tmp/sles12.iso