CoreOS as KVM Hypervisor

I’ve been looking to introduce a slim ideally immutable OS as the basis for Kubernetes and the occasional VM and in theory CoreOS seems to fit the bill. I’m struggling to get libvirtd running though, while the service starts it shuts down nearly immediately again.

Steps:

  • fresh install CoreOS
  • rpm-ostree override remove nfs-utils-coreos (otherwise conflict with nfs-utils)
  • rpm-ostree install virt-install libvirt-daemon-config-network libvirt-daemon-kvm qemu-kvm libguestfs-tools python3-libguestfs virt-top
  • systemctl enable --now libvirtd

Journal output:

Jun 27 06:08:35 fcostest systemd[1]: Starting libvirtd.service - Virtualization daemon...
Jun 27 06:08:35 fcostest systemd[1]: Started libvirtd.service - Virtualization daemon.
Jun 27 06:08:35 fcostest dnsmasq[1437]: started, version 2.86 cachesize 150
Jun 27 06:08:35 fcostest dnsmasq[1437]: compile time options: IPv6 GNU-getopt DBus no-UBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth cryptohash DNSSEC loop-detect inotify dumpfile
Jun 27 06:08:35 fcostest dnsmasq-dhcp[1437]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
Jun 27 06:08:35 fcostest dnsmasq-dhcp[1437]: DHCP, sockets bound exclusively to interface virbr0
Jun 27 06:08:35 fcostest dnsmasq[1437]: reading /etc/resolv.conf
Jun 27 06:08:35 fcostest dnsmasq[1437]: using nameserver 127.0.0.53#53
Jun 27 06:08:35 fcostest dnsmasq[1437]: read /etc/hosts - 2 addresses
Jun 27 06:08:35 fcostest dnsmasq[1437]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jun 27 06:08:35 fcostest dnsmasq-dhcp[1437]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Jun 27 06:10:35 fcostest systemd[1]: libvirtd.service: Deactivated successfully.
Jun 27 06:10:35 fcostest systemd[1]: libvirtd.service: Unit process 1437 (dnsmasq) remains running after unit stopped.
Jun 27 06:10:35 fcostest systemd[1]: libvirtd.service: Unit process 1438 (dnsmasq) remains running after unit stopped.

I found some posts mentioning that libvirt needs specific localizations installed, which doesn’t seem to be possible in CoreOS?
I’ve read that KubeVirt and others managed to containerize libvirt? Any pointers here maybe?

I’ve tried KubeVirt as well and while that is the long term goal I’d feel more comfortable starting out with a libvirt based approach for now…

Welcome to :fedora: with your very interesting idea!

Generally, I suggest to put forward your question at ask.Fedora since this is the first line support experienced in handling such issues. You can use the search function there since libvirt-related issues have been handled many times in the past. You may use extracts from your journal to find related threads.

It reminds me a bit of an issue we had before, although this was not related to CoreOS: IP address of VMs in gnome-boxes - Ask Fedora

That problem contained the same log flow, including the seemingly intended/successful deactivation of libvirt.

If you start a new thread, I suggest to add more information. At the best, and based upon the configuration you have, add the whole logs of a boot that ended up in the practically-deactivated libvirt, plus the final output of sudo systemctl status libvirtd

Of course as long as we do not know the origin for sure, every possibility should be kept in mind. But given your logs, I do not see a relation to locales.

The idea of KubeVirt is a bit different. It is not to containerize VMs or even containerize the VM daemon, but to put the VMs and the containers along with each other into a common environment (see the Why KubeVirt? section), so that, for example, a continuous transition phase becomes possible in which VMs can be used in the new container-focused environment and then replaced one by one over time. So it seems not related to your issue. So KubeVirt will not solve that issue.

1 Like

Thank you, I completely missed ask.fedora somehow will post there. Seems like not a ton of people are trying to use VMs on top of CoreOS but I’ll try my luck.

Fair point around providing more information / the complete log, thanks for pointing that out.

Re: KubeVirt - it at least let me run the 2,3 VMs I would need in addition to my Kubernetes workload, so that’s at least something! Installation of k3s + kubevirt was also straight forward and worked pretty much out of the box, so I still might consider that as an option.

Most folks seem to run something else as the hypervisor and then layer CoreOS VMs on top, I’d rather stick closer to the metal and run the BlueIris / TrueNAS VMs on top of CoreOS in addition to k3s.

Thanks again,
Mathias

I think this is expected. libvirtd is socket activated now and will shut itself down if there aren’t any VMs running.

From this page I see:

this will use a 2 minute timeout so they shut down if no guest/container are running.

What actual issue are you seeing?

1 Like

@mringhof you may check out the output of sudo systemctl status libvirtd.service (if you use a dedicated root account, remove the sudo) in conjunction with the output of sudo systemctl status libvirtd.socket → is the .socket “loaded” + “active (listening)” / “active (running)” without errors? Is the .service “loaded” and “inactive (dead)” without errors? I cannot say anything about differences at CoreOS, but at the normal (=non-rpm-ostree) Fedora, systemctl enable --now libvirtd should, in conjunction with the packages you installed, immediately make both the socket but also the service “active (running)” by default (@dustymabe may correct me at this point if this does not apply to CoreOS / rpm-ostree versions). Indeed, I would be curious to know about that :slight_smile:

And just to be on the same page: Have you tried to create + run a vm yet? If that does not work, what exactly happens and what do the logs of that action look like?

Nevertheless, I think you will get most feedback when putting forward all the given points and information at ask.fedora :wink: If you open a thread at ask.fp, you should add a link to the ask.fp thread in here, to facilitate any support to centralize there.

Found the error in a rather embarrassing turn of events. I missed the error on the client below “cannot connect, is libvirt running?” that said “polkit agent not enabled” which lead me down the correct route:
Installation libvirt should create a group of the same name, which it does but I can’t add users to it.
I didn’t realize that usermod -a -G libvirt runs without error but doesn’t actually add the user to the group. No libvirt group, no access to libvirt. I’m not entirely sure why adding to an existing group doesn’t work though, it does with groups created manually through groupadd.

In case someone stumbles over this thread here’s how I got libvirt working:

  • create a system group libvirt and add a user to it for example through ignition
variant: fcos
version: 1.4.0
passwd:
  groups:
    - name: libvirt
      system: true
  users:
    - name: core
      groups:
      - libvirt
  • remove the package nfs-utils-coreos to avoid conflict with nfs-utils that’s installed by libvirt
rpm-ostree override remove nfs-utils-coreos
  • install qemu + libvirt packages (see dnf groupinfo virtualization)
rpm-ostree install virt-install libvirt-daemon-config-network libvirt-daemon-kvm qemu-kvm libguestfs-tools python3-libguestfs virt-top
  • reboot
  • enable libvirt
systemctl enable --now libvirtd

Learned quite a bit about FCOS in that journey. Also to read the error messages completely…
/facepalm
I’ll ask more questions on asks.fedora as suggested if and when they come up.

Thanks,
Mathias

Hey @mringhof - This isn’t your fault. You’ve stumbled upon a really old incompatibility that we’re working to fix.

See

1 Like