Playing around with Fedora CoreOS using vagrant-libvirt on Fedora Silverblue

As a user of CoreOS/Container Linux for many years, and I’ve been eagerly awaiting Fedora CoreOS. I’ve even replaced my Desktop with Silverblue just to get a feel for things to come. For Container Linux I’ve made some advanced (read: complex) Vagrant projects to develop and test our deployment setups locally. Now that some build artifacts have shown up, I thought it was a good time to experiment and try to get them up and running in Vagrant on Silverblue.

Vagrant on Silverblue

Overlay libvirt

Libvirt needs to be installed on the host directly, so you need to overlay at least the ‘libvirt’ package.

rpm-ostree install virt-install virt-manager
systemctl reboot

To get rid of password prompts for each and every libvirt action, install the following polkit rule:

sudo tee /etc/polkit-1/rules.d/80-libvirt.rules <<EOF
polkit.addRule(function(action, subject) {
 if (action.id == "org.libvirt.unix.manage" && subject.local && subject.active && subject.isInGroup("wheel")) {
 return polkit.Result.YES;
 }
});
EOF

Note this requires you are in the wheel group, if you have an admin account on Silveblue that will be the case.

Libvirt should now be up and running:

$ LIBVIRT_DEFAULT_URI=qemu:///system virsh list
# Id   Name           State
# ------------------------------

Vagrant in a Container

Vagrant can run fine from a container. Create a container image with:

cat - > Dockerfile <<EOF
FROM docker://registry.fedoraproject.org/fedora:30
RUN dnf install -y openssh-clients vagrant vagrant-libvirt
CMD [ "/bin/bash" ]
EOF
podman build -t localhost/vagrant-container:latest .

This Vagrant container can be started interactivly with:

podman run --rm -it \
	--volume /run/libvirt:/run/libvirt \
	--volume "${HOME}:${HOME}:rslave" \
	--env "HOME=${HOME}" \
	--workdir "$(pwd)" \
	--net host \
	--privileged \
	--security-opt label=disable \
	localhost/vagrant-container:latest

By added the following alias to your ~/.bashrc, the vagrant commands can be executed transparantly using this container:

alias vagrant='podman run --rm -it \
        --volume /run/libvirt:/run/libvirt \
        --volume "${HOME}:${HOME}:rslave" \
        --env "HOME=${HOME}" \
        --workdir "$(pwd)" \
        --net host \
        --privileged \
        --security-opt label=disable \
        --entrypoint /usr/bin/vagrant \
        localhost/vagrant-container:latest'

Building a Fedora CoreOS Vagrant Box

To package up the latest qcow2 image artifact from the ci server, I made a little makefile/container and put it on github. Use the following commands to clone the repo and build the container image with the Makefile and it’s dependencies:

git clone https://github.com/basvdlei/fedora-coreos-vagrant-box-builder.git
cd fedora-coreos-vagrant-box-builder
podman build -t localhost/box-builder:latest .

When this container is started, it will download the latest qcow2 image and package it up as box into the /output volume:

mkdir output
podman run --rm -v "$(pwd)/output:/output:Z" localhost/box-builder:latest

It creates two files (fedora-coreos.box and fedora-coreos.json) in the output directory. Import them in Vagrant with:

cd output
vagrant box add fedora-coreos.json

To show the installed boxes:

vagrant box list
# fedora-coreos-preview (libvirt, 30.83)

Simple Vagrant Fedora CoreOS Project

Now that we have Libvirt, Vagrant and a Fedora CoreOS box, it’s finally time to start and boot a sample Fedora CoreOS project.

Create a project directory and put the following contents in file called Vagrant

require 'json'

ignition_file = File.join(File.dirname(__FILE__), 'config.ign')

config = {
  :ignition => {
    :version => "3.0.0",
  },
  :passwd => {
    :users => [{
      :name => 'core',
      :sshAuthorizedKeys => ['ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ== vagrant insecure public key'],
    }],
  },
}

File.open(ignition_file, "w") { |file| file.puts JSON.generate(config)}
# Systems with SELinux will need to relabel the file.
system("chcon system_u:object_r:virt_content_t:s0 #{ignition_file}")

Vagrant.configure("2") do |config|
  config.vm.box = 'fedora-coreos-preview'
  config.vm.provider :libvirt do |lv|
    lv.memory = 1024
    lv.cpus = 1
    lv.qemuargs :value => '-fw_cfg'
    lv.qemuargs :value => "name=opt/com.coreos/config,file=#{ignition_file}"
  end
end

When you run vagrant status (or any other vagrant command) it should create an config.ign file. This is the Ignition config the VM will be booted with. The example above only installs the default Vagrant insecure SSH key. Full Ignition specs can be found here.

To start up the VM type:

vagrant up

When it’s finished creating and booting the machine, you can login to it with:

vagrant ssh

If everything went as it should, you should get a login prompt:

Fedora 30.83 (CoreOS preview)
Tracker: https://github.com/coreos/fedora-coreos-tracker
WARNING: All aspects subject to change, highly experimental

[core@localhost ~]$ 

Cheers!

As a side-note, I did play around with cosa, which is awesome. But I really wanted to migrate my current Vagrant projects over.

18 Likes

Wow. Thanks so much @basvdlei for writing this up! If you’re interested in getting more involved with Fedora CoreOS please stop by the IRC channel #fedora-coreos and chat with us!

Very cool! I like how simple this makes it look to run vagrant from a container. Will give this a try myself (have used a podman+vagrant workflow for developing on Fedora Atomic Host, will be great to try with a Vagrant box for FCOS).

Thanks! I’ll be sure to do drop by, although I don’t know how well our time zones will line up.

This is very cool, @basvdlei ! I’m a new Silverblue user, and I want to try your podman vagrant container since I want to keep package layering to a minimum. However, following your instructions to the letter and running vagrant up on one of my usual projects failed at nfs mounting with The executable '/usr/sbin/service' Vagrant is trying to run was not found in the PATH variable.. Any ideas? I’m a complete n00b with Podman, sadly. (I’ve only used docker-compose a little, which probably doesn’t help.)

You probably need some special setting or package for nfs?

I generally don’t use the folder sync feature. If you also don’t need it, you can disable it in the Vagrant project:

Vagrant.configure("2") do |config|
  config.vm.synced_folder ".", "/vagrant", disabled: true
end

An alternative is to switch to Rsync. This will copy the given directory inside the Vagrant machine on startup.

Setting NFS sync up inside the Vagrant container is going to be interesting. Since Vagrant’s NFS folder sync requires root privileges and modifies the host system. You can override the behavior of Vagrant by settings some environment variables. I imagine that setting up NFS on the host and mocking the commands inside the Vagrant container could work.

1 Like

Thanks so much for the quick reply! Sounds like I need to read up on this, and keep using package layering for the time being. For now, my workflow is really dependent on nfs and synced folders. I do Rails development (mostly the frontend bits; erb templates and css), and modify files in a directory on my home partition constantly, keep the rails server running on a vagrant-libvirt guest in the background and reload the browser to test. Without synced folders, this would be a lot harder, and I’ve tried rsync before and it doesn’t really cut it. I could run Rails and Postgres themselves in containers locally and skip Vagrant completely, but I like knowing how they’ll behave in a pseudo-production environment, and Vagrant allows me to run Rails reliably on the whole Ubuntu Server stack (with Ansible as a bonus).

The main draw of Vagrant for me was the instant file sync in a reproducible environment. Not quite ready to give this up - or drastically change my habits - just yet. :slight_smile:

@evenreven

I am using GitHub - dustymabe/vagrant-sshfs: SSHFS synced folder implementation for Vagrant. within the container. The only gotcha is that I have to run vagrant from inside the vagrant container.

This is how I build the container using buildah

#!/bin/bash -x

container=$(buildah from "fedora:latest")

buildah run "$container" -- dnf install -y ansible openssh-server openssh-clients vagrant vagrant-libvirt vagrant-sshfs
buildah run "$container" -- dnf clean all

buildah commit "$container" "vagrant"

I use ansible to provision the vagrant box, you can remove it if not needed.

then my alias is

vagrant-container='podman run --rm -it \
        --volume /run/libvirt:/run/libvirt \
        --volume "${HOME}:${HOME}:rslave" \
        --env "HOME=${HOME}" \
        --workdir "$(pwd)" \
        --net host \
        --privileged \
        --security-opt label=disable \
        localhost/vagrant:latest'

That will give you a bash session inside the vagrant container, then you can simply do

vagrant up
vagrant ssh

.

1 Like

That is very cool! I have never used the buildah tool except through podman, so thanks so much for the build script. Trying an old project with sshfs now, and although it’s a bit early to tell if sshfs is slower than nfs in this scenario, it seems to be working well. I should do some real use-case testing with modifying CSS on the host and reloading browser. If sshfs struggles there should be a delay. If it’s near-instantaneous, I can remove the layered package. Thanks again, @cverna!

1 Like

@basvdlei - thanks for posting this. I have been struggling to get Fedora CoreOS working via Vagrant now that FCOS is out of preview. Your instructions worked beautifully (with a minor) tweak in the Dockerfile to us Fedora 31. Jeff Gerling has an early release of his Ansible for Kubernetes ebook out and I thought I could learn a lot about FCOS if I follow along with Jeff’s vagrant examples using FCOS instead of Debian. Or run into a brick wall! In any case, thank you very much for sharing.

Brad

1 Like

Thanks! Great to hear it’s working for you.

The version of Fedora used in the Vagrant container image should preferably match your base Fedora version, so the versions of libvirt inside/outside the container match up.

I’ve done a quick update of the vagrant box building image so it will use the stable stream by default. The box name is also changed from fedora-coreos-preview to fedora-coreos.

With this stable release I should really get back at testing FCOS myself again…