The 'right way' to deploy kubernetes on coreos virtual machines

There seems to be some significant variation in opinion on how to deploy k8s to coreos based VM. I am new to coreos and am struggling to find a good tutorial or even a high level thought piece on the process.

My intuition is that k8s should be installed via packages from rpm-ostree (not least because they would form part of the automatic updates for the OS?) and specified in the ign file for a template VM and then that template should be cloned and booted with additional node specific ign configurations for control planes, infrastructure, and application nodes.

In the first instance I would be grateful if someone could validate this general approach. I would be super excited if someone could outline the process in whatever detail they had inclination to offer or sign post resources that would cover the process.

I would be VERY interested in seeing a sample ign file that installed k8s components.

As an aside I use terraform to provision infrastructure, ansible to configure applications, VCenter for virtualisation . I am currently in the process of trying to deploy a k8s cluster to a single esxi host on a workstation.

You can take a look at what Typhoon is doing:

How to deploy Kubernetes on Fedora CoreOS depends on a lot of factors:

  • Do you want the kubernetes version to be updated at the same time as the OS or on its own schedule?
  • Do you rely on a specific version of Kubernetes?
  • Which container runtime do you want to use? containerd or cri-o?
  • etc.

so depending on your goals, the way to deploy it may vary widely.

1 Like

Thanks Timothée,

Certainly for testing purposes I am content for the most recent version of Kubernetes available in package manager and do not have any dependencies on specific versions with my current projects. In fact a benefit of CoreOs that attracts me is the fact that the services can be updated automatically to latest version within the os.

Whilst the runtime isn’t so important to me at this stage I would generally lean in to cri-o, but this preference is based mainly on commentary from third parties on stability.

Based on this what, in your opinion, would be the best way to provision Kubernetes at ignition stage? Could you point me to a sample ign file? I am happy to configure in ansible post-deployment - joining control planes to workers, extracting kubectl config etc.

My goal in the short term is to merely provision Kubernetes components at the point of initial boot with ign files - and make them operational shortly afterwards with playbooks - then round out the approach as I explore the possibilities of the os and identify specific configurations for projects once I have some infrastructure to experiment on.

Thanks again for your replies.