We’re currently in the process of migrating all of our nodes from CoreOS to Fedora CoreOS (FCOS), and one challenge that we’re facing is determining how to properly version control our updates.
Given that we have around ~20 CoreOS clusters spread across Dev, QA, and Prod, it sometimes will take us 2-4 weeks to complete an upgrade across all of the clusters, mainly due to a strict change management controls, limited maintenance window availability, and a staggered rollout approach.
With that said, our concern is that if we start an upgrade on a particular FCOS version, we want to ensure that the same version is deployed to all of our clusters. Given that Zincati polls the FCOS Cincinnati servers for the latest available release, our concern is that it is possible that we might encounter new FCOS versions being introduced mid-way through our internal upgrade cycle.
For CoreOS (Container Linux), we were able to work around this concern by implementing something ismiliar to the following gist where we’d download the desired CoreOS version’s update.gz file locally, create the necessary Omaha XML, and then point each CoreOS node’s update-engine service at these files:
Does anyone know if a similiar approach could be used for FCOS version controlling? If nothing exists, then I suppose the next best option is to look into deploying our own local FCOS Cincinnati server by reverse engineering something like:
Please let me know if you need any additional information.
Thanks so much!