I wanted to briefly mention some of the operational Datanommer work that Fedora Infra might be able to upstream from Hatlas, in case anyone is interested:
Switch to running TimescaleDB in a container.
Upgrade to Timescale/PG18. (Note: one does not merely upgrade across 5 major versions …)
Run in Kubernetes.
On my TODO: Use CNPG to manage this. I don’t anticipate any issues, though it’s probably way more useful for you than for me.
Split the storage so that recent data + indexes can live on fast nvme while older data (rarely queried) can live on cheap HDD.
All of this is documented in Making sure you're not a bot! . I probably don’t have time to upstream this myself at the moment but happy to chat / collaborate.
I wanted to briefly mention some of the operational Datanommer work that Fedora Infra might be able to upstream from Hatlas, in case anyone is interested:
Thanks, of course interested.
Switch to running TimescaleDB in a container.
I don’t think this makes sense unless we move it to openshift…
(see below)
Upgrade to Timescale/PG18. (Note: one does not merely upgrade across 5 major versions …)
Currently it’s running on a rhel9 box, so we want to stick with the
rhel9 supported postgres. If we move it to openshift we could change
that…
Run in Kubernetes.
On my TODO: Use CNPG to manage this. I don’t anticipate any issues, though it’s probably way more useful for you than for me.
This I would be interested in.
In the past the reason we didn’t do this (aside from personpower) is
that storage might be an issue (it’s very large as you know) and the
database handling in openshift was imature.
I don’t think those are problems anymore (aside from time). We could use
netapp storage, and I think the various containerized postgres apps have
gotten quite good.
Split the storage so that recent data + indexes can live on fast nvme while older data (rarely queried) can live on cheap HDD.
I don’t think this applies to us… the current datanommerdb is on a
host with all nvme storage, there’s no cheap HD around. ;(
If we moved it to openshift, I think we would want to back it with
netapp space (this would also allow us to take snapshots/replcate for
disaster recovery, etc).
FYI that rhel9 package is PG13, which hit upstream EOL in 2025.
Related, upgrading your PG version might be a pre-req to getting into k8s:
Timescale/PG13 containers do exist, but they predate the invention of safety. They run as root, so your cluster’s security requirements might not allow it (no idea). This will also create permissions headaches later when you migrate to newer non-root versions.
Since PG13 is EOL, these containers are no longer being updated. Their contents are also unlikely to be updated upstream, so a custom build probably won’t resolve any CVEs. (In case you’re required to pass container security scans.)
I performed the upgrade outside of k8s on a traditional host, by mapping the host directory into a series of containers (PG13 → PG16 → PG18). This would be tricky to do in k8s. So even if it’s not strictly required, I highly recommend upgrading on a traditional host before migrating to k8s. (Which is much easier with containers than with packages!)
Once you’re on a new-ish version in k8s, CNPG can handle future major version upgrades. Though FYI, there is still a manual component of upgrades where you have to use psql to tell Postgres to upgrade your database’s data structures to the new Timescale extension version. All the gory details are documented in our datanommer-ops repo.
Datanommer on Hatlas
If you want to crib from Hatlas k8s, the Datanommer big picture is here, though it’s using Kustomize which I don’t think you guys are using. Kustomize is about as close as you can get to raw yaml + a sprinkle of templating, so here’s the almost-raw StatefulSet.
TimescaleDB on CNPG
There are some minor differences between how Tiger Data builds the official Timescale containers and how CNPG expects them to be built. As a result, you do need to build your own Timescale containers to run them under CNPG. Details and instructions are in this Tiger Data blog post.
This is the only reason Hatlas isn’t using CNPG yet – because I still have a TODO to set up a FDWG team Quay account and request some Forge CI runners.
Hope all of that helps! You know where to find me.
FYI that rhel9 package is PG13, which hit upstream EOL in 2025.
Sure, but that doesn’t matter to us, since RHEL is supporting it.
Related, upgrading your PG version might be a pre-req to getting into k8s:
Timescale/PG13 containers do exist, but they predate the invention of safety. They run as root, so your cluster’s security requirements might not allow it (no idea). This will also create permissions headaches later when you migrate to newer non-root versions.
Since PG13 is EOL, these containers are no longer being updated. Their contents are also unlikely to be updated upstream, so a custom build probably won’t resolve any CVEs. (In case you’re required to pass container security scans.)
I performed the upgrade outside of k8s on a traditional host, by mapping the host directory into a series of containers (PG13 → PG16 → PG18). This would be tricky to do in k8s. So even if it’s not strictly required, I highly recommend upgrading on a traditional host before migrating to k8s. (Which is much easier with containers than with packages!)
Once you’re on a new-ish version in k8s, CNPG can handle future major version upgrades. Though FYI, there is still a manual component of upgrades where you have to use psql to tell Postgres to upgrade your database’s data structures to the new Timescale extension version. All the gory details are documented in our datanommer-ops repo.
Yeah, definitely will need upgrading to move to something else.
Even if we move from rhel9 → rhel10 we would have to do that anyhow.
Datanommer on Hatlas
If you want to crib from Hatlas k8s, the Datanommer big picture is here, though it’s using Kustomize which I don’t think you guys are using. Kustomize is about as close as you can get to raw yaml + a sprinkle of templating, so here’s the almost-raw StatefulSet.
TimescaleDB on CNPG
There are some minor differences between how Tiger Data builds the official Timescale containers and how CNPG expects them to be built. As a result, you do need to build your own Timescale containers to run them under CNPG. Details and instructions are in this Tiger Data blog post.
This is the only reason Hatlas isn’t using CNPG yet – because I still have a TODO to set up a FDWG team Quay account and request some Forge CI runners.
Hope all of that helps! You know where to find me.
