$0.02: IPA upgrade follow-up

Project Discussion
1

I’m not sure where to put this on a Sunday, but the follow-up plan that was stated elsewhere has bitten me in the past. I thought I’d try to share some of my scars here from that approach, in the hopes that it would save some future headaches.

The problem

During the recent IPA upgrade, it was found that several apps had hardcoded a specific server, e.g. ipa03.

The proposed solution

It sounds like the plan is to put DNS in front of IPA so that clients can point to a single endpoint to reach all nodes in the cluster.

I think that’s the correct thing to do, but I wanted to caution that DNS is likely only half the story, and DNS alone can lead to other failure scenarios.

Pain

Clients of all types are notoriously ill-behaved when it comes to respecting DNS TTL. So let’s say some clients currently have the IP for ipa02 but it goes down or needs maintenance. You temporarily remove that host from the DNS record, but many clients (which can include intermediate routers etc) will either hold onto that record until they’re rebooted, or until they’ve reached their own internal TTL which completely ignores the DNS TTL. It’s pretty difficult to test for this so usually nobody does it, and even if they do it’s a constant game of whack-a-mole.

This is why in the http world DNS-based routing for reliability isn’t really a thing any more – we use http load balancers instead. (If all clients were correct, we wouldn’t need load balancers!)

Non-refundable $0.02

In addition to the DNS approach, I think we need to stand up an haproxy load balancer in front of IPA to avoid future issues. haproxy can do TCP-based load balancing, and it has option ldap-check to allow it to health-check LDAP backends and automatically stop routing to them when the health check fails.

I know this is extra work up front. But this way clients only need the DNS for haproxy, which stays stable, and thus you sidestep any issues with client DNS caching. Plus, future maintenance (planned or otherwise) has reduced toil, as you simply let haproxy notice when nodes go away and handle doing the right thing.

image
image681×500 197 KB

Hopefully this doesn’t sound like me telling you how to do your job! Just trying to tell ghost stories before the ghosts haunt anyone else.

2 Likes
2

As the one who did most of the work on IPA migration to RHEL10 I want to give my opinion to this.

Where the problems started

So everything worked fine when I worked on ipa02 and ipa03. The problem started when I started to move ipa01 to RHEL10. As this one is hardcoded in multiple places.

There are places where it makes sense, for example for others IPA servers we need one reference server to create replicas from. I agree that for other machines we should have something in place that will help move the requests to other IPA servers. You mentioned the cons of DNS approach. We already have HAProxy in place, so I’m OK for it to being used for IPA servers.

3

I’m not sure where to put this on a Sunday, but the follow-up plan that was stated elsewhere has bitten me in the past. I thought I’d try to share some of my scars here from that approach, in the hopes that it would save some future headaches.

Thanks for the feedback!

The problem

During the recent IPA upgrade, it was found that several apps had hardcoded a specific server, e.g. ipa03.

The proposed solution

It sounds like the plan is to put DNS in front of IPA so that clients can point to a single endpoint to reach all nodes in the cluster.

I think that’s the correct thing to do, but I wanted to caution that DNS is likely only half the story, and DNS alone can lead to other failure scenarios.

Pain

Clients of all types are notoriously ill-behaved when it comes to respecting DNS TTL. So let’s say some clients currently have the IP for ipa02 but it goes down or needs maintenance. You temporarily remove that host from the DNS record, but many clients (which can include intermediate routers etc) will either hold onto that record until they’re rebooted, or until they’ve reached their own internal TTL which completely ignores the DNS TTL. It’s pretty difficult to test for this so usually nobody does it, and even if they do it’s a constant game of whack-a-mole.

This is why in the http world DNS-based routing for reliability isn’t really a thing any more – we use http load balancers instead. (If all clients were correct, we wouldn’t need load balancers!)

Non-refundable $0.02

In addition to the DNS approach, I think we need to stand up an haproxy load balancer in front of IPA to avoid future issues. haproxy can do TCP-based load balancing, and it has option ldap-check to allow it to health-check LDAP backends and automatically stop routing to them when the health check fails.

I know this is extra work up front. But this way clients only need the DNS for haproxy, which stays stable, and thus you sidestep any issues with client DNS caching. Plus, future maintenance (planned or otherwise) has reduced toil, as you simply let haproxy notice when nodes go away and handle doing the right thing.

The hard coding I was talking about was in our rdu3 datacenter.
All our instances there use our dns servers. There’s not any routers
holding anything or the like, and the ttl is 5min. Also it’s already in
dns… hosts use dns already for finding the ipa servers they should
use.

So, things we have that have hard coded one ipa server:

  • Making sure you're not a bot!
    This is the openshift/ipa-client role that gets used on tons of our
    openshift apps. It hard codes just one ipa server, if that one is down,
    they have problems.

I think (but need to test and confirm) that if we simply remove that
section entirely, kerberos will default to getting the list of servers
from dns and will iterate over them to find one that answers/is working.

Sure, there may be a delay if the one it goes to first isn’t answering,
but I think the delay is pretty small and this was designed to handle
this case.

But all this needs testing/confirmation for sure.

I think here we may need to adjust noggin to either find these via dns
too, or just iterate over them until a working one is reached.

If we stood up another haproxy(s), then those would become a point of
failure too and also add complexity, so I am not sure it would be a win.

Anyhow, thats my thoughts on it, might be missing something or the
like…

4

Standing up a load-balancer in front of IPA servers is not impossible
but it is not simple either.

Some old but still valid explainer of why:

load-balancing the IPA servers themselves is a bit harder than a
regular pool of kerberos enabled HTTP servers.