Bind9 stops responding for some clients

[Browne, Stuart]

Congratulations on finding the cause.

Sometimes, it's the simplest of things.

Stuart

From: bind-users [mailto:bind-users-bounces at lists.isc.org] On Behalf Of Gregory Sloop
Sent: Thursday, 6 June 2019 12:37 PM
To: bind-users at lists.isc.org
Subject: Re: Bind9 stops responding for some clients

Thanks for the idea.
I did resolve this a day or two ago.

The story is;
This server was a fairly recent replacement for an older Ubuntu setup. The new server as well as the old one are/were VM's - yet on different VM platforms. The old VM was turned off, and was marked never to start except unless manually started. [There were a few other things on the VM host that had yet to be migrated - so we didn't want it entirely off quite yet.]

The problem happened again in the last day or two - and packet captures showed that no packets were even arriving at the new VM.
Since there really wasn't anything that should be blocking that traffic, I checked the arp table on a problem client.
The arp table showed an "incorrect" MAC address for the current BIND server. [The MAC in the arp table didn't match the MAC for the new VM.]

While I didn't have the MAC address for the "old" deactivated server handy, it was the first obvious problem/solution to check.
Sure enough, after connection to the VM hypervisor, I could see that the "old" BIND vm was active.

I killed it, and service returned to normal.

So, the "solution" was pretty routine.
What made it more "interesting" and perhaps odd is how seemingly randomly the problem would crop up.
And it would only impact some clients, not all. There was no pattern that seemed to explain why some got the current/correct BIND server and others didn't. [The arp poisoning certainly wasn't anywhere nearly universal.]
And why was it so infrequent - it would go many days between issues.
I have to assume the bad VM had been up for some time, at least since the problems started.
There are quite a number of odd-ish other things too, but not worth detailing.

Probably it's just one of those "undefined" situations where you can't anticipate some predictable order to what happens when you screw it up. Rather than burn additional time trying to grok what was going on - it's simply best to say "don't do that - bad things happen, though I can't say what bad things will happen and in which logical order. They just will - so DON'T DO THAT!"

[And yeah, I obviously knew all about not doing that. But it happened anyway, in spite of specific steps to prevent it. I'm still not sure why.]

In the end, it's a somewhat complicated story with a very obvious cause - but it wasn't so clear at the outset.

TLDR version;
Don't run your old and new bind servers on the same IP address - ether by accident or intentionally. Bad stuff will happen!
It might be really odd, or it might be plain as day -  but in either case it won't be good! :)

Thanks all for the suggestions!
Here's hoping I don't need to ask for BIND assistance for another 20 years! :)

-Greg


I just randomly spotted this post, and thought I would toss in 2¢

How many nics and how many it's are on the servers?  Are the failing clients on the same subnet as the server?

--


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/bind-users/attachments/20190606/99837b9b/attachment.html>