change in reverse dns lookup behavior

[cool burn]

Hello,

We have an internal network of the form 10.x.x.x

We have two DNS servers (bind 9.2.1) that are
multi-homed, but are used by the internal network at
10.0.0.10 and 10.0.0.11

All of the internal servers have resolv.conf setup as:
nameserver 10.0.0.10
nameserver 10.0.0.11

This has worked perfectly for 8 months.

Today, we suddenly started getting timeouts in our
application server connecting to our db server.  Then,
I saw I was also getting very slow times to connect
using SSH.  I knew right away this was DNS related.

The DNS servers were still responding perfectly to
requests ("dig google.com" worked fine on all internal
servers, and was getting responses from both 10.0.0.10
and 10.0.0.11)

The problem is, they were giving extremely slow
responses to "dig -x 10.0.0.5", etc.

I realize now that the problem is, we had no reverse
DNS defined for these internal IP addreses.  I simply
added them into the hosts file on the servers that
needed them, and connection times between our
applications, and ssh connection times became instant.

My question is, why did we only just now start
noticing this behavior?

To be honest, I hadn't tried a "dig -x 10.0.0.5" in a
long, long time on any of the internal servers.  But,
I ssh among them throughout the day, every day.  Our
applications had no problems connecting, etc.

My only thought is that our two DNS servers somehow
had the failed reverse lookup cached, and today
decided to do live lookups on them. 

What I don't get, is why they didn't seem to be
cacheing the failed lookups anymore.

Repeated lookups of "dig -x 10.0.0.5" result in:
[root at db01 etc]# dig -x 10.0.0.5

; <<>> DiG 9.2.4 <<>> -x 10.0.0.5
;; global options:  printcmd
;; connection timed out; no servers could be reached
[root at db01 etc]#

Whereas:
[root at db01 etc]# dig -x 216.239.39.99

; <<>> DiG 9.2.4 <<>> -x 216.239.39.99
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id:
46492
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1,
ADDITIONAL: 0

;; QUESTION SECTION:
;99.39.239.216.in-addr.arpa.    IN      PTR

;; AUTHORITY SECTION:
39.239.216.in-addr.arpa. 54     IN      SOA    
ns1.google.com.
dns-admin.google.com. 2004031201 21600 3600 1038800 60

;; Query time: 1 msec
;; SERVER: 10.0.0.10#53(10.0.0.10)
;; WHEN: Thu May 12 13:03:03 2005
;; MSG SIZE  rcvd: 104

Any insight into this is much appreciated!

Thank you,
-Michael Thomas


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com