recursive queries fail with high load?

[Chris Michels]

I have 3 DNS servers running bind 9.3.2.  Two of them are failing to resolve
recursive queries.   Both of these servers have a higher load because they
are used by our spam filtering software.  I have increased the
recursive-clients option on both servers.  It seems like recursive queries
are just taking a long time and timing out.   What is going on here or where
should I be looking for what is wrong?

A dig of a random name returns:

[root at ruby named]# dig www.websudoku.com[1] @ns2.nau.edu

; <<>>DiG 9.2.4 <<>>www.websudoku.com[2] @ns2.nau.edu
; (1 server found)
;; global options:  printcmd
;; connection timed out; no servers could be reached

But if I set the timeout high it returns:

[root at ruby named]# dig +time=240 www.websudoku.com[3] @ns2.nau.edu 

; <<>>DiG 9.2.4 <<>>+time=240 www.websudoku.com[4] @ns2.nau.edu
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50071
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 0

;; QUESTION SECTION:
;www.websudoku.com.             IN      A

;; ANSWER SECTION:
www.websudoku.com[5].      3600    IN      CNAME   websudoku.com.
websudoku.com.          3600    IN      A       66.39.22.227

;; AUTHORITY SECTION:
websudoku.com.          3600    IN      NS      ns00.ns0.com.
websudoku.com.          3600    IN      NS      ns97.pair.com.

;; Query time: 20270 msec
;; SERVER: 134.114.138.3#53(134.114.138.3)
;; WHEN: Sun Feb 25 22:27:40 2007
;; MSG SIZE  rcvd: 112
rndc status shows:

number of zones: 746
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is OFF
recursive clients: 1441/5000
tcp clients: 0/1000
server is up and running

stats show a high percentage of failures

+++ Statistics Dump +++ (1172467505)
success 305925
referral 114
nxrrset 15781
nxdomain 335350
recursion 529820
failure 303309
--- Statistics Dump --- (1172467505)



-- 
Chris Michels -- Systems Programmer/Team Lead -- +1 928 523-6495
Northern Arizona University -- Flagstaff, AZ
PGP key: http://jan.ucc.nau.edu/~cvm[6] Team Info:
http://www4.nau.edu/its/sia[7]

"The significant problems we face cannot be solved at the same level of 
thinking we were at when we created them" -- Albert Einstein



--- Links ---
   1 http://www.websudoku.com
   2 http://www.websudoku.com
   3 http://www.websudoku.com
   4 http://www.websudoku.com
   5 http://www.websudoku.com
   6 http://jan.ucc.nau.edu/%7Ecvm
   7 http://www4.nau.edu/its/sia