Troubleshooting poor performing BIND

[Ray Van Dolson]

Hi all, I am trying to track down a performance issue on a BIND 9
server powered by Solaris 8 and a Sun Fire 280R with 2GB's memory.
This server handles about 15 queries per second more or less.

Currently it's set up as a caching server.  Other DNS servers forward
requests to it and it then forwards them out to the Internet via the
regular hints method (hints file contaisn the root servers).

This DNS server takes a *long* time to complete resolution especially
compared to other DNS servers here running BIND and in a similar
configuration but which are basically idle.

Obviously there are many factors as to the cause for this delay
including Internet latency... however, if I watch the packets as they
arrive on my DNS server I see that a query packet comes in from a
client and it is 4 seconds before the first query is forwarded out to
one of the root servers for that query.  This leads me to believe that
BIND is busy doing something and simply cannot send the query out until
it's free.

I'm trying to figure out *what* could be causing that 4 second delay.
The server itself is fairly idle... (5 minute load is around 0.10 and
CPU is 85-90% idle via vmstat).

I've read over the various debug logging options but am not clear which
one will best let me determine where the query is getting stuck.

BTW, this BIND server is a bit old (BIND 9.3.2).  Obviously it should
be upgraded, but it hasn't yet and will have to be brought up here and
planned out.

In the meantime what can I do to track this down?  Does this type of
symptom ring a bell with anyone?  15 queries per second doesn't seem
terribly busy, but maybe BIND is only able to handle X amount of
queries at a time and thus we've got a lot of waiting going on... 

Thanks in advance, and feel free to yell at me to include specific
information that I should have. :)

Ray