High IOWAIT when running multiple rndc addzone / delzone causing dropped queries

[Another Email]

Hello,

I was wondering if someone on this list can assist me in figuring this out.  I am trying to run the rndc addzone / delzone for many domains at once on a set of name servers.  When this is done the the load on the box goes very high, and the process just slows right down to a halt (dropping queries).

I am basically wondering if there are certain settings that I can change in order to run BIND more efficiently.


Linux kernel: 3.1.10  x86_64 Intel(R) Xeon(R) CPU E3110 @ 3.00GHz GenuineIntel GNU/Linux


At the time when I run the rndc addzone / delzone commands I will see this:
Cpu(s):  0.3%us,  1.8%sy,  0.0%ni,  9.7%id, 88.2%wa,  0.0%hi,  0.0%si,  0.0%st



I believe I have given named enough possible file handlers.
# lsof -n | grep named | wc -l
1232
# su - named
named at b1123 / $ ulimit -Hn
65536
named at b1123 / $ ulimit -Sn
65536


When I strace the different threads I notice that one thread is constantly redoing the 3bf305731dd26307.nzf and reading the JNL files.  Another thread is just spewing out as fast as possible the following:
futex(0x7ff72930e07c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME,
677931537, {1361978981, 779057000}, ffffffff) = -1 ETIMEDOUT (Connection
timed out)
futex(0x7ff72930e028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7ff72930e07c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME,
677931539, {1361978981, 891290000}, ffffffff) = -1 ETIMEDOUT (Connection
timed out)
futex(0x7ff72930e028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7ff72930e07c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME,
677931541, {1361978982, 44187000}, ffffffff) = -1 ETIMEDOUT (Connection
timed out)
futex(0x7ff72930e028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7ff72930e07c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME,
677931543, {1361978982, 100744000}, ffffffff) = -1 ETIMEDOUT (Connection
timed out)
futex(0x7ff72930e028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7ff72930e07c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME,
677931545, {1361978982, 266914000}, ffffffff) = 0
futex(0x7ff72930e028, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x7ff72930e028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7ff72930e07c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME,
677931547, {1361978982, 186681000}, ffffffff) = -1 ETIMEDOUT (Connection
timed out)
futex(0x7ff72930e028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7ff72930e07c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME,
677931549, {1361978982, 206752000}, ffffffff) = -1 ETIMEDOUT (Connection
timed out)
futex(0x7ff72930e028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7ff72930e07c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME,
677931551, {1361978982, 226846000}, ffffffff) = -1 ETIMEDOUT (Connection
timed out)
futex(0x7ff72930e028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7ff72930e07c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME,
677931553, {1361978982, 246940000}, ffffffff) = -1 ETIMEDOUT (Connection
timed out)
futex(0x7ff72930e028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7ff72930e07c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME,
677931555, {1361978982, 266914000}, ffffffff) = -1 ETIMEDOUT (Connection
timed out) 


Please let me know what you may think I need to do. 
Thank you!
-Ted