Query timeouts on FreeBSD 7 over network

[Vinny Abello]

(Posting to both BIND users and FreeBSD-Stable):

        I'm having a problem with query timeouts with BIND (I've tried versions 9.4.2, 9.4.2-P1, 9.4.2-P2, 9.5.0-P1) on FreeBSD 7.0-STABLE branch. I'm testing with dnsperf and find that about 0.01% of my queries are not being responded to if I run the test from another machine on the same network segment. They seem to happen in bursts from what I can tell. This is on a Dell PowerEdge 2850 with all the latest firmware. The NIC in the server is using the em driver and negotiates at 1000Mb/full. Cables are Cat5e or better. I've grabbed the latest source with csvup and rebuild a custom kernel and rebuilt world. Currently I'm running AMD64 but also tested i386. The interesting thing is if I use dnsperf from the local machine experiencing this problem, I do not experience any timeouts, so it would appear to be network related in some way. Netstat doesn't show any errors. No errors are showing up in my syslog either. I've tried with the ULE scheduler and 4BSD and tried with and without PREEMPTION turned on. Nothing makes a difference.

I've also tested against 2 Windows servers running BIND 9.4.2 on the same segment and do not have any timeouts (yes, these are being replaced by the FreeBSD ones once I get this issue fixed).

I'm pretty sure this is related to the OS or the em driver in some way, because if I disable all ICMP rate limiting and run an extended ping from the local firewall, I experience a very low amount of random packet loss in no pattern, unlike if you have the ICMP rate limiting enabled.

I'm hoping there are many FreeBSD gurus running BIND on these lists that might have experienced this on similar hardware or know the solution. Any help would be greatly appreciated. My only other thoughts are to drop the speed to 100Mbps or try swapping in a different type of NIC.

Also, are there any documented recommendations for sysctl values for FreeBSD when running BIND for optimal performance?

I've considered running CentOS instead as there typically seem to be less strange issues like this, but I totally prefer FreeBSD and would like to stick with it and resolve the problem. Obviously if ISC runs F.ROOT on FreeBSD and BIND, it must work well, right? :)

Thanks in advance for any help or tips on how others are running BIND on FreeBSD with similar hardware.

-Vinny