Recursive bind becomes unresponsive with high load

[Michael Brunnbauer]

Hello Mathew,

On Sat, Apr 02, 2016 at 12:02:38AM +0000, Mathew Ian Eis wrote:
> * You can check for dropped packets on the receive path with # netstat -u -s
> High numbers on "packet receive errors??? can indicate an overflow in the receive buffer - this is fixable by network stack tuning as Mike Mitchell suggests. 

This seems to be the case. Thank you very much!

Udp:
    176299751 packets received
    611578 packets to unknown port received.
    230666 packet receive errors
    263458543 packets sent
    RcvbufErrors: 0
    SndbufErrors: 0

> We changed the following:
> net.core.rmem_max = 16777216
> net.core.rmem_default = 16777216
> net.core.wmem_max = 16777216
> net.core.wmem_default = 16777216
> net.core.netdev_max_backlog = 5000
> net.unix.max_dgram_qlen = 100

I adopted these settings. Unfortunately, they do not stop the problem.

> * Try watching your incoming UDP packet buffers in tight intervals at the same time as top
> 
> # watch -n 0.1 'cat /proc/net/udp | grep ":0035 00000000:0000 "'

I can see the error count on the lo interface growing when the queue is at

 00000000:00040200

Is that 262656 in decimal? Why is this value is much lower than the value of
16777216 above?

> Does the named unresponsiveness coincide with the UDP rx_queue filling up and named dropping to 0% CPU usage?

Yes - although the CPU usage of named does not drop to 0%.

I watched the load around the time the problem occurs. There is still more 
than 30% CPU idle time and no significant IO (wa is below 5%). 

Am I running into the limits of my system here?

Regards,

Michael Brunnbauer

-- 
++  Michael Brunnbauer
++  netEstate GmbH
++  Geisenhausener Straße 11a
++  81379 München
++  Tel +49 89 32 19 77 80
++  Fax +49 89 32 19 77 89 
++  E-Mail brunni at netestate.de
++  http://www.netestate.de/
++
++  Sitz: München, HRB Nr.142452 (Handelsregister B München)
++  USt-IdNr. DE221033342
++  Geschäftsführer: Michael Brunnbauer, Franz Brunnbauer
++  Prokurist: Dipl. Kfm. (Univ.) Markus Hendel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <https://lists.isc.org/pipermail/bind-users/attachments/20160402/1b025031/attachment.bin>