Performance Problem Postmortem

[Simon Hobson]

Robert Hille wrote:

>My question now, is what exactly was causing the packets to go missing?
>
>The going theory is that there is a buffer somewhere between the NIC 
>and the dhcpd daemon that stores the packets and it is getting full 
>and dropping the packets. Is this reasonable? If not what else could 
>have been occurring? If it is right, what can I monitor to warn me 
>the next time this happens?

There probably is a buffer - but the main issue is that the server is 
single threaded. You need to take a close look at everything the 
server needs to do in order to respond to a query :

receive the packet
process it and pick a lease to offer or ack
log it to the leases file AND SYNC IT TO DISK
log it to syslog
do ddns
send packet to client

(not sure of the order of steps 4-5)

A delay in any of those steps will result in a delay responding to 
the client. If there are sufficient clients making requests, then any 
delay may result in buffer overflow and inbound packets getting lost. 
Also, if there is enough delay, then clients will time out and try 
again - thus making the problem worse.

Performance comes up regularly - you may wish to check the archives.

Things you can do :
Make syslog asynchronous - that removes one synchronous write to 
disk. By default it's sync and that means another hardware write 
before the call returns.

If you are brave, or have suitable battery backed hardware, put the 
leases file on a ram disk. It must not be on a 'normal' ram disk that 
would be lost in a system crash or power failure as that would leave 
you without a lease file. Generally look at the performance of your 
disk system as that will be a limiting factor on response rates.

Make sure DDNS is either working quickly, or turned off.

-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.