FW: Performance issue ( maybe ) corrected

[Bjarne Blichfeldt]

> -----Original Message-----
> From: Glenn Satchell
> Sent: 3. september 2010 18:24
> To: Users of ISC DHCP
> Subject: Re: FW: Performance issue ( maybe ) corrected
> 
> On 09/04/10 01:36, Simon Hobson wrote:
> > Bjarne Blichfeldt wrote:
> >
> >> During the problem I see something like (numbers added from memory):
> >> Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> >> Cpu1 : 0.3%us, 0.3%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> >> è Cpu2 : 0.0%us, 0.3%sy, 0.0%ni, 1.3%id, 98%wa, 0.0%hi, 0.0%si, 0.0%st
> >> Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> >> Mem: 2075208k total, 714656k used, 1360552k free, 189160k buffers
> >> Swap: 4128760k total, 100k used, 4128660k free, 318604k cached
> >
> > ...
> >
> >> The next I have done is to increase default lease time from 8 hours to
> >> 5 days.
> >
> > ...
> >
> >> 1352 subnets
> >> 792 pools
> >> around 7500 active leases
> >
> >
> > That works out at one lease renewal every around 1.5 to 2 seconds, not
> > exactly a heavy load. However, there are two things that can cripple the
> > server under heavy I/O :
> >
> > Log files. Configure syslog to use async logs for whatever dhcp logs end
> > up in. Otherwise, every discover, offer, request, and ack results in a
> > wait while the log entry is synced out.
> >
> > Leases file. Again, every offer or ack is preceded by writing to the
> > leases file. If the machine is liable to heavy I/O from other processes
> > then it may be worth putting the leases file on it's own storage so
> > dhcpd doesn't find itself stuck in an I/O queue.
> 
> I'd check the network connection on the server for errors. Maybe a
> faulty cable, switch port or NIC? Perhaps auto-negotiation of speed or
> duplex is not correct?

Good idea, but we have been checking that all day and everything seems fine. Also a quick scp
from dhcp1 to dhcp2 gives : "100%   98MB  24.6MB/s   00:04" , which is very close to 100MB line.
But I think I will remember this test if the problem returns.

> 
> The system looks to be very idle, and dhcpd doesn't impose much load.
> 
I agree, that is why this is weird and why I think there might be "something else".

Thank you for your input.


mvh,
Bjarne