Strange happening

[Peter Rathlev]

On Wed, 2013-07-17 at 13:05 -0400, Adam Moffett wrote:
> I assume the system was I/O bound so I guess this is not strictly 
> related to dhcpd, so flag me off topic and flame me if you want, but
> if it happens again I'm wondering if there's a way to determine what 
> processes are waiting for I/O and/or what they're waiting for
> specifically.

On Linux you could use `iotop` to find out what processes do a lot of
read or write. If dhcpd was practically the only thing running that
question almost answers itself, though since iotop teels you what the
read and write rates are it can also help you figure out if the I/O
capacity is adequate.

To figure out exactly what is being read and written you could run dhcpd
under strace, e.g.:

  strace -eopen,read,write -y dhcpd 2> strace-output.log

Beware that this means a lot of overhead and things will be very much
slower. If you have the RAM it might be a very good idea to place the
strace output file in e.g. /dev/shm/ since writing to it would otherwise
exacerbate the I/O bottleneck problem.

You could also use `lsof -n -b -p <pid>` to look at what files a
specific process has open. You need to catch this at the right time of
course, which can be tricky. We have previously used a small script to
look at vmstat output every 2 seconds and react to high "wait" load. You
can take a look at what we used here:

   http://ampere.rathlev.dk/alert-on-high-cpu-wait

Beware that it's just an example and probably needs some customization
to work on your system. It requires bash, mkfifo, vmstat, top, iostat,
iotop and mail. Our CentOS 5 and 6 servers run it without problems, but
it relies on the different tools to output things in a certain format
and might thus not work well on other platforms.

-- 
Peter