select() starvation?

[Jaye Mathisen]

I was watching innd with systat -netstat on FreeBSD
here last night.

16 incoming connections from Cidera.  I have
bumped up the size of the socket buffers to 512k, so I can
see connections that are backed up.

What seems to be happening, is that Cidera would send a big 
burst of news.  connections say 1-8 would fill almost
instantly.  THen innd would start draining the data, and in
the meaintime, 9-16 would fill.    However, watching it
once connection 1, maybe 2's recvq was empty, then I'd
see data being sucked out of other connections, until 1 and
2 got back up to having a bunch of data queue'd.  then
1-2 would drain, and in the meaintime, 10-16 are sitting
there with full socket buffers, and seemingly no work
being done to suck out the data.

So I'm wondering if when innd does the select() and gets
the list of fd's waiting to be read, does it just empty
the first one, then go back to select?  or does it
process all the fd's that came back as ready, and then 
go back into select.

Cause the first option kind of explains what I was seeing,
the 2nd, I shouldn't be seeing it.

I guess it would be easy to reproduce, just run innd on a
really slow machine. :)

If I run with the default 16k buffers, I can't keep up
with Cidera with 16 connections.  If I bump the recv buffers
to 512k, I kept up just fine for almost a day, but now
I see I'm starting to fall behind.

I am going to upgrade the CPU in the transit box, maybe that
will help.