innfeed crashing on Linux

[Mark Hittinger]

> Sep 28 15:46:29 innfeed[30295]: bigbird:0 Setting up a reopen callback
> [Crash here -aw]
>...
> This is a fairly recent behavior.  Any ideas???

Anne the reopen callback queues up a subtask to check on the connection
later - like its busy now or not responding now.

Innfeed is allocating some memory on the stack to hold the "note to self"
and then sticking the note on its wall of post-it notes.  I think the wall
might be filling up with too many :-)

Are you getting a core file?  It would be very helpful to get a stack backtrace
on the core file with gdb.

Anyway one idea is that you are running into the process stack limit.  If
you get in your Linux box and do the "limit" command under bash you'll see
something like this:

[bugs at hlt ~]$ limit
...
stacksize       8192 kbytes
...

Linux will bomb the program if the stack tries to grow beyond 8meg.  I have
seen programs with large data structures reach this limit and die unexpectedly.

If I remember right your site is processing a bunch of tiny files.   My
thinking is that your unique load footprint could cause the innfeed to get to
the 8meg of stack under the extreme condition of the destination host being
always unresponsive.

You could set up a seperate innfeed process for bigbird.  That way the others
would not restart.

If the stacksize limit is the culprit we might have to add a bit of code to
innfeed to kick up the limit.  Another trick would be to set the stacksize
to unlimited in the rc.news script, i.e.:

##  Start the show.
echo 'Starting innd.'
limit stacksize unlimited
eval ${WHAT} ${RFLAG} ${INNFLAGS}

This is just a guess and the core file (if there is one) backtrace would let
us make a better guess.

Good luck,

Mark Hittinger
bugs at pu.net