CNFS mmap allocation problem and kernel panic

[Antoine Delvaux]

Hello all,

I'm coming back with a problem we've seen a few times before (see thread 
'CNFS mmap memory error' in march) and are starting to see it quite 
frequently now.  We use INN-STABLE-20021007 in a feeder only setup (no 
readers, no overview).  We use CNFS buffers as storage method.  The OS is 
Debian stable with a 2.4.18 kernel build from Debian sources.  The box 
itself is a Dell 6400 with 2GB ram and 4 Pentium III Xeon 700MHz.  It is 
fully dedicated to INN with an apache server to access the stats.

This machine has been working great during two months but wasn't very 
loaded, only 10-20 peers exchanging full feed, workload between 1 and 2 (4 
processors box), output traffic around 120Mb/s.  However we were sometimes 
seeing the following lines in the logs (extract of innreport), at random 
occurences :

----
Unknown entries from news log file:

First 50 / 2870 lines (1.7%)
Sep 24 19:38:52 aotearoa innfeed[24368]: CNFS-sm: could not mmap token 
@0301423130000000000000274D45000002CB@ B10:0x4e9a8a0c:715: Cannot allocate 
memory
Sep 24 19:38:52 aotearoa innfeed[24368]: Could not retrieve 
@0301423130000000000000274D45000002CB@: mmap failed
----

In the last few weeks, we've started to add a few feeds to the box (10 at 
most) and began to run into troubles this week.  The box has crashed 4 
times this week with kernel panic related to memory access or virtual 
memory mapping.  innreport for two days ago was saying this :

----
Unknown entries from news log file:

First 50 / 346732 lines (0.0%)
Oct 6 01:18:21 aotearoa innfeed[31462]: CNFS-sm: could not mmap token 
@03014233390000000000002A37FF00000303@ B39:0x546ffe0c:771: Cannot allocate 
memory
Oct 6 01:18:21 aotearoa innfeed[31462]: Could not retrieve 
@03014233390000000000002A37FF00000303@: mmap failed
----

Quite a lot more mmap errors then before.

How can this kind of error be trouble shooted ?  How can INN crash the 
kernel ?  I'm a bit lost at where to look at.  Does anybody have ideas ?  
I'm going to compile the linux kernel with some debugging enabled and see 
what's the output.

If you want, you can access the box statistics here (load, feeds, 
innreport, etc.) : http://newsfeed.belnet.be/ (the crashing box is 
aotearoa).  This box is also the web server, so if it crashes, you don't 
have access to statistics...

Thanks for your time,

Antoine.

-- 
http://www.belnet.be                              Tel: + 32 2 790 33 33
BELNET Service Support Team                       Fax: + 32 2 790 33 34