INN not keeping up with incoming news

[Fabien Tassin]

According to Russ Allbery:
> 
> Alex Kiernan <alexk at demon.net> writes:
> 
> > We've just taken to locking the history.{hash,index} in core; its worked
> > wonders - we've gone from spending hours per day in HIS* to under 10
> > minutes (although I've no doubt that some of the time is hidden
> > elsewhere).
> 
> Here's something to try if you have a test box for it:  only mlock
> history.index and see if you still get the same win.  I wouldn't be
> surprised if you did.

I've tried to mlock() only history.index.

Before :
hiswrite 41992(5063)
hiswrite 40724(5485)
hiswrite 44661(5521)
hiswrite 40876(4972)
hiswrite 40325(4853)
hiswrite 44043(5292)
hiswrite 42554(5076)

After :
hiswrite 339(5054)
hiswrite 374(5513)
hiswrite 440(5917)
hiswrite 469(6115)
hiswrite 455(6024)

No other timer impacted. It is under Linux 2.4.1.
I'm still waiting to see if it make things faster...

> history.hash is already mmap'd, so I think you're already seeing a lot of
> the benefits there.  history.index on the other hand isn't mapped at all
> and INN normally writes to it without any buffering at all.  This is a big
> drawback of the new dbz code that needs to be fixed, after the new history
> API, and I think it's the main reason why people are seeing history writes
> be so slow.
> 
> When you lock it into memory, though, with Solaris and its unified buffer
> cache, I bet all of those disk writes are staying in memory.  I wonder if
> you wouldn't get similar performance by editing innd/his.c and specifying
> INCORE_MMAP for opt.pag_incore.

I'm also wondering it.

> > If anyone's interested, here's the code we're using (it doubtless has
> > some Solaris assumptions, I haven't tried building it anywhere else):
> 
> Shall I drop this into contrib?

yes, please.

-- 
Fabien Tassin -+- fta at sofaraway.org