INN not keeping up with incoming news

Alex Kiernan alexk at demon.net
Mon Feb 5 09:08:48 UTC 2001


Katsuhiro Kondou <kondou at nec.co.jp> writes:

> In article <72bssluy1v.fsf at nd1.eng.demon.net>,
> 	Alex Kiernan <alexk at demon.net> wrote;
> 
> } My suspicion is the history performance problem is platform dependent
> } - what platforms are people running on (we're using Solaris 2.6/8 and
> } seeing it)?
> 
> I think it depends on how much memory the server has.  Mine in
> production has 4GB, which Fabien said was not in the problem.
> While, on my testing box(1GB mem) it's in the case(hishave and
> hiswrite consumes 1/3 cpu time).

CPU or wall clock time? Profiling it here showed that it was all wall
clock & not CPU time.

>  I think it's due to looking
> thru mmapp'ed region(history.hash) everytime(in most case)
> dbzexists() and dbzstore() are called.  I think this could be
> improved by returning pointer of empty entry if dbzexists()
> returns FALSE.  The pointer is also cached on pre-commit cache
> so that it can be used by dbzstore().
> 

Not sure I understand...

What I thinks happening is that (at least on Solaris) is that the IO
load is causing the mmaped history file to be paged out, causing IO
wait when the mem* routines happen in the dbz code. I've hacked a
server here so that it locks the history.{hash,index} files in core to
demonstrate this...

Before:

history lookup 11:54:37.225 49.8% 16618668 0.006 2.580 8.002 
history write 04:36:32.542 19.3% 995949 7.404 16.660 33.993 

After:

history lookup 00:02:03.073 0.1% 20674300 0.003 0.006 0.403 
history write 00:07:40.148 0.5% 1206923 0.174 0.381 12.178 

Chucking memory at the problem probably works because the break point
moves far enough out that the kernel doesn't do completely the wrong
thing.

I'm going to clean up my mlock() code so that inndstart spawns a
helper process which innd can use to lock the history files into
memory.

> I know Russ is thinking history api, but I think not a few people
> cannot wait until it's done.

At a guess I'm 2-3 days from having a bunch of stuff which could be
committed as a history API (just duplicating what's there today
really), obviously thats not going to help with the problem, but it
should give us a framework for replacing the current code more
painlessly.

-- 
Alex Kiernan, Principal Engineer, Development, Thus PLC


More information about the inn-workers mailing list