INN not keeping up with incoming news

Alex Kiernan alexk at demon.net
Mon Feb 12 08:53:15 UTC 2001


Fabien Tassin <fta at sofaraway.org> writes:

> According to The Hermit Hacker:
> > 
> > FreeBSD 4.2-STABLe, and my stats are viewable at http://news.hub.org/~news
> > if anyone wants to look at the timings ...
> 
> There's a factor 4 comparing HIS to ART in your stats so you are definitly
> seeing the problem.
>  

[With apologies that much of this is probably Solaris specific, but
thats the kernel I know best]

We're running on Solaris, with the history .hash & .index files locked
in core (the box has 2GB of RAM, so its not too big a hit). My theory
is this (and the profiling seem to support it):

On Solaris the mmap()ed history file is as eligible for pageout as any
other page, since all I/O happens through the VM system[1] there is a
considerable load on the VM system (we need to free ~384 pages per
second). The application binaries, shared libraries and anonymous
application memory (malloc etc.) are generally ineligible[2], in
addition recently received articles are more likely to have been
touched in the page cache than the history pages[3].

By locking the history pages in core[4] I correct the behaviour of the
VM subsystem.

With this change I've taken a box which was struggling to take
70-80GB/day and feeding 120-150GB/day to a box which is taking
180-190GB/day[5] and feeding 300-500GB/day.

I suspect I could tune the kernel paging parameters some more to avoid
the locking in core, but keeping the history pages in core seems more
like the semantics I'm after.

I'll post the mlock code after I've tidied it some, to see if it helps
others.

[1] - on Solaris
[2] - priority_paging=1
[3] - the articles get written sequentially, then read by the
innfeeds, so their `used' bit will get touched often
[4] - which probably isn't what was really wanted, I'd like to just
give them a much longer lifetime
[5] - slightly less than a full feed, but the box in question has a
"challenged" I/O system, without that I'm confident it would take a
full feed.

-- 
Alex Kiernan, Principal Engineer, Development, Thus PLC


More information about the inn-workers mailing list