Slow expireover (again)

[Russ Allbery]

Mike Zanker <mike-sender-6677e0 at zanker.org> writes:

> OK, I might switch from timehash to timecaf for the next 7 days (I
> expire anything over 7 days) to see if makes a difference.

I know you said this previously, but I don't remember now... is the
slowdown related to an upgrade to a newer version of INN (and if so, from
what to what?) or did it just happen all of a sudden?

Thanks for the trace.  There are some obvious bogosities here, but I'm not
sure if they're related to the problems you're encountering (and I'm
guessing unfortunately probably not).  But the trace is interesting, since
there are some obvious changes we could make that would at least speed up
some of the overhead.

For example, notice that group.index is being opened and closed between
every single group expiration, which is really rather unnecessary (and
mmap isn't the world's fastest system call, so it may well be noticably
slower to do this).

The constant locks and unlocks are probably unavoidable if we want to let
innd keep accepting articles while expireover runs, though.

Overview files that are empty shouldn't be expired in the first place;
there's no point, and it would save a bunch of opens and renames.  And
knowing the university Usenet environment, I bet you have tons of empty
course newsgroups.

Setting the file descriptors to close on exec looks pretty painful in a
trace, although I'm not sure how much of a performance hit it really is.
It's an additional two system calls for every open, though, and Solaris's
system call interface, unlike Linux's, is pretty painful.  Those are only
in there for innd; we should be able to make this some sort of
configuration flag, maybe another flag to pass into OVopen() like the
existing modes.

I'll look at doing the above, but I'm guessing it's not going to be enough
to really resolve the problem.  I'd be curious to see a trace while it's
doing a group with articles in it.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>

    Please send questions to the list rather than mailing me directly.
     <http://www.eyrie.org/~eagle/faqs/questions.html> explains why.