Wow... expire still too slow

[Russ Allbery]

Pauline Middelink <middelink at polyware.nl> writes:

> Yes, tradspool.  It would be great to have this server running properly
> again, as it is now, it needs to much attention from my ample time.

> The reason I see tradspool is so slow, is because of the split between
> the actual removing of articles by expireover and the updating of the
> history by expire.

This is not the only reason why tradspool is slow, although yes, this is a
factor.  I'll be able to fix this, I think, once we have the ability to
update history records in-place, which should speed up expire by another
factor of two, roughly.

Another reason why tradspool is so slow is because fastrm was broken from
a speed perspective for backends that stored articles in a single file per
article.  This has been fixed in CURRENT for a while (I did the fixes for
my new reader machine, which is mostly tradspool), and I'm currently
working on backporting those changes to STABLE.  This was more difficult
than it might have otherwise been because I completely rewrote fastrm in
the process and made use of infrastructure in CURRENT which isn't in
STABLE, such as lib/error.c.

I think I finished backporting fastrm last night, so now I just need to
dig through my changes to the storage backends and the storage API.

> I assumed (when talking about the history API) that when this comes into
> effect, expireover will expire those articles in the history too,
> thereby making the bin/expire run effectivly useless.

It unfortunately isn't quite that simple, but yes, my intention is to
combine expire and expireover into a single program.  The best way to do
that isn't quite clear to me yet, but I've been thinking about it.

> It does not have much memory, but if it was swapping that much, vmstat
> would show it, right? It doesn't.

I don't trust vmstat.

You need enough memory to hold two complete copies of the history.hash
file in memory at the same time.  If you don't, expire will swap itself to
death and will take days to complete.

>> I'm running a pure tradspool server with a quarter again as many
>> articles as yours and it expires in three hours.  Not ideal, but
>> certainly much, much faster than what you're seeing.

> :( Howmuch RAM does your machine have?

320MB.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>