INN -BETA expire vs. traditional spool

Bill Davidsen davidsen at tmr.com
Fri Jun 30 15:26:20 UTC 2000


On 28 Jun 2000, Russ Allbery wrote:

> What I think needs to happen instead is that more of the expire process
> needs to be dependent on the underlying storage implementation.  For
> tradspool, what you want to do is exactly what older versions of INN did;
> write out the path names for each article you're expiring and then have
> fastrm do what it did before, namely just unlink them.  Looking at
> tradspool_cancel, the storage backend will deal with that just fine.
> fastrm should look at each line and decide from the first character if
> it's a path or a storage API token and do the appropriate thing.

I hate to say it, but I think it's time for expire to... well, expire. We
need to eliminate history as a flat file and go to "something else" which
allows removing files as they are overwritten, or as they age until it's
time for them to go.

Doing something big and slow on a regular basis is no longer practical for
large sites, and I think the time to really fix it is "soon," rather than
to try to make the current unscalable solution work. Improving the current
solution is like coding a bubble sort in assembler, it will still take far
more resources than are readily available, and cause load spikes.

A better history would allow grabbing articles by message-id or group, and
the data entry would include filename(s) if tradspool were being
employed. Tradspool would expire articles either based on age or total
disk space for {articles, article class, hierarchy, group}. Instead of a
spike, article would "evaporate' as they reached the expiration criteria.

Expire was a good idea when there was a time of low load for housekeeping.
That's gone, the volume is higher, and it's not used by humans in the USA,
but by people everywhere and both posting and sucking software. I figure
my consolidators need 300-500MB of RAM more for expire than for the rest
of the day. And even then the disk load on a good RAID system is such that
some drop in article rate accepted is seen.

If it weren't for reported problems with the BSD db stuff I would look at
that, until I'm sure it will keep working I don't want to be dependent.
Maybe someone solved the problems and I missed it?

That's my take, time to rethink history and expire, and I actually expect
to have some time after July to work on fun stuff!

-- 
bill davidsen <davidsen at tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.




More information about the inn-workers mailing list