Slow expireover (again)
hakehoe at avalon.net
Mon May 24 14:39:47 UTC 2004
-----BEGIN PGP SIGNED MESSAGE-----
On May 24, 2004, at 7:42, Bill Davidsen wrote:
> The problem is really that the whole expire is probably a bad idea at
> current volumes. Old articles accumulate all day and then we try to
> clean up all at once. That may have made sense in the days when the
> system was idle from 2-6am in most places, but it dosen't see to fit
> now. In the long run trying to improve it is a band-aid, like coding a
> bubble sort in assembler. Like the leap from file-per-article to CNFS,
> to get an order of magnitude improvement you need to change WHAT you
> not HOW you do it.
> I think it's time for a radical rethink of history and overview. The
> idea of overview in a database is appealing, but other than trying it I
> am just not able to even guess if it would be fast enough. I have some
> ideas on history, but until I find some time they will be just ideas.
> Hopefully in the next 6-12 months I will have time to try them.
I've been considering writing a history storage method to put
history in a BerkeleyDB database. This could be integrated with
ovdb, withthe goal of combining the expire and expireover operations
On my all-CNFS server with 15M history lines, nearly 19M overview keys,
and group-based-expiry off; the expire takes 14 minutes, but the
expireover takes several hours. If I extend history enough so that
when a token is removed from history (either from a cancel or the expire
run) the corresponding overview entries can be efficiently removed as
well: you wouldn't need a separate expireover run.
Once expire and expireover are combined, the next step might be to
make expiration a continuous process, instead of a batched operation.
Perhaps tie it in somehow with the self-expiring storage methods, such
that the storage method notifies the expire daemon when an article
expires (if that can be done without too much of a hit to performance).
- - h
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)
-----END PGP SIGNATURE-----
More information about the inn-workers