Optimization for the expireover procedure.

[Russ Allbery]

Kirill Berezin <kyb at online.ru> writes:
> Russ Allbery ?????:

>> expireover start Tue Oct 16 07:11:08 ESTEDT 2007:
>>     Article lines processed  2787588
>>     Articles dropped          113463
>>     Overview index dropped    118785
>> expireover end Tue Oct 16 07:56:46 ESTEDT 2007
>> lowmarkrenumber begin Tue Oct 16 07:56:46 ESTEDT 2007:
>> lowmarkrenumber end Tue Oct 16 07:56:46 ESTEDT 2007
>>         expirerm start Tue Oct 16 07:56:49 ESTEDT 2007
>>         expirerm end Tue Oct 16 08:14:58 ESTEDT 2007
>> expire begin Tue Oct 16 08:15:28 ESTEDT 2007: (-v1)
>>     Article lines processed  2999159
>>     Articles retained        2870176
>>     Entries expired           128983
>> expire end Tue Oct 16 09:46:13 ESTEDT 2007
>>         all done Tue Oct 16 09:46:13 ESTEDT 2007

> expireover start Wed Oct 17 03:03:01 MSD 2007: ( -z/news/log/expire.rm -Z/news/log/expire.lowmark)
> expireover end Wed Oct 17 11:45:37 MSD 2007
> lowmarkrenumber begin Wed Oct 17 11:45:37 MSD 2007: (/news/log/expire.lowmark)
> lowmarkrenumber end Wed Oct 17 11:45:38 MSD 2007
> expire begin Wed Oct 17 11:46:11 MSD 2007: (-v1)
>    Article lines processed 14841845
>    Articles retained       12560464
>    Entries expired          2281381
> expire end Wed Oct 17 11:53:27 MSD 2007
> 	all done Wed Oct 17 11:53:28 MSD 2007

> 8 hours to expireover and a lot of missing articles due to feeder disk
> space limits. We are using ovdb. We tried buffindexed, but there were a
> lot of mmaps over a large area, and as a result it was run even slower.

And I'm running tradindexed.  Maybe that's the problem.  :)  You're
storing roughly 7x as many articles as I am, but probably on much faster
hardware and it looks like on CNFS.  I wonder if tradindexed wouldn't be
quite a bit faster for you.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>