expire info in history when using CNFS

[Fabien Tassin]

According to Russ Allbery:
> 
> I'm not sure how much one is really saving, though; nightly expire is
> actually pretty fast for a transit machine.  My slowest pure-CNFS feeder
> machine only takes:

I agree.. but it requires *a lot* of RAM to be fast. My expire process usually
takes between 350 and 500MB of RAM. As my innd takes around 150MB and I have
two innfeed (350MB + 50MB), it is easy to start swaping, especially if
a control message arrives during expire forcing innd to flush its funnels
(forcing me to keep another instance of each innfeed for a time, sometime
several instances).

I've looked at the code and I only see two issues :
- make innfeed smaller by avoiding it to keep/make copies of articles
- make innd smarter regarding program feeds during reloads and control messages
(I think this point is in your todo list Russ).

> expire begin Sun Jan 21 04:34:05 PST 2001:
>     Article lines processed  3312954
>     Articles retained        2967846
>     Entries expired           345108
>     Old entries dropped      1070795
>     Old entries retained     4290381
> expire end Sun Jan 21 04:51:43 PST 2001
>         all done Sun Jan 21 04:55:00 PST 2001
> 
> 17 minutes to run expire, and the server is only paused for a small
> fraction of that time.

expire begin Sun Jan 21 00:08:27 CET 2001:
    Article lines processed  5197774
    Articles retained        3715771
    Entries expired          1482003
    Old entries dropped      1355592
    Old entries retained     8732245
expire end Sun Jan 21 00:18:49 CET 2001
        all done Sun Jan 21 00:21:04 CET 2001

with traffic half the average during this period. The whole news.daily
run takes around 15 minutes on my system.

[ 3 minutes are taken by innreport. For a transit server, I think it
can be reduced to a few seconds by using innpeer-stats with some 
extentions to create an ASCII report. No time yet but it is in my
todo list. ]

-- 
Fabien Tassin -+- fta at sofaraway.org