History File over 2GB

[bill davidsen]

In article <yllm6o2lwz.fsf at windlord.stanford.edu>,
Russ Allbery  <rra at stanford.edu> wrote:
| 
| Mark Hittinger <bugs at pu.net> writes:
| 
| > I'd like to see some record flags in history and overview data that
| > imply the record is deleted and should not be used.
| 
| We already have that for overview, pretty much.  Not entirely, but it
| would be relatively easy.

But that implies nnrpd doing more editing that is currently done,
doesn't it?
 
| > It would be nice during the day to have cancels flip such a "deleted"
| > bit in the overview record.
| 
| This is the problem.  Overview is indexed by newsgroup and article number.
| How do you get that for the message that was cancelled?

It's easy to do right, but hard to do both right and fast. If you don't
mind a lot of references you can have a single descriptor for each
article, and overview entries just point to them, with the descriptor
pointing to the article text. I did a BBS that way, and it works
perfectly with everything marked valid or not, added to a cleanup list,
etc. But what works on a 68000 running SysV4 with four 2400 baud modems
doesn't scale at all to usenet.

I see expire as the least scalable part of INN right now, sort of like
writing a check and hoping to cover it before it's due. And running
nnrpd as a daemon adds to the problem, it doesn't always notice when
expire is run, processes holding file open use tons of file space in
deleted but not released files, etc. I think we have to go database at
some point, so the cleanup can be done all the time. Going to cycbuffs
eliminated trying to unlink a million files a day, I think history and
overview must go that way eventually.
-- 
bill davidsen <davidsen at tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.