History File over 2GB

[Russ Allbery]

bill davidsen <davidsen at tmr.com> writes:
> Russ Allbery  <rra at stanford.edu> wrote:

> | We already have that for overview, pretty much.  Not entirely, but it
> | would be relatively easy.

> But that implies nnrpd doing more editing that is currently done,
> doesn't it?

No, nnrpd is already looking at the overview a line at a time.  We're not
using sendfile or anything to just send the overview to the client; among
other things, we have to handle out-of-order article arrival in the
presence of Xref slaving.

> It's easy to do right, but hard to do both right and fast. If you don't
> mind a lot of references you can have a single descriptor for each
> article, and overview entries just point to them, with the descriptor
> pointing to the article text.

We have that already in the form of the storage API tokens.  I don't see
how that helps.  Having overview point to something else isn't the
problem; the problem is having something else point to overview.

> And running nnrpd as a daemon adds to the problem, it doesn't always
> notice when expire is run, processes holding file open use tons of file
> space in deleted but not released files, etc.

nnrpd running as a daemon is certainly supposed to notice when expire
happens.  If it doesn't, that's a bug.  It's not that hard for nnrpd to
figure it out.  It already checks to see if the history file has changed
and reopens it, I thought.

> I think we have to go database at some point, so the cleanup can be done
> all the time. Going to cycbuffs eliminated trying to unlink a million
> files a day, I think history and overview must go that way eventually.

Note that I'm still using traditional spool for all text hierarchies
because I honestly thing it's better.  More reliable, more robust, and in
many ways just easier to deal with for anything that doesn't have sudden,
unpredictable usage spikes.  It takes expire about four hours to run, and
the server is fully available to clients during the entire period and
accepting new articles during most of that period.

I agree that the way that expire is currently being done is less than
ideal, but I don't see it as as large of a problem as you do.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>

    Please send questions to the list rather than mailing me directly.
     <http://www.eyrie.org/~eagle/faqs/questions.html> explains why.