History File over 2GB

[Russ Allbery]

bill davidsen <davidsen at tmr.com> writes:

> We might need a token to msgid table, and a table with Xref info per
> msgid. Then you could relatively quickly find the info to delete. But as
> much thrashing as that might seem, it still needs to be done sometime,
> either by expire or by a realtime (more or less) cleanup.

Well, no, it *doesn't* have to be done sometime, since there's the
existing expire system, which bases its decisions on the overview entries.
They contain the article tokens, so everything flows downstream.

Overview has to contain a pointer to the storage API tokens, since
overview is the one thing that's indexed by group and article number and
nnrpd needs to be able to find the articles.  History has to have a
pointer to the storage API token to support retrieval by message ID.
Those two are basically forced by the NNTP protocol.

So right now, we have an information flow that looks like this:

              ---->- History ->----
             /                     \
    Overview ----->----------->----- Storage

where you can either query history directly or query overview, and both of
them lead you to the storage API token.

Expire wants to do two basic things:  It wants to know from the storage
subsystem what articles have been automatically expired, and it wants to
know based on rules and the article arrival time when the article should
be deleted.  If overview asks storage, that works, since all the necessary
information is available to do that.  But right now, there's no way for
storage to tell overview, since storage has no way of knowing the keys
that overview uses (without doing something incredibly nasty, like finding
the article Xref header and breaking it apart).

Cancels are currently handled by retrieving the storage API token from
history and telling storage to delete the article without ever informing
overview.

You can get the overview keys for a new incoming article, but once the
article is stored in disk, you cannot get the overview entry for an
article if all you have is a message ID without again doing something
really ugly like retrieving the article and taking apart the Xref header.

Adding to the complexity, there can be multiple overview records for a
given article (one per newsgroup it was posted to), but only one history
or storage API entry (tradspool fakes multiple entries under the hood, but
as far as the storage API is concerned, there's only one).

Note that for crossposted articles, overview-driven expiration already
parses the Xref header to figure out whether to tell the storage subsystem
to delete the article, since it depends on whether it has expired out of
all of the groups to which it was posted or just some of them.  Bleh.

It's clear to me from an algorithmic standpoint that every time we parse
the Xref header to recover information, we're doing something
fundamentally wrong, whether that be in not having sufficient data
structures or in using the wrong algorithm.  That's a nasty bit of parsing
to recover data.

There are a few other conflicts that we run into.  One of them is that for
traditional expiration, you essentially treat every crosspost of an
article as a distinct article that's expired separately, but then
special-case the point at which they've all expired to remove the article
from the backend storage entirely.  Whereas if you're running
self-expiring methods exclusively, you treat articles as single entities
no matter how they're crossposted, and retain them for so long as they're
still in the storage subsystem.

If we had metadata stored in the storage subsystem that gave the groups
and article numbers to which the article was posted, it would close the
loop to a significant degree.  That would at least make it possible to
expire overview in real time like you're talking about since the
information would be available to the storage subsystem which is
overwriting or cancelling the articles.

I'm not sure that it's worthwhile deleting history in real time, since
having up-to-date history doesn't affect the operation of the server; if
someone retrieves something by message ID that isn't there any more, you
may have to make an extra trip to the storage subsystem, but that's all.
If we can get the locking right, offloading that into a separate process.

One of the problems that I always hit and that various parts of INN are
dealing with right now is that newsgroup names make horrible keys because
they're not fixed-length.  tradindexed uses an MD5 hash.  tradspool
assigns its own numbers to groups (and has various problems because of
that, although it also enables a neat group renaming hack).  If we could
generate somehow a way of assigning unique numerical keys to newsgroups,
it seems like that would be a significant advantage, although there are a
bunch of nasty issues related to that that have to be sorted out (many of
which tradspool is already dealing with).

And, of course, one of the fundamental problems with adding additional
metadata to the stored articles is that it invalidates all the existing
storage APIs; we'll either need to do a painful upgrade or design INN to
work either with or without that information and phase in new storage
subsystems.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>

    Please send questions to the list rather than mailing me directly.
     <http://www.eyrie.org/~eagle/faqs/questions.html> explains why.