Rebuild your Overview for Fun and/or Profit! Or something

Jaye Mathisen mrcpu at
Tue Dec 19 20:46:30 UTC 2000

With regards to below, would it be possible to just take
cycbuff.conf, parse it, generate new storage.conf and cycbuff.conf
files with a 1-1 mapping between the metacycbuff and the cycbuff,
ignoring the old names for things?


I have 4 buffers buff00-03

and 2 metacycbuffs

mb1 made up of buff00
mb2 made up of buff01-03.

I have a storage.conf that says

alt.* -> mb2, *,!alt.* -> mb1

Can I just generate 4 cycbuffs.conf containing

newmb buff00

newmb buff01

newmb buff02

newmb buff03

with  new storage.conf for #1 being
*,!alt.* -> newmb

and the storage.conf for #2 -> 4 being

alt.* -> newmb

(all other things being equal?)  Is this what you're saying below?  because I have 300GB's
fo news to re-import, and I'm thinking about just coding this up in a perl script for 
everybody to enjoy, if I understand the theory.
On Tue, Dec 19, 2000 at 12:07:43AM +0100, Reddy Crashalott wrote:
> [disclaimer: this is an invalid sender address; replies should be
>  posted to the list or perhaps better not posted at all]
> If, like me, you enjoy living dangerously, you've probably done stupid
> things like run out of disk space for your ovdb logs which has resulted
> in unrecoverable checkpoints so you've probably nuked them and as a
> result wound up with an apparently corrupt overview database that nothing
> seems to like very much, so you've wiped the whole entirety of your
> overview data to rebuild it all from ground up.  Or something.  Not that
> I've done this.  Well, not more than a few dozen times, anyway.
> Recovery of the overview data is pretty straightforward, with the simple
> makehistory command.  But you probably know you can speed things up by
> running several makehistory commands in parallel, to work on part of the
> spool at a time.
> The good thing about this is that you can recover the popular pr0n that
> people want so much in a matter of minutes, while the text froups take
> their good ole time.
> The way to do this, as you well know, is to use the INNCONF value to
> point to a different inn.conf file that points to a different pathetc
> that points to a different cycbuff.conf and storage.conf file.  In
> the storage.conf file you want to comment out all but at least one
> reference to the metacycbuff(s) you want to populate into overview, and
> in the cycbuff.conf you want to comment out all but the corresponding
> definition(s) of the metacycbuff(s), and also all the appropriate
> cycbuffs proper.  Repeat, lather, rinse, as needed until you've split
> your spool into all the parts that, if put together, form your spool.
> Then run makehistory with INNCONF pointed appropriately, in parallel,
> or in order of what you want to appear.
> Note that if you have a metacycbuff formed of more than one cycbuff,
> the contents of one will be put in the overview data first, then the
> next, and so on.  Normally, with a single buffer, articles start to
> appear and follow one after another.
> A further speedup can be obtained by splitting the metacycbuff
> definitions into two cycbuff.conf files (and INNCONF), one for
> each cycbuff, and makehistory'ing each in parallel.  This is
> particularly useful for text groups in multiple cycbuff metacycbuffs
> that otherwise would take many an hour to make all the passes.
> Question:  The recovery of the data proceeds from the first article
> in the froup to the most recent, by calling some `next'-like function
> in the S&M k0de, while at the same time, you could be writing fresh
> new incoming articles, leaving a gap of temporarily missing news.
> This also means that people who missed nabbing news right before you
> crashed the system have to wait for the articles they're missing to
> appear when they may well have the older articles that reappear first.
> I haven't tried to figure out the logic in this `next'-article call,
> but would it be possible in general to reverse the order of pulling
> articles, by grabbing the most recent article and working backwards?
> This would cause more recent (and more interesting) articles to show
> before the older ones arrive, and would speed up the time before
> users would be able to make use of the system again after a crash.
> It's just an idea I had while bored, waiting for makehistory to work
> its magic...  With these techniques, I'm able to be fully running again
> with a day of all the pr0n after some 15 minutes, with only the text
> articles from the last couple weeks taking longer than that.
> yr humble whatever

More information about the inn-workers mailing list