buffindexed: expireover needs _very_ much memory

[Heiko Schlichting]

Katsuhiro Kondou answered to Heiko Schlichting:
> } INN created the buffers 1-8 with a block size of 16384 bytes (Snapshot of
> } last week) and the buffers 9-17 with a block size of 8192 bytes (with the
> } snapshot of yesterday):
> :
> } [reader at Curry] (~): buffindexed de.test | head
> } base 16705(2), last 16672(10), cur 16705(4), prev 16705(2), next 16705(8)
> } de.test: low is 155198, high is 155936, count is 229, flag is 'y'
> }   gloc is 0x00000f92
> }     addr 0x423c000, ovblock 0x423e000, base 155070, baseoffset 0, len 16384, index 2, blocknum 16705
> }     addr 0x4240000, ovblock 0x4242000, base 155274, baseoffset 0, len 16384, index 4, blocknum 16705
> }     addr 0x4244000, ovblock 0x4246000, base 155478, baseoffset 0, len 16384, index 8, blocknum 16705
> }     addr 0x4248000, ovblock 0x4248000, base 155682, baseoffset 0, len 8192, index 9, blocknum 16672
> }     addr 0x424c000, ovblock 0x424c000, base 155886, baseoffset 0, len 8192, index 10, blocknum 16672
> 
> Hm, you've misused buffindexed.  OV_BLOCKSIZE must be same
> size for all buffers.  Or you'll see serious problem like
> this case.

Hmm. I never (re)defined OV_BLOCKSIZE and therefore this value should always
defined to 8192. Maybe my interpretation on 'len 16384' or 'len 8192' in the
above output is wrong. It seems to be gip->len, not the blocksize. Should I
expect this to be the same for all buffers?

But I doubt that this is the only reason for the growing of expireover. Last
week, with only 8 buffers which was created at the same time, I noticed that
the expireover process needs a lot more memory than 1GB RAM. I set
'noexpireover' for news.daily and tried expireover by hand today. But if
you think it could help to clear/reinitalize the overview buffers, I'll
do that after I found how to rebuild the overview database from the CNFS
spool. I know that rebuilding overview was possible in INN 1.x & 2.2 with
expireover -a but this program does not have an "-a" option in INN 2.3 and
I didn't succeed in finding it's replacement.

Heiko

Heiko Schlichting       | Freie Universitaet Berlin
heiko at FU-Berlin.DE      | Zentraleinrichtung fuer Datenverarbeitung (ZEDAT)
Telefon +49 30 838-4327 | Fabeckstrasse 32
Telefax +49 30 838-6721 | D-14195 Berlin