buffindexed: expireover needs _very_ much memory

[Heiko Schlichting]

Does someone noticed that expireover needs _very_ much memory when overview
method buffindexed is used?

[root at Curry] (~): ps -ef | grep expireover
  reader     367179     318848  0 04:35:59 pts/6   0:46 expireover
[root at Curry] (~): ls -lh /proc/367179
-rw-------    1 reader   news         747M Oct 13 04:35 /proc/367179

So, expireover uses 747 MBytes yet and it is still growing fast.

Using par (process activity reporter, a truss-like system call tracer)
it is shown that expireover calls mmap in a fast loop:

990678mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 29, 634699776) = 0x557f8000
990703mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 30, 634699776) = 0x557f4000
990727mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 31, 634699776) = 0x557f0000
990748mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 32, 634699776) = 0x557ec000
990775mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 33, 634699776) = 0x557e8000
990794mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 34, 634699776) = 0x557e4000
990814mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 35, 634699776) = 0x557e0000
990832mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 36, 634699776) = 0x557dc000
990855mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 29, 634961920) = 0x557d8000
990878mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 30, 634961920) = 0x557d4000
990903mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 31, 634961920) = 0x557d0000
990925mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 32, 634961920) = 0x557cc000
990946mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 33, 634961920) = 0x557c8000
990968mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 34, 634961920) = 0x557c4000
990982mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 35, 634961920) = 0x557c0000
990997mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 36, 634961920) = 0x557bc000
991025mS[  3]: mmap(0, 8192, PROT_READ, MAP_SHARED, 29, 635224064) = 0x557b8000
[...]

I'm a little bit surprised that this only happens for 8 file descriptors
because there are 17 buffers defined in buffindexed.conf. But I recognised
that a similar mmap loop already run until the offset matches the size of
the index buffers (here: 1 GB). During the first loop expireover is using
less than 150 MByte, which is no problem on this system. mmap() is not known
as broken on IRIX, the buffers are smaller than 2 GByte and are regualar
files in the filesystem.

Anyone noticed similar problems with buffindexed?

Here is the buffindexed.conf:

#
# overview buffer configuration file
#
# $Id: buffindexed.conf,v 1.1 1999/10/12 23:04:06 heiko Exp $
#
1:/news/reader/spool/overview/buf01:1048576
2:/news/reader/spool/overview/buf02:1048576
3:/news/reader/spool/overview/buf03:1048576
4:/news/reader/spool/overview/buf04:1048576
5:/news/reader/spool/overview/buf05:1048576
6:/news/reader/spool/overview/buf06:1048576
7:/news/reader/spool/overview/buf07:1048576
8:/news/reader/spool/overview/buf08:1048576
9:/news/reader/spool/overview/buf09:1048576
10:/news/reader/spool/overview/buf10:1048576
11:/news/reader/spool/overview/buf11:1048576
12:/news/reader/spool/overview/buf12:1048576
13:/news/reader/spool/overview/buf13:1048576
14:/news/reader/spool/overview/buf14:1048576
15:/news/reader/spool/overview/buf15:1048576
16:/news/reader/spool/overview/buf16:1048576
17:/news/reader/spool/overview/buf17:1048576

INN created the buffers 1-8 with a block size of 16384 bytes (Snapshot of
last week) and the buffers 9-17 with a block size of 8192 bytes (with the
snapshot of yesterday):

[reader at Curry] (~): buffindexed de.test | head
base 16705(2), last 16672(10), cur 16705(4), prev 16705(2), next 16705(8)
de.test: low is 155198, high is 155936, count is 229, flag is 'y'
  gloc is 0x00000f92
    addr 0x423c000, ovblock 0x423e000, base 155070, baseoffset 0, len 16384, index 2, blocknum 16705
    addr 0x4240000, ovblock 0x4242000, base 155274, baseoffset 0, len 16384, index 4, blocknum 16705
    addr 0x4244000, ovblock 0x4246000, base 155478, baseoffset 0, len 16384, index 8, blocknum 16705
    addr 0x4248000, ovblock 0x4248000, base 155682, baseoffset 0, len 8192, index 9, blocknum 16672
    addr 0x424c000, ovblock 0x424c000, base 155886, baseoffset 0, len 8192, index 10, blocknum 16672

Heiko

Heiko Schlichting       | Freie Universitaet Berlin
heiko at FU-Berlin.DE      | Zentraleinrichtung fuer Datenverarbeitung (ZEDAT)
Telefon +49 30 838-4327 | Fabeckstrasse 32
Telefax +49 30 838-6721 | D-14195 Berlin