buffindexed: expireover needs _very_ much memory

[Heiko Schlichting]

Katsuhiro Kondou wrote:
> To recreate overview data, you can do it with 'makehistory'.
> '-I -O -x -l 0' would be useful for you.

I recreate the buffers with mkfile and start 'makehistory -I -O -x -l 0'.
Now, 12 hours later, I stopped this process. I seems that it has a
similar problem as expireover because makehistory uses 1 GB RAM of the
system. Using par, it is shown that every ca. 30 seconds a mmap() call is
done - always with the same offset:

    0mS[3]: mmap(0, 16384, PROT_WRITE|PROT_READ, MAP_SHARED, 4, 378044416) = 0x47008000
22825mS[2]: fcntl(5, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=168}) OK
22825mS[2]: time() = 939827664
22825mS[2]: fcntl(5, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=168}) OK
28437mS[2]: mmap(0, 16384, PROT_WRITE|PROT_READ, MAP_SHARED, 5, 378044416) = 0x47004000
50621mS[2]: fcntl(6, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=168}) OK
50621mS[2]: time() = 939827692
50621mS[2]: fcntl(6, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=168}) OK
61688mS[2]: mmap(0, 16384, PROT_WRITE|PROT_READ, MAP_SHARED, 6, 378044416) = 0x47000000
78645mS[1]: fcntl(7, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=168}) OK
90541mS[3]: time() = 939827732
90542mS[3]: fcntl(7, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=168}) OK
101743mS[3]: mmap(0, 16384, PROT_WRITE|PROT_READ, MAP_SHARED, 7, 378044416) = 0x46ffc000

As this looks like it never comes to an end and uses about 1 GB RAM, I stopped
this. The server does not have so many articles yet so 10 hours is much longer
than expected in this environment.

The server is a slave server - does this break buffindexed overview? On a
slave server it is not garanteed that the articles come in the Xref order
because innfeed reorders the queues. But I doubt that a slave server is
so unusual that it should break buffindexed and during normal operation
of the slave the wrong Xref order works without problems. Just expireover
(and now makehistory) fails. If it is not the Xref order, what else could
be the problem? Any ideas?

Is there anything I should test on this server? If not, I see no other
solution as deleting spool and overview and start over.

Heiko

Heiko Schlichting       | Freie Universitaet Berlin
heiko at FU-Berlin.DE      | Zentraleinrichtung fuer Datenverarbeitung (ZEDAT)
Telefon +49 30 838-4327 | Fabeckstrasse 32
Telefax +49 30 838-6721 | D-14195 Berlin