makedbz and expire grow very big

Alex Kiernan alexk at demon.net
Fri Jan 4 09:40:46 UTC 2002


list-inn-workers at news.cistron.nl writes:

> In article <728zcz41qn.fsf at nd1.eng.demon.net>,
> Alex Kiernan  <alexk at demon.net> wrote:
> >list-inn-workers at news.cistron.nl (Miquel van Smoorenburg) writes:
> >
> >> makedbz becomes quite big:
> >> 345 MB? During nightly expire, it grows about this big as well. This
> >> box has 1GB of memory, so it isn't that much of a problem, but I've
> >> got other boxes with 192/256 MB and they will trash to death because
> >> of this.
> >> Looks like the history.hash or history.index files are built
> in-memory
> >> and only later on written out ?
> >
> >Thats the way -STABLE works too (either that or I completely
> >misunderstood the code when I rewrote it). You could hack line 1219 in
> >hisv6.c to be INCORE_MMAP and hope your VM dtrt (Solaris 8 most
> >definately doesn't, Linux I don't know).
> 
> Ok, so my guess was right ;). Looks like I never noticed it on other
> machines because they run tagged hash and the .pag file is only 74 MB. 
> I noticed it on the feeder that runs non-tagged last night because
> expire had been running for about 10 hours and was dead slow, the
> box went into swap and so on. Upgraded from kernel 2.4.15-pre6 to
> 2.4.15-pre8 and that appears to fix it (living on the edge ... )
> 

The real problem comes when the code switches from one hash table to
two (see the comments at the start of dbz.c). When that happens, set()
switches from mmap to many small pwrites, so you don't see a slight
slowdown, you see massive degradation.

Thinking about that, fixing set() (and friends) to mmap additional
tables probably wouldn't be too hard and should avoid the catastrophic
performance drop off (but you'd better have enough RAM/address
space!).

> BTW, I also run with largefile support enabled. That's probably
> also a reason that the process grows so big because of the
> 64-bit file offsets, right ?
> 

The .index will double, the .hash should say the same (if I remember
the way it works correctly).

-- 
Alex Kiernan, Principal Engineer, Development, Thus PLC


More information about the inn-workers mailing list