crash whilst expiring the overviewdb
John Leslie
john at jlc.net
Sat May 19 13:16:39 UTC 2001
On Fri, 23 Feb 2001 19:20:12 +0900 (JST)
Katsuhiro Kondou <kondou at ISC-Nospam.nec.co.jp> wrote:
> In article <5.0.2.1.0.20010222171753.05031410 at pop3.gradwell.net>,
> Peter Gradwell <peter at gradwell.com> wrote;
>}
>} We get the following in the daily usenet report:
>}
>} buffindexed.c:1590 Can't malloc 701472768 bytes: Cannot allocate
>} memoryExpire messages:
>} expireover start Wed Feb 14 03:08:03 GMT 2001: ( -z/var/log/news/expire.rm
>} -Z/var/log/news/expire.lowmark)
>} Article lines processed 21807
>} Articles dropped 0
>} Overview index dropped 0
>}
>} Any way, has anyone else experienced this problem?
I've been seeing it for a few days:
buffindexed.c:1590 Can't malloc 589668352 bytes: Cannot allocate memory
It occurs when expireover (after two hours) encounters the newsgroup
"control". I figure, since alt.* expireover'd OK, I have a while before
I will desperately need to fix this. ;^)
>} Is it a (known) bug? Is there a patch? Or, does anyone have any
>} pointers so that I can go and persuade some one with copious free time
>} to fix it!
The code shows:
]
] STATIC BOOL ovgroupmmap(GROUPENTRY *ge, int low, int high, BOOL needov) {
]...
]
] for (i = 0 ; i < Gibcount ; i++) {
] if (Gib[i].artnum == 0)
] continue;
] ov.index = Gib[i].index;
] ov.blocknum = Gib[i].blocknum;
] gdb = searchgdb(&ov);
] if (gdb != NULL)
] continue;
] ovbuff = getovbuff(ov);
] if (ovbuff == NULL)
] continue;
] gdb = NEW(GROUPDATABLOCK, 1);
] gdb->datablk = ov;
] gdb->next = NULL;
] insertgdb(&ov, gdb);
] count++;
] }
] if (count == 0)
] return TRUE;
] Gdb = NEW(char, count * OV_BLOCKSIZE);
> Can you try CURRENT which avoids this problem?
> --
> Katsuhiro Kondou
The corresponding code in CURRENT is rather different:
news at mozart:~> diff buffindexed.c.*|less
1c1
< STATIC BOOL ovgroupmmap(GROUPENTRY *ge, int low, int high, BOOL needov) {
---
> static bool ovgroupmmap(GROUPENTRY *ge, int low, int high, bool needov) {
6c6
< OFFSET_T offset, mmapoffset;
---
> off_t offset, mmapoffset;
11,12c11,12
< Gibcount = 0;
< if (high - low < 0)
---
> if (high - low < 0) {
> Gibcount = 0;
14,18c14,15
< i = 0;
< if (high - low + 1 < ge->count)
< Gibcount = high - low + 1;
< else
< Gibcount = ge->count;
---
> }
> Gibcount = ge->count;
34c31
< if ((addr = mmap((caddr_t) 0, len, PROT_READ, MAP_SHARED, ovbuff->fd, mmap
offset)) == (MMAP_PTR) -1) {
---
> if ((addr = mmap((caddr_t) 0, len, PROT_READ, MAP_SHARED, ovbuff->fd, mmap
offset)) == MAP_FAILED) {
40,44d36
< if (low > ovblock->ovindexhead.high || high < ovblock->ovindexhead.low) {
< ov = ovblock->ovindexhead.next;
< munmap(addr, len);
< continue;
< }
51,56c43,45
< if (ovblock->ovindex[i].artnum >= low && ovblock->ovindex[i].artnum <= h
igh) {
< if (Gibcount == count) {
< Gibcount += OV_FUDGE;
< RENEW(Gib, OVINDEX, Gibcount);
< }
< Gib[count++] = ovblock->ovindex[i];
---
> if (Gibcount == count) {
> Gibcount += OV_FUDGE;
> RENEW(Gib, OVINDEX, Gibcount);
57a47
> Gib[count++] = ovblock->ovindex[i];
67c57
< qsort((POINTER)Gib, Gibcount, sizeof(OVINDEX), INDEXcompare);
---
> qsort(Gib, Gibcount, sizeof(OVINDEX), INDEXcompare);
79c69
< if (Gib[i].artnum == 0)
---
> if (Gib[i].artnum == 0 || Gib[i].artnum < low || Gib[i].artnum > high)
91a82
> gdb->mmapped = FALSE;
97,98c88,90
< Gdb = NEW(char, count * OV_BLOCKSIZE);
< count = 0;
---
> if (count * OV_BLOCKSIZE > innconf->keepmmappedthreshold * 1024)
> /* large retrieval, mmap is done in ovsearch() */
> return TRUE;
107c99
< if ((gdb->addr = mmap((caddr_t) 0, gdb->len, PROT_READ, MAP_SHARED, ovbu
ff->fd, mmapoffset)) == (MMAP_PTR) -1) {
---
> if ((gdb->addr = mmap((caddr_t) 0, gdb->len, PROT_READ, MAP_SHARED, ovbu
ff->fd, mmapoffset)) == MAP_FAILED) {
114,117c106
< memcpy(&Gdb[count * OV_BLOCKSIZE], gdb->data, OV_BLOCKSIZE);
< munmap(gdb->addr, gdb->len);
< gdb->data = &Gdb[count * OV_BLOCKSIZE];
< count++;
---
> gdb->mmapped = TRUE;
... and, with that note about "mmap is done in ovsearch", I rather doubt
that there's an easy substitution to be done.
Frankly, my problem just doesn't seem immediate enough to risk
switching to CURRENT. :^(
I suspect the correct solution is to put "control" in its own tiny
cyclic-buffer, too small to run into this problem. Barring that -- or
before that -- I've got to wonder if the symptoms would disappear if
I just munged the "control" lowmark to pretend earlier articles were
already expired...
Any opinions?
--
John Leslie <john at jlc.net>
More information about the inn-workers
mailing list