crash whilst expiring the overviewdb

[John Leslie]

On Fri, 23 Feb 2001 19:20:12 +0900 (JST) 
Katsuhiro Kondou <kondou at ISC-Nospam.nec.co.jp> wrote:
> In article <5.0.2.1.0.20010222171753.05031410 at pop3.gradwell.net>,
> Peter Gradwell <peter at gradwell.com> wrote;
>} 
>} We get the following in the daily usenet report:
>} 
>} buffindexed.c:1590 Can't malloc 701472768 bytes: Cannot allocate
>} memoryExpire messages:
>} expireover start Wed Feb 14 03:08:03 GMT 2001: ( -z/var/log/news/expire.rm
>} -Z/var/log/news/expire.lowmark)
>} Article lines processed 21807
>} Articles dropped 0
>} Overview index dropped 0
>} 
>} Any way, has anyone else experienced this problem?

   I've been seeing it for a few days:

buffindexed.c:1590 Can't malloc 589668352 bytes: Cannot allocate memory

   It occurs when expireover (after two hours) encounters the newsgroup
"control". I figure, since alt.* expireover'd OK, I have a while before
I will desperately need to fix this. ;^)

>} Is it a (known) bug? Is there a patch? Or, does anyone have any
>} pointers so that I can go and persuade some one with copious free time
>} to fix it!

   The code shows:
] 
] STATIC BOOL ovgroupmmap(GROUPENTRY *ge, int low, int high, BOOL needov) {
]...
] 
] for (i = 0 ; i < Gibcount ; i++) {
]   if (Gib[i].artnum == 0)
]     continue;
]   ov.index = Gib[i].index;
]   ov.blocknum = Gib[i].blocknum;
]   gdb = searchgdb(&ov);
]   if (gdb != NULL)
]     continue;
]   ovbuff = getovbuff(ov);
]   if (ovbuff == NULL)
]     continue;
]   gdb = NEW(GROUPDATABLOCK, 1);
]   gdb->datablk = ov;
]   gdb->next = NULL;
]   insertgdb(&ov, gdb);
]   count++;
] }
] if (count == 0)   
]   return TRUE;
] Gdb = NEW(char, count * OV_BLOCKSIZE);

> Can you try CURRENT which avoids this problem?
> -- 
> Katsuhiro Kondou

   The corresponding code in CURRENT is rather different:

news at mozart:~> diff buffindexed.c.*|less
1c1
< STATIC BOOL ovgroupmmap(GROUPENTRY *ge, int low, int high, BOOL needov) {
---
> static bool ovgroupmmap(GROUPENTRY *ge, int low, int high, bool needov) {
6c6
<   OFFSET_T            offset, mmapoffset;
---
>   off_t                 offset, mmapoffset;
11,12c11,12
<   Gibcount = 0;
<   if (high - low < 0)
---
>   if (high - low < 0) {
>     Gibcount = 0;
14,18c14,15
<   i = 0;
<   if (high - low + 1 < ge->count)
<     Gibcount = high - low + 1;
<   else
<     Gibcount = ge->count;
---
>   }
>   Gibcount = ge->count;
34c31
<     if ((addr = mmap((caddr_t) 0, len, PROT_READ, MAP_SHARED, ovbuff->fd, mmap
offset)) == (MMAP_PTR) -1) {
---
>     if ((addr = mmap((caddr_t) 0, len, PROT_READ, MAP_SHARED, ovbuff->fd, mmap
offset)) == MAP_FAILED) {
40,44d36
<     if (low > ovblock->ovindexhead.high || high < ovblock->ovindexhead.low) {
<       ov = ovblock->ovindexhead.next;
<       munmap(addr, len);
<       continue;
<     }
51,56c43,45
<       if (ovblock->ovindex[i].artnum >= low && ovblock->ovindex[i].artnum <= h
igh) {
<       if (Gibcount == count) {
<         Gibcount += OV_FUDGE;
<         RENEW(Gib, OVINDEX, Gibcount);
<       }
<       Gib[count++] = ovblock->ovindex[i];
---
>       if (Gibcount == count) {
>       Gibcount += OV_FUDGE;
>       RENEW(Gib, OVINDEX, Gibcount);
57a47
>       Gib[count++] = ovblock->ovindex[i];
67c57
<   qsort((POINTER)Gib, Gibcount, sizeof(OVINDEX), INDEXcompare);
---
>   qsort(Gib, Gibcount, sizeof(OVINDEX), INDEXcompare);
79c69
<     if (Gib[i].artnum == 0)
---
>     if (Gib[i].artnum == 0 || Gib[i].artnum < low || Gib[i].artnum > high)
91a82
>     gdb->mmapped = FALSE;
97,98c88,90
<   Gdb = NEW(char, count * OV_BLOCKSIZE);
<   count = 0;
---
>   if (count * OV_BLOCKSIZE > innconf->keepmmappedthreshold * 1024)
>     /* large retrieval, mmap is done in ovsearch() */
>     return TRUE;
107c99
<       if ((gdb->addr = mmap((caddr_t) 0, gdb->len, PROT_READ, MAP_SHARED, ovbu
ff->fd, mmapoffset)) == (MMAP_PTR) -1) {
---
>       if ((gdb->addr = mmap((caddr_t) 0, gdb->len, PROT_READ, MAP_SHARED, ovbu
ff->fd, mmapoffset)) == MAP_FAILED) {
114,117c106
<       memcpy(&Gdb[count * OV_BLOCKSIZE], gdb->data, OV_BLOCKSIZE);
<       munmap(gdb->addr, gdb->len);
<       gdb->data = &Gdb[count * OV_BLOCKSIZE];
<       count++;
---
>       gdb->mmapped = TRUE;

... and, with that note about "mmap is done in ovsearch", I rather doubt
that there's an easy substitution to be done.

   Frankly, my problem just doesn't seem immediate enough to risk
switching to CURRENT. :^(

   I suspect the correct solution is to put "control" in its own tiny
cyclic-buffer, too small to run into this problem. Barring that -- or
before that -- I've got to wonder if the symptoms would disappear if
I just munged the "control" lowmark to pretend earlier articles were
already expired...

   Any opinions?

--
John Leslie <john at jlc.net>