problems with tradindexed and buffindexed (was: expireover needs _very_ much memory)

[Heiko Schlichting]

[part 1 of this mail is for the buffindexed hackers, 
 part 2 for tradindexed henchmen]

Katsuhiro Kondou wrote:
> That's very weird.  buffindexed may sit in forever loop.
> Can you see how this come from core which can be created
> by SIGABRT.

I reproduced the problem with some debug code added to makehistory.c.
There are 739824 articles in CNFS spool and the temporary file is created
and sorted without problems. FlushOverTmpFile() works fine until the
373085th article and makehistory needs 10 MByte RAM. But the next article
never returns from OVadd(), the systems is doing mmap() and needs about
1 GByte RAM. But the entries in the temp file are not corrupt and does not
look unusual for me.

These are the last successful and the next:
373085  FlushOverTmpFile        @03044655454E4630380000029B7C00000001@
373086  FlushOverTmpFile        @030144524549303100000001475200000001@
 
939352233
  @03044655454E4630380000029B7C00000001@
  =?iso-2022-jp?B?W1dBTlRdRERJIHBvY2tldBskQiROTCQ3QExzQzxLdhsoQg==?=
  "Muke" <cak43600 at pop17.odn.ne.jp>
  Fri, 8 Oct 1999 12:12:31 +0900
  <7tjn7g$bi1$1 at newsgw7.odn.ne.jp>
  20
  Xref: fu-berlin.de fj.fleamarket.misc:43358
939352234
  @030144524549303100000001475200000001@
  cmsg cancel
  <7tda18$2f1$63 at nslave1.tin.it>
  Cosmo Roadkill <cosmo.roadkill%bofh.int at rauug.mil.wi.us>
  08 Oct 1999 03:02:58 GMT
  <cancel.7tda18$2f1$63 at nslave1.tin.it>
  9
  Xref: fu-berlin.de control.cancel:4667350

(Line breaks and spaces are for the mailing list only. The format in the
 file is correct in one line and with tabs)

The ABRT signal created a core but it was not possible to analyze it. Seems
that my dbx can't handle core files with a size of 1 GByte.

> One problem I can imagine is that there is a too old article
> in cycbuff and its article number and the latest article
> number is too far.  Buffindexed prepares every room for
> index of articles between them regardless of their existence.
> This may happen for control.cancel or junk.  But even in
> this case buffindexed never behaves like yours.

Ok, it might be a problem of mmap(). I attached a debugger (cvd) to the
looping process and it does not seem come back to user code. It is always
in stripped system routines (mmap, fcntl and time). 


After that I tried to go back to tradindexed but a makehistory -I -O -x -l 0
does not work also (I can't believe it). With a setting of overcachesize
to 1024(*) in inn.conf, makehistory stops after article 206728 and reports

	tradindexed: group cache is full, waiting 10 seconds

via syslog, sleeps 10 seconds, reports the problem again, sleeps and so on.
Looking into the source of OV3cleancache(), I can't see any reason why a
sleep(10) could help here in any case - it will always sleep forever just
waking up to contact syslog. But this is a side effect.
Using the debugger, I can see that there are 1024 cache entries in the array
of hashlists but no entry has a refcount of 0 (or less). The refcounts are
often in the range of 30 to 50 which looks unusual for me.

I didn't traced it in full detail until now, but in tradindexed_add() 
[storage/ov3/ov3.c] this code is part of the for-loop:

        if (Cutofflow && ge->low > artnum)
            continue;

This is done after OV3opengroup() and if the condition is true, the
corresponding OV3closegroup() call is never made. Or am I wrong?
Consequence of the missing OV3closegroup() are the high refcount settings.
If this happens as often as the setting of overcachesize, OV3cleancache()
fails with the 'group cache is full' message and sleeps forever in 10 second
intervals.

And makehistory -I sets the OVCUTOFFLOW control flag which is normally
unset (and might be the reason why others didn't noticed this problem
during normal opration of INN 2.3).

I'm not sure, if I'm right with this assumptions. Maybe someone who really
understand these tradindexed overview things can take a look at this?

Heiko

Footnote:
(*) "overcachesize: 1024" is working. The system has a per process file
    descriptor limit of 15000 and can handle this properly. Decreasing
    overcachesize to 128 breaks here after 26405 articles (instead of
    206728 articles with 1024). Setting this to the number of groups
    in active might help but needs the double of this value as
    file descriptors and is not an acceptable solution.

Heiko Schlichting       | Freie Universitaet Berlin
heiko at FU-Berlin.DE      | Zentraleinrichtung fuer Datenverarbeitung (ZEDAT)
Telefon +49 30 838-4327 | Fabeckstrasse 32
Telefax +49 30 838-6721 | D-14195 Berlin