"expire" extremely slow

[Ernst Boetsch]

Hello *,


at the moment I am upgradeing our news server to a new host
and the INN version 2.4.3.  All is well (thanks for the new release
2.4.3) but  "expire"  is extremely slow:
"expire"  needs  ca. 1:45 h  for processing 2.391.821 history lines !

	expireover start Thu Apr 27 00:01:06 CEST 2006:
	( -z/local/news.log/expire.rm -Z/local/news.log/expire.lowmark)
	    Article lines processed  3059843
	    Articles dropped            4924
	    Overview index dropped      5251
	expireover end Thu Apr 27 00:16:47 CEST 2006
	lowmarkrenumber begin Thu Apr 27 00:16:47 CEST 2006:
	(/local/news.log/expire.lowmark)
	lowmarkrenumber end Thu Apr 27 00:16:47 CEST 2006
		expirerm start Thu Apr 27 00:16:47 CEST 2006
		expirerm end Thu Apr 27 00:17:18 CEST 2006
	expire begin Thu Apr 27 00:17:48 CEST 2006: (-v1)
	    Article lines processed  2391821
	    Articles retained        2385999
	    Entries expired             5822
	expire end Thu Apr 27 02:00:47 CEST 2006
		all done Thu Apr 27 02:00:47 CEST 2006


Environment:

 * Hardware:
    + Sun AMD/64
    + Storage:
	= Spool area (pathspool) is a dedicated area on a NAS filer
	  (appliance) which is mounted via NFS
	= The rest is on the local disc (no mirror, no RAID)

 * OS:  SUSE LINUX Enterprise Server 9 (x86_64)

 * Configuration of  INN-2.4.3:
    + Storage:	   tradspool for "newsgroups: *"
    + inn.conf:	   groupbaseexpiry = true
    + expire.ctl:  Format for groupbaseexpiry with wildmat lines


"strace ./expire -v1"  immediately shows the direct reason of the
slowness of "expire":  "expire"  performs a system call  "access(2)"
for  _every_  article in  "pathdb/history" !!!  Example:

   access("/local/news.spool/articles/comp/lang/c++/903692", R_OK) = 0


Subroutine  "EXPdoline(...)"  in  "inn-2.4.3/expire/expire.c"
explains this behaviour:

    if (innconf->groupbaseexpiry || SMprobe(SELFEXPIRE, token, NULL)) {
        if ((article = SMretrieve(*token, RETR_STAT)) == (ARTHANDLE *)NULL) {
            [ ... ]
        } else {
            [ ... ]
        }
    }

The  "SMretrieve(*token, RETR_STAT)"  leads to the  "access(2)".


Some days ago I had a test environment with only one (?) difference:
The spool area was also on the local disc.
Surprisingly  "expire"  finished within a few minutes:

	expire begin Mon Apr 24 00:22:51 CEST 2006: (-v1)
	    Article lines processed  2338203
	    Articles retained        2321951
	    Entries expired            16252
	expire end Mon Apr 24 00:32:46 CEST 2006


Is the slow behaviour of  "expire"  already known ?

Why does  "expire"  perform a  "access(2)"  for  _every_  article ?

What's to do  to prevent this behaviour ?


I could'nt find a hint in the man pages or in the FAQ.

I will try to implement a workaround in  "inn-2.4.3/expire/expire.c".
Unfortunately I'm not familiar with the internals of INN.
Therefore, any hint would be nice.


Kind regards,
    Ernst Boetsch

-- 
******************************************************************
Ernst N. Boetsch                     |  Leibniz Computing Centre
                                     |     of the Bavarian Academy
E-Mail:  boetsch _()_ lrz * de       |     of Sciences