expire question

[inn at hermes.ES.bss.boeing.com]

Hi all,

Our INN server runs on a dedicated Sun t1 server running Solaris 8 and
we are using a qtree on a Network Appliance filer to store the news spool.
Everything was working great under 2.2 but we needed the added flexibility
offered by the 2.3.x readers.conf file.  So I recently upgraded from INN 2.2 
to 2.3.2 over a month ago.  It was compiled on the server with gcc 2.95.2.
The history, overview, etc. files didn't like the change much so I did a full 
reset and deleted all the articles in the spool (no users were pointed to 
the server at that point so it wasn't a big deal), the history file, and the 
overview DBs.  Since the upgrade I've have an odd problem.

I don't want to flood everyone with a extra data, so I'll try to extract
what might be relevant (this technique always seems to fail because if
if the asker knew everything that was relevant there wouldn't be a problem,
but here goes...).  INN is configured for:
	From inn.conf:

		ovmethod:               tradindexed
		enableoverview:         true
		useoverchan:            false

	From storage.conf:

		method tradspool {
			newsgroups: *
			class: 1
		}

	From expire.ctl
		*:A:1:1:1
		alt.*:A:1:8:8

There is one newsgroup in alt that is pretty chatty so it's a good one to
use to monitor this problem (correction, there are probably a lot of
newsgroups in alt that fit that description, but I digress).  I spot 
checked the articles to make sure they were valid and it doesn't look like
it is being flooded with bogus articles.

The problem is that expire only expires a subset of articles we want it to.
This results in a news spool that slowly fills up.  That made me think that 
the history file had a problem and expire couldn't find the articles.  I saw
articles from the day after the 2.3.2 initial start-up (i.e. some of the
first articles received).  This implied expire didn't know about the articles
so I rebuilt the history file (this took about 18 hours, partial expire => big 
spool):

	ctlinnd throttle maintenance
	makehistory -b -f history.n -O -l 30000
	makedbz -s `wc -l < history` -f history.n
	mv history.n history
	mv history.n.dir history.dir
	mv history.n.hash history.hash
	mv history.n.index history.index
	ctlinnd go maintenance

The next news.daily deleted some articles but not all the articles I wanted
it to, and since then it still doesn't delete enough to keep the spool from 
growing.  Each expire deletes some articles but not all and there are still
articles going all the way back to the start-up date a month ago.

The test newsgroup has 7058 files in it's directory.  My news reader can see
3766 at the summary level.  When I try to see the old articles it only goes
back 8 days (which is what is set in expire).  The files in expire.list are
no longer in the spool (but the list is too short and it is under a MB in 
size).

I went through articles on the email list archive, FAQ and the newsgroup
and I saw similar problems a couple times, but none of the solutions seem
to fit how our configuration would cause these symptoms.

By Katsuhiro:
> I suspect overview was not created for all articles.
> This could be possible if you run overchan and it
> cannot keep up with innd.
>  
> You can manually delete hidden articles which I mean
> you cannot see thru news reader but find in spool.

By Katsuhiro:
> Can you read those articles thru nnrpd?  I suspect not.
> It tells those are not listed in overview data, if so.
> Then you need to recreate overview data by makehistory.

(I tried makehistory as described above).

By Tim:
> I've figured out the problem, or at least fixed the problem. Your remark
> about not having large files enabled reminded me to check how Berkeley DB
> 3.2.9 was compiled. I've originally compiled BDB 3.29 on a Solaris 2.5.1
> system without large file support. I've recompiled BDB 3.2.9 under Solaris
> 8 with large file support and installed it in a special location just for
> INN as most of our Sun servers are not running Solaris 8. I also recompiled
> inn-2.3.2 with large file support and linked it against the newly compiled
> BDB 3.2.9. I deleted the overview database and history files, recreated
> both with makehistory and makedbz, and fired up inn via rc.news. Voila! No
> problems, and she's running great so far.
> 
> Tim

By Miquel:
> Just remove them, using something like
> find /var/spool/news -type f -mtime +90 | xargs rm

The last one (using variations with +9, etc.) would fix the immediate problem 
but I suspect the problem would return as expire failed to delete all the
articles we want it to.

This seems (or seemed) like something I should be able to figure out, but
I'm at a point where diminishing returns on more investigation and running
out of spool space prod me to ask the net to make sure this isn't something
obvious to the experts.

Any hints or suggestions would be appreciated!

Thanks!

-Dave