expire question

Bernhard Kohl B.Kohl at rrz.uni-koeln.de
Mon Jul 2 07:22:47 UTC 2001


Mitteilung vom Freitag, 29. Juni 2001, 22:11:52:

It may be due to a broken "sort -u" command. I've had a similar
problem some months ago and with a recent patch from Sun the problem
went away.

ihEbbc> To: undisclosed-recipients:;
ihEbbc> Sender: inn-workers-bounce at isc.org
ihEbbc> Errors-to: inn-workers-bounce at isc.org
ihEbbc> X-original-sender: inn at hermes.ES.bss.boeing.com
ihEbbc> Precedence: bulk
ihEbbc> List-unsubscribe: <mailto:inn-workers-request at isc.org?Subject=unsubscribe>
ihEbbc> X-List-ID: <inn-workers.isc.org>

ihEbbc> Hi all,

ihEbbc> Our INN server runs on a dedicated Sun t1 server running Solaris 8 and
ihEbbc> we are using a qtree on a Network Appliance filer to store the news spool.
ihEbbc> Everything was working great under 2.2 but we needed the added flexibility
ihEbbc> offered by the 2.3.x readers.conf file.  So I recently upgraded from INN 2.2 
ihEbbc> to 2.3.2 over a month ago.  It was compiled on the server with gcc 2.95.2.
ihEbbc> The history, overview, etc. files didn't like the change much so I did a full 
ihEbbc> reset and deleted all the articles in the spool (no users were pointed to 
ihEbbc> the server at that point so it wasn't a big deal), the history file, and the 
ihEbbc> overview DBs.  Since the upgrade I've have an odd problem.

ihEbbc> I don't want to flood everyone with a extra data, so I'll try to extract
ihEbbc> what might be relevant (this technique always seems to fail because if
ihEbbc> if the asker knew everything that was relevant there wouldn't be a problem,
ihEbbc> but here goes...).  INN is configured for:
ihEbbc>         From inn.conf:

ihEbbc>                 ovmethod:               tradindexed
ihEbbc>                 enableoverview:         true
ihEbbc>                 useoverchan:            false

ihEbbc>         From storage.conf:

ihEbbc>                 method tradspool {
ihEbbc>                         newsgroups: *
ihEbbc>                         class: 1
ihEbbc>                 }

ihEbbc>         From expire.ctl
ihEbbc>                 *:A:1:1:1
ihEbbc>                 alt.*:A:1:8:8

ihEbbc> There is one newsgroup in alt that is pretty chatty so it's a good one to
ihEbbc> use to monitor this problem (correction, there are probably a lot of
ihEbbc> newsgroups in alt that fit that description, but I digress).  I spot 
ihEbbc> checked the articles to make sure they were valid and it doesn't look like
ihEbbc> it is being flooded with bogus articles.

ihEbbc> The problem is that expire only expires a subset of articles we want it to.
ihEbbc> This results in a news spool that slowly fills up.  That made me think that 
ihEbbc> the history file had a problem and expire couldn't find the articles.  I saw
ihEbbc> articles from the day after the 2.3.2 initial start-up (i.e. some of the
ihEbbc> first articles received).  This implied expire didn't know about the articles
ihEbbc> so I rebuilt the history file (this took about 18 hours, partial expire => big 
ihEbbc> spool):

ihEbbc>         ctlinnd throttle maintenance
ihEbbc>         makehistory -b -f history.n -O -l 30000
ihEbbc>         makedbz -s `wc -l < history` -f history.n
ihEbbc>         mv history.n history
ihEbbc>         mv history.n.dir history.dir
ihEbbc>         mv history.n.hash history.hash
ihEbbc>         mv history.n.index history.index
ihEbbc>         ctlinnd go maintenance

ihEbbc> The next news.daily deleted some articles but not all the articles I wanted
ihEbbc> it to, and since then it still doesn't delete enough to keep the spool from 
ihEbbc> growing.  Each expire deletes some articles but not all and there are still
ihEbbc> articles going all the way back to the start-up date a month ago.

ihEbbc> The test newsgroup has 7058 files in it's directory.  My news reader can see
ihEbbc> 3766 at the summary level.  When I try to see the old articles it only goes
ihEbbc> back 8 days (which is what is set in expire).  The files in expire.list are
ihEbbc> no longer in the spool (but the list is too short and it is under a MB in 
ihEbbc> size).

ihEbbc> I went through articles on the email list archive, FAQ and the newsgroup
ihEbbc> and I saw similar problems a couple times, but none of the solutions seem
ihEbbc> to fit how our configuration would cause these symptoms.

ihEbbc> By Katsuhiro:
>> I suspect overview was not created for all articles.
>> This could be possible if you run overchan and it
>> cannot keep up with innd.
>> You can manually delete hidden articles which I mean
>> you cannot see thru news reader but find in spool.

ihEbbc> By Katsuhiro:
>> Can you read those articles thru nnrpd?  I suspect not.
>> It tells those are not listed in overview data, if so.
>> Then you need to recreate overview data by makehistory.

ihEbbc> (I tried makehistory as described above).

ihEbbc> By Tim:
>> I've figured out the problem, or at least fixed the problem. Your remark
>> about not having large files enabled reminded me to check how Berkeley DB
>> 3.2.9 was compiled. I've originally compiled BDB 3.29 on a Solaris 2.5.1
>> system without large file support. I've recompiled BDB 3.2.9 under Solaris
>> 8 with large file support and installed it in a special location just for
>> INN as most of our Sun servers are not running Solaris 8. I also recompiled
>> inn-2.3.2 with large file support and linked it against the newly compiled
>> BDB 3.2.9. I deleted the overview database and history files, recreated
>> both with makehistory and makedbz, and fired up inn via rc.news. Voila! No
>> problems, and she's running great so far.
>> Tim

ihEbbc> By Miquel:
>> Just remove them, using something like
>> find /var/spool/news -type f -mtime +90 | xargs rm

ihEbbc> The last one (using variations with +9, etc.) would fix the immediate problem 
ihEbbc> but I suspect the problem would return as expire failed to delete all the
ihEbbc> articles we want it to.

ihEbbc> This seems (or seemed) like something I should be able to figure out, but
ihEbbc> I'm at a point where diminishing returns on more investigation and running
ihEbbc> out of spool space prod me to ask the net to make sure this isn't something
ihEbbc> obvious to the experts.

ihEbbc> Any hints or suggestions would be appreciated!

ihEbbc> Thanks!

ihEbbc> -Dave

Mit freundlichen Grüßen,
Bernhard Kohl                            mailto:B.Kohl at rrz.uni-koeln.de

More information about the inn-workers mailing list