INN 2.4.3 and news.daily errors with buffindexed overview storage

Mike Brudenell pmb1 at york.ac.uk
Tue Oct 17 16:57:40 UTC 2006


Greetings -

On 17 Oct 2006, at 17:23, Jeffrey M. Vinocur wrote:

> On Tue, 17 Oct 2006, Mike Brudenell wrote:
>
>>      /usr/local/news/bin/news.daily expireover lowmark
>
> Incidentally, I see that I'm using "news.daily expireover lowmark  
> delayrm"
> (the reason I added delayrm is lost in the mists of time).  I  
> suppose you
> could try that just for kicks and see if somehow it makes a  
> difference; it
> shouldn't hurt anything.

I seem to recall that 'delayrm' was to prevent expire unlinking  
article files stored in the traditional filestore layout and instead  
produce a list of the files to be deleted.  This was then fed to the  
fastrm program, which sorted the input stream and then performed the  
unlinks (to minimise re-reading of directories).

======================================================================

>> expire.ctl
>> ----------
>> /remember/:10
>> *:A:1:32:90
>> york:A:1:32:90
>> york.*:A:1:32:90
>> york.test:A:0.1:0.1:0.1
>
> None of this actually matters since you're not passing -N to  
> expire, but
> it's kinda silly to have expiration times substantially shorter  
> than the
> frequency with which you run news.daily (e.g. 24 hours in your case),
> since it means the user-visible behavior will change drastically  
> depending
> on the time of day when the post was made.

The short lifetime was deliberate... our users were turning york.test  
into a chat forum so I enforced a very short lifespan on posts.   
Basically anything posted before the last 0.1 of a day got  
unceremoniously deleted, with articles made in the last few hours of  
the day being allowed to stay on until news.daily is next run.

Once the current problems are sorted out I'll look at reintroducing  
the -N option, but didn't want to muddy things in the meantime.

======================================================================

>> Output from contrib/showtoken.in for some of the tokens
>> -------------------------------------------------------
>> echo @03016362303200000000001E5FA900000001@ | perl contrib/ 
>> showtoken.in
>> @03016362303200000000001E5FA900000001@ type=cnfs class=1 buffer=cb02
>> offset=3cbf5200 cycnum=1
>>
>> echo @03016362303200000000001E754800000001@ | perl contrib/ 
>> showtoken.in
>> @03016362303200000000001E754800000001@ type=cnfs class=1 buffer=cb02
>> offset=3cea9000 cycnum=1
>>
>> echo @03016362303200000000001E87EB00000001@ | perl contrib/ 
>> showtoken.in
>> @03016362303200000000001E87EB00000001@ type=cnfs class=1 buffer=cb02
>> offset=3d0fd600 cycnum=1
>>
>> echo @03016362303200000000001EE68900000001@ | perl contrib/ 
>> showtoken.in
>> @03016362303200000000001EE68900000001@ type=cnfs class=1 buffer=cb02
>> offset=3dcd1200 cycnum=1
>>
>> echo @03016362303200000000001F2D3C00000001@ | perl contrib/ 
>> showtoken.in
>> @03016362303200000000001F2D3C00000001@ type=cnfs class=1 buffer=cb02
>> offset=3e5a7800 cycnum=1
>
> Okay, so they are stored in cycbuffs as expected.  Can you use
> ~news/bin/sm to check on a few of the articles and find out if there's
> anything that would make them need to be expired sooner (e.g. an  
> Expires
> header), or anything else they have in common?

I checked all the above and none have an Expires header.  The posting  
dates are all very recent, and nothing else obvious might be causing  
it from the content of the message.

I'm running Cleanfeed but can't see any sign of the affected message  
ids in the news.notice log either, which is where it logs articles it  
rejects.



> Also, if there are any
> articles you don't mind deleting, you could try `sm -r` on one to  
> see if
> you get the same error message (about being read-only) as expire does.

I picked another token from the list and checked I could retrieve it  
with "sm" OK (I could).  Then I tried using sm to remove it (which  
appeared to work) and then retrieve it again (the article now  
couldn't be found):

% sm -r @0301636230330000000000022F8F00000001@
%
% sm @0301636230330000000000022F8F00000001@
sm: could not retrieve @0301636230330000000000022F8F00000001@

So I *can* remove articles from the cycbuffs by hand but, it seems,  
news.daily can't.  Weird!

A thought:
Is it worth me running news.daily (with a "norotate") by hand to see  
if that works or complains?


>> Does this imply the other cycbuffs aren't being used?
>
> Oh, you put the SEQUENTIAL flag in your cycbuff.conf, so INN is  
> using the
> cycbuffs one at a time.  (That's a sensible thing to do, unless your
> server is under a lot of load and you think distributing the disk I/O
> would help.)  FYI, ~news/bin/cnfsstat is useful for checking on the  
> in-use
> status of your cycbuffs; it gives you a little more detail than ls -l.

Ah-ha, I see.  That makes sense as the cycbuffs are stored on a  
single disk (possibly a pair mirrored: I can't remember!):

======================================================================

% cnfsstat
Class YORK  for groups matching "york,york.*"
Buffer cb00, size:  1.00 GBytes, position:   318 kBytes  0.00 cycles
   Newest: 2006-10-17  0:00:02,    0 days, 17:53:21 ago

Class DEFAULT  for groups matching "*"
Buffer cb01, size:  1.00 GBytes, position:   272 kBytes  1.00 cycles
   Newest: 2006-10-09 23:30:26,    7 days, 18:22:57 ago
Buffer cb02, size:  1.00 GBytes, position:   272 kBytes  1.00 cycles
   Newest: 2006-10-13 21:31:24,    3 days, 20:21:59 ago
Buffer cb03, size:  1.00 GBytes, position:   905 MBytes  0.88 cycles
   Newest: 2006-10-17 17:53:03,    0 days,  0:00:20 ago
Buffer cb04, size:  1.00 GBytes, position:   272 kBytes  0.00 cycles
   Newest: 2006-10-05 16:38:14,   12 days,  1:15:09 ago
Buffer cb05, size:  1.00 GBytes, position:   272 kBytes  0.00 cycles
   Newest: 2006-10-05 16:38:14,   12 days,  1:15:09 ago
Buffer cb06, size:  1.00 GBytes, position:   272 kBytes  0.00 cycles
   Newest: 2006-10-05 16:38:14,   12 days,  1:15:09 ago
Buffer cb07, size:  1.00 GBytes, position:   272 kBytes  0.00 cycles
   Newest: 2006-10-05 16:38:14,   12 days,  1:15:09 ago
Buffer cb08, size:  1.00 GBytes, position:   272 kBytes  0.00 cycles
   Newest: 2006-10-05 16:38:14,   12 days,  1:15:09 ago
Buffer cb09, size:  1.00 GBytes, position:   272 kBytes  0.00 cycles
   Newest: 2006-10-05 16:38:14,   12 days,  1:15:09 ago
Buffer cb10, size:  1.00 GBytes, position:   272 kBytes  0.00 cycles
   Newest: 2006-10-05 16:38:14,   12 days,  1:15:09 ago

What a lot of useful utilities there seem to be lurking in the depths  
of INN nowadays!  (Shows how long it is since I had to do serious  
work on an installation. :-)


Cheers,
Mike B-)

-- 
The Computing Service, University of York, Heslington, York Yo10 5DD, UK
Tel:+44-1904-433811  FAX:+44-1904-433740

* Unsolicited commercial e-mail is NOT welcome at this e-mail address. *




More information about the inn-workers mailing list