Optimization for the expireover procedure.

Kirill Berezin kyb at online.ru
Wed Oct 17 07:00:29 UTC 2007


Bill Davidsen ?????:
> Julien ÉLIE wrote:
>   
>> Hi Kirill,
>>
>>
>>   
>>     
>>> But in the case of
>>> expiry the main keys are time of the arrival and group number.
>>>     
>>>       
>> So expireover will not remove articles which no longer exist in
>> the news spool if it is based upon such information.
>> And articles stored in CNFS buffers will be expired from the overview
>> even though there are still available from the spool.
>>
>> But well, these behaviours could be tweaked.
>>
>>
>>   
>>     
>>> For example we can use a list of pointers to overview
>>> records sorted according to arrival date or even expiration date ( this
>>> is a little bit tricky). I believe this will be much more faster.
>>>     
>>>       
>> I think that it is not the arrival date which matters but the Date: header
>> (correct me if I am wrong).
>> As for scheduled expiration date, it cannot be determined since expire.ctl
>> can change after the moment it is computed.  And there is also the
>> Expires: header to take into account.
>>
>>
>> There is also expire to optimize (not only expireover).
>>
>>   
>>     
> We discussed this long ago, making a list of articles to expire and the 
> time at which that should happen. All based on the rules and policies in 
> place at the moment the article arrives. This can save a great deal of 
> time depending on how the information is stored for both articles and 
> overview. And with CNFS stores it depends on policy, do you expire when 
> and article wraps in the store (keep as long as possible) or when your 
> policy says to remove it? Actually there's a very good reason to expire 
> from policy, it keeps your policy from visibly changing under the users 
> depending on article size and volume.
>   
Good point.
> But the most desirable change would be to spread the expire load over 
> all uptime, so that the server avoids a load spike when expire runs. 
This could be the best solution. Just select an old article using some 
kind of expiration policy and overwrite it with a new one. However to 
follow a some kind of policy we still need to run a cleanup after a while.
> To 
> that end the ability to set expire time at the moment the article 
> arrives is a benefit, and frankly most sites don't change their policy 
> all that often, and even if a recalculation of all the expire times were 
> needed when the policy changed, it would still probably be a win.
>   
I hope on it.


More information about the inn-workers mailing list