Optimization for the expireover procedure.

[Bill Davidsen]

Julien ÉLIE wrote:
> Hi Kirill,
>
>
>   
>> But in the case of
>> expiry the main keys are time of the arrival and group number.
>>     
>
> So expireover will not remove articles which no longer exist in
> the news spool if it is based upon such information.
> And articles stored in CNFS buffers will be expired from the overview
> even though there are still available from the spool.
>
> But well, these behaviours could be tweaked.
>
>
>   
>> For example we can use a list of pointers to overview
>> records sorted according to arrival date or even expiration date ( this
>> is a little bit tricky). I believe this will be much more faster.
>>     
>
> I think that it is not the arrival date which matters but the Date: header
> (correct me if I am wrong).
> As for scheduled expiration date, it cannot be determined since expire.ctl
> can change after the moment it is computed.  And there is also the
> Expires: header to take into account.
>
>
> There is also expire to optimize (not only expireover).
>
>   
We discussed this long ago, making a list of articles to expire and the 
time at which that should happen. All based on the rules and policies in 
place at the moment the article arrives. This can save a great deal of 
time depending on how the information is stored for both articles and 
overview. And with CNFS stores it depends on policy, do you expire when 
and article wraps in the store (keep as long as possible) or when your 
policy says to remove it? Actually there's a very good reason to expire 
from policy, it keeps your policy from visibly changing under the users 
depending on article size and volume.

But the most desirable change would be to spread the expire load over 
all uptime, so that the server avoids a load spike when expire runs. To 
that end the ability to set expire time at the moment the article 
arrives is a benefit, and frankly most sites don't change their policy 
all that often, and even if a recalculation of all the expire times were 
needed when the policy changed, it would still probably be a win.

-- 
bill davidsen <davidsen at tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979