Optimization for the expireover procedure.
kyb at online.ru
Wed Oct 17 07:00:29 UTC 2007
Bill Davidsen ?????:
> Julien ÉLIE wrote:
>> Hi Kirill,
>>> But in the case of
>>> expiry the main keys are time of the arrival and group number.
>> So expireover will not remove articles which no longer exist in
>> the news spool if it is based upon such information.
>> And articles stored in CNFS buffers will be expired from the overview
>> even though there are still available from the spool.
>> But well, these behaviours could be tweaked.
>>> For example we can use a list of pointers to overview
>>> records sorted according to arrival date or even expiration date ( this
>>> is a little bit tricky). I believe this will be much more faster.
>> I think that it is not the arrival date which matters but the Date: header
>> (correct me if I am wrong).
>> As for scheduled expiration date, it cannot be determined since expire.ctl
>> can change after the moment it is computed. And there is also the
>> Expires: header to take into account.
>> There is also expire to optimize (not only expireover).
> We discussed this long ago, making a list of articles to expire and the
> time at which that should happen. All based on the rules and policies in
> place at the moment the article arrives. This can save a great deal of
> time depending on how the information is stored for both articles and
> overview. And with CNFS stores it depends on policy, do you expire when
> and article wraps in the store (keep as long as possible) or when your
> policy says to remove it? Actually there's a very good reason to expire
> from policy, it keeps your policy from visibly changing under the users
> depending on article size and volume.
> But the most desirable change would be to spread the expire load over
> all uptime, so that the server avoids a load spike when expire runs.
This could be the best solution. Just select an old article using some
kind of expiration policy and overwrite it with a new one. However to
follow a some kind of policy we still need to run a cleanup after a while.
> that end the ability to set expire time at the moment the article
> arrives is a benefit, and frankly most sites don't change their policy
> all that often, and even if a recalculation of all the expire times were
> needed when the policy changed, it would still probably be a win.
I hope on it.
More information about the inn-workers