Expireover running really slow

[Kirill Berezin]

Frankly speaking, I don't know. For servers under my administration 
upgrade or replacement  is not an easy  procedure, and I have to deal 
with corrections to the source mainly. I think that it is worth to make 
a tests before acquiring anything. It looks like a seek time for a disk 
plays a lot, and a placement of overview files also worth to check.
Expiryover looks through every group for articles to expire and it can 
take a lot of reads before a last article to be removed will be reached. 
Optimization for this procedure is a quite bit tricky. Meanwhile check 
your expiration policy, I have found a fast performance degrade 
extending of a artcutoff from 10 days to 30 . May be some intervals can 
be easily shortened?

For example, we have a two identical servers for readers, each has about 
12000000 overview records total, 2000000 for expiration every day, total 
volume is about 300GB per day. We use an ovdb expiration method. With 10 
days cutoff expiration ends after 8 hours, and with 30 days it ends 
after 12 hours.

kirill


Kristian Grønfeldt Sørensen wrote:
> Hi. 
>
> Thanks for the answer. As I'm already using buffindexed overview there's
> nothing more to do there.
> Do you think that buying a couple of 15K SCSI disks and putting them in
> RAID 1, to host the filesystem with the overview files would bring down
> expireover runtime to an acceptable (maybe 3-4 hours instead of 16)?
>
> Regards 
> Kristian Sørensen
>
> On Wed, 2008-01-09 at 09:41 +0300, Kirill Berezin wrote:
>   
>> *Hie.*
>> This is a real problem for running innd with large storages. Performance 
>> degrades because overview records are highly scattered over big area, 
>> ovdb does not help. An expiration for ovdb works much faster for first 
>> couple of weeks, but later it will be slower an slower. Fortunately a 
>> separate expiration index can help, but no such feature available and 
>> implementation requires a lot of changes for existing code. It seems a 
>> best solution for a moment is to use buffindexed method and to use as 
>> much as possible separate physical drives for overview storage.
>>
>> kirill
>>
>>
>> Kristian Grønfeldt Sørensen wrote:
>>     
>>> Hi.
>>>
>>> I've set up a new inn server as a replacement for our old one. The
>>> server is running inn 2.4.3-1 from Debian stable.
>>> My problem is that expireover is taking approximately 16 hours to
>>> finish. I find this quite strange as the new server is more powerful
>>> than the old one, which did not have this problem. 
>>>
>>> Since the new server has more disk space, there will be more articles
>>> stored in the spool, but the old server can finish expireover within an
>>> hour or so. 
>>>
>>> My setup is:
>>>
>>> article-spool and overview files is on two separate LVM logical volumes
>>> with XFS on top of them. All articles are stored in a total of 280 CNFS
>>> buffers. 
>>> Overview is buffindexed in a total of 82 files.
>>> The LVM volumes is hosted on a 4-disk RAID-5 array on an Intel SRCS28X
>>> sata controller. Benchmarking with bonnie++ on this device reveals read
>>> speeds of around 140 MB/s and write speeds of around 110 MB/s, so the
>>> underlying hardware should not be any problem.
>>>
>>> However during expireover the box spends ~60% if its time in IOWait
>>> mode, which indicates to me that the problem is related to the disk
>>> device.
>>>
>>> Using iostat, I can see that constant reading of approximately 1MB/s
>>> and, and almost no write requests. The LV hosting the article spool has
>>> almost no reads but around 1 MB/s but in a very bursty fashion - always
>>> 4 MB at a time. 
>>>
>>> Does any of you have any ideas for optimizing expireover performance for
>>> this system? I've tried playing around with readahead-policies and
>>> enabling/disabling write caching with no result.
>>>
>>> Regarde
>>>
>>> Kristian Sørensen
>>>   
>>>       
>>