Expireover running really slow

[Kristian Grønfeldt Sørensen]

Hi. Expiration does not happen after fixed time periods as I use CNFS
exclusively for article storage.

The server currently holds around 31000000 articles and my guess is that
maybe 5000000 expires every day.

Regards

Kristian

On Wed, 2008-01-09 at 14:22 +0300, Kirill Berezin wrote: 
> Frankly speaking, I don't know. For servers under my administration 
> upgrade or replacement  is not an easy  procedure, and I have to deal 
> with corrections to the source mainly. I think that it is worth to make 
> a tests before acquiring anything. It looks like a seek time for a disk 
> plays a lot, and a placement of overview files also worth to check.
> Expiryover looks through every group for articles to expire and it can 
> take a lot of reads before a last article to be removed will be reached. 
> Optimization for this procedure is a quite bit tricky. Meanwhile check 
> your expiration policy, I have found a fast performance degrade 
> extending of a artcutoff from 10 days to 30 . May be some intervals can 
> be easily shortened?
> 
> For example, we have a two identical servers for readers, each has about 
> 12000000 overview records total, 2000000 for expiration every day, total 
> volume is about 300GB per day. We use an ovdb expiration method. With 10 
> days cutoff expiration ends after 8 hours, and with 30 days it ends 
> after 12 hours.
> 
> kirill
> 
> 
> Kristian Grønfeldt Sørensen wrote:
> > Hi. 
> >
> > Thanks for the answer. As I'm already using buffindexed overview there's
> > nothing more to do there.
> > Do you think that buying a couple of 15K SCSI disks and putting them in
> > RAID 1, to host the filesystem with the overview files would bring down
> > expireover runtime to an acceptable (maybe 3-4 hours instead of 16)?
> >
> > Regards 
> > Kristian Sørensen
> >
> > On Wed, 2008-01-09 at 09:41 +0300, Kirill Berezin wrote:
> >   
> >> *Hie.*
> >> This is a real problem for running innd with large storages. Performance 
> >> degrades because overview records are highly scattered over big area, 
> >> ovdb does not help. An expiration for ovdb works much faster for first 
> >> couple of weeks, but later it will be slower an slower. Fortunately a 
> >> separate expiration index can help, but no such feature available and 
> >> implementation requires a lot of changes for existing code. It seems a 
> >> best solution for a moment is to use buffindexed method and to use as 
> >> much as possible separate physical drives for overview storage.
> >>
> >> kirill
> >>
> >>
> >> Kristian Grønfeldt Sørensen wrote:
> >>     
> >>> Hi.
> >>>
> >>> I've set up a new inn server as a replacement for our old one. The
> >>> server is running inn 2.4.3-1 from Debian stable.
> >>> My problem is that expireover is taking approximately 16 hours to
> >>> finish. I find this quite strange as the new server is more powerful
> >>> than the old one, which did not have this problem. 
> >>>
> >>> Since the new server has more disk space, there will be more articles
> >>> stored in the spool, but the old server can finish expireover within an
> >>> hour or so. 
> >>>
> >>> My setup is:
> >>>
> >>> article-spool and overview files is on two separate LVM logical volumes
> >>> with XFS on top of them. All articles are stored in a total of 280 CNFS
> >>> buffers. 
> >>> Overview is buffindexed in a total of 82 files.
> >>> The LVM volumes is hosted on a 4-disk RAID-5 array on an Intel SRCS28X
> >>> sata controller. Benchmarking with bonnie++ on this device reveals read
> >>> speeds of around 140 MB/s and write speeds of around 110 MB/s, so the
> >>> underlying hardware should not be any problem.
> >>>
> >>> However during expireover the box spends ~60% if its time in IOWait
> >>> mode, which indicates to me that the problem is related to the disk
> >>> device.
> >>>
> >>> Using iostat, I can see that constant reading of approximately 1MB/s
> >>> and, and almost no write requests. The LV hosting the article spool has
> >>> almost no reads but around 1 MB/s but in a very bursty fashion - always
> >>> 4 MB at a time. 
> >>>
> >>> Does any of you have any ideas for optimizing expireover performance for
> >>> this system? I've tried playing around with readahead-policies and
> >>> enabling/disabling write caching with no result.
> >>>
> >>> Regarde
> >>>
> >>> Kristian Sørensen
> >>>   
> >>>       
> >>     
> 
> 
>