optimizing overview expiry (or: overview without expiry)

[Harald Welte]

On Mon, Feb 04, 2002 at 12:32:27PM +0900, Katsuhiro Kondou wrote:

> } b) Include a pointer to the overview information 'ovtoken' into 
> }    CNFSARTICLEHEADER.  This way, at the time cnfs overwrites an article,
> }    it could directly call into the overview code and tell it to remove
> }    information about this article.
> 
> You mean CNFSARTHEADER will be variable length, since
> I think the token needs to include all crossposted
> newsgroups?  And innd needs to read the header before
> writing new article.  I'm not sure how the perfomance
> gets down here.

ouch. I didn't think about crosspostings. variable size CNFSARTHEADER is
of course no option :(  I need to re-think this.  One way would be to have
a fixed size array (thus making the number of possible crossposts a compile-
time default).  This is an ugly hack and sucks. 

What if the overview would contain a linked list about all crossposted
overview entries of one article?  This is fixed-size since it just adds
a linked list header with two pointers to every overview entry.

> }   member, and call into the overview code, which removes overview info for
> }   this article
> 
> I think it's necessary to consider how overview data
> will be removed at the same time.  Just marked as
> expired in index?  Or really delete the data?  The
> former should be fast, but you still need to purge
> old data by expireover.  The latter, I believe, it'll
> be much slower to receiving ariticle.

I was thinking about the two options as well.  just marking it as deleted
and then running some 'defragment' as a cronjob is one idea.

Even better would be if the overview block bitmap indicated if a block is
used or not.  Then once we write a new overview entry, we can look in the
bitmap to find a sufficient number of 'deleted' (i.e. free) overview blocks
and re-use them.  This way there would no 'collapsing' be needed anymore.

> } What do you expect with regard to performance?  Would people be interested
> } if I'd implement this?
> 
> It's interesting, but how do you think of storage methods
> other than cnfs?

To be honest:  I wouldn't care too much.  We could make this new overview method
bount to cnfs, not to be used with any other storage methods.  It's not very
clean, I know...

> Katsuhiro Kondou

-- 
Live long and prosper
- Harald Welte / laforge at gnumonks.org               http://www.gnumonks.org/
============================================================================
GCS/E/IT d- s-: a-- C+++ UL++++$ P+++ L++++$ E--- W- N++ o? K- w--- O- M+ 
V-- PS++ PE-- Y++ PGP++ t+ 5-- !X !R tv-- b+++ !DI !D G+ e* h--- r++ y+(*)