History File over 2GB

[bill davidsen]

In article <3D747BCE.2040705 at meowing.net>,
greg andruk  <gja at meowing.net> wrote:
| 
| bill davidsen wrote:
| 
| > | The rollover stuff might be better noticed and flagged by readers than innd.
| > 
| > The problem being that readers are not all smart.
| 
| Oh, I think we're on the same page then.  I mean INN readers (primarily 
| nnrpd) might notice the situation, and inform the overview scribbler 
| (inn or overchan), or even make the change itself.  We've had so many 
| flavors of overview in the past few years that I've really lost track of 
| who can safely change stuff.

Okay, that could be done. I envisioned having any delete queue a request
to the overview system (treated as another black box) which would de
whatever is needed for any given overview implementation.

| In the words of the esteemed philospher Ellen Feiss, it would be.. 
| like... a bummer... if you're the first reader to run across the 
| rollover, but it should average out.

If the queue write for the deleter was fast it might not matter. That's
another problem, you need to do things in the right order. Before you
delete the article from history you need to go get the group and article
information you need to delete the overview entries. That seems to be
only in the article right now, there's no Xref database for delete which
would let you just queue the msgid and get the actual overview entries
later.
 
| Well... one of the nice things about the fixed record size we now seem 
| to have is that updates are almost practical to do, including deletes. 
| What I don't particularly care for, though, is the idea that innd would 
| be performing random access on the history text. Pushing history work 
| into a thread or three would be attractive, but it's not really clear if 
| all the various Unixoid systems really have their threading acts 
| together yet. It's certainly looks better than it did a couple years 
| ago, but...

If we assume a bit of locking and mmap we need not use pthreads,
although they seem to work in Linux, Solaris and BSD, which might be
enough. However, passing information between processes can be done, we
do it for overchan now. No reason why deletes can't be done the same
way, hand off the msgid and some flags and let the garbage collector
handle it as needed. And the GC could prioritize between "this article
HAS gone away" like cycbuff rollover, and "please MAKE this article go
away" like cancel.

We have occasionally mentioned running multiple copies of overchan, here
and in groups, a few threaded overview deletors wouldn't be a bad thing
either, so deletes wouldn't be serialized on disk i/o.

| > As you might guess I have problems with users complaining when an
| > article is in overview and not online, or when LISTGROUP and XOVER don't
| > agree (and sometimes neighter is correct).
| 
| Yeah, but checking should be quite a bit cheaper than it was in the 2.2 
| days. I suspect that a lot of gain can be had from small tweaks to 
| what's already in there; again, I haven't got too deep into the current 
| overview k0deZ, but overall they look pretty nice.

-- 
bill davidsen <davidsen at tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.