how does history file get too large?

[bill davidsen]

In article <401712E7.11A42445 at unidata.ucar.edu>,
Anne Wilson  <anne at unidata.ucar.edu> wrote:
| 
| Thanks, Mark!  I'll try setting the /remember/:n to something smaller.  Also, the -c option to innd may be useful.  
| 
| What I'd like to do is to keep history information as long as I can without getting this error.  (That way, if a duplicate shows up after some awful long time, it would still be rejected.  Although, the likelihood of that is probably pretty small.)
| 
| If there was something that would just check the size of the file and warn me if it's getting too big, then I could deal with it before it becomes a problem.  news.daily?  expire?  Or, maybe that happened and I missed it?  Just seems like there should be
| a better way to deal with this problem besides the crash and burn approach.  (The only error message was "ME source lost . Exiting")
| 
| I'm imagining a utility that would scan the timestamps in the history file and get the oldest one.  And/or you could give it a timestamp and have it delete anything older than that time.    
| 
| If I was to write such a script, could I just delete expired entries from the history file, remake the indexes and overview, and have INN run successfully?
| 
| I *am* one of the high volume sites - we relay and recieve only scientific data.  

What are you running for version and o/s which doesn't support large
files? That's the real problem here, most o/s have allowed large files
for some years.

Warning, you first want to be sure that you don't delete entries for
anything which is still in spool. That will *really* bite you over time.

Given that, it's pretty simple to write a program, let's call it a perl
script because the source is very easy, and do thus:

- calculate what you need to delete

  if you're growing history by 400MB/day, and you want a sane safty net,
pick a target size like 1.4GB (max - 1.5days). If the history file is
smaller than that you're done.

- pause inn

  inn won't like what you're about to do.

- calculate how much you need to remove

  remove = current - target

- run a program to start dropping history lines

  Drop lines for articles not on-line until the target is met. Copy
everything else to another history file (history.NEW).

- rename the files and run makedbz

  history => history.OLD
  history.NEW => history
  makedbz

You're done.

================ I made this up, untested ================

#!/usr/bin/perl

$DropGoal = shift @ARGV;

while (<>) {
  if (m/@/) {
    print $_;
  } else {
    $dropped += length($_);
    last if $dropped >= $DropGoal;
  }
}

# copy the rest w/o checks
print $_ while <>;
exit 0;
================================================================

  Do remember there are other ways to do it, setting remember is
probably better, but you DID say you wanted to check and recover...
-- 
bill davidsen <davidsen at tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.