how does history file get too large?
bill davidsen
davidsen at tmr.com
Wed Jan 28 17:20:29 UTC 2004
In article <401712E7.11A42445 at unidata.ucar.edu>,
Anne Wilson <anne at unidata.ucar.edu> wrote:
|
| Thanks, Mark! I'll try setting the /remember/:n to something smaller. Also, the -c option to innd may be useful.
|
| What I'd like to do is to keep history information as long as I can without getting this error. (That way, if a duplicate shows up after some awful long time, it would still be rejected. Although, the likelihood of that is probably pretty small.)
|
| If there was something that would just check the size of the file and warn me if it's getting too big, then I could deal with it before it becomes a problem. news.daily? expire? Or, maybe that happened and I missed it? Just seems like there should be
| a better way to deal with this problem besides the crash and burn approach. (The only error message was "ME source lost . Exiting")
|
| I'm imagining a utility that would scan the timestamps in the history file and get the oldest one. And/or you could give it a timestamp and have it delete anything older than that time.
|
| If I was to write such a script, could I just delete expired entries from the history file, remake the indexes and overview, and have INN run successfully?
|
| I *am* one of the high volume sites - we relay and recieve only scientific data.
What are you running for version and o/s which doesn't support large
files? That's the real problem here, most o/s have allowed large files
for some years.
Warning, you first want to be sure that you don't delete entries for
anything which is still in spool. That will *really* bite you over time.
Given that, it's pretty simple to write a program, let's call it a perl
script because the source is very easy, and do thus:
- calculate what you need to delete
if you're growing history by 400MB/day, and you want a sane safty net,
pick a target size like 1.4GB (max - 1.5days). If the history file is
smaller than that you're done.
- pause inn
inn won't like what you're about to do.
- calculate how much you need to remove
remove = current - target
- run a program to start dropping history lines
Drop lines for articles not on-line until the target is met. Copy
everything else to another history file (history.NEW).
- rename the files and run makedbz
history => history.OLD
history.NEW => history
makedbz
You're done.
================ I made this up, untested ================
#!/usr/bin/perl
$DropGoal = shift @ARGV;
while (<>) {
if (m/@/) {
print $_;
} else {
$dropped += length($_);
last if $dropped >= $DropGoal;
}
}
# copy the rest w/o checks
print $_ while <>;
exit 0;
================================================================
Do remember there are other ways to do it, setting remember is
probably better, but you DID say you wanted to check and recover...
--
bill davidsen <davidsen at tmr.com>
CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
More information about the inn-workers
mailing list