New makehistory(8)

Russ Allbery rra at
Fri Apr 28 11:03:43 UTC 2000

May as well send this out for general information and comment,
particularly since I missed the nightly snapshot.

=head1 NAME

makehistory - Initialize or rebuild INN history database


B<makehistory> [B<-abFIOux>] [B<-f> I<filename>] [B<-l> I<count>]
[B<-T> I<tmpdir>]


B<makehistory> rebuilds the history(5) text file, which contains a list of
message IDs of articles already seen by the server.  It can also be used
to rebuild the overview database.  Note that the dbz(3) indexes for the
history file are rebuilt by makedbz(8), not by B<makehistory> as in
earlier versions of INN.

The default name of the history text file is I<pathdb>/history; to specify
a different name, use the B<-f> flag.

By default, B<makehistory> will scan the entire spool, using the storage
manager, and write a history line for every article.  If you want to only
write history lines for articles not already in the history file, use
B<-u> (this only makes sense in combination with the B<-f> flag so that
you can then append the additional entries to your history file).  To also
generate overview information, use the B<-O> flag.

=head1 OPTIONS

=over 4

=item B<-a>

Append to the history file rather than generating a new one.  If you
append to the main history file, make sure innd(8) is throttled or not
running or you can corrupt the history.

=item B<-b>

Delete any messages found in the spool that do not have valid Message-ID
headers in them.

=item B<-f> I<filename>

Rather than writing directly to I<pathdb>/history, instead write to
I<filename>.  This option must be used if B<-u> is used.

=item B<-F>

Fork a separate process to flush overview data to disk rather than doing
it directly.  The advantage of this is that it allows B<makehistory> to
continue to collect more data from the spool while the first batch of data
is being written to the overview database.  The disadvantage is that up to
twice as much temporary disk space will be used for the generated overview
data.  This option only makes sense in combination with B<-O>.

=item B<-I>

Don't store overview data for articles numbered lower than the lowest
article number in active(5).  This is useful if there are old articles on
disk for whatever reason that shouldn't be available to readers or put
into the overview database.

=item B<-l> I<count>

This option specifies how many articles to process before writing the
accumulated overview information out to the overview database.  The
default is C<100000>.  A setting of C<0> won't flush overview data until
B<makehistory> has completely finished.  B<makehistory> will run faster
with larger settings, but note that the larger this setting the more
temporary space will be needed to store temporary overview data.  At a
rough estimate, about 300 * I<count> bytes of temporary space will be
required.  See the description of the B<-T> option for how to specify the
temporary storage location.

=item B<-O>

Create the overview database as well as the history file.  Overview
information is only required if the server supports readers; it is not
needed for a transit-only server.  If you are using the C<buffindexed>
overview storage method, erase all of your overview buffers before running
B<makehistory> with B<-O>.

=item B<-T> I<tmpdir>

If B<-O> is given, B<makehistory> needs a location to write temporary
overview data.  By default, it uses I<pathtmp>, set in inn.conf(5), but if
this option is given, the provided I<tmpdir> is used instead.  This is
also used for temporary files created by sort(1) (which is invoked in the
process of writing overview information since sorted overview information
writes faster).  By default, sort usually uses /var/tmp; see the man page
for sort on your system to be sure.

=item B<-u>

This option enables "update" mode.  In this mode, B<makehistory> will only
generate history (and overview if B<-O> is given) data for articles not
already found in the history file.  This can be useful after a system
crash, particularly in combination with B<-b> to delete any mangled
articles.  Note that B<-u> will not throttle innd(8) even if it's running,
and the B<-f> option is mandatory to specify the file to which the
additional history entries are written.  If B<-O> is given, overview
information will be updated as B<makehistory> is running; this is safe as
the overview storage methods lock against multiple updates.

The most useful thing to do with the new history entries, after
B<makehistory> finishes, is to throttle innd with ctlinnd(8), append the
history entries to the end of the history file, run makedbz(8) to rebuild
the indexes, and then unthrottle innd.

=item B<-x>

If this option is given, B<makehistory> won't write out history file
entries.  This is useful mostly for building overview without generating
a new history file.



Here's a typical example of rebuilding the entire history and overview
database, removing broken articles in the news spool.  This uses the
default temporary file locations and should be done while innd isn't
running or is throttled.

    makehistory -b -f history.n -O -l 300000 -I

This will rebuild the overview (if using C<buffindexed>, erase the
existing overview buffers before running this command) and leave a new
history file in history.n in I<pathdb>.  To preserve all of the history
entries from the old history file that correspond to rejected articles or
expired articles, use:

    cd /usr/local/news/db
    awk 'NF == 2 { print }' < history >> history.n

(replacing the path with your I<pathdb>, if it isn't the default).  Then
look over the new history file for problems and run:

    makedbz -s `wc -l < history` -f history.n

Then rename all of the files matching C<history.n.*> to C<history.*>,
replacing the current history database and indexes.  After that, it's safe
to unthrottle innd.

For a simpler example:

    makehistory -bu -f history.n -I -O

will scan the spool, removing broken articles and generating history and
overview entries for articles missing from history.  After running this,
you can throttle innd, append F<history.n> to F<history>, run B<makedbz>,
and unthrottle innd as described under B<-u>.

=head1 FILES

=over 4

=item inn.conf

Read for I<pathdb> and I<pathtmp>.

=item I<pathdb>/history

This and its associated indexes is used as the current history database if
the B<-u> option is given (to check found articles against to see if a
history entry should be generated).  This is also the default output file
for B<makehistory>.

=item I<pathtmp>

Where temporary files are written unless B<-T> is given.


=head1 HISTORY

Originally written by Rich $alz <rsalz at> for InterNetNews and
updated by various other people since.

$Id: makehistory.pod,v 1.1 2000/04/28 10:45:49 rra Exp $

=head1 SEE ALSO

dbz(3), active(5), history(5), inn.conf(5), ctlinnd(8), innd(8),


Russ Allbery (rra at             <>

More information about the inn-workers mailing list