Overview methods and rebuild documentation

Wed Dec 23 19:19:14 UTC 2020

Hi all,

We'll need instructions to help people migrate to ovsqlite.  Here is a 
suggestion of wording for our documentation.
Bo's implementation of the first new overview storage method since INN 
2.3.0 is a major milestone that needs care!

Please tell any additions or wording changes you find appropriate.


Also, we currently compare the 4 overview storage methods in 3 
documents:  FAQ, INSTALL and inn.conf(5).
I would suggest to have the same wording at the 3 places, instead of 
different ones, if you're OK with that (or even better, only once and 
the 2 other places point to it - for instance to inn.conf, that is 
currently the one with the less explanations!).


Global suggestion for inn.conf, concatenating the arguments currently in 
the 3 documents:

buffindexed
Stores overview data and index information into preconfigured large 
files like CNFS.  Fast at writing, the buffindexed overview storage 
method can keep up with a large feed more easily and never consumes 
additional disk space beyond that allocated to these buffers.  The 
downside is that these buffers are hard to recover in case of corruption 
and somewhat slower for readers.  See the buffindexed.conf(5) man page 
for more details, and notably how to create the buffers.

ovdb
Stores overview information into a S<Berkeley DB> database.  This method 
is fast and very robust, but may require more disk space, unless 
compression is enabled.  See the ovdb(5) man page for more details.

ovsqlite
Stores overview information into an SQLite database, known for its 
long-term stability and compatibility.  This method is fast and very 
robust, but may require more disk space, unless compression is enabled. 
  Being the most recent of all methods, ovsqlite has not been as widely 
tested as the others yet, though already reliable enough to be used in 
production.  See the ovsqlite(5) man page for more details.

tradindexed
Uses two files per newsgroup, one containing the overview data and one
containing the index.  Fast for readers, but slow to write to because it 
has to update two files for each incoming article.  Its main advantage 
is to be the best tested, the most reliable and the method with the best 
recovery tools.

--- doc/pod/makehistory.pod	(révision 10457)
+++ doc/pod/makehistory.pod	(copie de travail)
@@ -24,7 +24,9 @@
  manager, and write a history line for every article.  To also generate
  overview information, use the B<-O> flag.

-WARNING:  If you're trying to rebuild the overview database, be sure to
+=head1 OVERVIEW REBUILD
+
+I<WARNING>:  If you're trying to rebuild the overview database, be sure to
  stop innd(8) and delete or zero out the existing database before you start
  for the best results.  An overview rebuild should not be done while the
  server is running.  Unless the existing overview is deleted, you may end
@@ -37,6 +39,52 @@
  rest of the server by running B<ovdb_init>; see ovdb_init(8) for more
  details.

+Similarly, if I<ovmethod> in F<inn.conf> is C<ovsqlite>, you must
+have the B<ovsqlite-server> process running while rebuilding overview.
+See ovsqlite-server(8) for more details and how to start it by hand.
+
+Rebuilding overview data is as straight-forward as:
+
+=over 4
+
+=item 1.
+
+Setting the new overview storage method in the I<ovmethod> parameter
+in F<inn.conf>.
+
+=item 2.
+
+Checking that its configuration file is correctly installed in
+I<pathetc> and fits your needs (F<buffindexed.conf>, F<ovdb.conf> or
+F<ovsqlite.conf>).  Note that the tradindexed overview storage method
+does not have a configuration file.
+
+=item 3.
+
+Making sure that INN is stopped.
+
+=item 4.
+
+Making sure that the directory specified by the I<pathoverview> parameter
+in F<inn.conf> exists and is empty.  Otherwise, rename the current one
+(to backup existing overview data) and re-create I<pathoverview> as
+the news user.
+
+=item 5.
+
+Starting B<ovdb_init> or B<ovsqlite-server> as the news user if the
+new overview storage method is respectively ovdb or ovsqlite.
+
+=item 6.
+
+Running C<makehistory -O -x -F> and waiting for the command to finish.
+
+=item 7.
+
+Starting INN and checking the logs to make sure everything is fine.
+
+=back
+

=> OK with that new "OVERVIEW REBUILD" section ?









===================================================================
--- doc/FAQ	(révision 10457)
+++ doc/FAQ	(copie de travail)
@@ -452,13 +452,19 @@
  clients.

  Any INN server that supports readers must therefore have an overview
-method configured.  There are three different methods to choose from:
-tradindexed, which is the slowest but the best tested and most reliable
-and the method with the best recovery tools; buffindexed, which is fast at
-writing because it uses preconfigured large buffers like CNFS, but which
-is harder to recover; and the experimental ovdb overview method, which
-stores overview information in a BerkeleyDB database.
+method configured.  There are four different methods to choose from:

+  - buffindexed, which is fast at writing because it uses preconfigured
+    large buffers like CNFS, but which is hard to recover;
+  - ovdb, which stores overview information in a Berkeley DB database
+    and supports compression;
+  - ovsqlite, implemented in INN 2.7.0, which stores overview information
+    in an SQLite database and supports compression, but still not as
+    widely tested as the other overview mechanisms (all introduced with
+    INN 2.3.0);
+  - tradindexed, which is the slowest but the best tested, the most
+    reliable and the method with the best recovery tools.
+
  ------------------------------

=> suggestion to point to inn.conf documentation instead





--- doc/pod/install.pod	(révision 10457)
+++ doc/pod/install.pod	(copie de travail)
@@ -666,11 +684,6 @@

  =over 4

-=item tradindexed
-
-It is very fast for readers, but it has to update two files for each
-incoming article and can be quite slow to write.
-
  =item buffindexed

  It can keep up with a large feed more easily, since it uses large buffers
@@ -682,10 +695,23 @@

  =item ovdb

-It stores overview data in a S<Berkeley DB> database; it's fast and 
very robust
,
-but may require more disk space.  See the ovdb(5) man page for more
-information on it.
+It stores overview data in a S<Berkeley DB> database; it's fast and very
+robust, but may require more disk space, unless compression is enabled.
+See the ovdb(5) man page for more information on it.

+=item ovsqlite
+
+It stores overview data in an SQLite database, known for its long-term
+stability and compatibility.  It's fast and very robust, but may require
+more disk space, unless compression is enabled.  See the ovsqlite(5)
+man page for more information on it.
+
+=item tradindexed
+
+It is very fast for readers, but it has to update two files for each
+incoming article and can be quite slow to write.  Robust and well-tested,
+with the best recovery tools.
+
  =back


=> suggestion to point to inn.conf documentation instead







--- doc/pod/inn.conf.pod	(révision 10457)
+++ doc/pod/inn.conf.pod	(copie de travail)
@@ -590,8 +590,9 @@
  =item I<ovmethod>

  Which overview storage method to use.  Currently supported values are
-C<tradindexed>, C<buffindexed>, and C<ovdb>.  There is no default value;
-this parameter must be set if I<enableoverview> is true (the default).
+C<buffindexed>, C<ovdb>, C<ovsqlite> and C<tradindexed>.  There is no
+default value; this parameter must be set if I<enableoverview> is true
+(the default).

  =over 4

@@ -601,15 +602,19 @@
  preconfigured files defined in F<buffindexed.conf>.  C<buffindexed> never
  consumes additional disk space beyond that allocated to these buffers.

+=item C<ovdb>
+
+Stores data into a S<Berkeley DB> database.  See the ovdb(5) man page.
+
+=item C<ovsqlite>
+
+Stores data into an SQLite database.  See the ovsqlite(5) man page.
+
  =item C<tradindexed>

  Uses two files per newsgroup, one containing the overview data and one
  containing the index.  Fast for readers, but slow to write to.

-=item C<ovdb>
-
-Stores data into a S<Berkeley DB> database.  See the ovdb(5) man page.
-
  =back


=> suggestion to have the comparison only here





--- doc/pod/checklist.pod	(révision 10457)
+++ doc/pod/checklist.pod	(copie de travail)
@@ -92,7 +92,7 @@

  You probably want B<--with-perl>.  If you're not using NetBSD with
  cycbuffs or OpenBSD, perhaps B<--with-tagged-hash>.  You might want to
-compile in TLS/SSL and S<Berkeley DB>, if your system supports them.  You
+compile in TLS/SSL and SQLite, if your system supports them.  You
  will need to have the relevant external libraries to compile (depending
  on whether you use OpenSSL for TLS/SSL access to your news server, GnuPG
  to verify the authenticity of Usenet control messages, Perl, Python, 
etc.).


=> suggestion to now prefer SQLite over Berkeley DB in CHECKLIST

-- 
Julien ÉLIE

« Sol lucet omnibus. »
