INN commit: trunk/doc (5 files)

[INN Commit]

    Date: Wednesday, January 20, 2021 @ 03:49:11
  Author: iulius
Revision: 10524

Compare space usage of our 4 overview methods

Modified:
  trunk/doc/FAQ
  trunk/doc/pod/buffindexed.conf.pod
  trunk/doc/pod/install.pod
  trunk/doc/pod/ovdb.pod
  trunk/doc/pod/ovsqlite.pod

--------------------------+
 FAQ                      |   16 ++++++++++++++++
 pod/buffindexed.conf.pod |    7 +++++++
 pod/install.pod          |   16 ++++++++++++++++
 pod/ovdb.pod             |   21 +++++++++++----------
 pod/ovsqlite.pod         |    9 ++++++++-
 5 files changed, 58 insertions(+), 11 deletions(-)

Modified: FAQ
===================================================================
--- FAQ	2021-01-17 21:52:00 UTC (rev 10523)
+++ FAQ	2021-01-20 11:49:11 UTC (rev 10524)
@@ -481,6 +481,22 @@
     incoming article.  Its main advantage is to be the best tested,
     the most reliable and the method with the best recovery tools.
 
+Here are a few elements that can be helpful in choosing the right
+overview method for your needs and estimating the associated storage
+size.  In 2020, the volume for a full-text Usenet feed is about 18,000
+articles per day, with peaks to 1,200 articles per hour.  Article storage
+size is about 65 MB per day.
+
+As for overview storage size, if you have 5 million articles, you'll
+need at least 3.25 GB of disk space for buffindexed, 5.5 GB for ovdb
+(4.5 GB if compressed), 4.65 GB for ovsqlite (2 GB if compressed),
+and 3.10 GB for tradindexed.
+
+If you store more header fields in overview data than the standard
+ones, the space needed to store overview data will be superior than
+these estimates.  (This is configured in the extraoverviewadvertised
+and extraoverviewhidden inn.conf parameters.)
+
 ------------------------------
 
 Subject: 2.5. What are deferrals (NNTP code 431)?

Modified: pod/buffindexed.conf.pod
===================================================================
--- pod/buffindexed.conf.pod	2021-01-17 21:52:00 UTC (rev 10523)
+++ pod/buffindexed.conf.pod	2021-01-20 11:49:11 UTC (rev 10524)
@@ -68,6 +68,13 @@
 care (other than restarting B<innd> after modifying F<buffindexed.conf>).
 If you need to rebuild overview, you should zero all of the buffers first.
 
+We recommend not to reserve too much spare space in existing buffers, so
+that to minimize the duration of the B<expireover> process, and to just
+add new buffers when space left is low (see the result of C<inndf -no>).
+Plan on needing at least S<0.65 KB> for every article in your spool
+(not counting crossposts).  So, if you have 5 million articles, you'll
+need at least S<3.25 GB> of disk space for buffindexed.
+
 =head1 CREATING BUFFERS
 
 There are two methods to create a new buffindexed buffer:

Modified: pod/install.pod
===================================================================
--- pod/install.pod	2021-01-17 21:52:00 UTC (rev 10523)
+++ pod/install.pod	2021-01-20 11:49:11 UTC (rev 10524)
@@ -729,6 +729,22 @@
 
 =back
 
+Here are a few elements that can be helpful in choosing the right
+overview method for your needs and estimating the associated storage
+size.  In 2020, the volume for a full-text Usenet feed is about 18,000
+articles per day, with peaks to 1,200 articles per hour.  Article storage
+size is about S<65 MB> per day.
+
+As for overview storage size, if you have 5 million articles, you'll
+need at least S<3.25 GB> of disk space for buffindexed, S<5.5 GB> for ovdb
+(S<4.5 GB> if compressed), S<4.65 GB> for ovsqlite (S<2 GB> if compressed),
+and S<3.10 GB> for tradindexed.
+
+If you store more header fields in overview data than the standard
+ones, the space needed to store overview data will be superior
+than these estimates.  (This is configured in F<inn.conf>, via the
+I<extraoverviewadvertised> and I<extraoverviewhidden> parameters.)
+
 =head1 Configuring INN
 
 All documentation from this point on assumes that you have set up the news

Modified: pod/ovdb.pod
===================================================================
--- pod/ovdb.pod	2021-01-17 21:52:00 UTC (rev 10523)
+++ pod/ovdb.pod	2021-01-20 11:49:11 UTC (rev 10524)
@@ -60,16 +60,17 @@
 libraries are used, one or both of the options B<--with-bdb-include>
 and B<--with-bdb-lib> can be given to configure with a path.
 
-The ovdb database may take up more disk space for a given spool
-than the other overview methods.  Plan on needing at least S<1.1 KB>
-for every article in your spool (not counting crossposts).  So, if
-you have 5 million articles, you'll need at least S<5.5 GB> of disk
-space for ovdb.  With compression enabled, this estimate changes to
-S<0.7 KB> per article.  See the L<COMPRESSION> section below.  Plus,
-you'll need additional space for transaction logs: at least S<100 MB>.
-By default, the transaction logs go in the same directory as the database.
-To improve performance, they can be placed on a different disk S<--
-see> the L<DB_CONFIG> section.
+The ovdb database may take up more disk space for a given spool than the
+other overview methods.  Plan on needing at least S<1.1 KB> for every
+article in your spool (not counting crossposts).  So, if you have 5
+million articles, you'll need at least S<5.5 GB> of disk space for ovdb.
+With compression enabled, this estimate changes to S<0.9 KB> per article,
+so you'll need at least S<4.5 GB> of disk space for 5 million articles.
+See the L<COMPRESSION> section below.  Plus, you'll need additional space
+for transaction logs: at least S<100 MB>.  By default, the transaction
+logs go in the same directory as the database.  To improve performance,
+they can be placed on a different disk S<-- see> the L<DB_CONFIG>
+section.
 
 =head1 CONFIGURATION
 

Modified: pod/ovsqlite.pod
===================================================================
--- pod/ovsqlite.pod	2021-01-17 21:52:00 UTC (rev 10523)
+++ pod/ovsqlite.pod	2021-01-20 11:49:11 UTC (rev 10524)
@@ -29,6 +29,13 @@
 installed.  You may have to install a separate package with a name similar
 to C<libsqlite3-dev> to get the required header files.
 
+Plan on needing at least S<0.93 KB> for every article in your spool
+(not counting crossposts).  So, if you have 5 million articles, you'll
+need at least S<4.65 GB> of disk space for ovsqlite.  With compression
+enabled, this estimate changes to S<0.4 KB> per article, so you'll
+need at least S<2 GB> of disk space for 5 million articles.  Plus,
+you'll need additional space for transaction logs (a few MB).
+
 =head1 CONFIGURATION
 
 To select ovsqlite as your overview method, set the I<ovmethod> parameter
@@ -52,7 +59,7 @@
 If INN was built with zlib support and this parameter is true, ovsqlite
 will compress overview records whenever this saves space.  This parameter
 is consulted only when creating a new database.  Enabling compression
-saves about S<70 %> of disk space on typical overview data. The default
+saves about S<55 %> of disk space on standard overview data.  The default
 value is false.
 
 =item I<pagesize>