Call for testing new expiration and ovsqlite features

[Julien ÉLIE]

Hi all,

There had been several useful improvements these last weeks, thanks to 
Kevin's great work.
We would be interested in having more wide-testing and feedback from 
people running large news spools (billion of articles), lots of several 
concurrent readers or encountering long expire/expireover runs.

The improvements are in latest CURRENT (2.8) snapshots, starting from 
inn-CURRENT-20260528.tar.gz and inn-2.8-20260527.tar.gz:
   https://downloads.isc.org/isc/inn/snapshots/daily/
   https://downloads.isc.org/isc/inn/snapshots/

I'll merge them soon to the STABLE branch, and provide a release 
candidate for the upcoming INN 2.7.4 release.


 From changelog about Kevin's work:

     * The speed of expireover has been greatly improved by the
       implementation of a Bloom filter (a probabilistic data structure used
       to efficiently test whether an element is a member of a set, without
       any false negative).  Instead of checking every article in the
       overview database against the history file to detect orphaned 
entries,
       expireover now sequentially reads the history file at startup to 
build
       a Bloom filter, and makes its checks in that optimized data 
structure.
       On large news spools of billions of articles, expireover could take
       several weeks to complete; it now runs in a few minutes!  This 
feature
       is activated by default and controlled by the new *expirebloomfp*
       parameter in inn.conf.

     * The speed of expire has also been drastically improved for large news
       spools thanks to the recording of every cancellation alongside the
       existing storage and overview deletions, so that expire can just drop
       matching history entries without doing per-article retrieval system
       calls, thus saving a good amount of time at billion-article scale.
       This feature is not enabled by default.  See the documentation of
       *expiretombstone* in inn.conf for more details, as well as
       *nnrpdcheckart* which also makes use of the cancel tombstone log 
file.

     * Added four new parameters to ovsqlite.conf for the ovsqlite overview
       method: *mmapsize* to have SQLite read data via mmap instead of
       copying into its own page cache, *readercachesize* to set the SQLite
       in-memory page cache size for each nnrpd reader process, *walmode* to
       enable SQLite's Write-Ahead Logging journal mode and improve read
       performace when multiple nnrpd processes access the overview database
       simultaneously, and *walcheckpointthreshold* to parameterize when to
       perform checkpoints.  See the ovsqlite(5) manual page for more
       details.




-- 
Julien ÉLIE

« Petite annonce : Astronaute recherche femme lunatique. »