From julien at trigofacile.com  Thu Jul 16 21:02:39 2026
From: julien at trigofacile.com (=?UTF-8?Q?Julien_=C3=89LIE?=)
Date: Thu, 16 Jul 2026 23:02:39 +0200
Subject: Call for testing new expiration and ovsqlite features
In-Reply-To: <b9286e4c-0600-4b0c-94f9-f56e7a04e66b@trigofacile.com>
References: <1ab654c7-2f4b-4a8a-9a21-3efaf52acc14@trigofacile.com>
 <811c5a2e-c50e-4f08-81d9-9240ca09cedb@blueworldhosting.com>
 <7537d7ca-15c8-496c-aa7b-a068c47b38e1@trigofacile.com>
 <b9286e4c-0600-4b0c-94f9-f56e7a04e66b@trigofacile.com>
Message-ID: <6f380c72-c5e7-4a13-82e2-3b4db269ae8b@trigofacile.com>

Hi Jesse, and all,

>>> I'm very excited about these changes, hats off to everyone involved!
> 
> Do you happen to have found a bit of time to test these new features?
> 
> In addition, Kevin contributed a few days ago a new SQLite-based history 
> method:
>  ? https://github.com/InterNetNews/inn/pull/349
> 
> It would be worthwhile testing too, notably to see whether it performs 
> better on large spools.? If you are disposed to give it a try in a 
> testing instance, you can apply the patch (or I can generate a tarball 
> with the patch for you).

The new history storage method using SQLite is available in latest 
snapshots.  You may want to try it :)

See the hissqlite-convert manual page to convert a hisv6 database to 
hissqlite.
Basically, assuming pathdb is /news/db:
1 - Stop INN.
2 - Update to the latest CURRENT version.
3 - Run: hissqlite-convert /news/db/history /news/db/history
4 - If it went fine (history.sqlite generated, and no errors during the 
conversion), change hismethod to hissqlite in inn.conf.
5 - Start INN.
6 - Enjoy!



Also, did you try the improvements to expire/expireover?

-- 
Julien ?LIE

??H?te-toi de bien vivre et songe que chaque jour est ? lui seul une
   vie.?? (S?n?que)


From julien at trigofacile.com  Thu Jul 16 21:13:12 2026
From: julien at trigofacile.com (=?UTF-8?Q?Julien_=C3=89LIE?=)
Date: Thu, 16 Jul 2026 23:13:12 +0200
Subject: New pathhistory setting for the history database directory
Message-ID: <ab45e38a-bb51-41f7-8793-dd34a4557cc6@trigofacile.com>

Hi all,

In tomorrow's snapshot (inn-CURRENT-20260717.tar.gz), there will be the possibility to set pathhistory in inn.conf for the directory of the history database.
This is a great contribution from Kevin Bowling.

You may want to have your history database in a dedicated partition, so feel free to migrate your history files to a new directory (while INN is not running) and set pathhistory in inn.conf to that directory.
One advantage is to set the following parameter appropriately:

hissqlitepagesize
     The SQLite page size in bytes, applied when the database is  created
     (it  cannot  be  changed  afterwards without rebuilding).  Must be a
     power of two between 512 and 65536.  The default is "4096", matching
     the common filesystem block size so that a random single-row  insert
     or point lookup on the random-MD5 keys touches exactly one block (no
     read-modify-write).   For  best  results, the page size should match
     the underlying storage  block  size:  on  ZFS,  create  a  dedicated
     history  dataset  with  "recordsize=4k"  and  point  the pathhistory
     parameter at it.  A small page on a  coarse-recordsize  ZFS  dataset
     (the  128k  default)  forces read-modify-write on every random write
     and will perform worse, not better.

-- 
Julien ?LIE

??? l'?cole, en alg?bre, j'?tais du genre Einstein. Mais plut?t Franck
   qu'Albert.?? (Philippe Geluck)


From julien at trigofacile.com  Thu Jul 16 21:17:26 2026
From: julien at trigofacile.com (=?UTF-8?Q?Julien_=C3=89LIE?=)
Date: Thu, 16 Jul 2026 23:17:26 +0200
Subject: Switching to ovsqlite / hissqlite by default?
Message-ID: <d59a94f0-c6c0-407d-b223-22927c0bd552@trigofacile.com>

Hi all,

What do you think about changing the default tradindexed and hisv6 
settings in the inn.conf sample file shipped with INN so that they are 
determined at configure time, for the next major INN 2.8.0 release?

If SQLite is usable, then it defaults to ovsqlite and hissqlite 
(otherwise, it remains tradindexed and hisv6) at installation time.

-- 
Julien ?LIE

??? Ce serait une joyeuse id?e de faire une temp?te dans nos cerveaux,
     n'est-il pas??
   ? Il veut parler d'une r?union de travail?!?? (Ast?rix)


From julien at trigofacile.com  Thu Jul 16 21:30:47 2026
From: julien at trigofacile.com (=?UTF-8?Q?Julien_=C3=89LIE?=)
Date: Thu, 16 Jul 2026 23:30:47 +0200
Subject: New pathhistory setting for the history database directory
In-Reply-To: <ab45e38a-bb51-41f7-8793-dd34a4557cc6@trigofacile.com>
References: <ab45e38a-bb51-41f7-8793-dd34a4557cc6@trigofacile.com>
Message-ID: <cc542f72-23da-49f5-973c-8aae6072d8b1@trigofacile.com>

About it, the default pathhistory value for new installations remains 
pathdb in the shipped inn.conf file.  It can be set at configure time 
with the --with-history-dir option, or manually changed after the 
installation.

I think it is fine to keep the history database in its legacy directory. 
  One has to elaborate a special configuration to benefit from having a 
separate directory for it.
If someone believes it would be better to ship <prefix>/history by 
default instead of <prefix>/db in the sample inn.conf file, do not 
hesitate to tell.

-- 
Julien ?LIE

??? Ce serait une joyeuse id?e de faire une temp?te dans nos cerveaux,
     n'est-il pas??
   ? Il veut parler d'une r?union de travail?!?? (Ast?rix)


From julien at trigofacile.com  Fri Jul 17 07:45:26 2026
From: julien at trigofacile.com (=?UTF-8?Q?Julien_=C3=89LIE?=)
Date: Fri, 17 Jul 2026 09:45:26 +0200
Subject: Call for testing new expiration and ovsqlite features
In-Reply-To: <6f380c72-c5e7-4a13-82e2-3b4db269ae8b@trigofacile.com>
References: <1ab654c7-2f4b-4a8a-9a21-3efaf52acc14@trigofacile.com>
 <811c5a2e-c50e-4f08-81d9-9240ca09cedb@blueworldhosting.com>
 <7537d7ca-15c8-496c-aa7b-a068c47b38e1@trigofacile.com>
 <b9286e4c-0600-4b0c-94f9-f56e7a04e66b@trigofacile.com>
 <6f380c72-c5e7-4a13-82e2-3b4db269ae8b@trigofacile.com>
Message-ID: <c1f5853c-f087-466a-b39e-55fee0edcc56@trigofacile.com>

Hi Jesse,

>> Do you happen to have found a bit of time to test these new features?

I hope we have not lost some of your previous responses.  (I do not 
always check Mailman auto-discard notifications.)
I see that you are subscribed to this mailing list with the e-mail 
address jesse.rehmer@ but are sending messages with jrehmer@, which 
makes your messages be discarded.  FWIW, I have just added your jrehmer@ 
mail to the whitelist :)


> I have several scenarios for performance testing hissqlite and the
> bloom filter for expire/expireover, but need a bit of time to get
> testing in motion. My current focus is getting the "complete" Usenet
> archive (>5TB of articles from 1986 forward) online. Once I have
> this dataset in a ready state, I will clone it for testing hissqlite
> and the bloom filter. Based on the current runtime, I expect it to
> take another 5-7 days to have the spool in its entirety.
OK, thanks!  Keep us informed.


> Will also do tests on a fresh server comparing hisv6 against
> hissqlite for write/ingestion performance. Based on comments I saw
> from Kevin's benchmarks, I don't think there is a performance gain
> in this scenario, but is worth testing. Currently the server spends
> slightly more time writing history than to overview, which feels
> backwards:> ----> Jul 16 16:26:22 archive innd: [ID 702911 news.notice] ME time 
600000
> hishave 4001(521335) hiswrite 272940(521335) hissync 16574(2) idle
> 12463(255268) artclean 1142(521335) artwrite 9962(521325) artcncl
> 356(588) hisgrep/artcncl 85(593) hishave/artcncl 0(5) artlog/artcncl
> 0(5) overv 261973(521325) perl 50(521335) python 50(521335) nntpread
> 1868(255268) artparse 6108(766077) artlog 1480(521656) datamove
> 163(295799)> ---->
> Based on what I'm seeing from my current operation, history writes
> seem to be the least performant operation in INN.>
> This is probably worthy of a separate thread, but are there any
> additional filesystem performance tuning that should be done with
> INN on ZFS? I've understand the recommendation for a 4k recordsize
> when using hissqlite, and currently use ovsqlite on a separate ZFS
> dataset that is set to 64k recordsize (matching what I've set in
> ovsqlite.conf), but what about for CNFS buffers? Do you think there
> would be any gain to changing the recordsize for CNFS (currently
> using the default 128k recordsize).
I admit I do not know what other recommendations there would be for ZFS. 
  I relay the question on the mailing-list!

-- 
Julien ?LIE

??J'ai ?t? un homme dans un corps de femme. Avant ma naissance, bien
   entendu.?? (Mike Bent)


From kevin.bowling at kev009.com  Fri Jul 17 19:54:53 2026
From: kevin.bowling at kev009.com (Kevin Bowling)
Date: Fri, 17 Jul 2026 12:54:53 -0700
Subject: Call for testing new expiration and ovsqlite features
In-Reply-To: <c1f5853c-f087-466a-b39e-55fee0edcc56@trigofacile.com>
References: <1ab654c7-2f4b-4a8a-9a21-3efaf52acc14@trigofacile.com>
 <811c5a2e-c50e-4f08-81d9-9240ca09cedb@blueworldhosting.com>
 <7537d7ca-15c8-496c-aa7b-a068c47b38e1@trigofacile.com>
 <b9286e4c-0600-4b0c-94f9-f56e7a04e66b@trigofacile.com>
 <6f380c72-c5e7-4a13-82e2-3b4db269ae8b@trigofacile.com>
 <c1f5853c-f087-466a-b39e-55fee0edcc56@trigofacile.com>
Message-ID: <CAK7dMtAM2V7DgJHrqxOAWG0h+y6nyg5eCEDSquu-gLW8TDWX2w@mail.gmail.com>

On Fri, Jul 17, 2026 at 12:45?AM Julien ?LIE <julien at trigofacile.com> wrote:
>
> Hi Jesse,
>
> >> Do you happen to have found a bit of time to test these new features?
>
> I hope we have not lost some of your previous responses.  (I do not
> always check Mailman auto-discard notifications.)
> I see that you are subscribed to this mailing list with the e-mail
> address jesse.rehmer@ but are sending messages with jrehmer@, which
> makes your messages be discarded.  FWIW, I have just added your jrehmer@
> mail to the whitelist :)
>
>
> > I have several scenarios for performance testing hissqlite and the
> > bloom filter for expire/expireover, but need a bit of time to get
> > testing in motion. My current focus is getting the "complete" Usenet
> > archive (>5TB of articles from 1986 forward) online. Once I have
> > this dataset in a ready state, I will clone it for testing hissqlite
> > and the bloom filter. Based on the current runtime, I expect it to
> > take another 5-7 days to have the spool in its entirety.
> OK, thanks!  Keep us informed.

csiph.com is running w/hissqlite, the conversion took about 1.5hrs
limited by older SAS flash SSDs.  The expected benefit is observed on
my heavy read side, and no need for the innd pause and rebuild history
every night is an operational/availability win.

The write amplification bugs me and I don't know what to do about it
right now.  It's not a big problem during steady state, but conversion
and especially flood filling will create a ton of extra I/O.  On fast
NVMe you may get something like 100k history writes/sec, I wouldn't
think that would serve as an ultimate bottleneck like Jesse is seeing
with hisv6 but we need some empirical data.  It might also make
hissqlite not great for HDDs.. only SSD.

Jesse, playing around with history-bench would be useful for me in
addition to your real world testing.

>
> > Will also do tests on a fresh server comparing hisv6 against
> > hissqlite for write/ingestion performance. Based on comments I saw
> > from Kevin's benchmarks, I don't think there is a performance gain
> > in this scenario, but is worth testing. Currently the server spends
> > slightly more time writing history than to overview, which feels
> > backwards:> ----> Jul 16 16:26:22 archive innd: [ID 702911 news.notice] ME time
> 600000
> > hishave 4001(521335) hiswrite 272940(521335) hissync 16574(2) idle
> > 12463(255268) artclean 1142(521335) artwrite 9962(521325) artcncl
> > 356(588) hisgrep/artcncl 85(593) hishave/artcncl 0(5) artlog/artcncl
> > 0(5) overv 261973(521325) perl 50(521335) python 50(521335) nntpread
> > 1868(255268) artparse 6108(766077) artlog 1480(521656) datamove
> > 163(295799)> ---->
> > Based on what I'm seeing from my current operation, history writes
> > seem to be the least performant operation in INN.>
> > This is probably worthy of a separate thread, but are there any
> > additional filesystem performance tuning that should be done with
> > INN on ZFS? I've understand the recommendation for a 4k recordsize
> > when using hissqlite, and currently use ovsqlite on a separate ZFS
> > dataset that is set to 64k recordsize (matching what I've set in
> > ovsqlite.conf), but what about for CNFS buffers? Do you think there
> > would be any gain to changing the recordsize for CNFS (currently
> > using the default 128k recordsize).
> I admit I do not know what other recommendations there would be for ZFS.
>   I relay the question on the mailing-list!

Probably worth setting primarycache=all for pathoverview and
pathhistory datasets.  It is critical that the record size matches the
SQLite pagesize.  atime off, lz4 are good all around defaults.  Both
datasets should ideally be on flash storage.  Flash is less critical
for the actual spool, and it doesn't need specific tuning, atime=off
and lz4 defaults would be good.

Also which OS are you running zfs on?

> --
> Julien ?LIE
>
> ? J'ai ?t? un homme dans un corps de femme. Avant ma naissance, bien
>    entendu. ? (Mike Bent)
>
> --
> inn-workers mailing list
> inn-workers at lists.isc.org
> https://lists.isc.org/mailman/listinfo/inn-workers

