ovsqlite

Bo Lindbergh 2bfjdsla52kztwejndzdstsxl9athp at gmail.com
Sun Dec 20 10:00:16 UTC 2020


Quoth Julien ÉLIE <julien at trigofacile.com>:
> I see that compression is disabled by default in ovsqlite.conf; is it still nowadays the best configuration?  (ovdb also disables it by default but we may have a different behaviour, if better, for ovsqlite)

No idea.  I just copied that from ovdb.

> Commits are done by default after 10 000 transactions.  Maybe a silly question but how does ovsqlite behaves if INN (or better say ovsqlite-server) crashes or is killed without exiting properly?  (do we loose up to 10 000 overview changes since last commit?)

You'd lose at most the last 10000 articles, or the last 10 seconds of changes,
whichever is smaller.  (I have no idea how many articles a typical INN site
needs to store each second.)

The problem is that committing a transaction takes several hundred
times as long as inserting a single row within a transaction,
so throughput correlates strongly with transaction size.

I chose the default after measuring how long "makehistory -O -x" took
using various settings.  If it seems too high for daily use,
we could lower it and add a note in the documentation about temporarily
raising it when rebuilding overview data.

> With ovsqlite, are article counts in newsgroups always accurate? (I think so)

That's what the code attempts to do, at least.  :-)

> I see that the removal of a newsgroup is deferred until the next expiration, which is a good thing (preventing an error in a control message or manually with ctlinnd).

Actually, it's because removing thousands of rows from the article table
could take some time and shouldn't be allowed to block regular operations.

Adding a new group does NOT undelete a previous one with the same name.
I checked carefully what ovdb does in this situation and implemented
the closest equivalent.

Manual undeletion would be a lot easier with ovsqlite than with ovdb,
since you get a free SQL command line tool with each SQLite installation.

> In order to make searches faster (responses to HDR and XPAT when requesting header fields stored in overview), wouldn't it be useful to also store each header body (at least the mandatory ones in overview.fmt) in a separate column?  or would the gain be meaningless?

The current overview method interface has no way to specify a particular
overview field; it's all or nothing.  I did look into storing the subset
needed for expiration separately, but since the helper functions
(OVhisthasmsgid and OVgroupbasedexpire) expect the full overview data
as their input, I abandoned that idea.

Using separate columns would negate the advantage of compression too.


/Bo Lindbergh



More information about the inn-workers mailing list