ovsqlite

Julien ÉLIE julien at trigofacile.com
Tue Dec 22 19:37:17 UTC 2020


Hi Bo,

>> ovsqlite/ovsqlite-server.c: Dans la fonction « unpack_length »:
>> ovsqlite/ovsqlite-server.c:501:20: erreur: décalage vers la gauche d'une valeur négative [-Werror=shift-negative-value]
>>  501 |     length = c&~(~0<<(8-lenlen));
>>      |                    ^~
> 
> Marking some constants as unsigned should fix that
> (included in the attached patch).

Yes, indeed.


>> For this last one, it seems that SQLITE_PREPARE_PERSISTENT was introduced in SQLite 3.20.0.
>> Should INN require that recent version (August 2017) at configure time?
>> Or #if SQLITE_VERSION_NUMBER >= 3020000 added there?
> 
> The prepare flags argument is useful but not essential.  Don't change
> the version requirement; add workarounds to sqlite-helper.[ch]
> (patch attached).

Works like a charm!


I've rebuilt my overview data earlier this day, to switch to ovsqlite :)

It is as straight-forward as:
1- install ovsqlite.conf and eventually change its default settings 
(though ovsqlite also works without that file, and just complains in the 
logs)
2- run "ovsqlite-server"
3- run "makehistory -O -x -F"



I have 3,249,249 articles in my overview (according to the latest run of 
expireover).


As for sizes:
-rw-r--r--  1 news news 5,1G déc.  22 17:36 ovsqlite.db
-rw-r--r--  1 news news  22M déc.  22 17:36 ovsqlite.db-journal

My previous tradindexed overview was 2,9 GB in size (sum of all 
subdirectories).  It seems that ovsqlite with uncompressed overview data 
takes almost twice as much space, but that's not a big deal for me.

For benchmarking comparison, I rebuilt it with compression on:

-rw-r--r--  1 news news 1,6G déc.  22 18:26 ovsqlite.db
-rw-r--r--  1 news news  17M déc.  22 18:26 ovsqlite.db-journal

ovsqlite then performs better than tradindexed.
The ratio between compressed and uncompressed ovsqlite data is about 1/3.




However, I think I have spotted an issue with ovsqlite.
I had 30 "cannot write overview data" errors from makehistory.  To make 
sure it ovsqlite-related, I did 4 rebuilds:
1/ uncompressed ovsqlite => 30 write errors
2/ compressed ovsqlite => 30 write errors (exactly the same)
3/ uncompressed ovsqlite, with a modified makehistory so as to keep 
overview data => 30 write errors (exactly the same)
4/ tradindexed => 0 error


For instance, this one:

fr.rec.photo.numerique:	1493646864	0 
@04020059073E000600000000000000000000@ 
=?iso-8859-1?Q?Forcer_r=E9solution_16:9_sous_XP_pour_affichage_sur_TV_LCD?= 
"Alf92" <alf921 at gmail.com>	Wed, 30 Mar 2011 00:53:33 +0200 
<imtntk$t4h$1 at dont-email.me>		2545	34	Xref: news.trigofacile.com 
fr.rec.photo.numerique:87727 fr.rec.son-image.video.materiel:4634 
microsoft.public.fr.windowsxp:86375	injection-info: 
mx01.eternal-september.org; posting-host="1VpS/qOBiAXpLThHEQ8yMA"; 
logging-data="29841"; 
mail-complaints-to="abuse at eternal-september.org"; 
posting-account="U2FsdGVkX19016eiCDPk9wyjdfTtK+5G"		Path: 
news2.trigofacile.com!news.trigofacile.com!weretis.net!feeder4.news.weretis.net!news-transit.tcx.org.uk!eternal-september.org!feeder.eternal-september.org!.POSTED!not-for-mail 
Xref: news.trigofacile.com fr.rec.photo.numerique:87727 
fr.rec.son-image.video.materiel:4634 microsoft.public.fr.windowsxp:86375	


This article appears in my tradindexed overview (in 
fr.rec.son-image.video.materiel.DAT), just rebuilt.
However, it is not in my ovsqlite overview.
I think the issue comes from the fact that the fr.rec.photo.numerique 
newsgroup no longer exists.  Yet, one of the newsgroup it was 
crossposted to still exists.

350|5350|5827|384|0|0|fr.rec.son-image.video.materiel|y
select * from artinfo where artnum=4634 and groupid=350;
=> no article


I also have an example of crosspost where the 1st group still exists, 
but not the 2nd.


I've not checked all 30 write errors, but the ones I did are similar 
crossposts to a removed newsgroup from active.



Full log (for future reference for me):
makehistory: cannot write overview data 
"@030346523300000000000000001000000002@"
makehistory: cannot write overview data 
"@030346523400000000000000001000000002@"
makehistory: cannot write overview data 
"@04020059073900DB00000000000000000000@"
makehistory: cannot write overview data 
"@04020059073900DC00000000000000000000@"
makehistory: cannot write overview data 
"@04020059073900DD00000000000000000000@"
makehistory: cannot write overview data 
"@04020059073900DE00000000000000000000@"
makehistory: cannot write overview data 
"@04020059073900DF00000000000000000000@"
makehistory: cannot write overview data 
"@04020059073900E000000000000000000000@"
makehistory: cannot write overview data 
"@04020059073900E100000000000000000000@"
makehistory: cannot write overview data 
"@04020059073900E200000000000000000000@"
makehistory: cannot write overview data 
"@04020059073900E300000000000000000000@"
makehistory: cannot write overview data 
"@04020059073900EC00000000000000000000@"
makehistory: cannot write overview data 
"@04020059073900ED00000000000000000000@"
makehistory: cannot write overview data 
"@04020059073900D600000000000000000000@"
makehistory: cannot write overview data 
"@04020059073A00BF00000000000000000000@"
makehistory: cannot write overview data 
"@04020059073E000600000000000000000000@"
makehistory: cannot write overview data 
"@04020059073E000700000000000000000000@"
makehistory: cannot write overview data 
"@04020059073E000800000000000000000000@"
makehistory: cannot write overview data 
"@04020059073E000900000000000000000000@"
makehistory: cannot write overview data 
"@0201590739421D5100000000000000000000@"
makehistory: cannot write overview data 
"@020159073974356600000000000000000000@"
makehistory: cannot write overview data 
"@0201590739C646D900000000000000000000@"
makehistory: cannot write overview data 
"@020159073974357100000000000000000000@"
makehistory: cannot write overview data 
"@020159073974356A00000000000000000000@"
makehistory: cannot write overview data 
"@020159073A19651800000000000000000000@"
makehistory: cannot write overview data 
"@020159073E7436B300000000000000000000@"
makehistory: cannot write overview data 
"@020159073CD9F9E100000000000000000000@"
makehistory: cannot write overview data 
"@05000000024B000000000000020000000000@"
makehistory: cannot write overview data 
"@05000000024B00000000000001F900000000@"
makehistory: cannot write overview data 
"@05000000024B000000000000020400000000@"


-- 
Julien ÉLIE

« Ce serait un tunnel pour aller de la Gaule en Bretagne. » (Astérix)


More information about the inn-workers mailing list