ovsqlite
Julien ÉLIE
julien at trigofacile.com
Tue Dec 22 19:37:17 UTC 2020
Hi Bo,
>> ovsqlite/ovsqlite-server.c: Dans la fonction « unpack_length »:
>> ovsqlite/ovsqlite-server.c:501:20: erreur: décalage vers la gauche d'une valeur négative [-Werror=shift-negative-value]
>> 501 | length = c&~(~0<<(8-lenlen));
>> | ^~
>
> Marking some constants as unsigned should fix that
> (included in the attached patch).
Yes, indeed.
>> For this last one, it seems that SQLITE_PREPARE_PERSISTENT was introduced in SQLite 3.20.0.
>> Should INN require that recent version (August 2017) at configure time?
>> Or #if SQLITE_VERSION_NUMBER >= 3020000 added there?
>
> The prepare flags argument is useful but not essential. Don't change
> the version requirement; add workarounds to sqlite-helper.[ch]
> (patch attached).
Works like a charm!
I've rebuilt my overview data earlier this day, to switch to ovsqlite :)
It is as straight-forward as:
1- install ovsqlite.conf and eventually change its default settings
(though ovsqlite also works without that file, and just complains in the
logs)
2- run "ovsqlite-server"
3- run "makehistory -O -x -F"
I have 3,249,249 articles in my overview (according to the latest run of
expireover).
As for sizes:
-rw-r--r-- 1 news news 5,1G déc. 22 17:36 ovsqlite.db
-rw-r--r-- 1 news news 22M déc. 22 17:36 ovsqlite.db-journal
My previous tradindexed overview was 2,9 GB in size (sum of all
subdirectories). It seems that ovsqlite with uncompressed overview data
takes almost twice as much space, but that's not a big deal for me.
For benchmarking comparison, I rebuilt it with compression on:
-rw-r--r-- 1 news news 1,6G déc. 22 18:26 ovsqlite.db
-rw-r--r-- 1 news news 17M déc. 22 18:26 ovsqlite.db-journal
ovsqlite then performs better than tradindexed.
The ratio between compressed and uncompressed ovsqlite data is about 1/3.
However, I think I have spotted an issue with ovsqlite.
I had 30 "cannot write overview data" errors from makehistory. To make
sure it ovsqlite-related, I did 4 rebuilds:
1/ uncompressed ovsqlite => 30 write errors
2/ compressed ovsqlite => 30 write errors (exactly the same)
3/ uncompressed ovsqlite, with a modified makehistory so as to keep
overview data => 30 write errors (exactly the same)
4/ tradindexed => 0 error
For instance, this one:
fr.rec.photo.numerique: 1493646864 0
@04020059073E000600000000000000000000@
=?iso-8859-1?Q?Forcer_r=E9solution_16:9_sous_XP_pour_affichage_sur_TV_LCD?=
"Alf92" <alf921 at gmail.com> Wed, 30 Mar 2011 00:53:33 +0200
<imtntk$t4h$1 at dont-email.me> 2545 34 Xref: news.trigofacile.com
fr.rec.photo.numerique:87727 fr.rec.son-image.video.materiel:4634
microsoft.public.fr.windowsxp:86375 injection-info:
mx01.eternal-september.org; posting-host="1VpS/qOBiAXpLThHEQ8yMA";
logging-data="29841";
mail-complaints-to="abuse at eternal-september.org";
posting-account="U2FsdGVkX19016eiCDPk9wyjdfTtK+5G" Path:
news2.trigofacile.com!news.trigofacile.com!weretis.net!feeder4.news.weretis.net!news-transit.tcx.org.uk!eternal-september.org!feeder.eternal-september.org!.POSTED!not-for-mail
Xref: news.trigofacile.com fr.rec.photo.numerique:87727
fr.rec.son-image.video.materiel:4634 microsoft.public.fr.windowsxp:86375
This article appears in my tradindexed overview (in
fr.rec.son-image.video.materiel.DAT), just rebuilt.
However, it is not in my ovsqlite overview.
I think the issue comes from the fact that the fr.rec.photo.numerique
newsgroup no longer exists. Yet, one of the newsgroup it was
crossposted to still exists.
350|5350|5827|384|0|0|fr.rec.son-image.video.materiel|y
select * from artinfo where artnum=4634 and groupid=350;
=> no article
I also have an example of crosspost where the 1st group still exists,
but not the 2nd.
I've not checked all 30 write errors, but the ones I did are similar
crossposts to a removed newsgroup from active.
Full log (for future reference for me):
makehistory: cannot write overview data
"@030346523300000000000000001000000002@"
makehistory: cannot write overview data
"@030346523400000000000000001000000002@"
makehistory: cannot write overview data
"@04020059073900DB00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900DC00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900DD00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900DE00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900DF00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900E000000000000000000000@"
makehistory: cannot write overview data
"@04020059073900E100000000000000000000@"
makehistory: cannot write overview data
"@04020059073900E200000000000000000000@"
makehistory: cannot write overview data
"@04020059073900E300000000000000000000@"
makehistory: cannot write overview data
"@04020059073900EC00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900ED00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900D600000000000000000000@"
makehistory: cannot write overview data
"@04020059073A00BF00000000000000000000@"
makehistory: cannot write overview data
"@04020059073E000600000000000000000000@"
makehistory: cannot write overview data
"@04020059073E000700000000000000000000@"
makehistory: cannot write overview data
"@04020059073E000800000000000000000000@"
makehistory: cannot write overview data
"@04020059073E000900000000000000000000@"
makehistory: cannot write overview data
"@0201590739421D5100000000000000000000@"
makehistory: cannot write overview data
"@020159073974356600000000000000000000@"
makehistory: cannot write overview data
"@0201590739C646D900000000000000000000@"
makehistory: cannot write overview data
"@020159073974357100000000000000000000@"
makehistory: cannot write overview data
"@020159073974356A00000000000000000000@"
makehistory: cannot write overview data
"@020159073A19651800000000000000000000@"
makehistory: cannot write overview data
"@020159073E7436B300000000000000000000@"
makehistory: cannot write overview data
"@020159073CD9F9E100000000000000000000@"
makehistory: cannot write overview data
"@05000000024B000000000000020000000000@"
makehistory: cannot write overview data
"@05000000024B00000000000001F900000000@"
makehistory: cannot write overview data
"@05000000024B000000000000020400000000@"
--
Julien ÉLIE
« Ce serait un tunnel pour aller de la Gaule en Bretagne. » (Astérix)
More information about the inn-workers
mailing list