time spent writing history..

[Fabien Tassin]

According to Reddy Crashalott:
> 
> :: > I wonder why innd spend so much time to write into the history.
> :: > These numbers are for a full feed (1.3M articles and 220 GB that day).
> :: 
> :: Because, I think, one of the changes in INN 2.0 was to change from one
> :: text file and one memory mapped table of offsets to a memory mapped table
> :: indicating presence or absence of a particular hash and a separate table
> :: of offsets into the text file.  It cuts down on memory usage and it makes
> :: lookups faster, but the table of offsets isn't memory mapped and writes
> :: are done with pwrite, which is seeking all over the disk and not buffering
> :: a bunch of writes.
> 
> *Z-Z-Z-Z-Zip*
> 
>    INND timer:
>    
>     Code region             Time      Pct  Invoked  ms Min ms Avg ms Max
>     history grep        00:00:00.000  0.0%        0  0.000  0.000  0.000
>     history lookup      00:03:30.496  0.2% 14279616  0.010  0.015  0.020
>     history sync        00:00:03.281  0.0%    11574  0.023  0.283  5.818
>     history write       00:04:54.013  0.3%   817600  0.149  0.360  1.022
>     idle                17:10:53.713 71.6% 47914650  0.778  1.291  2.259

>    Precommit cache stats:
>    
>            Precommit cache lookups Precommit cache hits Hit rate
>            36504070                            23042161    63.1%
> 
> So, total of some 36 1/2 million history lookups in a few minutes.
> 
> History.hash and history.index file, 21 days retention.

hmm.. looks good. Time to port the precommit artifice to INN 2.*.
Russ, do think it is still possible ?

I'll try to see if I can do it but if someone already have something,
please, speak now ! :)

-- 
Fabien Tassin -+- fta at sofaraway.org