time spent writing history..

[Russ Allbery]

Fabien Tassin <fta at sofaraway.org> writes:

> Code region              Time    Pct    Invoked  Min(ms)  Avg(ms)    Max(ms)
> article cleanup  01:03:22.687   4.4%    1349312    1.207    2.818      4.229
> article write    03:38:32.563  15.2%    1316215    4.628    9.962     14.299
> history lookup   01:33:00.892   6.5%   15994074    0.008    0.349      2.731
> history sync     00:00:09.114   0.0%      26704    0.000    0.341     76.367
> history write    03:35:37.066  15.0%    1327232    8.507    9.747     21.893

> I wonder why innd spend so much time to write into the history.
> These numbers are for a full feed (1.3M articles and 220 GB that day).

Because, I think, one of the changes in INN 2.0 was to change from one
text file and one memory mapped table of offsets to a memory mapped table
indicating presence or absence of a particular hash and a separate table
of offsets into the text file.  It cuts down on memory usage and it makes
lookups faster, but the table of offsets isn't memory mapped and writes
are done with pwrite, which is seeking all over the disk and not buffering
a bunch of writes.

I'm not sure how to improve the tradeoffs.  :/

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>