CURRENT branch

Russ Allbery rra at
Mon May 18 07:24:47 UTC 2009

Julien ÉLIE <julien at> writes:

> Yes, that's it.  If an overview data contains article numbers:
> 2 - 3 - 9 - 6 - 4
> in that order.  Then remapping is done when an article whose number is
> < 9 is written.  It should not happen in normal overview use since the
> numbers are increasing.

Right, it would normally only happen in the Xref slaving case.

What was confusing me was that this shouldn't make any difference for
the overview index.  In the normal case, the overview index will have
holes in it for the articles between 2 and 9, and when we write the
entries for 6 and 4 later on, the mmaped memory will pick them up.  No
remapping should be required.

The part that I was missing is that we're getting misses on the *data*
mmap, not the index mmap.  And of course that makes sense now.  What I'd
missed when writing the code originally is that we need to remap the
data file on *any* write, not just on writes beyond the high water mark,
since of course the overview data is written in the order received
(unlike the index).

So I still think that your patch is doing too much work (although it
probably doesn't matter a great deal), but not in the way that I'd
thought.  I believe that in the out-of-order write case, it's only
necessary to remap the data, not the index.  In other words, instead of:

    if ((end > data->high && high > data->high) || data->remapoutoforder) {
        data->high = high;
        data->remapoutoforder = false;

I suspect all that's needed is:

    if ((end > data->high && high > data->high) || data->remapoutoforder) {
        data->remapoutoforder = false;
    if (end > data->high && high > data->high) {
        data->high = high;

Or, even more simply, just set a flag whenever we do any writes to the
overview at all and always remap the data in that case, but keep the
previous logic for when to remap the index.

But given that this case only affects clients that both read and write
to overview, this minor optimization probably doesn't matter that much
(and doesn't make our behavior with OVSTATICSEARCH any better; the data
remapping is the part that matters there anyway).

> When there is a rebase, remapping will currently be done and also if
> there is an outstanding search.

Yes, indeed, I should have checked that.

> I do not see well what you prefer to do (except for filling overview
> holes when possible).

There won't ever be useful holes in the data file since it's rewritten
entirely during nightly expire (and hence always packed).

Russ Allbery (rra at             <>

    Please send questions to the list rather than mailing me directly.
     <> explains why.

More information about the inn-workers mailing list