XOVER ranges

[Jeffrey M. Vinocur]

On Fri, 4 Apr 2008, Jeffrey M. Vinocur wrote:

> On Thu, 3 Apr 2008, Russ Allbery wrote:
> 
> > >   (2) Change CMDgetrange() to not adjust the watermarks
> > 
> > I think 2 is clearly the right solution if all of the overview backends
> > can handle ranges outside the current watermarks.  

Okay, buffindexed it looks like checks for this situation and updates the
current watermarks itself.  I don't think OVDB catches the stale 
watermarks, but it certainly shouldn't break if we ask for articles higher 
than the high watermark.


> > >       I'm still worried there might be a bug lurking in the
> > >       unmap_file/map_file that can take place in the middle of the search,
> > >       since SendIOv may still have references to data in the old location.
> > 
> > Oh, yes.  Given that tradindexed advertises OVSTATICSEARCH false, it
> > shouldn't be doing that.  Good catch.  It either needs to advertise
> > OVSTATICSEARCH true to force nnrpd to copy the data or it needs to never
> > remap in the middle of a search.  I think the latter is more correct.
> 
> I just wish it was actually the cause of the strange behavior I'm seeing!

Okay, this is really interesting.  I just added some debugging, and 
despite the fact that CMDgetrange() ought to be masking the ability to 
request articles past the (stale) high watermark, there apparently is a 
code path that gets there because I'm seeing remapping of both the index 
file and data file happening in routine use.  So I bet this is the cause 
of our problem.

Interestingly, it only seems to happen with some clients (slrn) and I'm 
not sure why -- possibly other clients send GROUP again when they're 
trying to refresh for newer articles, rather than just making repeated 
XOVER requests.  I think that would do it.
 

> I thought about just updating the open code to remap both if it found a
> stale watermark, but I suppose that potentially we could get in trouble
> there if there's a write to the data file that hasn't flushed yet?

For kicks, I just tried remapping the data file every time the opensearch 
wants to remap the index file, and that eliminated the data remapping that 
had been happening mid-search.

The only reason I can see not to make that change is that it might prompt 
unnecessary I/O in case we weren't actually going to need, but in all of 
the cases that I'm seeing, both files are always ending up remapped 
anyway.


Then we have the issue of what to do if the index file points to a part of 
the data file that is not mapped.  Right now we try remap_data, and if it 
fails warn of corruption.  Maybe we could just warn of corrupion and see 
if anybody ever encounters it?


-- 
Jeffrey M. Vinocur
jeff at litech.org