INN not keeping up with incoming news

[Fabien Tassin]

According to Alex Kiernan:
> 
> > According to Alex Kiernan:
> > > 
> > > We've just taken to locking the history.{hash,index} in core; its
> > > worked wonders
> > 
> > without touching to the INN code ?
> > 
> 
> Without touching the INN code.

yes, I know it could work but should we not try to obtain a similar result
by doing the right thing (c) in INN ? I mean, using INCORE_MMAP, avoiding
holes in history by not using ftruncate(), mmaping history.index, etc.
 
> The reason it works (and this is Solaris specific, but may apply to
> every OS with a page cache, depending on exactly how they deal with
> mmap and IO):

I think this will work for Linux and FreeBSD too.
 
> The files are MAP_SHARED, so shareable amongst all processes. When the
> hash file is used in inn (mmaped in dbz), you just need to take minor
> page faults in the caller to link them into the memory map. When you
> use the index file (not mmaped in dbz), you enter the kernel and
> because all IO happens via mmap you take minor page faults to link
> them into the kernel memory map, whereon we copy to user
> land. Modifying INN to mmap the index file so you avoid this copy
> looks like an optimisation.

agreed.
 
> > > If anyone's interested, here's the code we're using (it doubtless has
> > > some Solaris assumptions, I haven't tried building it anywhere else):
> > 
> > I haven't tried your program yet but I wonder if this job should not be
> > done in dbz.c automatically.
> 
> The problem is you need to be root to use mlock() - I played with the
> idea of helper processes which INN communicates with (so I could avoid
> the polling), but in the end keeping it simple was what I went with.

this will be easier with a threaded implementation.. once it will be there ;)
 
> > I also wonder about its behaviour during and after
> > expire/shutdown/crashes/sync etc. If the files are never committed on disks,
> > the risk is very high to corrupt or lose everything.
> > 
> 
> The crash risk is partially already there - on exactly what basis does
> each platform flush modified mmaped pages, how each platform then
> manages syncs from pages which are mlocked is another part of the
> problem (AFAICT Solaris manages both problems in the same way, the
> mlocked pages are just ignored by the page scanner when stealing
> pages).
> 
> I guess we could reduce the risk by msyncing the pages every time the
> code wakes up.
> 
> Also your text history file is complete, I'm not playing games with
> that one, so you can always rebuild.
> 
> I'm not sure I understand what risk you see on expire/shutdown/sync?

I was trying to figure what will happen when the history files are swapped
after a rebuild. I don't remember what were the problems I saw with
shutdown/sync..

-- 
Fabien Tassin -+- fta at sofaraway.org