INN not keeping up with incoming news

[Alex Kiernan]

Fabien Tassin <fta at sofaraway.org> writes:

> > | >   Interestingly, the hiswrite is large, but it went down more than the
> > | > artwrite. Don't understand that one. Still tuning, I want to make the
> > | > initial article buffer size larger, and see if that helps. My first try
> > | > broke something, so I have to look at the code a bit more carefully. I
> > | > have less than 200 streams, so a MB/stream is not an issue for memory,
> > | > as I usually have at least a GB free (4GB machine).
> > | 
> > | I have far less memory on that server (around 1.5 GB) but most of the time,
> > | only a third is in use. 
> > 
> >   My art/hiswrite dropped from 22-25ms to 14-18ms, idle time went for
> > <1% to 15-30% (several servers). I'm going to try locking hist cache in
> > memory if it isn't already, I see no reason why it would page, given 1GB
> > free memory, but I see no reason not to try it, either!
> 
> idle is always >60% here. art/hiswrite avg is around 8ms but I assume hiswrite
> should be lower than 3ms (3.827/8.240/20.995ms today). It's not easy to locate
> bottlenecks on production servers so I'll write a tool to bench the history
> code separately once it will be in a lib with an API.
> 

We've just taken to locking the history.{hash,index} in core; its
worked wonders - we've gone from spending hours per day in HIS* to
under 10 minutes (although I've no doubt that some of the time is
hidden elsewhere). Thinking about how you do page cleaning in an OS, I
can imagine that the INN history hash is a hard problem to solve when
its mmap()ed. There's an interesting comment from Matt Dillon I turned
up:

http://groups.google.com/groups?start=10&hl=en&lr=&safe=off&th=d0e8de5661dacd45&rnum=15&seld=955735264&ic=1

If anyone's interested, here's the code we're using (it doubtless has
some Solaris assumptions, I haven't tried building it anywhere else):

/* $Id: mlockfile.c,v 1.1 2001/02/13 16:16:51 alexk Exp $ */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <fcntl.h>
#include <poll.h>
#include <sysexits.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <sys/stropts.h>

struct mlock {
    const char *path;
    struct stat st;
    void *base;
};

char *progname;

void
lock_files(struct mlock *ml)
{
    int i;

    for (; ml->path != NULL; ++ml) {
	int fd;

	fd = open(ml->path, O_RDWR);
	if (fd == -1) {
	    fprintf(stderr, "%s: can't open `%s' - %s\n",
		    progname, ml->path, strerror(errno));
	}
	else {
	    struct stat st;

	    /* check if size, inode or device of the path have
	     * changed, if so unlock the previous file & lock the new
	     * one */
	    if (fstat(fd, &st) != 0) {
		fprintf(stderr, "%s: can't stat `%s' - %s\n",
			progname, ml->path, strerror(errno));
	    }
	    else if (ml->st.st_ino != st.st_ino ||
		     ml->st.st_dev != st.st_dev ||
		     ml->st.st_size != st.st_size) {
		void *p;

		if (ml->base != MAP_FAILED)
		    munmap(ml->base, ml->st.st_size);

		/* free everything here, so in case of failure we try
		 * again next time */
		ml->st.st_ino = 0;
		ml->st.st_dev = 0;
		ml->st.st_size = 0;

		ml->base = mmap(NULL, st.st_size, PROT_READ|PROT_WRITE,
				MAP_SHARED, fd, 0);

		if (p == MAP_FAILED) {
		    fprintf(stderr, "%s: can't mmap `%s' - %s\n",
			    progname, ml->path, strerror(errno));
		}
		else {
		    if (mlock(ml->base, st.st_size) != 0) {
			fprintf(stderr, "%s: can't mlock `%s' - %s\n",
				progname, ml->path, strerror(errno));
		    }
		    else {
			ml->st = st;
		    }
		}
	    }
	}
	close (fd);
    }
}

int
main(int argc, char *argv[])
{
    int didit = 0;
    struct mlock *ml;
    int i;

    progname = *argv;

    /* construct list of pathnames which we're to operate on, zero out
     * the "cookies" so we lock it in core first time through */
    ml = malloc(argc * sizeof ml);
    for (i = 0; --argc; ++i) {
	int fd;

	ml[i].path = *++argv;
	ml[i].st.st_ino = 0;
	ml[i].st.st_dev = 0;
	ml[i].st.st_size = 0;
	ml[i].base = MAP_FAILED;
    }
    ml[i].path = NULL;

    /* loop over the list of paths, sleeping 60s between iterations */
    for (;;) {
	lock_files(ml);
	poll(NULL, 0, 60000);
    }
    return EX_OSERR;
}


-- 
Alex Kiernan, Principal Engineer, Development, Thus PLC