After reading the Russ Allberry's posting today, I realized that I had to send patches to inn-patches list. It's quite long time since I contributed anything to INN, and I didn't aware the rule. Anyway, here's the buffindexed jumbo patch. The patch includes three files: 1. buffindexed.c which is an update to the Kondou's original version 1.40.2.8 2002/01/17, which can be found on inn-STABLE-20021027 2. shmem.c and shmem.h : This is a shared memory and semaphore utility. As I have told on inn-workers, I used shared memory and semaphore to resolve the file locking problem of the ovbuffhead. The group files locking is unchanged because I think just one process, i.e., either innd or overchan will update the group data. I increased the version number. Also, I decided not to support compatibility in the default update. The incompatibility occurs in the following two defines: #ifdef KEEP_COMPATIBILTY #define OV_HDR_PAGESIZE 16384 #define OV_BEFOREBITF (1 * OV_BLOCKSIZE) #else #define OV_HDR_PAGESIZE 4096 #define OV_BEFOREBITF 256 #endif The reason I decided not to support compatibility is that the shared memory will be resident for all buffers. The original version mmaps one buffer at a time. This means the new version consumes more memory than the previous one. To save memory space, I reduced OV_BEFOREBITF from 8192 to 256. Most of this area is unused. Also, I've reduced OV_HDR_PAGESIZE from 16384 to 4096. The new value is the Linux page size. If this is big, the round-up effect will increase the bitfield size which is again unused. If you define KEEP_COMPATIBILITY, old overview DB may be used as is. However, I've noticed that there are many inconsitent blocks in old overview data, and this new program may fail. It tries to fix the corrupted overview. Ok. Here is the change summary since I released the very first version. ovflushhead() has been changed to used pwrite istead of the two separate system calls, lseek and write. Registered buffindexed_close() as an atexit function. This makes sure that the in memory data is synced to the disk even if the overchan is killed by a signal. Increased OVBUFF_SYNC_COUNT to icdsynccount * 10. Fixed low number bug in buffindexed_expiregroup(). It was reporting low number as is if the article count is zero and low == high. The correct reporting syntax, in this case, is high+1. Tries to fix corrupted overview such as from power failure. The type of fix includes (1) trying to allocate block which has been already occupied, (2) trying to free block which has been already released(no fix needed in this case), (3) unoccupied block which must have been occupied. These three cases are reported via syslog in normal mode. In OV_DEBUG the program aborts, without fixing so that programmer can diagnose the problem without altering the block status. Removed syslog message of smcGetShmemBuffer(). It is making news log file huge on a busy nnrp server. -- Attached file included as plaintext by Ecartis -- diff -uNr buffindexed.orig/buffindexed.c buffindexed/buffindexed.c --- buffindexed.orig/buffindexed.c Thu Jan 17 10:41:32 2002 +++ buffindexed/buffindexed.c Mon Dec 23 09:41:11 2002 @@ -2,6 +2,32 @@ ** ** Overview buffer and index method. */ + +/* +** Buffindexed using shared memory on ovbuff by Sang-yong Suh +** +** Id: buffindexed.c,v 1.2 2002/12/22 12:00:04 sysuh Exp +** +** During the recent discussions in inn-workers, Alex Kiernan found +** that INN LockRange() is not working for MMAPed file. This explains +** why buffindexed has long outstanding bugs such as "could not MMAP...". +** +** This version corrects the file locking error by using shared memory. +** The bitfield of each buffer file is loaded into memory, and is shared +** by all programs such as innd, expireover, makehistory, and overchan. +** The locking problem is handled by semaphore. +*/ + +/* + * Yes. I know that it violates INN coding style. However, this allows + * me to compile this new version without reconfiguring INN. + * If all goes well, shmem.c should go to $INN/lib, and shmem.h should + * go to $INN/include. + */ +#include "shmem.h" +#include "shmem.c" + + #include "config.h" #include "clibrary.h" #include @@ -30,7 +56,15 @@ #include "buffindexed.h" -#define OVBUFF_MAGIC "ovbuff" +#define OVBUFF_MAGIC "ovbuff" +#define OVBUFF_VERSION 2 + +/* +** Because ovbuff bitfields are residing in memory, we don't have to +** do file write for each update. Instead we'll do it at every +** OVBUFF_SYNC_COUNT updates. +*/ +#define OVBUFF_SYNC_COUNT (innconf->icdsynccount * 10 + 1) /* ovbuff header */ #define OVBUFFMASIZ 8 @@ -40,9 +74,28 @@ #define OVMAXCYCBUFFNAME 8 +/* +** XXX The preferred value of shared memory version of buffindexed. +** +** OV_HDR_PAGESIZE 4096 +** OV_BEFOREBITF 256 +** +** Turing on the above change will result no upward compatibility. +** The benifit of the change is saving (8192 - 256) bytes per +** buffindexed file. +*/ + +/*#define KEEP_COMPATIBILITY*/ + +#ifdef KEEP_COMPATIBILTY #define OV_HDR_PAGESIZE 16384 -#define OV_BLOCKSIZE 8192 #define OV_BEFOREBITF (1 * OV_BLOCKSIZE) +#else +#define OV_HDR_PAGESIZE 4096 +#define OV_BEFOREBITF 256 +#endif + +#define OV_BLOCKSIZE 8192 #define OV_FUDGE 1024 /* ovblock pointer */ @@ -61,13 +114,18 @@ char useda[OVBUFFLASIZ]; /* ASCII version of used */ char freea[OVBUFFLASIZ]; /* ASCII version of free */ char updateda[OVBUFFLASIZ]; /* ASCII version of updated */ + /* + * The following parts will be always synced + */ + int version; /* magic version number */ + unsigned int freeblk; /* next free block number */ + unsigned int usedblk; /* number of used blocks */ } OVBUFFHEAD; /* ovbuff info */ typedef struct _OVBUFF { unsigned int index; /* ovbuff index */ char path[OVBUFFPASIZ]; /* Path to file */ - int magicver; /* Magic version number */ int fd; /* file descriptor for this ovbuff */ OFFSET_T len; /* Length of writable area, in @@ -83,10 +141,10 @@ header */ caddr_t bitfield; /* Bitfield for ovbuff block in use */ - BOOL needflush; /* true if OVBUFFHEAD is needed - to be flushed */ + int dirty; /* OVBUFFHEAD dirty count */ struct _OVBUFF *next; /* next ovbuff */ int nextchunk; /* next chunk */ + smcd_t *smc; /* shared memory control data */ #ifdef OV_DEBUG struct ov_trace_array *trace; #endif /* OV_DEBUG */ @@ -255,101 +313,107 @@ STATIC char *Gdb; #ifdef MMAP_MISSES_WRITES +#define PWRITE(fd, buf, nbyte, offset) mmapwrite(fd, buf, nbyte, offset) +#else +#define PWRITE(fd, buf, nbyte, offset) pwrite(fd, buf, nbyte, offset) +#endif + +#ifdef MMAP_MISSES_WRITES /* With HP/UX, you definitely do not want to mix mmap-accesses of a file with read()s and write()s of the same file */ STATIC OFFSET_T mmapwrite(int fd, void *buf, OFFSET_T nbyte, OFFSET_T offset) { - int pagefudge, len; - OFFSET_T mmapoffset; - caddr_t addr; + int pagefudge, len; + OFFSET_T mmapoffset; + caddr_t addr; + + pagefudge = offset % pagesize; + mmapoffset = offset - pagefudge; + len = pagefudge + nbyte; - pagefudge = offset % pagesize; - mmapoffset = offset - pagefudge; - len = pagefudge + nbyte; - - if ((addr = mmap((caddr_t) 0, len, PROT_READ|PROT_WRITE, MAP_SHARED, fd, mmapoffset)) == (caddr_t) -1) { - return -1; - } - memcpy(addr+pagefudge, buf, nbyte); - munmap(addr, len); - return nbyte; + if ((addr = mmap((caddr_t) 0, len, PROT_READ|PROT_WRITE, MAP_SHARED, fd, mmapoffset)) == (caddr_t) -1) { + return -1; + } + memcpy(addr+pagefudge, buf, nbyte); + munmap(addr, len); + return nbyte; } #endif /* MMAP_MISSES_WRITES */ STATIC BOOL ovparse_part_line(char *l) { - char *p; - struct stat sb; - OFFSET_T len, base; - int tonextblock; - OVBUFF *ovbuff, *tmp = ovbufftab; - - /* ovbuff partition name */ - if ((p = strchr(l, ':')) == NULL || p - l <= 0 || p - l > OVMAXCYCBUFFNAME - 1) { - syslog(L_ERROR, "%s: bad index in line '%s'", LocalLogName, l); - return FALSE; - } - *p = '\0'; - ovbuff = NEW(OVBUFF, 1); - ovbuff->index = strtoul(l, NULL, 10); - for (; tmp != (OVBUFF *)NULL; tmp = tmp->next) { - if (tmp->index == ovbuff->index) { - syslog(L_ERROR, "%s: dupulicate index in line '%s'", LocalLogName, l); - DISPOSE(ovbuff); - return FALSE; - } - } - l = ++p; - - /* Path to ovbuff partition */ - if ((p = strchr(l, ':')) == NULL || p - l <= 0 || p - l > OVBUFFPASIZ - 1) { - syslog(L_ERROR, "%s: bad pathname in line '%s'", LocalLogName, l); - DISPOSE(ovbuff); - return FALSE; - } - *p = '\0'; - memset(ovbuff->path, '\0', OVBUFFPASIZ); - strcpy(ovbuff->path, l); - if (stat(ovbuff->path, &sb) < 0) { - syslog(L_ERROR, "%s: file '%s' does not exist, ignoring '%d'", - LocalLogName, ovbuff->path, ovbuff->index); - DISPOSE(ovbuff); - return FALSE; - } - l = ++p; + char *p; + struct stat sb; + OFFSET_T len, base; + int tonextblock; + OVBUFF *ovbuff, *tmp = ovbufftab; + + /* ovbuff partition name */ + if ((p = strchr(l, ':')) == NULL || p - l <= 0 || p - l > OVMAXCYCBUFFNAME - 1) { + syslog(L_ERROR, "%s: bad index in line '%s'", LocalLogName, l); + return FALSE; + } + *p = '\0'; + ovbuff = NEW(OVBUFF, 1); + ovbuff->index = strtoul(l, NULL, 10); + for (; tmp != (OVBUFF *)NULL; tmp = tmp->next) { + if (tmp->index == ovbuff->index) { + syslog(L_ERROR, "%s: dupulicate index in line '%s'", LocalLogName, l); + DISPOSE(ovbuff); + return FALSE; + } + } + l = ++p; + + /* Path to ovbuff partition */ + if ((p = strchr(l, ':')) == NULL || p - l <= 0 || p - l > OVBUFFPASIZ - 1) { + syslog(L_ERROR, "%s: bad pathname in line '%s'", LocalLogName, l); + DISPOSE(ovbuff); + return FALSE; + } + *p = '\0'; + memset(ovbuff->path, '\0', OVBUFFPASIZ); + strcpy(ovbuff->path, l); + if (stat(ovbuff->path, &sb) < 0) { + syslog(L_ERROR, "%s: file '%s' does not exist, ignoring '%d'", + LocalLogName, ovbuff->path, ovbuff->index); + DISPOSE(ovbuff); + return FALSE; + } + l = ++p; - /* Length/size of symbolic partition */ - len = strtoul(l, NULL, 10) * (OFFSET_T)1024; /* This value in KB in decimal */ - /* - ** The minimum article offset will be the size of the bitfield itself, - ** len / (blocksize * 8), plus however many additional blocks the OVBUFFHEAD - ** external header occupies ... then round up to the next block. - */ - base = len / (OV_BLOCKSIZE * 8) + OV_BEFOREBITF; - tonextblock = OV_HDR_PAGESIZE - (base & (OV_HDR_PAGESIZE - 1)); - ovbuff->base = base + tonextblock; - if (S_ISREG(sb.st_mode) && (len != sb.st_size || ovbuff->base > sb.st_size)) { - if (len != sb.st_size) - syslog(L_NOTICE, "%s: length mismatch '%ld' for index '%d' (%ld bytes)", - LocalLogName, len, ovbuff->index, sb.st_size); - if (ovbuff->base > sb.st_size) - syslog(L_NOTICE, "%s: length must be at least '%ld' for index '%d' (%ld bytes)", - LocalLogName, ovbuff->base, ovbuff->index, sb.st_size); - DISPOSE(ovbuff); - return FALSE; - } - ovbuff->len = len; - ovbuff->fd = -1; - ovbuff->next = (OVBUFF *)NULL; - ovbuff->needflush = FALSE; - ovbuff->bitfield = (caddr_t)NULL; - ovbuff->nextchunk = 1; - - if (ovbufftab == (OVBUFF *)NULL) - ovbufftab = ovbuff; - else { - for (tmp = ovbufftab; tmp->next != (OVBUFF *)NULL; tmp = tmp->next); - tmp->next = ovbuff; - } - return TRUE; + /* Length/size of symbolic partition */ + len = strtoul(l, NULL, 10) * (OFFSET_T)1024; /* This value in KB in decimal */ + /* + ** The minimum article offset will be the size of the bitfield itself, + ** len / (blocksize * 8), plus however many additional blocks the OVBUFFHEAD + ** external header occupies ... then round up to the next block. + */ + base = len / (OV_BLOCKSIZE * 8) + OV_BEFOREBITF; + tonextblock = OV_HDR_PAGESIZE - (base & (OV_HDR_PAGESIZE - 1)); + ovbuff->base = base + tonextblock; + if (S_ISREG(sb.st_mode) && (len != sb.st_size || ovbuff->base > sb.st_size)) { + if (len != sb.st_size) + syslog(L_NOTICE, "%s: length mismatch '%ld' for index '%d' (%ld bytes)", + LocalLogName, len, ovbuff->index, sb.st_size); + if (ovbuff->base > sb.st_size) + syslog(L_NOTICE, "%s: length must be at least '%ld' for index '%d' (%ld bytes)", + LocalLogName, ovbuff->base, ovbuff->index, sb.st_size); + DISPOSE(ovbuff); + return FALSE; + } + ovbuff->len = len; + ovbuff->fd = -1; + ovbuff->next = (OVBUFF *)NULL; + ovbuff->dirty = 0; + ovbuff->bitfield = (caddr_t)NULL; + ovbuff->nextchunk = 1; + + if (ovbufftab == (OVBUFF *)NULL) + ovbufftab = ovbuff; + else { + for (tmp = ovbufftab; tmp->next != (OVBUFF *)NULL; tmp = tmp->next); + tmp->next = ovbuff; + } + return TRUE; } /* @@ -357,355 +421,413 @@ */ STATIC BOOL ovbuffread_config(void) { - char *config, *from, *to, **ctab = (char **)NULL; - int ctab_free = 0; /* Index to next free slot in ctab */ - int ctab_i; - - if ((config = ReadInFile(cpcatpath(innconf->pathetc, _PATH_OVBUFFCONFIG), - (struct stat *)NULL)) == NULL) { - syslog(L_ERROR, "%s: cannot read %s", LocalLogName, - cpcatpath(innconf->pathetc, _PATH_OVBUFFCONFIG), NULL); + char *config, *from, *to, **ctab = (char **)NULL; + int ctab_free = 0; /* Index to next free slot in ctab */ + int ctab_i; + + if ((config = ReadInFile(cpcatpath(innconf->pathetc, _PATH_OVBUFFCONFIG), + (struct stat *)NULL)) == NULL) + { + syslog(L_ERROR, "%s: cannot read %s", LocalLogName, + cpcatpath(innconf->pathetc, _PATH_OVBUFFCONFIG), NULL); + DISPOSE(config); + return FALSE; + } + for (from = to = config; *from; ) { + if (*from == '#') { /* Comment line? */ + while (*from && *from != '\n') + from++; /* Skip past it */ + from++; + continue; /* Back to top of loop */ + } + if (*from == '\n') { /* End or just a blank line? */ + from++; + continue; /* Back to top of loop */ + } + if (ctab_free == 0) + ctab = NEW(char *, 1); + else + RENEW(ctab, char *, ctab_free+1); + /* If we're here, we've got the beginning of a real entry */ + ctab[ctab_free++] = to = from; + while (1) { + if (*from && *from == '\\' && *(from + 1) == '\n') { + from += 2; /* Skip past backslash+newline */ + while (*from && isspace((int)*from)) + from++; + continue; + } + if (*from && *from != '\n') + *to++ = *from++; + if (*from == '\n') { + *to++ = '\0'; + from++; + break; + } + if (! *from) + break; + } + } + for (ctab_i = 0; ctab_i < ctab_free; ctab_i++) { + if (!ovparse_part_line(ctab[ctab_i])) { + DISPOSE(config); + DISPOSE(ctab); + return FALSE; + } + } DISPOSE(config); - return FALSE; - } - for (from = to = config; *from; ) { - if (*from == '#') { /* Comment line? */ - while (*from && *from != '\n') - from++; /* Skip past it */ - from++; - continue; /* Back to top of loop */ - } - if (*from == '\n') { /* End or just a blank line? */ - from++; - continue; /* Back to top of loop */ + DISPOSE(ctab); + if (ovbufftab == (OVBUFF *)NULL) { + syslog(L_ERROR, "%s: no buffindexed defined", LocalLogName); + return FALSE; } - if (ctab_free == 0) - ctab = NEW(char *, 1); - else - RENEW(ctab, char *, ctab_free+1); - /* If we're here, we've got the beginning of a real entry */ - ctab[ctab_free++] = to = from; - while (1) { - if (*from && *from == '\\' && *(from + 1) == '\n') { - from += 2; /* Skip past backslash+newline */ - while (*from && isspace((int)*from)) - from++; - continue; - } - if (*from && *from != '\n') - *to++ = *from++; - if (*from == '\n') { - *to++ = '\0'; - from++; - break; - } - if (! *from) - break; - } - } - for (ctab_i = 0; ctab_i < ctab_free; ctab_i++) { - if (!ovparse_part_line(ctab[ctab_i])) { - DISPOSE(config); - DISPOSE(ctab); - return FALSE; - } - } - DISPOSE(config); - DISPOSE(ctab); - if (ovbufftab == (OVBUFF *)NULL) { - syslog(L_ERROR, "%s: no buffindexed defined", LocalLogName); - return FALSE; - } - return TRUE; + return TRUE; } STATIC char hextbl[] = {'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'}; STATIC char *offt2hex(OFFSET_T offset, BOOL leadingzeros) { - static char buf[24]; - char *p; + static char buf[24]; + char *p; - if (sizeof(OFFSET_T) <= 4) { - sprintf(buf, (leadingzeros) ? "%016lx" : "%lx", offset); - } else { - int i; + if (sizeof(OFFSET_T) <= 4) { + sprintf(buf, (leadingzeros) ? "%016lx" : "%lx", offset); + } else { + int i; - for (i = 0; i < OVBUFFLASIZ; i++) - buf[i] = '0'; /* Pad with zeros to start */ - for (i = OVBUFFLASIZ - 1; i >= 0; i--) { - buf[i] = hextbl[offset & 0xf]; - offset >>= 4; - } - } - if (!leadingzeros) { - for (p = buf; *p == '0'; p++) + for (i = 0; i < OVBUFFLASIZ; i++) + buf[i] = '0'; /* Pad with zeros to start */ + for (i = OVBUFFLASIZ - 1; i >= 0; i--) { + buf[i] = hextbl[offset & 0xf]; + offset >>= 4; + } + } + if (!leadingzeros) { + for (p = buf; *p == '0'; p++) ; - if (*p != '\0') - return p; - else - return p - 1; /* We converted a "0" and then bypassed all the zeros */ - } else - return buf; + if (*p != '\0') + return p; + else + return p - 1; /* We converted a "0" and then bypassed all the zeros */ + } else + return buf; } STATIC OFFSET_T hex2offt(char *hex) { - if (sizeof(OFFSET_T) <= 4) { - OFFSET_T rpofft; - sscanf(hex, "%lx", &rpofft); - return rpofft; - } else { - char diff; - OFFSET_T n = (OFFSET_T) 0; - - for (; *hex != '\0'; hex++) { - if (*hex >= '0' && *hex <= '9') - diff = '0'; - else if (*hex >= 'a' && *hex <= 'f') - diff = 'a' - 10; - else if (*hex >= 'A' && *hex <= 'F') - diff = 'A' - 10; - else { - /* - ** We used to have a syslog() message here, but the case - ** where we land here because of a ":" happens, er, often. - */ - break; - } - n += (*hex - diff); - if (isalnum((int)*(hex + 1))) - n <<= 4; + if (sizeof(OFFSET_T) <= 4) { + OFFSET_T rpofft; + sscanf(hex, "%lx", &rpofft); + return rpofft; + } else { + char diff; + OFFSET_T n = (OFFSET_T) 0; + + for (; *hex != '\0'; hex++) { + if (*hex >= '0' && *hex <= '9') + diff = '0'; + else if (*hex >= 'a' && *hex <= 'f') + diff = 'a' - 10; + else if (*hex >= 'A' && *hex <= 'F') + diff = 'A' - 10; + else { + /* + ** We used to have a syslog() message here, but the case + ** where we land here because of a ":" happens, er, often. + */ + break; + } + n += (*hex - diff); + if (isalnum((int)*(hex + 1))) + n <<= 4; + } + return n; } - return n; - } } STATIC void ovreadhead(OVBUFF *ovbuff) { - OVBUFFHEAD rpx; - char buff[OVBUFFLASIZ+1]; - - memcpy(&rpx, ovbuff->bitfield, sizeof(OVBUFFHEAD)); - strncpy((char *)buff, rpx.useda, OVBUFFLASIZ); - buff[OVBUFFLASIZ] = '\0'; - ovbuff->usedblk = (unsigned int)hex2offt((char *)buff); - strncpy((char *)buff, rpx.freea, OVBUFFLASIZ); - buff[OVBUFFLASIZ] = '\0'; - ovbuff->freeblk = (unsigned int)hex2offt((char *)buff); - return; + OVBUFFHEAD *x = (OVBUFFHEAD *)ovbuff->bitfield; + ovbuff->freeblk = x->freeblk; + ovbuff->usedblk = x->usedblk; + return; } STATIC void ovflushhead(OVBUFF *ovbuff) { - OVBUFFHEAD rpx; + OVBUFFHEAD rpx; - if (!ovbuff->needflush) + /* skip time consuming data conversion and write call */ + if (ovbuff->dirty < OVBUFF_SYNC_COUNT) { + OVBUFFHEAD *x = (OVBUFFHEAD *)ovbuff->bitfield; + x->freeblk = ovbuff->freeblk; + x->usedblk = ovbuff->usedblk; + return; + } + + memset(&rpx, 0, sizeof(OVBUFFHEAD)); + ovbuff->updated = time(NULL); + strncpy(rpx.magic, OVBUFF_MAGIC, strlen(OVBUFF_MAGIC)); + strncpy(rpx.path, ovbuff->path, OVBUFFPASIZ); + /* Don't use sprintf() directly ... the terminating '\0' causes grief */ + strncpy(rpx.indexa, offt2hex(ovbuff->index, TRUE), OVBUFFLASIZ); + strncpy(rpx.lena, offt2hex(ovbuff->len, TRUE), OVBUFFLASIZ); + strncpy(rpx.totala, offt2hex(ovbuff->totalblk, TRUE), OVBUFFLASIZ); + strncpy(rpx.useda, offt2hex(ovbuff->usedblk, TRUE), OVBUFFLASIZ); + strncpy(rpx.freea, offt2hex(ovbuff->freeblk, TRUE), OVBUFFLASIZ); + strncpy(rpx.updateda, offt2hex(ovbuff->updated, TRUE), OVBUFFLASIZ); + rpx.version = OVBUFF_VERSION; + rpx.freeblk = ovbuff->freeblk; + rpx.usedblk = ovbuff->usedblk; + memcpy(ovbuff->bitfield, &rpx, sizeof(OVBUFFHEAD)); + + if( pwrite(ovbuff->fd, ovbuff->bitfield, ovbuff->base, 0) != ovbuff->base ) + syslog(L_ERROR, "%s: ovflushhead: cant flush on %s: %m", + LocalLogName, ovbuff->path); + ovbuff->dirty = 0; return; - memset(&rpx, 0, sizeof(OVBUFFHEAD)); - ovbuff->updated = time(NULL); - strncpy(rpx.magic, OVBUFF_MAGIC, strlen(OVBUFF_MAGIC)); - strncpy(rpx.path, ovbuff->path, OVBUFFPASIZ); - /* Don't use sprintf() directly ... the terminating '\0' causes grief */ - strncpy(rpx.indexa, offt2hex(ovbuff->index, TRUE), OVBUFFLASIZ); - strncpy(rpx.lena, offt2hex(ovbuff->len, TRUE), OVBUFFLASIZ); - strncpy(rpx.totala, offt2hex(ovbuff->totalblk, TRUE), OVBUFFLASIZ); - strncpy(rpx.useda, offt2hex(ovbuff->usedblk, TRUE), OVBUFFLASIZ); - strncpy(rpx.freea, offt2hex(ovbuff->freeblk, TRUE), OVBUFFLASIZ); - strncpy(rpx.updateda, offt2hex(ovbuff->updated, TRUE), OVBUFFLASIZ); - memcpy(ovbuff->bitfield, &rpx, sizeof(OVBUFFHEAD)); -#if defined (MMAP_NEEDS_MSYNC) -#if defined (HAVE_MSYNC_3_ARG) - msync(ovbuff->bitfield, ovbuff->base, MS_ASYNC); -#else - msync(ovbuff->bitfield, ovbuff->base); -#endif -#endif - ovbuff->needflush = FALSE; - return; } +/* +** Lock and unlock shared memory buffers +*/ STATIC BOOL ovlock(OVBUFF *ovbuff, LOCKTYPE type) { - return LockRange(ovbuff->fd, type, TRUE, 0, sizeof(OVBUFFHEAD)); + int ret; + smcd_t *smc = ovbuff->smc; + + if( type == LOCK_WRITE ) { + ret = smcGetExclusiveLock( smc ); + smc->locktype = (int)LOCK_WRITE; + } else if( type == LOCK_READ ) { + ret = smcGetExclusiveLock( smc ); + smc->locktype = (int)LOCK_READ; + } else if( smc->locktype == (int)LOCK_WRITE ) { + ret = smcReleaseExclusiveLock( smc ); + } else { + ret = smcReleaseSharedLock( smc ); + } + return (ret == 0); } STATIC BOOL ovbuffinit_disks(void) { - OVBUFF *ovbuff = ovbufftab; - char buf[64]; - OVBUFFHEAD *rpx; - int i, fd; - OFFSET_T tmpo; + OVBUFF *ovbuff = ovbufftab; + char buf[64]; + OVBUFFHEAD *rpx; + int i, fd; + OFFSET_T tmpo; + smcd_t *smc; + STATIC BOOL atexit_registered = FALSE; - /* - ** Discover the state of our ovbuffs. If any of them are in icky shape, - ** duck shamelessly & return FALSE. - */ - for (; ovbuff != (OVBUFF *)NULL; ovbuff = ovbuff->next) { - if (ovbuff->fd < 0) { - if ((fd = open(ovbuff->path, ovbuffmode & OV_WRITE ? O_RDWR : O_RDONLY)) < 0) { - syslog(L_ERROR, "%s: ERROR opening '%s' : %m", LocalLogName, ovbuff->path); - return FALSE; - } else { - CloseOnExec(fd, 1); - ovbuff->fd = fd; - } - } - if ((ovbuff->bitfield = - mmap((caddr_t) 0, ovbuff->base, ovbuffmode & OV_WRITE ? (PROT_READ | PROT_WRITE) : PROT_READ, - MAP_SHARED, ovbuff->fd, (off_t) 0)) == (MMAP_PTR) -1) { - syslog(L_ERROR, - "%s: ovinitdisks: mmap for %s offset %d len %d failed: %m", - LocalLogName, ovbuff->path, 0, ovbuff->base); - return FALSE; - } - rpx = (OVBUFFHEAD *)ovbuff->bitfield; - ovlock(ovbuff, LOCK_WRITE); - if (strncmp(rpx->magic, OVBUFF_MAGIC, strlen(OVBUFF_MAGIC)) == 0) { - ovbuff->magicver = 1; - if (strncmp(rpx->path, ovbuff->path, OVBUFFPASIZ) != 0) { - syslog(L_ERROR, "%s: Path mismatch: read %s for buffindexed %s", - LocalLogName, rpx->path, ovbuff->path); - ovbuff->needflush = TRUE; - } - strncpy(buf, rpx->indexa, OVBUFFLASIZ); - buf[OVBUFFLASIZ] = '\0'; - i = hex2offt(buf); - if (i != ovbuff->index) { - syslog(L_ERROR, "%s: Mismatch: index '%d' for buffindexed %s", - LocalLogName, i, ovbuff->path); - ovlock(ovbuff, LOCK_UNLOCK); - return FALSE; - } - strncpy(buf, rpx->lena, OVBUFFLASIZ); - buf[OVBUFFLASIZ] = '\0'; - tmpo = hex2offt(buf); - if (tmpo != ovbuff->len) { - syslog(L_ERROR, "%s: Mismatch: read 0x%s length for buffindexed %s", - LocalLogName, offt2hex(tmpo, FALSE), ovbuff->path); - ovlock(ovbuff, LOCK_UNLOCK); - return FALSE; - } - strncpy(buf, rpx->totala, OVBUFFLASIZ); - buf[OVBUFFLASIZ] = '\0'; - ovbuff->totalblk = hex2offt(buf); - strncpy(buf, rpx->useda, OVBUFFLASIZ); - buf[OVBUFFLASIZ] = '\0'; - ovbuff->usedblk = hex2offt(buf); - strncpy(buf, rpx->freea, OVBUFFLASIZ); - buf[OVBUFFLASIZ] = '\0'; - ovbuff->freeblk = hex2offt(buf); - ovflushhead(ovbuff); - Needunlink = FALSE; - } else { - ovbuff->totalblk = (ovbuff->len - ovbuff->base)/OV_BLOCKSIZE; - if (ovbuff->totalblk < 1) { - syslog(L_ERROR, "%s: too small length '%d' for buffindexed %s", - LocalLogName, ovbuff->len, ovbuff->path); - ovlock(ovbuff, LOCK_UNLOCK); - return FALSE; - } - ovbuff->magicver = 1; - ovbuff->usedblk = 0; - ovbuff->freeblk = 0; - ovbuff->updated = 0; - ovbuff->needflush = TRUE; - syslog(L_NOTICE, - "%s: No magic cookie found for buffindexed %d, initializing", - LocalLogName, ovbuff->index); - ovflushhead(ovbuff); + /* + * Register the exit callback to sync the disk + */ + if( !atexit_registered ) { + atexit( buffindexed_close ); + atexit_registered = TRUE; } + + /* + ** Discover the state of our ovbuffs. If any of them are in icky shape, + ** duck shamelessly & return FALSE. + */ + for (; ovbuff != (OVBUFF *)NULL; ovbuff = ovbuff->next) { + if (ovbuff->fd < 0) { + if ((fd = open(ovbuff->path, ovbuffmode & OV_WRITE ? + O_RDWR : O_RDONLY)) < 0) + { + syslog(L_ERROR, "%s: ERROR opening '%s' : %m", + LocalLogName, ovbuff->path); + return FALSE; + } else { + CloseOnExec(fd, 1); + ovbuff->fd = fd; + } + } + /* get shared memory buffer */ + smc = smcGetShmemBuffer( ovbuff->path, ovbuff->base ); + if ( !smc ) { + /* No shared memory exists, create one. */ + smc = smcCreateShmemBuffer( ovbuff->path, ovbuff->base ); + if ( !smc ) { + syslog(L_ERROR, + "%s: ovinitdisks: cant create shmem for %s len %d: %m", + LocalLogName, ovbuff->path, ovbuff->base); + return FALSE; + } + + /* try to read the bitfield */ + read( ovbuff->fd, smc->addr, ovbuff->base ); + } + + ovbuff->smc = smc; + ovbuff->bitfield = smc->addr; + /* lock the buffer */ + ovlock(ovbuff, LOCK_WRITE); + + rpx = (OVBUFFHEAD *)ovbuff->bitfield; + if (strncmp(rpx->magic, OVBUFF_MAGIC, strlen(OVBUFF_MAGIC)) == 0) { + if (strncmp(rpx->path, ovbuff->path, OVBUFFPASIZ) != 0) { + syslog(L_ERROR, "%s: Path mismatch: read %s for buffindexed %s", + LocalLogName, rpx->path, ovbuff->path); + ovbuff->dirty = 1; + } + strncpy(buf, rpx->indexa, OVBUFFLASIZ); + buf[OVBUFFLASIZ] = '\0'; + i = hex2offt(buf); + if (i != ovbuff->index) { + syslog(L_ERROR, "%s: Mismatch: index '%d' for buffindexed %s", + LocalLogName, i, ovbuff->path); + ovlock(ovbuff, LOCK_UNLOCK); + return FALSE; + } + strncpy(buf, rpx->lena, OVBUFFLASIZ); + buf[OVBUFFLASIZ] = '\0'; + tmpo = hex2offt(buf); + if (tmpo != ovbuff->len) { + syslog(L_ERROR, "%s: Mismatch: read 0x%s length for buffindexed %s", + LocalLogName, offt2hex(tmpo, FALSE), ovbuff->path); + ovlock(ovbuff, LOCK_UNLOCK); + return FALSE; + } + strncpy(buf, rpx->totala, OVBUFFLASIZ); + buf[OVBUFFLASIZ] = '\0'; + ovbuff->totalblk = hex2offt(buf); + + if ( rpx->version == 0 ) { + /* no binary data available. use character data */ + strncpy(buf, rpx->useda, OVBUFFLASIZ); + buf[OVBUFFLASIZ] = '\0'; + ovbuff->usedblk = hex2offt(buf); + strncpy(buf, rpx->freea, OVBUFFLASIZ); + buf[OVBUFFLASIZ] = '\0'; + ovbuff->freeblk = hex2offt(buf); + } else { + /* use binary data. The first reason is the speed. + and the second reason is the other partner is not + synced. + */ + ovbuff->usedblk = rpx->usedblk; + ovbuff->freeblk = rpx->freeblk; + } + ovflushhead(ovbuff); + Needunlink = FALSE; + } else { + ovbuff->totalblk = (ovbuff->len - ovbuff->base)/OV_BLOCKSIZE; + if (ovbuff->totalblk < 1) { + syslog(L_ERROR, "%s: too small length '%d' for buffindexed %s", + LocalLogName, ovbuff->len, ovbuff->path); + ovlock(ovbuff, LOCK_UNLOCK); + return FALSE; + } + rpx->version = OVBUFF_VERSION; + ovbuff->usedblk = 0; + ovbuff->freeblk = 0; + ovbuff->updated = 0; + ovbuff->dirty = 1; + syslog(L_NOTICE, + "%s: No magic cookie found for buffindexed %d, initializing", + LocalLogName, ovbuff->index); + ovflushhead(ovbuff); + } #ifdef OV_DEBUG - ovbuff->trace = NEW(struct ov_trace_array, ovbuff->totalblk); - memset(ovbuff->trace, '\0', sizeof(struct ov_trace_array) * ovbuff->totalblk); + ovbuff->trace = NEW(struct ov_trace_array, ovbuff->totalblk); + memset(ovbuff->trace, '\0', sizeof(struct ov_trace_array) * ovbuff->totalblk); #endif /* OV_DEBUG */ - ovlock(ovbuff, LOCK_UNLOCK); - } - return TRUE; + ovlock(ovbuff, LOCK_UNLOCK); + } + return TRUE; } -STATIC int ovusedblock(OVBUFF *ovbuff, int blocknum, BOOL set_operation, BOOL setbitvalue) { - OFFSET_T longoffset; - int bitoffset; /* From the 'left' side of the long */ - ULONG bitlong, mask; - - longoffset = blocknum / (sizeof(long) * 8); - bitoffset = blocknum % (sizeof(long) * 8); - bitlong = *((ULONG *) ovbuff->bitfield + (OV_BEFOREBITF / sizeof(long)) +STATIC int ovusedblock(OVBUFF *ovbuff, int blocknum, BOOL set_operation, + BOOL setbitvalue) +{ + OFFSET_T longoffset; + int bitoffset; /* From the 'left' side of the long */ + ULONG bitlong, mask; + + longoffset = blocknum / (sizeof(long) * 8); + bitoffset = blocknum % (sizeof(long) * 8); + bitlong = *((ULONG *) ovbuff->bitfield + (OV_BEFOREBITF / sizeof(long)) + longoffset); - if (set_operation) { - if (setbitvalue) { - mask = onarray[bitoffset]; - bitlong |= mask; - } else { - mask = offarray[bitoffset]; - bitlong &= mask; - } - *((ULONG *) ovbuff->bitfield + (OV_BEFOREBITF / sizeof(long)) - + longoffset) = bitlong; - return 2; /* XXX Clean up return semantics */ - } - /* It's a read operation */ - mask = onarray[bitoffset]; - /* return bitlong & mask; doesn't work if sizeof(ulong) > sizeof(int) */ - if ( bitlong & mask ) return 1; else return 0; + if (set_operation) { + if (setbitvalue) { + mask = onarray[bitoffset]; + bitlong |= mask; + } else { + mask = offarray[bitoffset]; + bitlong &= mask; + } + *((ULONG *) ovbuff->bitfield + (OV_BEFOREBITF / sizeof(long)) + + longoffset) = bitlong; + return 2; /* XXX Clean up return semantics */ + } + /* It's a read operation */ + mask = onarray[bitoffset]; + /* return bitlong & mask; doesn't work if sizeof(ulong) > sizeof(int) */ + if ( bitlong & mask ) return 1; else return 0; } +/* +** Update freeblk and nextchunk to the next ovblock. Set freeblk to +** tatalblk, if no space available. +*/ STATIC void ovnextblock(OVBUFF *ovbuff) { - int i, j, last, lastbit, left; - ULONG mask = 0x80000000; - ULONG *table; - - last = ovbuff->totalblk/(sizeof(long) * 8); - if ((left = ovbuff->totalblk % (sizeof(long) * 8)) != 0) { - last++; - } - table = ((ULONG *) ovbuff->bitfield + (OV_BEFOREBITF / sizeof(long))); - for (i = ovbuff->nextchunk ; i < last ; i++) { - if (i == last - 1 && left != 0) { - for (j = 1 ; j < left ; j++) { - mask |= mask >> 1; - } - if ((table[i] & mask) != mask) - break; - } else { - if ((table[i] ^ ~0) != 0) - break; + int i, j, last, lastbit, left; + ULONG mask = 0x80000000; + ULONG *table; + + last = ovbuff->totalblk/(sizeof(long) * 8); + if ((left = ovbuff->totalblk % (sizeof(long) * 8)) != 0) { + last++; + } + table = ((ULONG *) ovbuff->bitfield + (OV_BEFOREBITF / sizeof(long))); + for (i = ovbuff->nextchunk ; i < last ; i++) { + if (i == last - 1 && left != 0) { + for (j = 1 ; j < left ; j++) { + mask |= mask >> 1; + } + if ((table[i] & mask) != mask) + break; + } else { + if ((table[i] ^ ~0) != 0) + break; + } + } + if (i == last) { + for (i = 0 ; i < ovbuff->nextchunk ; i++) { + if ((table[i] ^ ~0) != 0) + break; + } + if (i == ovbuff->nextchunk) { + ovbuff->freeblk = ovbuff->totalblk; + return; + } } - } - if (i == last) { - for (i = 0 ; i < ovbuff->nextchunk ; i++) { - if ((table[i] ^ ~0) != 0) - break; + if ((i - 1) >= 0 && (last - 1 == i) && left != 0) { + lastbit = left; + } else { + lastbit = sizeof(long) * 8; } - if (i == ovbuff->nextchunk) { - ovbuff->freeblk = ovbuff->totalblk; - return; - } - } - if ((i - 1) >= 0 && (last - 1 == i) && left != 0) { - lastbit = left; - } else { - lastbit = sizeof(long) * 8; - } - for (j = 0 ; j < lastbit ; j++) { - if ((table[i] & onarray[j]) == 0) - break; - } - if (j == lastbit) { - ovbuff->freeblk = ovbuff->totalblk; + for (j = 0 ; j < lastbit ; j++) { + if ((table[i] & onarray[j]) == 0) + break; + } + if (j == lastbit) { + ovbuff->freeblk = ovbuff->totalblk; + return; + } + ovbuff->freeblk = i * sizeof(long) * 8 + j; + ovbuff->nextchunk = i + 1; + if (i == last) + ovbuff->nextchunk = 0; return; - } - ovbuff->freeblk = i * sizeof(long) * 8 + j; - ovbuff->nextchunk = i + 1; - if (i == last) - ovbuff->nextchunk = 0; - return; } STATIC OVBUFF *getovbuff(OV ov) { - OVBUFF *ovbuff = ovbufftab; - for (; ovbuff != (OVBUFF *)NULL; ovbuff = ovbuff->next) { - if (ovbuff->index == ov.index) - return ovbuff; - } - return NULL; + OVBUFF *ovbuff = ovbufftab; + for (; ovbuff != (OVBUFF *)NULL; ovbuff = ovbuff->next) { + if (ovbuff->index == ov.index) + return ovbuff; + } + return NULL; } #ifdef OV_DEBUG @@ -713,80 +835,120 @@ #else STATIC OV ovblocknew(void) { #endif /* OV_DEBUG */ - static OVBUFF *ovbuffnext = NULL; - OVBUFF *ovbuff; - OV ov; + static OVBUFF *ovbuffnext = NULL; + OVBUFF *ovbuff; + OV ov; + BOOL done = FALSE; +#ifdef OV_DEBUG + int recno; + struct ov_trace_array *trace; +#endif /* OV_DEBUG */ + + if (ovbuffnext == NULL) + ovbuffnext = ovbufftab; + + /* We will try to recover broken overview due to power failure */ + /* Recovering is not activated for OV_DEBUG mode */ + +retry: + for (ovbuff = ovbuffnext ; ovbuff != (OVBUFF *)NULL ; + ovbuff = ovbuff->next) + { + ovlock(ovbuff, LOCK_WRITE); + ovreadhead(ovbuff); + + if (ovbuff->totalblk != ovbuff->usedblk && + ovbuff->freeblk == ovbuff->totalblk) + { + ovnextblock(ovbuff); + } + if (ovbuff->totalblk == ovbuff->usedblk || + ovbuff->freeblk == ovbuff->totalblk) + { + /* no space left for this ovbuff */ + ovlock(ovbuff, LOCK_UNLOCK); + continue; + } + break; + } + if (ovbuff == NULL) { + for (ovbuff = ovbufftab ; ovbuff != ovbuffnext ; + ovbuff = ovbuff->next) + { + ovlock(ovbuff, LOCK_WRITE); + ovreadhead(ovbuff); + + if (ovbuff->totalblk == ovbuff->usedblk || + ovbuff->freeblk == ovbuff->totalblk) + { + /* no space left for this ovbuff */ + ovlock(ovbuff, LOCK_UNLOCK); + continue; + } + break; + } + if (ovbuff == ovbuffnext) { + Nospace = TRUE; + return ovnull; + } + } + #ifdef OV_DEBUG - int recno; - struct ov_trace_array *trace; -#endif /* OV_DEBUG */ + recno = ((char *)ge - (char *)&GROUPentries[0])/sizeof(GROUPENTRY); + if (ovusedblock(ovbuff, ovbuff->freeblk, FALSE, TRUE)) { + syslog(L_FATAL, "%s: 0x%08x trying to occupy new block(%d, %d)," + " but already occupied", + LocalLogName, recno, ovbuff->index, ovbuff->freeblk); + buffindexed_close(); + abort(); + } + trace = &ovbuff->trace[ovbuff->freeblk]; + if (trace->ov_trace == NULL) { + trace->ov_trace = NEW(struct ov_trace, OV_TRACENUM); + trace->max = OV_TRACENUM; + memset(trace->ov_trace, '\0', sizeof(struct ov_trace) * OV_TRACENUM); + } else if (trace->cur + 1 == trace->max) { + trace->max += OV_TRACENUM; + RENEW(trace->ov_trace, struct ov_trace, trace->max); + memset(&trace->ov_trace[trace->cur], '\0', + sizeof(struct ov_trace) * (trace->max - trace->cur)); + } + if (trace->ov_trace[trace->cur].occupied != 0) { + trace->cur++; + } + trace->ov_trace[trace->cur].gloc.recno = recno; + trace->ov_trace[trace->cur].occupied = time(NULL); + done = TRUE; +#endif /* OV_DEBUG */ + + ov.index = ovbuff->index; + ov.blocknum = ovbuff->freeblk; + +#ifndef OV_DEBUG + if (ovusedblock(ovbuff, ovbuff->freeblk, FALSE, TRUE)) { + syslog(L_NOTICE, "%s: fixing invalid free block(%d, %d)." + " This may happen on a corrupted overview", + LocalLogName, ovbuff->index, ovbuff->freeblk); + } else + done = TRUE; +#endif - if (ovbuffnext == NULL) - ovbuffnext = ovbufftab; - for (ovbuff = ovbuffnext ; ovbuff != (OVBUFF *)NULL ; ovbuff = ovbuff->next) { - ovlock(ovbuff, LOCK_WRITE); - ovreadhead(ovbuff); - if (ovbuff->totalblk != ovbuff->usedblk && ovbuff->freeblk == ovbuff->totalblk) { - ovnextblock(ovbuff); - } - if (ovbuff->totalblk == ovbuff->usedblk || ovbuff->freeblk == ovbuff->totalblk) { - /* no space left for this ovbuff */ - ovlock(ovbuff, LOCK_UNLOCK); - continue; - } - break; - } - if (ovbuff == NULL) { - for (ovbuff = ovbufftab ; ovbuff != ovbuffnext ; ovbuff = ovbuff->next) { - ovlock(ovbuff, LOCK_WRITE); - ovreadhead(ovbuff); - if (ovbuff->totalblk == ovbuff->usedblk || ovbuff->freeblk == ovbuff->totalblk) { - /* no space left for this ovbuff */ - ovlock(ovbuff, LOCK_UNLOCK); - continue; - } - break; - } - if (ovbuff == ovbuffnext) { - Nospace = TRUE; - return ovnull; - } - } -#ifdef OV_DEBUG - recno = ((char *)ge - (char *)&GROUPentries[0])/sizeof(GROUPENTRY); - if (ovusedblock(ovbuff, ovbuff->freeblk, FALSE, TRUE)) { - syslog(L_FATAL, "%s: 0x%08x trying to occupy new block(%d, %d), but already occupied", LocalLogName, recno, ovbuff->index, ovbuff->freeblk); - buffindexed_close(); - abort(); - } - trace = &ovbuff->trace[ovbuff->freeblk]; - if (trace->ov_trace == NULL) { - trace->ov_trace = NEW(struct ov_trace, OV_TRACENUM); - trace->max = OV_TRACENUM; - memset(trace->ov_trace, '\0', sizeof(struct ov_trace) * OV_TRACENUM); - } else if (trace->cur + 1 == trace->max) { - trace->max += OV_TRACENUM; - RENEW(trace->ov_trace, struct ov_trace, trace->max); - memset(&trace->ov_trace[trace->cur], '\0', sizeof(struct ov_trace) * (trace->max - trace->cur)); - } - if (trace->ov_trace[trace->cur].occupied != 0) { - trace->cur++; - } - trace->ov_trace[trace->cur].gloc.recno = recno; - trace->ov_trace[trace->cur].occupied = time(NULL); -#endif /* OV_DEBUG */ - ov.index = ovbuff->index; - ov.blocknum = ovbuff->freeblk; - ovusedblock(ovbuff, ov.blocknum, TRUE, TRUE); - ovnextblock(ovbuff); - ovbuff->usedblk++; - ovbuff->needflush = TRUE; - ovflushhead(ovbuff); - ovlock(ovbuff, LOCK_UNLOCK); - ovbuffnext = ovbuff->next; - if (ovbuffnext == NULL) - ovbuffnext = ovbufftab; - return ov; + /* mark it as allocated */ + ovusedblock(ovbuff, ov.blocknum, TRUE, TRUE); + + ovnextblock(ovbuff); + ovbuff->usedblk++; + ovbuff->dirty++; + ovflushhead(ovbuff); + ovlock(ovbuff, LOCK_UNLOCK); + ovbuffnext = ovbuff->next; + if (ovbuffnext == NULL) + ovbuffnext = ovbufftab; + + if ( !done ) + goto retry; + + return ov; } #ifdef OV_DEBUG @@ -794,382 +956,411 @@ #else STATIC void ovblockfree(OV ov) { #endif /* OV_DEBUG */ - OVBUFF *ovbuff; + OVBUFF *ovbuff; #ifdef OV_DEBUG - int recno; - struct ov_trace_array *trace; + int recno; + struct ov_trace_array *trace; #endif /* OV_DEBUG */ - if (ov.index == NULLINDEX) - return; - if ((ovbuff = getovbuff(ov)) == NULL) - return; - ovlock(ovbuff, LOCK_WRITE); + if (ov.index == NULLINDEX) + return; + if ((ovbuff = getovbuff(ov)) == NULL) + return; + + /* + * XXX - isn't it a late locking? Some one may release the + * ov.blocknum before we have the lock. To protect this, + * do not run multiple instances of expireover-like procs simultaneously. + */ + ovlock(ovbuff, LOCK_WRITE); #ifdef OV_DEBUG - recno = ((char *)ge - (char *)&GROUPentries[0])/sizeof(GROUPENTRY); - if (!ovusedblock(ovbuff, ov.blocknum, FALSE, FALSE)) { - syslog(L_FATAL, "%s: 0x%08x trying to free block(%d, %d), but already freed", LocalLogName, recno, ov.index, ov.blocknum); - buffindexed_close(); - abort(); - } - trace = &ovbuff->trace[ov.blocknum]; - if (trace->ov_trace == NULL) { - trace->ov_trace = NEW(struct ov_trace, OV_TRACENUM); - trace->max = OV_TRACENUM; - memset(trace->ov_trace, '\0', sizeof(struct ov_trace) * OV_TRACENUM); - } else if (trace->cur + 1 == trace->max) { - trace->max += OV_TRACENUM; - RENEW(trace->ov_trace, struct ov_trace, trace->max); - memset(&trace->ov_trace[trace->cur], '\0', sizeof(struct ov_trace) * (trace->max - trace->cur)); - } - if (trace->ov_trace[trace->cur].freed != 0) { + recno = ((char *)ge - (char *)&GROUPentries[0])/sizeof(GROUPENTRY); + if (!ovusedblock(ovbuff, ov.blocknum, FALSE, FALSE)) { + syslog(L_FATAL, "%s: 0x%08x trying to free block(%d, %d), but" + " already freed", LocalLogName, recno, ov.index, ov.blocknum); + buffindexed_close(); + abort(); + } + trace = &ovbuff->trace[ov.blocknum]; + if (trace->ov_trace == NULL) { + trace->ov_trace = NEW(struct ov_trace, OV_TRACENUM); + trace->max = OV_TRACENUM; + memset(trace->ov_trace, '\0', sizeof(struct ov_trace) * OV_TRACENUM); + } else if (trace->cur + 1 == trace->max) { + trace->max += OV_TRACENUM; + RENEW(trace->ov_trace, struct ov_trace, trace->max); + memset(&trace->ov_trace[trace->cur], '\0', + sizeof(struct ov_trace) * (trace->max - trace->cur)); + } + if (trace->ov_trace[trace->cur].freed != 0) { + trace->cur++; + } + trace->ov_trace[trace->cur].freed = time(NULL); + trace->ov_trace[trace->cur].gloc.recno = recno; trace->cur++; - } - trace->ov_trace[trace->cur].freed = time(NULL); - trace->ov_trace[trace->cur].gloc.recno = recno; - trace->cur++; -#endif /* OV_DEBUG */ - ovusedblock(ovbuff, ov.blocknum, TRUE, FALSE); - ovreadhead(ovbuff); - if (ovbuff->freeblk == ovbuff->totalblk) - ovbuff->freeblk = ov.blocknum; - ovbuff->usedblk--; - ovbuff->needflush = TRUE; - ovflushhead(ovbuff); - ovlock(ovbuff, LOCK_UNLOCK); - return; +#endif /* OV_DEBUG */ + +#ifndef OV_DEBUG + if (!ovusedblock(ovbuff, ov.blocknum, FALSE, FALSE)) { + syslog(L_NOTICE, "%s: trying to free block(%d, %d), but" + " already freed. This can happen on a corrupted overview", + LocalLogName, ov.index, ov.blocknum); + } +#endif + + ovusedblock(ovbuff, ov.blocknum, TRUE, FALSE); + ovreadhead(ovbuff); + if (ovbuff->freeblk == ovbuff->totalblk) + ovbuff->freeblk = ov.blocknum; + ovbuff->usedblk--; + ovbuff->dirty++; + ovflushhead(ovbuff); + ovlock(ovbuff, LOCK_UNLOCK); + return; } BOOL buffindexed_open(int mode) { - char dirname[1024]; - char *groupfn; - struct stat sb; - int i, flag = 0; - static int uninitialized = 1; - ULONG on, off; - - if (uninitialized) { - on = 1; - off = on; - off ^= ULONG_MAX; - for (i = (longsize * 8) - 1; i >= 0; i--) { - onarray[i] = on; - offarray[i] = off; - on <<= 1; - off = on; - off ^= ULONG_MAX; - } - uninitialized = 0; - } - ovbuffmode = mode; - if (pagesize == 0) { + char dirname[1024]; + char *groupfn; + struct stat sb; + int i, flag = 0; + static int uninitialized = 1; + ULONG on, off; + + if (uninitialized) { + on = 1; + off = on; + off ^= ULONG_MAX; + for (i = (longsize * 8) - 1; i >= 0; i--) { + onarray[i] = on; + offarray[i] = off; + on <<= 1; + off = on; + off ^= ULONG_MAX; + } + uninitialized = 0; + } + ovbuffmode = mode; + if (pagesize == 0) { #if defined(HAVE_GETPAGESIZE) - pagesize = getpagesize(); + pagesize = getpagesize(); #elif defined(_SC_PAGESIZE) - if ((pagesize = sysconf(_SC_PAGESIZE)) < 0) { - syslog(L_ERROR, "%s: sysconf(_SC_PAGESIZE) failed: %m", LocalLogName); - return FALSE; - } + if ((pagesize = sysconf(_SC_PAGESIZE)) < 0) { + syslog(L_ERROR, "%s: sysconf(_SC_PAGESIZE) failed: %m", LocalLogName); + return FALSE; + } #else - pagesize = 16384; + pagesize = 16384; #endif - if ((pagesize > OV_HDR_PAGESIZE) || (OV_HDR_PAGESIZE % pagesize)) { - syslog(L_ERROR, "%s: OV_HDR_PAGESIZE (%d) is not a multiple of pagesize (%d)", LocalLogName, OV_HDR_PAGESIZE, pagesize); - return FALSE; - } - } - memset(&groupdatablock, '\0', sizeof(groupdatablock)); - if (!ovbuffread_config()) { - return FALSE; - } - Needunlink = TRUE; - if (!ovbuffinit_disks()) { - return FALSE; - } - - strcpy(dirname, innconf->pathdb); - groupfn = NEW(char, strlen(dirname) + strlen("/group.index") + 1); - strcpy(groupfn, dirname); - strcat(groupfn, "/group.index"); - if (Needunlink && unlink(groupfn) == 0) { - syslog(L_NOTICE, "%s: all buffers are brandnew, unlink '%s'", LocalLogName, groupfn); - } - GROUPfd = open(groupfn, ovbuffmode & OV_WRITE ? O_RDWR | O_CREAT : O_RDONLY, 0660); - if (GROUPfd < 0) { - syslog(L_FATAL, "%s: Could not create %s: %m", LocalLogName, groupfn); - DISPOSE(groupfn); - return FALSE; - } + if ((pagesize > OV_HDR_PAGESIZE) || (OV_HDR_PAGESIZE % pagesize)) { + syslog(L_ERROR, "%s: OV_HDR_PAGESIZE (%d) is not a multiple of pagesize (%d)", LocalLogName, OV_HDR_PAGESIZE, pagesize); + return FALSE; + } + } + memset(&groupdatablock, '\0', sizeof(groupdatablock)); + if (!ovbuffread_config()) { + return FALSE; + } + Needunlink = TRUE; + if (!ovbuffinit_disks()) { + return FALSE; + } - if (fstat(GROUPfd, &sb) < 0) { - syslog(L_FATAL, "%s: Could not fstat %s: %m", LocalLogName, groupfn); - DISPOSE(groupfn); - close(GROUPfd); - return FALSE; - } - if (sb.st_size > sizeof(GROUPHEADER)) { - if (mode & OV_READ) - flag |= PROT_READ; - if (mode & OV_WRITE) { - /* - * Note: below mapping of groupheader won't work unless we have - * both PROT_READ and PROT_WRITE perms. - */ - flag |= PROT_WRITE|PROT_READ; + strcpy(dirname, innconf->pathdb); + groupfn = NEW(char, strlen(dirname) + strlen("/group.index") + 1); + strcpy(groupfn, dirname); + strcat(groupfn, "/group.index"); + + if (Needunlink && unlink(groupfn) == 0) { + syslog(L_NOTICE, "%s: all buffers are brandnew, unlink '%s'", + LocalLogName, groupfn); } - GROUPcount = (sb.st_size - sizeof(GROUPHEADER)) / sizeof(GROUPENTRY); - if ((GROUPheader = (GROUPHEADER *)mmap(0, GROUPfilesize(GROUPcount), flag, - MAP_SHARED, GROUPfd, 0)) == (GROUPHEADER *) -1) { - syslog(L_FATAL, "%s: Could not mmap %s in buffindexed_open: %m", LocalLogName, groupfn); - DISPOSE(groupfn); - close(GROUPfd); - return FALSE; + +/* + if (Needunlink) { + syslog(L_ERROR, "%s: Needunlink", LocalLogName); + exit(1); } - GROUPentries = (GROUPENTRY *)((char *)GROUPheader + sizeof(GROUPHEADER)); - } else { - GROUPcount = 0; - if (!GROUPexpand(mode)) { - DISPOSE(groupfn); - close(GROUPfd); - return FALSE; +*/ + + GROUPfd = open(groupfn, ovbuffmode & OV_WRITE ? + O_RDWR | O_CREAT : O_RDONLY, 0660); + if (GROUPfd < 0) { + syslog(L_FATAL, "%s: Could not create %s: %m", LocalLogName, groupfn); + DISPOSE(groupfn); + return FALSE; } - } - CloseOnExec(GROUPfd, 1); - DISPOSE(groupfn); - Cutofflow = FALSE; + if (fstat(GROUPfd, &sb) < 0) { + syslog(L_FATAL, "%s: Could not fstat %s: %m", LocalLogName, groupfn); + DISPOSE(groupfn); + close(GROUPfd); + return FALSE; + } + if (sb.st_size > sizeof(GROUPHEADER)) { + if (mode & OV_READ) + flag |= PROT_READ; + if (mode & OV_WRITE) { + /* + * Note: below mapping of groupheader won't work unless we have + * both PROT_READ and PROT_WRITE perms. + */ + flag |= PROT_WRITE|PROT_READ; + } + GROUPcount = (sb.st_size - sizeof(GROUPHEADER)) / sizeof(GROUPENTRY); + if ((GROUPheader = (GROUPHEADER *)mmap(0, GROUPfilesize(GROUPcount), flag, + MAP_SHARED, GROUPfd, 0)) == (GROUPHEADER *) -1) { + syslog(L_FATAL, "%s: Could not mmap %s in buffindexed_open: %m", LocalLogName, groupfn); + DISPOSE(groupfn); + close(GROUPfd); + return FALSE; + } + GROUPentries = (GROUPENTRY *)((char *)GROUPheader + sizeof(GROUPHEADER)); + } else { + GROUPcount = 0; + if (!GROUPexpand(mode)) { + DISPOSE(groupfn); + close(GROUPfd); + return FALSE; + } + } + CloseOnExec(GROUPfd, 1); + + DISPOSE(groupfn); + Cutofflow = FALSE; - return TRUE; + return TRUE; } STATIC GROUPLOC GROUPfind(char *group, BOOL Ignoredeleted) { - HASH grouphash; - unsigned int i; - GROUPLOC loc; - - grouphash = Hash(group, strlen(group)); - memcpy(&i, &grouphash, sizeof(i)); - - loc = GROUPheader->hash[i % GROUPHEADERHASHSIZE]; - GROUPremapifneeded(loc); - - while (!GROUPLOCempty(loc)) { - if (GROUPentries[loc.recno].deleted == 0 || Ignoredeleted) { - if (memcmp(&grouphash, &GROUPentries[loc.recno].hash, sizeof(HASH)) == 0) { - return loc; - } - } - loc = GROUPentries[loc.recno].next; - } - return GROUPemptyloc; + HASH grouphash; + unsigned int i; + GROUPLOC loc; + + grouphash = Hash(group, strlen(group)); + memcpy(&i, &grouphash, sizeof(i)); + + loc = GROUPheader->hash[i % GROUPHEADERHASHSIZE]; + GROUPremapifneeded(loc); + + while (!GROUPLOCempty(loc)) { + if (GROUPentries[loc.recno].deleted == 0 || Ignoredeleted) { + if (memcmp(&grouphash, &GROUPentries[loc.recno].hash, sizeof(HASH)) == 0) { + return loc; + } + } + loc = GROUPentries[loc.recno].next; + } + return GROUPemptyloc; } BOOL buffindexed_groupstats(char *group, int *lo, int *hi, int *count, int *flag) { - GROUPLOC gloc; + GROUPLOC gloc; - gloc = GROUPfind(group, FALSE); - if (GROUPLOCempty(gloc)) { - return FALSE; - } - GROUPlock(gloc, LOCK_READ); - if (lo != NULL) - *lo = GROUPentries[gloc.recno].low; - if (hi != NULL) - *hi = GROUPentries[gloc.recno].high; - if (count != NULL) - *count = GROUPentries[gloc.recno].count; - if (flag != NULL) - *flag = GROUPentries[gloc.recno].flag; - GROUPlock(gloc, LOCK_UNLOCK); - return TRUE; + gloc = GROUPfind(group, FALSE); + if (GROUPLOCempty(gloc)) { + return FALSE; + } + GROUPlock(gloc, LOCK_READ); + if (lo != NULL) + *lo = GROUPentries[gloc.recno].low; + if (hi != NULL) + *hi = GROUPentries[gloc.recno].high; + if (count != NULL) + *count = GROUPentries[gloc.recno].count; + if (flag != NULL) + *flag = GROUPentries[gloc.recno].flag; + GROUPlock(gloc, LOCK_UNLOCK); + return TRUE; } STATIC void setinitialge(GROUPENTRY *ge, HASH grouphash, char *flag, GROUPLOC next, ARTNUM lo, ARTNUM hi) { - ge->hash = grouphash; - if (lo != 0) - ge->low = lo; - ge->high = hi; - ge->expired = ge->deleted = ge->count = 0; - ge->flag = *flag; - ge->baseindex = ge->curindex = ge->curdata = ovnull; - ge->curindexoffset = ge->curoffset = 0; - ge->next = next; -} - -BOOL buffindexed_groupadd(char *group, ARTNUM lo, ARTNUM hi, char *flag) { - unsigned int i; - HASH grouphash; - GROUPLOC gloc; - GROUPENTRY *ge; - void *handle; + ge->hash = grouphash; + if (lo != 0) + ge->low = lo; + ge->high = hi; + ge->expired = ge->deleted = ge->count = 0; + ge->flag = *flag; + ge->baseindex = ge->curindex = ge->curdata = ovnull; + ge->curindexoffset = ge->curoffset = 0; + ge->next = next; +} + +BOOL buffindexed_groupadd(char *group, ARTNUM lo, ARTNUM hi, char *flag) +{ + unsigned int i; + HASH grouphash; + GROUPLOC gloc; + GROUPENTRY *ge; + void *handle; #ifdef OV_DEBUG - struct ov_name_table *ntp; + struct ov_name_table *ntp; #endif /* OV_DEBUG */ - gloc = GROUPfind(group, TRUE); - if (!GROUPLOCempty(gloc)) { + gloc = GROUPfind(group, TRUE); + if (!GROUPLOCempty(gloc)) { + ge = &GROUPentries[gloc.recno]; + if (GROUPentries[gloc.recno].deleted != 0) { + grouphash = Hash(group, strlen(group)); + setinitialge(ge, grouphash, flag, ge->next, lo, hi); + } else { + ge->flag = *flag; + } + return TRUE; + } + grouphash = Hash(group, strlen(group)); + memcpy(&i, &grouphash, sizeof(i)); + i = i % GROUPHEADERHASHSIZE; + GROUPlockhash(LOCK_WRITE); + gloc = GROUPnewnode(); ge = &GROUPentries[gloc.recno]; - if (GROUPentries[gloc.recno].deleted != 0) { - grouphash = Hash(group, strlen(group)); - setinitialge(ge, grouphash, flag, ge->next, lo, hi); - } else { - ge->flag = *flag; + setinitialge(ge, grouphash, flag, GROUPheader->hash[i], lo, hi); + GROUPheader->hash[i] = gloc; +#ifdef OV_DEBUG + ntp = NEW(struct ov_name_table, 1); + memset(ntp, '\0', sizeof(struct ov_name_table)); + ntp->name = COPY(group); + ntp->recno = gloc.recno; + if (name_table == NULL) + name_table = ntp; + else { + ntp->next = name_table; + name_table = ntp; } - return TRUE; - } - grouphash = Hash(group, strlen(group)); - memcpy(&i, &grouphash, sizeof(i)); - i = i % GROUPHEADERHASHSIZE; - GROUPlockhash(LOCK_WRITE); - gloc = GROUPnewnode(); - ge = &GROUPentries[gloc.recno]; - setinitialge(ge, grouphash, flag, GROUPheader->hash[i], lo, hi); - GROUPheader->hash[i] = gloc; -#ifdef OV_DEBUG - ntp = NEW(struct ov_name_table, 1); - memset(ntp, '\0', sizeof(struct ov_name_table)); - ntp->name = COPY(group); - ntp->recno = gloc.recno; - if (name_table == NULL) - name_table = ntp; - else { - ntp->next = name_table; - name_table = ntp; - } #endif /* OV_DEBUG */ - GROUPlockhash(LOCK_UNLOCK); - return TRUE; + GROUPlockhash(LOCK_UNLOCK); + return TRUE; } STATIC BOOL GROUPfilesize(int count) { - return (count * sizeof(GROUPENTRY)) + sizeof(GROUPHEADER); + return (count * sizeof(GROUPENTRY)) + sizeof(GROUPHEADER); } /* Check if the given GROUPLOC refers to GROUPENTRY that we don't have mmap'ed, ** if so then see if the file has been grown by another writer and remmap */ STATIC BOOL GROUPremapifneeded(GROUPLOC loc) { - struct stat sb; + struct stat sb; - if (loc.recno < GROUPcount) - return TRUE; + if (loc.recno < GROUPcount) + return TRUE; - if (fstat(GROUPfd, &sb) < 0) - return FALSE; + if (fstat(GROUPfd, &sb) < 0) + return FALSE; - if (GROUPfilesize(GROUPcount) >= sb.st_size) - return TRUE; + if (GROUPfilesize(GROUPcount) >= sb.st_size) + return TRUE; - if (GROUPheader) { - if (munmap((void *)GROUPheader, GROUPfilesize(GROUPcount)) < 0) { - syslog(L_FATAL, "%s: Could not munmap group.index in GROUPremapifneeded: %m", LocalLogName); - return FALSE; + if (GROUPheader) { + if (munmap((void *)GROUPheader, GROUPfilesize(GROUPcount)) < 0) { + syslog(L_FATAL, "%s: Could not munmap group.index in GROUPremapifneeded: %m", LocalLogName); + return FALSE; + } } - } - GROUPcount = (sb.st_size - sizeof(GROUPHEADER)) / sizeof(GROUPENTRY); - GROUPheader = (GROUPHEADER *)mmap(0, GROUPfilesize(GROUPcount), + GROUPcount = (sb.st_size - sizeof(GROUPHEADER)) / sizeof(GROUPENTRY); + GROUPheader = (GROUPHEADER *)mmap(0, GROUPfilesize(GROUPcount), PROT_READ | PROT_WRITE, MAP_SHARED, GROUPfd, 0); - if (GROUPheader == (GROUPHEADER *) -1) { - syslog(L_FATAL, "%s: Could not mmap group.index in GROUPremapifneeded: %m", LocalLogName); - return FALSE; - } - GROUPentries = (GROUPENTRY *)((char *)GROUPheader + sizeof(GROUPHEADER)); - return TRUE; + if (GROUPheader == (GROUPHEADER *) -1) { + syslog(L_FATAL, "%s: Could not mmap group.index in GROUPremapifneeded: %m", LocalLogName); + return FALSE; + } + GROUPentries = (GROUPENTRY *)((char *)GROUPheader + sizeof(GROUPHEADER)); + return TRUE; } /* This function does not need to lock because it's callers are expected to do so */ STATIC BOOL GROUPexpand(int mode) { - int i; - int flag = 0; + int i; + int flag = 0; - if (GROUPheader) { - if (munmap((void *)GROUPheader, GROUPfilesize(GROUPcount)) < 0) { - syslog(L_FATAL, "%s: Could not munmap group.index in GROUPexpand: %m", LocalLogName); - return FALSE; - } - } - GROUPcount += 1024; - if (ftruncate(GROUPfd, GROUPfilesize(GROUPcount)) < 0) { - syslog(L_FATAL, "%s: Could not extend group.index: %m", LocalLogName); - return FALSE; - } - if (mode & OV_READ) - flag |= PROT_READ; - if (mode & OV_WRITE) { - /* - * Note: below check of magic won't work unless we have both PROT_READ - * and PROT_WRITE perms. - */ - flag |= PROT_WRITE|PROT_READ; - } - GROUPheader = (GROUPHEADER *)mmap(0, GROUPfilesize(GROUPcount), + if (GROUPheader) { + if (munmap((void *)GROUPheader, GROUPfilesize(GROUPcount)) < 0) { + syslog(L_FATAL, "%s: Could not munmap group.index in GROUPexpand: %m", LocalLogName); + return FALSE; + } + } + GROUPcount += 1024; + if (ftruncate(GROUPfd, GROUPfilesize(GROUPcount)) < 0) { + syslog(L_FATAL, "%s: Could not extend group.index: %m", LocalLogName); + return FALSE; + } + if (mode & OV_READ) + flag |= PROT_READ; + if (mode & OV_WRITE) { + /* + * Note: below check of magic won't work unless we have both PROT_READ + * and PROT_WRITE perms. + */ + flag |= PROT_WRITE|PROT_READ; + } + GROUPheader = (GROUPHEADER *)mmap(0, GROUPfilesize(GROUPcount), flag, MAP_SHARED, GROUPfd, 0); - if (GROUPheader == (GROUPHEADER *) -1) { - syslog(L_FATAL, "%s: Could not mmap group.index in GROUPexpand: %m", LocalLogName); - return FALSE; - } - GROUPentries = (GROUPENTRY *)((char *)GROUPheader + sizeof(GROUPHEADER)); - if (GROUPheader->magic != GROUPHEADERMAGIC) { - GROUPheader->magic = GROUPHEADERMAGIC; - GROUPLOCclear(&GROUPheader->freelist); - for (i = 0; i < GROUPHEADERHASHSIZE; i++) - GROUPLOCclear(&GROUPheader->hash[i]); - } - /* Walk the new entries from the back to the front, adding them to the freelist */ - for (i = GROUPcount - 1; (GROUPcount - 1024) <= i; i--) { - GROUPentries[i].next = GROUPheader->freelist; - GROUPheader->freelist.recno = i; - } - return TRUE; + if (GROUPheader == (GROUPHEADER *) -1) { + syslog(L_FATAL, "%s: Could not mmap group.index in GROUPexpand: %m", LocalLogName); + return FALSE; + } + GROUPentries = (GROUPENTRY *)((char *)GROUPheader + sizeof(GROUPHEADER)); + if (GROUPheader->magic != GROUPHEADERMAGIC) { + GROUPheader->magic = GROUPHEADERMAGIC; + GROUPLOCclear(&GROUPheader->freelist); + for (i = 0; i < GROUPHEADERHASHSIZE; i++) + GROUPLOCclear(&GROUPheader->hash[i]); + } + /* Walk the new entries from the back to the front, adding them to the freelist */ + for (i = GROUPcount - 1; (GROUPcount - 1024) <= i; i--) { + GROUPentries[i].next = GROUPheader->freelist; + GROUPheader->freelist.recno = i; + } + return TRUE; } STATIC GROUPLOC GROUPnewnode(void) { - GROUPLOC loc; + GROUPLOC loc; - /* If we didn't find any free space, then make some */ - if (GROUPLOCempty(GROUPheader->freelist)) { - if (!GROUPexpand(ovbuffmode)) { - return GROUPemptyloc; - } - } - assert(!GROUPLOCempty(GROUPheader->freelist)); - loc = GROUPheader->freelist; - GROUPheader->freelist = GROUPentries[GROUPheader->freelist.recno].next; - return loc; + /* If we didn't find any free space, then make some */ + if (GROUPLOCempty(GROUPheader->freelist)) { + if (!GROUPexpand(ovbuffmode)) { + return GROUPemptyloc; + } + } + assert(!GROUPLOCempty(GROUPheader->freelist)); + loc = GROUPheader->freelist; + GROUPheader->freelist = GROUPentries[GROUPheader->freelist.recno].next; + return loc; } BOOL buffindexed_groupdel(char *group) { - GROUPLOC gloc; - GROUPENTRY *ge; - void *handle; - - gloc = GROUPfind(group, FALSE); - if (GROUPLOCempty(gloc)) { + GROUPLOC gloc; + GROUPENTRY *ge; + void *handle; + + gloc = GROUPfind(group, FALSE); + if (GROUPLOCempty(gloc)) { + return TRUE; + } + GROUPlock(gloc, LOCK_WRITE); + ge = &GROUPentries[gloc.recno]; + ge->deleted = time(NULL); + HashClear(&ge->hash); + GROUPlock(gloc, LOCK_UNLOCK); return TRUE; - } - GROUPlock(gloc, LOCK_WRITE); - ge = &GROUPentries[gloc.recno]; - ge->deleted = time(NULL); - HashClear(&ge->hash); - GROUPlock(gloc, LOCK_UNLOCK); - return TRUE; } STATIC void GROUPLOCclear(GROUPLOC *loc) { - loc->recno = -1; + loc->recno = -1; } STATIC BOOL GROUPLOCempty(GROUPLOC loc) { - return (loc.recno < 0); + return (loc.recno < 0); } STATIC BOOL GROUPlockhash(LOCKTYPE type) { - return LockRange(GROUPfd, type, TRUE, 0, sizeof(GROUPHEADER)); + return LockRange(GROUPfd, type, TRUE, 0, sizeof(GROUPHEADER)); } STATIC BOOL GROUPlock(GROUPLOC gloc, LOCKTYPE type) { - return LockRange(GROUPfd, + return LockRange(GROUPfd, type, TRUE, sizeof(GROUPHEADER) + (sizeof(GROUPENTRY) * gloc.recno), @@ -1181,240 +1372,278 @@ #else STATIC BOOL ovsetcurindexblock(GROUPENTRY *ge) { #endif /* OV_DEBUG */ - OVBUFF *ovbuff; - OV ov; - int delta, i; - OVINDEXHEAD ovindexhead; - - /* there is no index */ -#ifdef OV_DEBUG - ov = ovblocknew(georig ? georig : ge); -#else - ov = ovblocknew(); -#endif /* OV_DEBUG */ - if (ov.index == NULLINDEX) { - syslog(L_ERROR, "%s: ovsetcurindexblock could not get new block", LocalLogName); - return FALSE; - } - if ((ovbuff = getovbuff(ov)) == NULL) { - syslog(L_ERROR, "%s: ovsetcurindexblock could not get ovbuff block for new, %d, %d", LocalLogName, ov.index, ov.blocknum); - return FALSE; - } - ovindexhead.next = ovnull; - ovindexhead.low = 0; - ovindexhead.high = 0; -#ifdef MMAP_MISSES_WRITES - if (mmapwrite(ovbuff->fd, &ovindexhead, sizeof(OVINDEXHEAD), ovbuff->base + ov.blocknum * OV_BLOCKSIZE) != sizeof(OVINDEXHEAD)) { -#else - if (pwrite(ovbuff->fd, &ovindexhead, sizeof(OVINDEXHEAD), ovbuff->base + ov.blocknum * OV_BLOCKSIZE) != sizeof(OVINDEXHEAD)) { -#endif /* MMAP_MISSES_WRITES */ - syslog(L_ERROR, "%s: could not write index record index '%d', blocknum '%d': %m", LocalLogName, ge->curindex.index, ge->curindex.blocknum); - return TRUE; - } - if (ge->baseindex.index == NULLINDEX) { - ge->baseindex = ov; - } else { - if ((ovbuff = getovbuff(ge->curindex)) == NULL) - return FALSE; -#ifdef OV_DEBUG - if (!ovusedblock(ovbuff, ge->curindex.blocknum, FALSE, FALSE)) { - syslog(L_FATAL, "%s: block(%d, %d) not occupied (index)", LocalLogName, ovbuff->index, ge->curindex.blocknum); - abort(); - } -#endif /* OV_DEBUG */ - ovindexhead.next = ov; - ovindexhead.low = ge->curlow; - ovindexhead.high = ge->curhigh; -#ifdef MMAP_MISSES_WRITES - if (mmapwrite(ovbuff->fd, &ovindexhead, sizeof(OVINDEXHEAD), ovbuff->base + ge->curindex.blocknum * OV_BLOCKSIZE) != sizeof(OVINDEXHEAD)) { -#else - if (pwrite(ovbuff->fd, &ovindexhead, sizeof(OVINDEXHEAD), ovbuff->base + ge->curindex.blocknum * OV_BLOCKSIZE) != sizeof(OVINDEXHEAD)) { -#endif /* MMAP_MISSES_WRITES */ - syslog(L_ERROR, "%s: could not write index record index '%d', blocknum '%d': %m", LocalLogName, ge->curindex.index, ge->curindex.blocknum); - return FALSE; - } - } - ge->curindex = ov; - ge->curindexoffset = 0; - ge->curlow = 0; - ge->curhigh = 0; - return TRUE; -} + OVBUFF *ovbuff; + OV ov; + int delta, i; + OVINDEXHEAD ovindexhead; -#ifdef OV_DEBUG -STATIC BOOL ovaddrec(GROUPENTRY *ge, ARTNUM artnum, TOKEN token, char *data, int len, time_t arrived, time_t expires, GROUPENTRY *georig) { -#else -STATIC BOOL ovaddrec(GROUPENTRY *ge, ARTNUM artnum, TOKEN token, char *data, int len, time_t arrived, time_t expires) { -#endif /* OV_DEBUG */ - OV ov; - OVINDEX ie; - OVBUFF *ovbuff; - OVINDEXHEAD ovindexhead; - BOOL needupdate = FALSE; -#ifdef OV_DEBUG - int recno; -#endif /* OV_DEBUG */ - - Nospace = FALSE; - if (OV_BLOCKSIZE < len) { - syslog(L_ERROR, "%s: overview data must be under %d", LocalLogName, OV_BLOCKSIZE); - return FALSE; - } - if (ge->curdata.index == NULLINDEX) { - /* no data block allocated */ + /* there is no index */ #ifdef OV_DEBUG ov = ovblocknew(georig ? georig : ge); #else ov = ovblocknew(); #endif /* OV_DEBUG */ if (ov.index == NULLINDEX) { - syslog(L_ERROR, "%s: ovaddrec could not get new block", LocalLogName); - return FALSE; + syslog(L_ERROR, "%s: ovsetcurindexblock could not get new block", + LocalLogName); + return FALSE; } if ((ovbuff = getovbuff(ov)) == NULL) { - syslog(L_ERROR, "%s: ovaddrec could not get ovbuff block for new, %d, %d, %d", LocalLogName, ov.index, ov.blocknum, artnum); - return FALSE; + syslog(L_ERROR, "%s: ovsetcurindexblock could not get ovbuff block for new, %d, %d", LocalLogName, ov.index, ov.blocknum); + return FALSE; } - ge->curdata = ov; - ge->curoffset = 0; - } else if ((ovbuff = getovbuff(ge->curdata)) == NULL) - return FALSE; - else if (OV_BLOCKSIZE - ge->curoffset < len) { - /* too short to store data, allocate new block */ + ovindexhead.next = ovnull; + ovindexhead.low = 0; + ovindexhead.high = 0; + if (PWRITE(ovbuff->fd, &ovindexhead, sizeof(OVINDEXHEAD), + ovbuff->base + ov.blocknum * OV_BLOCKSIZE) != sizeof(OVINDEXHEAD)) + { + syslog(L_ERROR, "%s: could not write index record index '%d'," + " blocknum '%d': %m", LocalLogName, ge->curindex.index, + ge->curindex.blocknum); + return TRUE; + } + if (ge->baseindex.index == NULLINDEX) { + ge->baseindex = ov; + } else { + if ((ovbuff = getovbuff(ge->curindex)) == NULL) + return FALSE; + if (!ovusedblock(ovbuff, ge->curindex.blocknum, FALSE, FALSE)) { +#ifdef OV_DEBUG + syslog(L_FATAL, "%s: block(%d, %d) not occupied (index)", + LocalLogName, ovbuff->index, ge->curindex.blocknum); + abort(); +#else + syslog(L_ERROR, "%s: block(%d, %d) not occupied (index)", + LocalLogName, ovbuff->index, ge->curindex.blocknum); + /* fix it */ + ovusedblock(ovbuff, ge->curindex.blocknum, TRUE, TRUE); +#endif /* OV_DEBUG */ + } + ovindexhead.next = ov; + ovindexhead.low = ge->curlow; + ovindexhead.high = ge->curhigh; + if (PWRITE(ovbuff->fd, &ovindexhead, sizeof(OVINDEXHEAD), + ovbuff->base + ge->curindex.blocknum * OV_BLOCKSIZE) != + sizeof(OVINDEXHEAD)) + { + syslog(L_ERROR, "%s: could not write index record index '%d'," + " blocknum '%d': %m", LocalLogName, + ge->curindex.index, ge->curindex.blocknum); + return FALSE; + } + } + ge->curindex = ov; + ge->curindexoffset = 0; + ge->curlow = 0; + ge->curhigh = 0; + return TRUE; +} + #ifdef OV_DEBUG - ov = ovblocknew(georig ? georig : ge); +STATIC BOOL ovaddrec(GROUPENTRY *ge, ARTNUM artnum, TOKEN token, char *data, + int len, time_t arrived, time_t expires, GROUPENTRY *georig) #else - ov = ovblocknew(); +STATIC BOOL ovaddrec(GROUPENTRY *ge, ARTNUM artnum, TOKEN token, char *data, + int len, time_t arrived, time_t expires) #endif /* OV_DEBUG */ - if (ov.index == NULLINDEX) { - syslog(L_ERROR, "%s: ovaddrec could not get new block", LocalLogName); - return FALSE; - } - if ((ovbuff = getovbuff(ov)) == NULL) { - syslog(L_ERROR, "%s: ovaddrec could not get ovbuff block for new, %d, %d, %d", LocalLogName, ov.index, ov.blocknum, artnum); - return FALSE; +{ + OV ov; + OVINDEX ie; + OVBUFF *ovbuff; + OVINDEXHEAD ovindexhead; + BOOL needupdate = FALSE; +#ifdef OV_DEBUG + int recno; +#endif /* OV_DEBUG */ + + Nospace = FALSE; + if (OV_BLOCKSIZE < len) { + syslog(L_ERROR, "%s: overview data must be under %d", + LocalLogName, OV_BLOCKSIZE); + return FALSE; } - ge->curdata = ov; - ge->curoffset = 0; - } -#ifdef OV_DEBUG - if (!ovusedblock(ovbuff, ge->curdata.blocknum, FALSE, FALSE)) { - syslog(L_FATAL, "%s: block(%d, %d) not occupied", LocalLogName, ovbuff->index, ge->curdata.blocknum); - buffindexed_close(); - abort(); - } -#endif /* OV_DEBUG */ -#ifdef MMAP_MISSES_WRITES - if (mmapwrite(ovbuff->fd, data, len, ovbuff->base + ge->curdata.blocknum * OV_BLOCKSIZE + ge->curoffset) != len) { + if (ge->curdata.index == NULLINDEX) { + /* no data block allocated */ +#ifdef OV_DEBUG + ov = ovblocknew(georig ? georig : ge); #else - if (pwrite(ovbuff->fd, data, len, ovbuff->base + ge->curdata.blocknum * OV_BLOCKSIZE + ge->curoffset) != len) { -#endif /* MMAP_MISSES_WRITES */ - syslog(L_ERROR, "%s: could not append overview record index '%d', blocknum '%d': %m", LocalLogName, ge->curdata.index, ge->curdata.blocknum); - return FALSE; - } - memset(&ie, '\0', sizeof(ie)); - ie.artnum = artnum; - ie.len = len; - ie.index = ge->curdata.index; - ie.blocknum = ge->curdata.blocknum; - ie.offset = ge->curoffset; - ie.token = token; - ie.arrived = arrived; - ie.expires = expires; - - if (ge->baseindex.index == NULLINDEX || ge->curindexoffset == OVINDEXMAX) { -#ifdef OV_DEBUG - if (!ovsetcurindexblock(ge, georig)) { -#else - if (!ovsetcurindexblock(ge)) { -#endif /* OV_DEBUG */ - syslog(L_ERROR, "%s: could not set current index", LocalLogName); - return FALSE; - } - } - if ((ovbuff = getovbuff(ge->curindex)) == NULL) - return FALSE; -#ifdef OV_DEBUG - if (!ovusedblock(ovbuff, ge->curindex.blocknum, FALSE, FALSE)) { - syslog(L_FATAL, "%s: block(%d, %d) not occupied (index)", LocalLogName, ovbuff->index, ge->curindex.blocknum); - buffindexed_close(); - abort(); - } + ov = ovblocknew(); #endif /* OV_DEBUG */ -#ifdef MMAP_MISSES_WRITES - if (mmapwrite(ovbuff->fd, &ie, sizeof(ie), ovbuff->base + ge->curindex.blocknum * OV_BLOCKSIZE + sizeof(OVINDEXHEAD) + sizeof(ie) * ge->curindexoffset) != sizeof(ie)) { -#else - if (pwrite(ovbuff->fd, &ie, sizeof(ie), ovbuff->base + ge->curindex.blocknum * OV_BLOCKSIZE + sizeof(OVINDEXHEAD) + sizeof(ie) * ge->curindexoffset) != sizeof(ie)) { -#endif /* MMAP_MISSES_WRITES */ - syslog(L_ERROR, "%s: could not write index record index '%d', blocknum '%d': %m", LocalLogName, ge->curindex.index, ge->curindex.blocknum); - return TRUE; - } - if ((ge->curlow <= 0) || (ge->curlow > artnum)) { - ge->curlow = artnum; - needupdate = TRUE; - } - if ((ge->curhigh <= 0) || (ge->curhigh < artnum)) { - ge->curhigh = artnum; - needupdate = TRUE; - } - if (needupdate) { - ovindexhead.next = ovnull; - ovindexhead.low = ge->curlow; - ovindexhead.high = ge->curhigh; -#ifdef MMAP_MISSES_WRITES - if (mmapwrite(ovbuff->fd, &ovindexhead, sizeof(OVINDEXHEAD), ovbuff->base + ge->curindex.blocknum * OV_BLOCKSIZE) != sizeof(OVINDEXHEAD)) { + if (ov.index == NULLINDEX) { + syslog(L_ERROR, "%s: ovaddrec could not get new block", + LocalLogName); + return FALSE; + } + if ((ovbuff = getovbuff(ov)) == NULL) { + syslog(L_ERROR, "%s: ovaddrec could not get ovbuff block for" + " new, %d, %d, %d", + LocalLogName, ov.index, ov.blocknum, artnum); + return FALSE; + } + ge->curdata = ov; + ge->curoffset = 0; + } else if ((ovbuff = getovbuff(ge->curdata)) == NULL) + return FALSE; + else if (OV_BLOCKSIZE - ge->curoffset < len) { + /* too short to store data, allocate new block */ +#ifdef OV_DEBUG + ov = ovblocknew(georig ? georig : ge); #else - if (pwrite(ovbuff->fd, &ovindexhead, sizeof(OVINDEXHEAD), ovbuff->base + ge->curindex.blocknum * OV_BLOCKSIZE) != sizeof(OVINDEXHEAD)) { -#endif /* MMAP_MISSES_WRITES */ - syslog(L_ERROR, "%s: could not write index record index '%d', blocknum '%d': %m", LocalLogName, ge->curindex.index, ge->curindex.blocknum); - return TRUE; + ov = ovblocknew(); +#endif /* OV_DEBUG */ + if (ov.index == NULLINDEX) { + syslog(L_ERROR, "%s: ovaddrec could not get new block", + LocalLogName); + return FALSE; + } + if ((ovbuff = getovbuff(ov)) == NULL) { + syslog(L_ERROR, "%s: ovaddrec could not get ovbuff block for" + " new, %d, %d, %d", + LocalLogName, ov.index, ov.blocknum, artnum); + return FALSE; + } + ge->curdata = ov; + ge->curoffset = 0; + } + /* curdata.blocknum must be allocated */ + if (!ovusedblock(ovbuff, ge->curdata.blocknum, FALSE, FALSE)) { +#ifdef OV_DEBUG + syslog(L_FATAL, "%s: block(%d, %d) not occupied", + LocalLogName, ovbuff->index, ge->curdata.blocknum); + buffindexed_close(); + abort(); +#else + syslog(L_ERROR, "%s: block(%d, %d) not occupied", + LocalLogName, ovbuff->index, ge->curdata.blocknum); + /* fix it */ + ovusedblock(ovbuff, ge->curdata.blocknum, TRUE, TRUE); +#endif /* OV_DEBUG */ + } + if (PWRITE(ovbuff->fd, data, len, ovbuff->base + + ge->curdata.blocknum * OV_BLOCKSIZE + ge->curoffset) != len) + { + syslog(L_ERROR, "%s: could not append overview record index '%d'," + " blocknum '%d': %m", + LocalLogName, ge->curdata.index, ge->curdata.blocknum); + return FALSE; } - } - if ((ge->low <= 0) || (ge->low > artnum)) - ge->low = artnum; - if ((ge->high <= 0) || (ge->high < artnum)) - ge->high = artnum; - ge->curindexoffset++; - ge->curoffset += len; - ge->count++; - return TRUE; + memset(&ie, '\0', sizeof(ie)); + ie.artnum = artnum; + ie.len = len; + ie.index = ge->curdata.index; + ie.blocknum = ge->curdata.blocknum; + ie.offset = ge->curoffset; + ie.token = token; + ie.arrived = arrived; + ie.expires = expires; + + if (ge->baseindex.index == NULLINDEX || ge->curindexoffset == OVINDEXMAX) { +#ifdef OV_DEBUG + if (!ovsetcurindexblock(ge, georig)) { +#else + if (!ovsetcurindexblock(ge)) { +#endif /* OV_DEBUG */ + syslog(L_ERROR, "%s: could not set current index", LocalLogName); + return FALSE; + } + } + if ((ovbuff = getovbuff(ge->curindex)) == NULL) + return FALSE; + if (!ovusedblock(ovbuff, ge->curindex.blocknum, FALSE, FALSE)) { +#ifdef OV_DEBUG + syslog(L_FATAL, "%s: block(%d, %d) not occupied (index)", + LocalLogName, ovbuff->index, ge->curindex.blocknum); + buffindexed_close(); + abort(); +#else + syslog(L_ERROR, "%s: block(%d, %d) not occupied (index)", + LocalLogName, ovbuff->index, ge->curindex.blocknum); + /* fix this */ + ovusedblock(ovbuff, ge->curindex.blocknum, TRUE, TRUE); +#endif /* OV_DEBUG */ + } + if (PWRITE(ovbuff->fd, &ie, sizeof(ie), + ovbuff->base + ge->curindex.blocknum * OV_BLOCKSIZE + + sizeof(OVINDEXHEAD) + sizeof(ie) * ge->curindexoffset) != sizeof(ie)) + { + syslog(L_ERROR, "%s: could not write index record index '%d'," + " blocknum '%d': %m", + LocalLogName, ge->curindex.index, ge->curindex.blocknum); + return TRUE; + } + if ((ge->curlow <= 0) || (ge->curlow > artnum)) { + ge->curlow = artnum; + needupdate = TRUE; + } + if ((ge->curhigh <= 0) || (ge->curhigh < artnum)) { + ge->curhigh = artnum; + needupdate = TRUE; + } + if (needupdate) { + ovindexhead.next = ovnull; + ovindexhead.low = ge->curlow; + ovindexhead.high = ge->curhigh; + if (PWRITE(ovbuff->fd, &ovindexhead, sizeof(OVINDEXHEAD), + ovbuff->base + ge->curindex.blocknum * OV_BLOCKSIZE) != + sizeof(OVINDEXHEAD)) + { + syslog(L_ERROR, "%s: could not write index record index '%d'," + " blocknum '%d': %m", + LocalLogName, ge->curindex.index, ge->curindex.blocknum); + return TRUE; + } + } + if ((ge->low <= 0) || (ge->low > artnum)) + ge->low = artnum; + if ((ge->high <= 0) || (ge->high < artnum)) + ge->high = artnum; + ge->curindexoffset++; + ge->curoffset += len; + ge->count++; + return TRUE; } -BOOL buffindexed_add(char *group, ARTNUM artnum, TOKEN token, char *data, int len, time_t arrived, time_t expires) { - GROUPLOC gloc; - GROUPENTRY *ge; +BOOL buffindexed_add(char *group, ARTNUM artnum, TOKEN token, char *data, + int len, time_t arrived, time_t expires) +{ + GROUPLOC gloc; + GROUPENTRY *ge; - if (len > OV_BLOCKSIZE) { - syslog(L_ERROR, "%s: overview data is too large %d", LocalLogName, len); - return TRUE; - } + if (len > OV_BLOCKSIZE) { + syslog(L_ERROR, "%s: overview data is too large %d", LocalLogName, len); + return TRUE; + } - gloc = GROUPfind(group, FALSE); - if (GROUPLOCempty(gloc)) { - return TRUE; - } - GROUPlock(gloc, LOCK_WRITE); - /* prepend block(s) if needed. */ - ge = &GROUPentries[gloc.recno]; - if (Cutofflow && ge->low > artnum) { - GROUPlock(gloc, LOCK_UNLOCK); - return TRUE; - } -#ifdef OV_DEBUG - if (!ovaddrec(ge, artnum, token, data, len, arrived, expires, NULL)) { -#else - if (!ovaddrec(ge, artnum, token, data, len, arrived, expires)) { -#endif /* OV_DEBUG */ - if (Nospace) { - GROUPlock(gloc, LOCK_UNLOCK); - syslog(L_ERROR, "%s: no space left for buffer, adding '%s'", LocalLogName, group); - return FALSE; + gloc = GROUPfind(group, FALSE); + if (GROUPLOCempty(gloc)) { + return TRUE; } - syslog(L_ERROR, "%s: could not add overview for '%s'", LocalLogName, group); - } - GROUPlock(gloc, LOCK_UNLOCK); - return TRUE; + GROUPlock(gloc, LOCK_WRITE); + /* prepend block(s) if needed. */ + ge = &GROUPentries[gloc.recno]; + if (Cutofflow && ge->low > artnum) { + GROUPlock(gloc, LOCK_UNLOCK); + return TRUE; + } +#ifdef OV_DEBUG + if (!ovaddrec(ge, artnum, token, data, len, arrived, expires, NULL)) { +#else + if (!ovaddrec(ge, artnum, token, data, len, arrived, expires)) { +#endif /* OV_DEBUG */ + if (Nospace) { + GROUPlock(gloc, LOCK_UNLOCK); + syslog(L_ERROR, "%s: no space left for buffer, adding '%s'", + LocalLogName, group); + return FALSE; + } + syslog(L_ERROR, "%s: could not add overview for '%s'", + LocalLogName, group); + } + GROUPlock(gloc, LOCK_UNLOCK); + + return TRUE; } BOOL buffindexed_cancel(TOKEN token) { @@ -1426,702 +1655,748 @@ #else STATIC void freegroupblock(void) { #endif /* OV_DEBUG */ - GROUPDATABLOCK *gdb; - int i; - OV ov; - GIBLIST *giblist; + GROUPDATABLOCK *gdb; + int i; + OV ov; + GIBLIST *giblist; - for (giblist = Giblist ; giblist != NULL ; giblist = giblist->next) { + for (giblist = Giblist ; giblist != NULL ; giblist = giblist->next) { #ifdef OV_DEBUG - ovblockfree(giblist->ov, ge); + ovblockfree(giblist->ov, ge); #else - ovblockfree(giblist->ov); + ovblockfree(giblist->ov); #endif /* OV_DEBUG */ - } - for (i = 0 ; i < GROUPDATAHASHSIZE ; i++) { - for (gdb = groupdatablock[i] ; gdb != NULL ; gdb = gdb->next) { + } + for (i = 0 ; i < GROUPDATAHASHSIZE ; i++) { + for (gdb = groupdatablock[i] ; gdb != NULL ; gdb = gdb->next) { #ifdef OV_DEBUG - ovblockfree(gdb->datablk, ge); + ovblockfree(gdb->datablk, ge); #else - ovblockfree(gdb->datablk); + ovblockfree(gdb->datablk); #endif /* OV_DEBUG */ + } } - } } STATIC void ovgroupunmap(void) { - GROUPDATABLOCK *gdb, *gdbnext; - int i; - GIBLIST *giblist, *giblistnext; - - for (i = 0 ; i < GROUPDATAHASHSIZE ; i++) { - for (gdb = groupdatablock[i] ; gdb != NULL ; gdb = gdbnext) { - gdbnext = gdb->next; - DISPOSE(gdb); - } - groupdatablock[i] = NULL; - } - for (giblist = Giblist ; giblist != NULL ; giblist = giblistnext) { - giblistnext = giblist->next; - DISPOSE(giblist); - } - Giblist = NULL; - if (Gib != NULL) { - DISPOSE(Gib); - Gib = NULL; - } - if (Gdb != NULL) { - DISPOSE(Gdb); - Gdb = NULL; - } + GROUPDATABLOCK *gdb, *gdbnext; + int i; + GIBLIST *giblist, *giblistnext; + + for (i = 0 ; i < GROUPDATAHASHSIZE ; i++) { + for (gdb = groupdatablock[i] ; gdb != NULL ; gdb = gdbnext) { + gdbnext = gdb->next; + DISPOSE(gdb); + } + groupdatablock[i] = NULL; + } + for (giblist = Giblist ; giblist != NULL ; giblist = giblistnext) { + giblistnext = giblist->next; + DISPOSE(giblist); + } + Giblist = NULL; + if (Gib != NULL) { + DISPOSE(Gib); + Gib = NULL; + } + if (Gdb != NULL) { + DISPOSE(Gdb); + Gdb = NULL; + } } STATIC void insertgdb(OV *ov, GROUPDATABLOCK *gdb) { - gdb->next = groupdatablock[(ov->index + ov->blocknum) % GROUPDATAHASHSIZE]; - groupdatablock[(ov->index + ov->blocknum) % GROUPDATAHASHSIZE] = gdb; - return; + gdb->next = groupdatablock[(ov->index + ov->blocknum) % GROUPDATAHASHSIZE]; + groupdatablock[(ov->index + ov->blocknum) % GROUPDATAHASHSIZE] = gdb; + return; } STATIC GROUPDATABLOCK *searchgdb(OV *ov) { - GROUPDATABLOCK *gdb; + GROUPDATABLOCK *gdb; - gdb = groupdatablock[(ov->index + ov->blocknum) % GROUPDATAHASHSIZE]; - for (; gdb != NULL ; gdb = gdb->next) { - if (ov->index == gdb->datablk.index && ov->blocknum == gdb->datablk.blocknum) - break; - } - return gdb; + gdb = groupdatablock[(ov->index + ov->blocknum) % GROUPDATAHASHSIZE]; + for (; gdb != NULL ; gdb = gdb->next) { + if (ov->index == gdb->datablk.index && ov->blocknum == gdb->datablk.blocknum) + break; + } + return gdb; } STATIC int INDEXcompare(CPOINTER p1, CPOINTER p2) { - OVINDEX *oi1; - OVINDEX *oi2; + OVINDEX *oi1; + OVINDEX *oi2; - oi1 = (OVINDEX *)p1; - oi2 = (OVINDEX *)p2; - return oi1->artnum - oi2->artnum; + oi1 = (OVINDEX *)p1; + oi2 = (OVINDEX *)p2; + return oi1->artnum - oi2->artnum; } STATIC BOOL ovgroupmmap(GROUPENTRY *ge, int low, int high, BOOL needov) { - OV ov = ge->baseindex; - OVBUFF *ovbuff; - GROUPDATABLOCK *gdb; - int pagefudge, base, limit, i, count, len; - OFFSET_T offset, mmapoffset; - OVBLOCK *ovblock; - caddr_t addr; - GIBLIST *giblist; - - Gibcount = 0; - if (high - low < 0) - return TRUE; - i = 0; - if (high - low + 1 < ge->count) - Gibcount = high - low + 1; - else - Gibcount = ge->count; - if (Gibcount == 0) - return TRUE; - Gib = NEW(OVINDEX, Gibcount); - count = 0; - while (ov.index != NULLINDEX) { - ovbuff = getovbuff(ov); - if (ovbuff == NULL) { - syslog(L_ERROR, "%s: ovgroupmmap ovbuff is null(ovindex is %d, ovblock is %d", LocalLogName, ov.index, ov.blocknum); - ovgroupunmap(); - return FALSE; - } - offset = ovbuff->base + (ov.blocknum * OV_BLOCKSIZE); - pagefudge = offset % pagesize; - mmapoffset = offset - pagefudge; - len = pagefudge + OV_BLOCKSIZE; - if ((addr = mmap((caddr_t) 0, len, PROT_READ, MAP_SHARED, ovbuff->fd, mmapoffset)) == (MMAP_PTR) -1) { - syslog(L_ERROR, "%s: ovgroupmmap could not mmap index block: %m", LocalLogName); - ovgroupunmap(); - return FALSE; - } - ovblock = (OVBLOCK *)(addr + pagefudge); - if (low > ovblock->ovindexhead.high || high < ovblock->ovindexhead.low) { - ov = ovblock->ovindexhead.next; - munmap(addr, len); - continue; - } - if (ov.index == ge->curindex.index && ov.blocknum == ge->curindex.blocknum) { - limit = ge->curindexoffset; - } else { - limit = OVINDEXMAX; - } - for (i = 0 ; i < limit ; i++) { - if (ovblock->ovindex[i].artnum >= low && ovblock->ovindex[i].artnum <= high) { - if (Gibcount == count) { - Gibcount += OV_FUDGE; - RENEW(Gib, OVINDEX, Gibcount); - } - Gib[count++] = ovblock->ovindex[i]; - } - } - giblist = NEW(GIBLIST, 1); - giblist->ov = ov; - giblist->next = Giblist; - Giblist = giblist; - ov = ovblock->ovindexhead.next; - munmap(addr, len); - } - Gibcount = count; - qsort((POINTER)Gib, Gibcount, sizeof(OVINDEX), INDEXcompare); - /* Remove duplicates. */ - for (i = 0; i < Gibcount - 1; i++) { - if (Gib[i].artnum == Gib[i+1].artnum) { - /* lower position is removed */ - Gib[i].artnum = 0; + OV ov = ge->baseindex; + OVBUFF *ovbuff; + GROUPDATABLOCK *gdb; + int pagefudge, base, limit, i, count, len; + OFFSET_T offset, mmapoffset; + OVBLOCK *ovblock; + caddr_t addr; + GIBLIST *giblist; + + Gibcount = 0; + if (high - low < 0) + return TRUE; + i = 0; + if (high - low + 1 < ge->count) + Gibcount = high - low + 1; + else + Gibcount = ge->count; + if (Gibcount == 0) + return TRUE; + Gib = NEW(OVINDEX, Gibcount); + count = 0; + while (ov.index != NULLINDEX) { + ovbuff = getovbuff(ov); + if (ovbuff == NULL) { + syslog(L_ERROR, "%s: ovgroupmmap ovbuff is null(ovindex is %d, ovblock is %d", LocalLogName, ov.index, ov.blocknum); + ovgroupunmap(); + return FALSE; + } + offset = ovbuff->base + (ov.blocknum * OV_BLOCKSIZE); + pagefudge = offset % pagesize; + mmapoffset = offset - pagefudge; + len = pagefudge + OV_BLOCKSIZE; + if ((addr = mmap((caddr_t) 0, len, PROT_READ, MAP_SHARED, ovbuff->fd, mmapoffset)) == (MMAP_PTR) -1) { + syslog(L_ERROR, "%s: ovgroupmmap could not mmap index block: %m", LocalLogName); + ovgroupunmap(); + return FALSE; + } + ovblock = (OVBLOCK *)(addr + pagefudge); + if (low > ovblock->ovindexhead.high || high < ovblock->ovindexhead.low) + { + ov = ovblock->ovindexhead.next; + munmap(addr, len); + continue; + } + if (ov.index == ge->curindex.index && + ov.blocknum == ge->curindex.blocknum) + { + limit = ge->curindexoffset; + } else { + limit = OVINDEXMAX; + } + for (i = 0 ; i < limit ; i++) { + if (ovblock->ovindex[i].artnum >= low && + ovblock->ovindex[i].artnum <= high) + { + if (Gibcount == count) { + Gibcount += OV_FUDGE; + RENEW(Gib, OVINDEX, Gibcount); + } + Gib[count++] = ovblock->ovindex[i]; + } + } + giblist = NEW(GIBLIST, 1); + giblist->ov = ov; + giblist->next = Giblist; + Giblist = giblist; + ov = ovblock->ovindexhead.next; + munmap(addr, len); + } + Gibcount = count; + qsort((POINTER)Gib, Gibcount, sizeof(OVINDEX), INDEXcompare); + /* Remove duplicates. */ + for (i = 0; i < Gibcount - 1; i++) { + if (Gib[i].artnum == Gib[i+1].artnum) { + /* lower position is removed */ + Gib[i].artnum = 0; + } + } + if (!needov) + return TRUE; + count = 0; + for (i = 0 ; i < Gibcount ; i++) { + if (Gib[i].artnum == 0) + continue; + ov.index = Gib[i].index; + ov.blocknum = Gib[i].blocknum; + gdb = searchgdb(&ov); + if (gdb != NULL) + continue; + ovbuff = getovbuff(ov); + if (ovbuff == NULL) + continue; + gdb = NEW(GROUPDATABLOCK, 1); + gdb->datablk = ov; + gdb->next = NULL; + insertgdb(&ov, gdb); + count++; + } + if (count == 0) + return TRUE; + Gdb = NEW(char, count * OV_BLOCKSIZE); + count = 0; + for (i = 0 ; i < GROUPDATAHASHSIZE ; i++) { + for (gdb = groupdatablock[i] ; gdb != NULL ; gdb = gdb->next) { + ov = gdb->datablk; + ovbuff = getovbuff(ov); + offset = ovbuff->base + (ov.blocknum * OV_BLOCKSIZE); + pagefudge = offset % pagesize; + mmapoffset = offset - pagefudge; + gdb->len = pagefudge + OV_BLOCKSIZE; + if ((gdb->addr = mmap((caddr_t) 0, gdb->len, PROT_READ, MAP_SHARED, ovbuff->fd, mmapoffset)) == (MMAP_PTR) -1) { + syslog(L_ERROR, "%s: ovgroupmmap could not mmap data block: %m", LocalLogName); + DISPOSE(gdb); + ovgroupunmap(); + return FALSE; + } + gdb->data = gdb->addr + pagefudge; + memcpy(&Gdb[count * OV_BLOCKSIZE], gdb->data, OV_BLOCKSIZE); + munmap(gdb->addr, gdb->len); + gdb->data = &Gdb[count * OV_BLOCKSIZE]; + count++; + } } - } - if (!needov) return TRUE; - count = 0; - for (i = 0 ; i < Gibcount ; i++) { - if (Gib[i].artnum == 0) - continue; - ov.index = Gib[i].index; - ov.blocknum = Gib[i].blocknum; - gdb = searchgdb(&ov); - if (gdb != NULL) - continue; - ovbuff = getovbuff(ov); - if (ovbuff == NULL) - continue; - gdb = NEW(GROUPDATABLOCK, 1); - gdb->datablk = ov; - gdb->next = NULL; - insertgdb(&ov, gdb); - count++; - } - if (count == 0) - return TRUE; - Gdb = NEW(char, count * OV_BLOCKSIZE); - count = 0; - for (i = 0 ; i < GROUPDATAHASHSIZE ; i++) { - for (gdb = groupdatablock[i] ; gdb != NULL ; gdb = gdb->next) { - ov = gdb->datablk; - ovbuff = getovbuff(ov); - offset = ovbuff->base + (ov.blocknum * OV_BLOCKSIZE); - pagefudge = offset % pagesize; - mmapoffset = offset - pagefudge; - gdb->len = pagefudge + OV_BLOCKSIZE; - if ((gdb->addr = mmap((caddr_t) 0, gdb->len, PROT_READ, MAP_SHARED, ovbuff->fd, mmapoffset)) == (MMAP_PTR) -1) { - syslog(L_ERROR, "%s: ovgroupmmap could not mmap data block: %m", LocalLogName); - DISPOSE(gdb); - ovgroupunmap(); - return FALSE; - } - gdb->data = gdb->addr + pagefudge; - memcpy(&Gdb[count * OV_BLOCKSIZE], gdb->data, OV_BLOCKSIZE); - munmap(gdb->addr, gdb->len); - gdb->data = &Gdb[count * OV_BLOCKSIZE]; - count++; - } - } - return TRUE; } STATIC void *ovopensearch(char *group, int low, int high, BOOL needov) { - GROUPLOC gloc; - GROUPENTRY *ge; - OVSEARCH *search; + GROUPLOC gloc; + GROUPENTRY *ge; + OVSEARCH *search; + + gloc = GROUPfind(group, FALSE); + if (GROUPLOCempty(gloc)) + return NULL; - gloc = GROUPfind(group, FALSE); - if (GROUPLOCempty(gloc)) - return NULL; - - ge = &GROUPentries[gloc.recno]; - if (low < ge->low) - low = ge->low; - if (high > ge->high) - high = ge->high; - - if (!ovgroupmmap(ge, low, high, needov)) { - return NULL; - } - - search = NEW(OVSEARCH, 1); - search->hi = high; - search->lo = low; - search->cur = 0; - search->group = COPY(group); - search->needov = needov; - search->gloc = gloc; - return (void *)search; + ge = &GROUPentries[gloc.recno]; + if (low < ge->low) + low = ge->low; + if (high > ge->high) + high = ge->high; + + if (!ovgroupmmap(ge, low, high, needov)) { + return NULL; + } + + search = NEW(OVSEARCH, 1); + search->hi = high; + search->lo = low; + search->cur = 0; + search->group = COPY(group); + search->needov = needov; + search->gloc = gloc; + return (void *)search; } void *buffindexed_opensearch(char *group, int low, int high) { - GROUPLOC gloc; - void *handle; + GROUPLOC gloc; + void *handle; - gloc = GROUPfind(group, FALSE); - if (GROUPLOCempty(gloc)) { - return NULL; - } - GROUPlock(gloc, LOCK_WRITE); - if ((handle = ovopensearch(group, low, high, TRUE)) == NULL) - GROUPlock(gloc, LOCK_UNLOCK); - return(handle); + gloc = GROUPfind(group, FALSE); + if (GROUPLOCempty(gloc)) { + return NULL; + } + GROUPlock(gloc, LOCK_WRITE); + if ((handle = ovopensearch(group, low, high, TRUE)) == NULL) + GROUPlock(gloc, LOCK_UNLOCK); + return(handle); } BOOL ovsearch(void *handle, ARTNUM *artnum, char **data, int *len, TOKEN *token, time_t *arrived, time_t *expires) { - OVSEARCH *search = (OVSEARCH *)handle; - OVBLOCK *ovblock; - OV srchov; - GROUPDATABLOCK *gdb; - - if (search->cur == Gibcount) { - return FALSE; - } - while (Gib[search->cur].artnum == 0) { + OVSEARCH *search = (OVSEARCH *)handle; + OVBLOCK *ovblock; + OV srchov; + GROUPDATABLOCK *gdb; + + if (search->cur == Gibcount) { + return FALSE; + } + while (Gib[search->cur].artnum == 0) { + search->cur++; + if (search->cur == Gibcount) + return FALSE; + } + + if (search->needov) { + if (Gib[search->cur].index == NULLINDEX) { + if (len) + *len = 0; + if (artnum) + *artnum = Gib[search->cur].artnum; + } else { + if (artnum) + *artnum = Gib[search->cur].artnum; + if (len) + *len = Gib[search->cur].len; + if (arrived) + *arrived = Gib[search->cur].arrived; + if (expires) + *expires = Gib[search->cur].expires; + if (data) { + srchov.index = Gib[search->cur].index; + srchov.blocknum = Gib[search->cur].blocknum; + gdb = searchgdb(&srchov); + if (gdb == NULL) { + if (len) + *len = 0; + search->cur++; + return TRUE; + } + *data = gdb->data + Gib[search->cur].offset; + } + } + } + if (token) { + if (Gib[search->cur].index == NULLINDEX && !search->needov) { + search->cur++; + return FALSE; + } + *token = Gib[search->cur].token; + } search->cur++; - if (search->cur == Gibcount) - return FALSE; - } - - if (search->needov) { - if (Gib[search->cur].index == NULLINDEX) { - if (len) - *len = 0; - if (artnum) - *artnum = Gib[search->cur].artnum; - } else { - if (artnum) - *artnum = Gib[search->cur].artnum; - if (len) - *len = Gib[search->cur].len; - if (arrived) - *arrived = Gib[search->cur].arrived; - if (expires) - *expires = Gib[search->cur].expires; - if (data) { - srchov.index = Gib[search->cur].index; - srchov.blocknum = Gib[search->cur].blocknum; - gdb = searchgdb(&srchov); - if (gdb == NULL) { - if (len) - *len = 0; - search->cur++; - return TRUE; - } - *data = gdb->data + Gib[search->cur].offset; - } - } - } - if (token) { - if (Gib[search->cur].index == NULLINDEX && !search->needov) { - search->cur++; - return FALSE; - } - *token = Gib[search->cur].token; - } - search->cur++; - return TRUE; + return TRUE; } BOOL buffindexed_search(void *handle, ARTNUM *artnum, char **data, int *len, TOKEN *token, time_t *arrived) { - return(ovsearch(handle, artnum, data, len, token, arrived, NULL)); + return(ovsearch(handle, artnum, data, len, token, arrived, NULL)); } STATIC void ovclosesearch(void *handle, BOOL freeblock) { - OVSEARCH *search = (OVSEARCH *)handle; + OVSEARCH *search = (OVSEARCH *)handle; #ifdef OV_DEBUG - GROUPENTRY *ge; - GROUPLOC gloc; + GROUPENTRY *ge; + GROUPLOC gloc; #endif /* OV_DEBUG */ - if (freeblock) { + if (freeblock) { #ifdef OV_DEBUG - gloc = GROUPfind(search->group, FALSE); - if (!GROUPLOCempty(gloc)) { - ge = &GROUPentries[gloc.recno]; - freegroupblock(ge); - } + gloc = GROUPfind(search->group, FALSE); + if (!GROUPLOCempty(gloc)) { + ge = &GROUPentries[gloc.recno]; + freegroupblock(ge); + } #else - freegroupblock(); + freegroupblock(); #endif /* OV_DEBUG */ - } - ovgroupunmap(); - DISPOSE(search->group); - DISPOSE(search); - return; + } + ovgroupunmap(); + DISPOSE(search->group); + DISPOSE(search); + return; } void buffindexed_closesearch(void *handle) { - OVSEARCH *search = (OVSEARCH *)handle; - GROUPLOC gloc; + OVSEARCH *search = (OVSEARCH *)handle; + GROUPLOC gloc; - gloc = search->gloc; - ovclosesearch(handle, FALSE); - GROUPlock(gloc, LOCK_UNLOCK); + gloc = search->gloc; + ovclosesearch(handle, FALSE); + GROUPlock(gloc, LOCK_UNLOCK); } BOOL buffindexed_getartinfo(char *group, ARTNUM artnum, char **data, int *len, TOKEN *token) { - GROUPLOC gloc; - void *handle; - BOOL retval; + GROUPLOC gloc; + void *handle; + BOOL retval; - gloc = GROUPfind(group, FALSE); - if (GROUPLOCempty(gloc)) { - return FALSE; - } - GROUPlock(gloc, LOCK_WRITE); - if (!(handle = ovopensearch(group, artnum, artnum, FALSE))) { + gloc = GROUPfind(group, FALSE); + if (GROUPLOCempty(gloc)) { + return FALSE; + } + GROUPlock(gloc, LOCK_WRITE); + if (!(handle = ovopensearch(group, artnum, artnum, FALSE))) { + GROUPlock(gloc, LOCK_UNLOCK); + return FALSE; + } + retval = buffindexed_search(handle, NULL, data, len, token, NULL); + ovclosesearch(handle, FALSE); GROUPlock(gloc, LOCK_UNLOCK); - return FALSE; - } - retval = buffindexed_search(handle, NULL, data, len, token, NULL); - ovclosesearch(handle, FALSE); - GROUPlock(gloc, LOCK_UNLOCK); - return retval; -} - -BOOL buffindexed_expiregroup(char *group, int *lo) { - void *handle; - GROUPENTRY newge, *ge; - GROUPLOC gloc, next; - char *data; - int i, j, len; - TOKEN token; - ARTNUM artnum, low, high; - ARTHANDLE *ah; - char flag; - HASH hash; - time_t arrived, expires; - - if (group == NULL) { - for (i = 0 ; i < GROUPheader->freelist.recno ; i++) { - gloc.recno = i; - GROUPlock(gloc, LOCK_WRITE); - ge = &GROUPentries[gloc.recno]; - if (ge->expired >= OVrealnow || ge->count == 0) { - GROUPlock(gloc, LOCK_UNLOCK); - continue; - } - if (!ovgroupmmap(ge, ge->low, ge->high, TRUE)) { - GROUPlock(gloc, LOCK_UNLOCK); - syslog(L_ERROR, "%s: could not mmap overview for hidden groups(%d)", LocalLogName, i); - continue; - } - for (j = 0 ; j < Gibcount ; j++) { - if (Gib[j].artnum == 0) - continue; - /* this may be duplicated, but ignore it in this case */ - OVEXPremove(Gib[j].token, TRUE, NULL, 0); - } + return retval; +} + +BOOL buffindexed_expiregroup(char *group, int *lo) +{ + void *handle; + GROUPENTRY newge, *ge; + GROUPLOC gloc, next; + char *data; + int i, j, len; + TOKEN token; + ARTNUM artnum, low, high; + ARTHANDLE *ah; + char flag; + HASH hash; + time_t arrived, expires; + + if (group == NULL) { + for (i = 0 ; i < GROUPheader->freelist.recno ; i++) { + gloc.recno = i; + GROUPlock(gloc, LOCK_WRITE); + ge = &GROUPentries[gloc.recno]; + if (ge->expired >= OVrealnow || ge->count == 0) { + GROUPlock(gloc, LOCK_UNLOCK); + continue; + } + if (!ovgroupmmap(ge, ge->low, ge->high, TRUE)) { + GROUPlock(gloc, LOCK_UNLOCK); + syslog(L_ERROR, "%s: could not mmap overview for hidden" + " groups(%d)", LocalLogName, i); + continue; + } + for (j = 0 ; j < Gibcount ; j++) { + if (Gib[j].artnum == 0) + continue; + /* this may be duplicated, but ignore it in this case */ + /* XXX - do we need this? */ + OVEXPremove(Gib[j].token, TRUE, NULL, 0); + } #ifdef OV_DEBUG - freegroupblock(ge); + freegroupblock(ge); #else - freegroupblock(); + freegroupblock(); #endif - ovgroupunmap(); - ge->expired = time(NULL); - ge->count = 0; - GROUPlock(gloc, LOCK_UNLOCK); + ovgroupunmap(); + ge->expired = time(NULL); + ge->count = 0; + GROUPlock(gloc, LOCK_UNLOCK); + } + return TRUE; } - return TRUE; - } - gloc = GROUPfind(group, FALSE); - if (GROUPLOCempty(gloc)) { - return FALSE; - } - GROUPlock(gloc, LOCK_WRITE); - ge = &GROUPentries[gloc.recno]; - if (ge->count == 0) { - if (lo) - *lo = ge->low; + gloc = GROUPfind(group, FALSE); + if (GROUPLOCempty(gloc)) { + return FALSE; + } + GROUPlock(gloc, LOCK_WRITE); + ge = &GROUPentries[gloc.recno]; + if (ge->count == 0) { + if ( ge->low < ge->high ) { + syslog(L_NOTICE, "%s: group %s updating low %d to %d", + LocalLogName, group, ge->low, ge->high); + ge->low = ge->high; + } + if (lo) + *lo = ge->low + 1; + ge->expired = time(NULL); + GROUPlock(gloc, LOCK_UNLOCK); + return TRUE; + } + flag = ge->flag; + hash = ge->hash; + next = ge->next; + low = ge->low; + high = ge->high; + + newge.low = 0; + setinitialge(&newge, hash, &flag, next, 0, high); + if ((handle = ovopensearch(group, low, high, TRUE)) == NULL) { + ge->expired = time(NULL); + GROUPlock(gloc, LOCK_UNLOCK); + syslog(L_ERROR, "%s: could not open overview for '%s'", + LocalLogName, group); + return FALSE; + } + while (ovsearch(handle, &artnum, &data, &len, &token, &arrived, &expires)) { + ah = NULL; + if (len == 0) + continue; + if (!SMprobe(EXPENSIVESTAT, &token, NULL) || OVstatall) { + if ((ah = SMretrieve(token, RETR_STAT)) == NULL) + continue; + SMfreearticle(ah); + } else { + if (!OVhisthasmsgid(data)) + continue; + } + if (innconf->groupbaseexpiry && OVgroupbasedexpire(token, group, + data, len, arrived, expires)) + { + continue; + } +#ifdef OV_DEBUG + if (!ovaddrec(&newge, artnum, token, data, len, arrived, expires, ge)) { +#else + if (!ovaddrec(&newge, artnum, token, data, len, arrived, expires)) { +#endif /* OV_DEBUG */ + ovclosesearch(handle, TRUE); + ge->expired = time(NULL); + GROUPlock(gloc, LOCK_UNLOCK); + syslog(L_ERROR, "%s: could not add new overview for '%s'", + LocalLogName, group); + return FALSE; + } + } + if (newge.low == 0) + /* no article for the group */ + newge.low = newge.high; + *ge = newge; + if (lo) { + if (ge->count == 0) { + if (ge->low < ge->high) + ge->low = ge->high; + /* lomark should be himark + 1, if no article for the group */ + *lo = ge->low + 1; + } else + *lo = ge->low; + } + ovclosesearch(handle, TRUE); ge->expired = time(NULL); GROUPlock(gloc, LOCK_UNLOCK); return TRUE; - } - flag = ge->flag; - hash = ge->hash; - next = ge->next; - low = ge->low; - high = ge->high; - - newge.low = 0; - setinitialge(&newge, hash, &flag, next, 0, high); - if ((handle = ovopensearch(group, low, high, TRUE)) == NULL) { - ge->expired = time(NULL); - GROUPlock(gloc, LOCK_UNLOCK); - syslog(L_ERROR, "%s: could not open overview for '%s'", LocalLogName, group); - return FALSE; - } - while (ovsearch(handle, &artnum, &data, &len, &token, &arrived, &expires)) { - ah = NULL; - if (len == 0) - continue; - if (!SMprobe(EXPENSIVESTAT, &token, NULL) || OVstatall) { - if ((ah = SMretrieve(token, RETR_STAT)) == NULL) - continue; - SMfreearticle(ah); - } else { - if (!OVhisthasmsgid(data)) - continue; - } - if (innconf->groupbaseexpiry && OVgroupbasedexpire(token, group, data, len, arrived, expires)) - continue; -#ifdef OV_DEBUG - if (!ovaddrec(&newge, artnum, token, data, len, arrived, expires, ge)) { -#else - if (!ovaddrec(&newge, artnum, token, data, len, arrived, expires)) { -#endif /* OV_DEBUG */ - ovclosesearch(handle, TRUE); - ge->expired = time(NULL); - GROUPlock(gloc, LOCK_UNLOCK); - syslog(L_ERROR, "%s: could not add new overview for '%s'", LocalLogName, group); - return FALSE; - } - } - if (newge.low == 0) - /* no article for the group */ - newge.low = newge.high; - *ge = newge; - if (lo) { - if (ge->count == 0) - /* lomark should be himark + 1, if no article for the group */ - *lo = ge->low + 1; - else - *lo = ge->low; - } - ovclosesearch(handle, TRUE); - ge->expired = time(NULL); - GROUPlock(gloc, LOCK_UNLOCK); - return TRUE; } BOOL buffindexed_ctl(OVCTLTYPE type, void *val) { - int total, used, *i; - OVBUFF *ovbuff = ovbufftab; - OVSORTTYPE *sorttype; - - switch (type) { - case OVSPACE: - for (total = used = 0 ; ovbuff != (OVBUFF *)NULL ; ovbuff = ovbuff->next) { - ovlock(ovbuff, LOCK_READ); - ovreadhead(ovbuff); - total += ovbuff->totalblk; - used += ovbuff->usedblk; - ovlock(ovbuff, LOCK_UNLOCK); + int total, used, *i; + OVBUFF *ovbuff = ovbufftab; + OVSORTTYPE *sorttype; + + switch (type) { + case OVSPACE: + for (total = used = 0 ; ovbuff != (OVBUFF *)NULL ; ovbuff = ovbuff->next) { + ovlock(ovbuff, LOCK_READ); + ovreadhead(ovbuff); + total += ovbuff->totalblk; + used += ovbuff->usedblk; + ovlock(ovbuff, LOCK_UNLOCK); + } + i = (int *)val; + *i = (used * 100) / total; + return TRUE; + case OVSORT: + sorttype = (OVSORTTYPE *)val; + *sorttype = OVNOSORT; + return TRUE; + case OVCUTOFFLOW: + Cutofflow = *(BOOL *)val; + return TRUE; + case OVSTATICSEARCH: + i = (int *)val; + *i = FALSE; + return TRUE; + default: + return FALSE; } - i = (int *)val; - *i = (used * 100) / total; - return TRUE; - case OVSORT: - sorttype = (OVSORTTYPE *)val; - *sorttype = OVNOSORT; - return TRUE; - case OVCUTOFFLOW: - Cutofflow = *(BOOL *)val; - return TRUE; - case OVSTATICSEARCH: - i = (int *)val; - *i = FALSE; - return TRUE; - default: - return FALSE; - } } void buffindexed_close(void) { - struct stat sb; - OVBUFF *ovbuffnext, *ovbuff = ovbufftab; + struct stat sb; + OVBUFF *ovbuffnext, *ovbuff = ovbufftab; #ifdef OV_DEBUG - FILE *F = NULL; - PID_T pid; - char *path = NULL; - int i,j; - struct ov_trace_array *trace; - struct ov_name_table *ntp; + FILE *F = NULL; + PID_T pid; + char *path = NULL; + int i,j; + struct ov_trace_array *trace; + struct ov_name_table *ntp; #endif /* OV_DEBUG */ #ifdef OV_DEBUG - for (; ovbuff != (OVBUFF *)NULL; ovbuff = ovbuff->next) { - for (i = 0 ; i < ovbuff->totalblk ; i++) { - trace = &ovbuff->trace[i]; - if (trace->ov_trace == NULL) - continue; - for (j = 0 ; j <= trace->cur && j < trace->max ; j++) { - if (trace->ov_trace[j].occupied != 0 || - trace->ov_trace[j].freed != 0) { - if (F == NULL) { - path = NEW(char, strlen(innconf->pathtmp) + 10); - pid = getpid(); - sprintf(path, "%s/%d", innconf->pathtmp, pid); - if ((F = fopen(path, "w")) == NULL) { - syslog(L_ERROR, "%s: could not open %s: %m", LocalLogName, path); - break; - } - } - fprintf(F, "%d: % 6d, % 2d: 0x%08x, % 10d, % 10d\n", ovbuff->index, i, j, - trace->ov_trace[j].gloc.recno, - trace->ov_trace[j].occupied, - trace->ov_trace[j].freed); - } - } - } - } - if ((ntp = name_table) != NULL) { - if (F == NULL) { - path = NEW(char, strlen(innconf->pathtmp) + 10); - pid = getpid(); - sprintf(path, "%s/%d", innconf->pathtmp, pid); - if ((F = fopen(path, "w")) == NULL) { - syslog(L_ERROR, "%s: could not open %s: %m", LocalLogName, path); - } - } - if (F != NULL) { - while(ntp) { - fprintf(F, "0x%08x: %s\n", ntp->recno, ntp->name); - ntp = ntp->next; - } - } - } - if (F != NULL) - fclose(F); - if (path != NULL) - DISPOSE(path); + for (; ovbuff != (OVBUFF *)NULL; ovbuff = ovbuff->next) { + for (i = 0 ; i < ovbuff->totalblk ; i++) { + trace = &ovbuff->trace[i]; + if (trace->ov_trace == NULL) + continue; + for (j = 0 ; j <= trace->cur && j < trace->max ; j++) { + if (trace->ov_trace[j].occupied != 0 || + trace->ov_trace[j].freed != 0) + { + if (F == NULL) { + path = NEW(char, strlen(innconf->pathtmp) + 10); + pid = getpid(); + sprintf(path, "%s/%d", innconf->pathtmp, pid); + if ((F = fopen(path, "w")) == NULL) { + syslog(L_ERROR, "%s: could not open %s: %m", LocalLogName, path); + break; + } + } + fprintf(F, "%d: % 6d, % 2d: 0x%08x, % 10d, % 10d\n", + ovbuff->index, i, j, + trace->ov_trace[j].gloc.recno, + trace->ov_trace[j].occupied, + trace->ov_trace[j].freed); + } + } + } + } + if ((ntp = name_table) != NULL) { + if (F == NULL) { + path = NEW(char, strlen(innconf->pathtmp) + 10); + pid = getpid(); + sprintf(path, "%s/%d", innconf->pathtmp, pid); + if ((F = fopen(path, "w")) == NULL) { + syslog(L_ERROR, "%s: could not open %s: %m", LocalLogName, path); + } + } + if (F != NULL) { + while(ntp) { + fprintf(F, "0x%08x: %s\n", ntp->recno, ntp->name); + ntp = ntp->next; + } + } + } + if (F != NULL) + fclose(F); + if (path != NULL) + DISPOSE(path); #endif /* OV_DEBUG */ - if (fstat(GROUPfd, &sb) < 0) - return; - close(GROUPfd); + if (fstat(GROUPfd, &sb) < 0) + return; + close(GROUPfd); - if (GROUPheader) { - if (munmap((void *)GROUPheader, GROUPfilesize(GROUPcount)) < 0) { - syslog(L_FATAL, "%s: could not munmap group.index in buffindexed_close: %m", LocalLogName); - return; + if (GROUPheader) { + if (munmap((void *)GROUPheader, GROUPfilesize(GROUPcount)) < 0) { + syslog(L_FATAL, "%s: could not munmap group.index in buffindexed_close: %m", LocalLogName); + return; + } + } + + ovbuff = ovbufftab; + for (; ovbuff != (OVBUFF *)NULL; ovbuff = ovbuffnext) { + if( ovbuff->dirty ) { + ovbuff->dirty = OVBUFF_SYNC_COUNT; + ovflushhead( ovbuff ); + } + smcClose( ovbuff->smc ); + ovbuffnext = ovbuff->next; + DISPOSE(ovbuff); } - } - for (; ovbuff != (OVBUFF *)NULL; ovbuff = ovbuffnext) { - ovbuffnext = ovbuff->next; - DISPOSE(ovbuff); - } - ovbufftab = NULL; + ovbufftab = NULL; } #ifdef DEBUG +/* +** clear shared memory segments +*/ +STATIC void myexit( void ) { + OVBUFF *ovbuff; + + for( ovbuff=ovbufftab; ovbuff; ovbuff=ovbuff->next ) { + smcClose( ovbuff->smc ); + } +} + STATIC int countgdb(void) { - int i, count = 0; - GROUPDATABLOCK *gdb; + int i, count = 0; + GROUPDATABLOCK *gdb; - for (i = 0 ; i < GROUPDATAHASHSIZE ; i++) { - for (gdb = groupdatablock[i] ; gdb != NULL ; gdb = gdb->next) - count++; - } - return count; + for (i = 0 ; i < GROUPDATAHASHSIZE ; i++) { + for (gdb = groupdatablock[i] ; gdb != NULL ; gdb = gdb->next) + count++; + } + return count; } main(int argc, char **argv) { - char *group, flag[2], buff[OV_BLOCKSIZE]; - int lo, hi, count, flags, i; - OVSEARCH *search; - OVBLOCK *ovblock; - OVINDEX *ovindex; - OVBUFF *ovbuff; - GROUPENTRY *ge; - GROUPLOC gloc; - GIBLIST *giblist; - - if (argc != 2) { - fprintf(stderr, "only one argument can be specified\n"); - exit(1); - } - /* if innconf isn't already read in, do so. */ - if (innconf == NULL) { - if (ReadInnConf() < 0) { - fprintf(stderr, "reading inn.conf failed\n"); - exit(1); - } - } - if (!buffindexed_open(OV_READ)) { - fprintf(stderr, "buffindexed_open failed\n"); - exit(1); - } - fprintf(stdout, "GROUPheader->freelist.recno is %d(0x%08x)\n", GROUPheader->freelist.recno, GROUPheader->freelist.recno); - group = argv[1]; - if (isdigit(*group)) { - gloc.recno = atoi(group); - ge = &GROUPentries[gloc.recno]; - fprintf(stdout, "left articles are %d for %d, last expiry is %d\n", ge->count, gloc.recno, ge->expired); - if (ge->count == 0) { - GROUPlock(gloc, LOCK_UNLOCK); - exit(0); + char *group, flag[2], buff[OV_BLOCKSIZE]; + int lo, hi, count, flags, i; + OVSEARCH *search; + OVBLOCK *ovblock; + OVINDEX *ovindex; + OVBUFF *ovbuff; + GROUPENTRY *ge; + GROUPLOC gloc; + GIBLIST *giblist; + + if (argc != 2) { + fprintf(stderr, "only one argument can be specified\n"); + exit(1); + } + + /* if innconf isn't already read in, do so. */ + if (innconf == NULL) { + if (ReadInnConf() < 0) { + fprintf(stderr, "reading inn.conf failed\n"); + exit(1); + } + } + + atexit( myexit ); + openlog( "buffindexed", LOG_PID, LOG_NEWS ); + if (!buffindexed_open(OV_READ)) { + fprintf(stderr, "buffindexed_open failed\n"); + exit(1); + } + fprintf(stdout, "GROUPheader->freelist.recno is %d(0x%08x)\n", GROUPheader->freelist.recno, GROUPheader->freelist.recno); + group = argv[1]; + if (isdigit(*group)) { + gloc.recno = atoi(group); + ge = &GROUPentries[gloc.recno]; + fprintf(stdout, "left articles are %d for %d, last expiry is %d\n", ge->count, gloc.recno, ge->expired); + if (ge->count == 0) { + GROUPlock(gloc, LOCK_UNLOCK); + exit(0); + } + if (!ovgroupmmap(ge, ge->low, ge->high, TRUE)) { + fprintf(stderr, "ovgroupmmap failed\n"); + GROUPlock(gloc, LOCK_UNLOCK); + } + for (giblist = Giblist, i = 0 ; giblist != NULL ; giblist = giblist->next, i++); + fprintf(stdout, "%d index block(s)\n", i); + fprintf(stdout, "%d data block(s)\n", countgdb()); + for (giblist = Giblist ; giblist != NULL ; giblist = giblist->next) { + fprintf(stdout, " % 8d(% 5d)\n", giblist->ov.blocknum, giblist->ov.index); + } + for (i = 0 ; i < Gibcount ; i++) { + if (Gib[i].artnum == 0) + continue; + if (Gib[i].index == NULLINDEX) + fprintf(stdout, " %d empty\n"); + else { + fprintf(stdout, " %d %d\n", Gib[i].offset, Gib[i].len); + } + } + ovgroupunmap(); + GROUPlock(gloc, LOCK_UNLOCK); + exit(0); + } + gloc = GROUPfind(group, FALSE); + if (GROUPLOCempty(gloc)) { + fprintf(stderr, "gloc is null\n"); } - if (!ovgroupmmap(ge, ge->low, ge->high, TRUE)) { - fprintf(stderr, "ovgroupmmap failed\n"); - GROUPlock(gloc, LOCK_UNLOCK); + GROUPlock(gloc, LOCK_READ); + ge = &GROUPentries[gloc.recno]; + fprintf(stdout, "base %d(%d), cur %d(%d), expired at %s\n", ge->baseindex.blocknum, ge->baseindex.index, ge->curindex.blocknum, ge->curindex.index, ge->expired == 0 ? "none\n" : ctime(&ge->expired)); + if (!buffindexed_groupstats(group, &lo, &hi, &count, &flags)) { + fprintf(stderr, "buffindexed_groupstats failed for group %s\n", group); + exit(1); + } + flag[0] = (char)flags; + flag[1] = '\0'; + fprintf(stdout, "%s: low is %d, high is %d, count is %d, flag is '%s'\n", group, lo, hi, count, flag); + if ((search = (OVSEARCH *)ovopensearch(group, lo, hi, TRUE)) == NULL) { + fprintf(stderr, "ovopensearch failed for group %s\n", group); + exit(1); } + fprintf(stdout, " gloc is %d(0x%08x)\n", search->gloc.recno, search->gloc.recno); for (giblist = Giblist, i = 0 ; giblist != NULL ; giblist = giblist->next, i++); fprintf(stdout, "%d index block(s)\n", i); fprintf(stdout, "%d data block(s)\n", countgdb()); for (giblist = Giblist ; giblist != NULL ; giblist = giblist->next) { - fprintf(stdout, " % 8d(% 5d)\n", giblist->ov.blocknum, giblist->ov.index); + fprintf(stdout, " % 8d(% 5d)\n", giblist->ov.blocknum, giblist->ov.index); } for (i = 0 ; i < Gibcount ; i++) { - if (Gib[i].artnum == 0) - continue; - if (Gib[i].index == NULLINDEX) - fprintf(stdout, " %d empty\n"); - else { - fprintf(stdout, " %d %d\n", Gib[i].offset, Gib[i].len); - } - } - ovgroupunmap(); - GROUPlock(gloc, LOCK_UNLOCK); - exit(0); - } - gloc = GROUPfind(group, FALSE); - if (GROUPLOCempty(gloc)) { - fprintf(stderr, "gloc is null\n"); - } - GROUPlock(gloc, LOCK_READ); - ge = &GROUPentries[gloc.recno]; - fprintf(stdout, "base %d(%d), cur %d(%d), expired at %s\n", ge->baseindex.blocknum, ge->baseindex.index, ge->curindex.blocknum, ge->curindex.index, ge->expired == 0 ? "none\n" : ctime(&ge->expired)); - if (!buffindexed_groupstats(group, &lo, &hi, &count, &flags)) { - fprintf(stderr, "buffindexed_groupstats failed for group %s\n", group); - exit(1); - } - flag[0] = (char)flags; - flag[1] = '\0'; - fprintf(stdout, "%s: low is %d, high is %d, count is %d, flag is '%s'\n", group, lo, hi, count, flag); - if ((search = (OVSEARCH *)ovopensearch(group, lo, hi, TRUE)) == NULL) { - fprintf(stderr, "ovopensearch failed for group %s\n", group); - exit(1); - } - fprintf(stdout, " gloc is %d(0x%08x)\n", search->gloc.recno, search->gloc.recno); - for (giblist = Giblist, i = 0 ; giblist != NULL ; giblist = giblist->next, i++); - fprintf(stdout, "%d index block(s)\n", i); - fprintf(stdout, "%d data block(s)\n", countgdb()); - for (giblist = Giblist ; giblist != NULL ; giblist = giblist->next) { - fprintf(stdout, " % 8d(% 5d)\n", giblist->ov.blocknum, giblist->ov.index); - } - for (i = 0 ; i < Gibcount ; i++) { - if (Gib[i].artnum == 0) - continue; - if (Gib[i].index == NULLINDEX) - fprintf(stdout, " %d empty\n"); - else { - fprintf(stdout, " %d %d\n", Gib[i].offset, Gib[i].len); - } - } - { - ARTNUM artnum; - char *data; - int len; - TOKEN token; - while (buffindexed_search((void *)search, &artnum, &data, &len, &token, NULL)) { - if (len == 0) - fprintf(stdout, "%d: len is 0\n", artnum); - else { - memcpy(buff, data, len); - buff[len] = '\0'; - fprintf(stdout, "%d: %s\n", artnum, buff); - } + if (Gib[i].artnum == 0) + continue; + if (Gib[i].index == NULLINDEX) + fprintf(stdout, " %d empty\n"); + else { + fprintf(stdout, " %d %d\n", Gib[i].offset, Gib[i].len); + } + } + { + ARTNUM artnum; + char *data; + int len; + TOKEN token; + while (buffindexed_search((void *)search, &artnum, &data, + &len, &token, NULL)) + { + if (len == 0) + fprintf(stdout, "%d: len is 0\n", artnum); + else { + memcpy(buff, data, len); + buff[len] = '\0'; + fprintf(stdout, "%d: %s\n", artnum, buff); + } + } } - } } #endif /* DEBUG */ diff -uNr buffindexed.orig/shmem.c buffindexed/shmem.c --- buffindexed.orig/shmem.c Thu Jan 1 09:00:00 1970 +++ buffindexed/shmem.c Sun Dec 22 22:06:29 2002 @@ -0,0 +1,380 @@ +/* +** $Id: shmem.c,v 1.2 2002/12/22 13:06:25 sysuh Exp $ +*/ + +/* shared memory control utility */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "shmem.h" + +#ifndef MAP_FAILED + #define MAP_FAILED ((caddr_t)-1) +#endif + +#ifndef L_NOTICE +#define L_NOTICE LOG_NOTICE +#endif + +#ifndef L_ERROR +#define L_ERROR LOG_ERR +#endif + +#ifndef L_FATAL +#define L_FATAL LOG_CRIT +#endif + +static int smcGetSemaphore(const char *name) +{ + key_t kt = ftok( (char *)name, 0 ); + int id = semget(kt, 0, S_IRWXU|S_IRWXG|S_IRWXO); + + if (id < 0) { + syslog( L_ERROR, "semget failed to get semaphore for %s: %m", name ); + } + return id; +} + +static int smcCreateSemaphore(const char *name) +{ + key_t kt = ftok( (char *)name, 0 ); + int id = semget(kt, 2, IPC_CREAT|S_IRWXU|S_IRWXG|S_IRWXO); + + if (id < 0) { + if (errno == EACCES || errno == EINVAL) { + /* looks like a wrong semaphore exists. remove it. */ + id = semget(kt, 0, S_IRWXU|S_IRWXG|S_IRWXO); + if (id < 0) { + /* couldn't even retrieve it. */ + syslog( L_ERROR, "cant get semaphore using %s", name ); + return id; + } + /* try to remove it */ +#ifdef SEMCTL_NEEDS_UNION + { + union semun semArg; + semArg.val = 1; + if (semctl(id, 0, IPC_RMID, semArg) < 0) { + syslog( L_FATAL, "cant remove semaphore %s", name ); + exit(1); + } + } +#else + if (semctl(id, 0, IPC_RMID, NULL) < 0) { + syslog( L_FATAL, "cant remove semaphore %s", name ); + exit(1); + } +#endif + /* and retry creating it */ + id = semget(kt, 2, IPC_CREAT|S_IRWXU|S_IRWXG|S_IRWXO); + } + } + if (id < 0) { + syslog( L_ERROR, "cant create semaphore using %s", name ); + } + return id; +} + +int smcGetExclusiveLock(smcd_t *this) +{ + struct sembuf sops[3] = { + {0, 0, SEM_UNDO}, /* wait for exclusive lock. */ + {0, 1, SEM_UNDO}, /* lock */ + {1, 0, SEM_UNDO} /* wait for shared lock */ + }; + + /* Get a lock for the buffer. Try again if it fails because our + SIGHUP may interrupt this semop() call */ + if (semop(this->semap, sops, 3) < 0 && + semop(this->semap, sops, 3) < 0) + { + syslog( L_ERROR, "semop failed to getExclusiveLock: %m" ); + return(-1); + } + return(0); +} + +int smcGetSharedLock(smcd_t *this) +{ + struct sembuf sops[2] = { + {0, 0, SEM_UNDO}, /* wait for exclusive lock. */ + {1, 1, SEM_UNDO} /* increase access count */ + }; + + /* Get a lock for the buffer. Try again if it fails because our + SIGHUP may interrupt this semop() call */ + if (semop(this->semap, sops, 2) < 0 && + semop(this->semap, sops, 2) < 0) + { + syslog( L_ERROR, "semop failed to getSharedLock: %m" ); + return(-1); + } + return(0); +} + +int smcReleaseSharedLock(smcd_t *this) +{ + struct sembuf sops = { 1, -1, SEM_UNDO|IPC_NOWAIT }; + + /* Release the lock */ + if (semop(this->semap, &sops, 1) < 0) { + syslog( L_ERROR, "semop failed to release shared lock: %m" ); + return(-1); + } + return(0); +} + +int smcReleaseExclusiveLock(smcd_t *this) +{ + struct sembuf sops = { 0, -1, SEM_UNDO|IPC_NOWAIT }; + + /* Release the lock */ + if (semop(this->semap, &sops, 1) < 0) { + syslog( L_ERROR, "semop failed to release exclusive lock: %m" ); + return(-1); + } + return(0); +} + +/* +** Get an existing shared memory buffer +*/ +smcd_t* smcGetShmemBuffer(const char *name, int size) +{ + int shmid, semap; + smcd_t *this; + caddr_t addr; + key_t fk = ftok( (char *)name, 0 ); + + /* create shared memory buffer */ + shmid = shmget(fk, size, S_IRWXU|S_IRGRP|S_IROTH); + if (shmid < 0) { + /* this is normal */ + return NULL; + } + + /* attach to shared memory buffer */ + if ((addr = (caddr_t)shmat(shmid,0,0)) == MAP_FAILED) { + syslog( L_ERROR, "cant attach shared memory" ); + if (shmctl(shmid, IPC_RMID, 0) < 0) { + syslog( L_ERROR, "cant remove shared memory" ); + } + return NULL; + } + + /* Get control semaphore */ + if ((semap = smcGetSemaphore(name)) < 0) { + syslog( L_ERROR, "failed to get semaphore for key %s", name ); + if (shmdt(addr) < 0) + syslog( L_ERROR, "cant detatch shared memory" ); + if (shmctl(shmid, IPC_RMID, 0) < 0) + syslog( L_ERROR, "cant remove shared memory" ); + return NULL; + } + + this = malloc( sizeof(smcd_t) ); + this->addr = addr;; + this->size = size; + this->shmid = shmid; + this->semap = semap; + +#if 0 + /* turing on the following line will give you a huge log file + * especially on a busy nnrp servers + */ + syslog( L_NOTICE, "got shmid %d semap %d addr %8.8x size %lu", + shmid, semap, addr, size ); +#endif + + return this; +} + +/* +** Create a shared memory buffer +*/ +smcd_t* smcCreateShmemBuffer(const char *name, int size) +{ + int shmid, semap; + smcd_t *this; + caddr_t addr; + key_t fk = ftok( (char *)name, 0 ); + + /* create shared memory buffer */ + shmid = shmget(fk, size, IPC_CREAT|S_IRWXU|S_IRGRP|S_IROTH); + if (shmid < 0) { + /* try to get existing segment */ + shmid = shmget(fk, 4, S_IRWXU|S_IRGRP|S_IROTH); + if (shmid >= 0) { + syslog( L_ERROR, "shmem segment already exists name %s", name ); + /* try to delete old segment */ + if (shmctl(shmid, IPC_RMID, 0) < 0) { + syslog( L_ERROR, "cant delete old memory segment" ); + return NULL; + } + syslog( L_NOTICE, "recreating another shmem segment." ); + shmid = shmget(fk, size, IPC_CREAT|S_IRWXU|S_IRGRP|S_IROTH); + } + } + if (shmid < 0) { + syslog( L_ERROR, "cant create shared memory segment" ); + return NULL; + } + + /* attach to shared memory buffer */ + if ((addr = (caddr_t)shmat(shmid,0,0)) == MAP_FAILED) { + syslog( L_ERROR, "cant attach shared memory" ); + if (shmctl(shmid, IPC_RMID, 0) < 0) { + syslog( L_ERROR, "cant remove shared memory" ); + } + return NULL; + } + /* clear the data */ + memset( addr, 0, size ); + + /* Create control semaphore */ + if ((semap = smcCreateSemaphore(name)) < 0) { + syslog( L_ERROR, "failed to create semaphore for %s", name ); + if (shmdt(addr) < 0) + syslog( L_ERROR, "cant detatch shared memory" ); + if (shmctl(shmid, IPC_RMID, 0) < 0) + syslog( L_ERROR, "cant remove shared memory" ); + return NULL; + } + + this = malloc( sizeof(smcd_t) ); + this->addr = addr; + this->size = size; + this->shmid = shmid; + this->semap = semap; + + syslog( L_NOTICE, "created shmid %d semap %d addr %8.8x size %lu", + shmid, semap, addr, size ); + + return this; +} + +void smcClose( smcd_t *this ) +{ + struct shmid_ds buf; + + if (this->addr != MAP_FAILED) { + /* detach shared memory segment */ + if (shmdt(this->addr) < 0) { + syslog( L_ERROR, "cant detach shared memory segment: %m" ); + } + this->addr = MAP_FAILED; + } + + /* delete shm if no one has attached it */ + if ( shmctl(this->shmid, IPC_STAT, &buf) < 0) { + syslog( L_ERROR, "cant stat shmid %s", this->shmid ); + } else if ( buf.shm_nattch == 0 ) { + if (shmctl(this->shmid, IPC_RMID, 0) < 0) { + syslog( L_ERROR, "cant delete shmid %d", this->shmid ); + } else { + syslog( L_NOTICE, "shmid %d deleted", this->shmid ); + } + } + free( this ); +} + +#ifdef _TEST_ + +/* Check if the testfile exists. + If the file is absent + create one with size 1M, and fill the contents with all zero. + for (i=0; i<100; i++) + add 1 to the content; +*/ +static const char* testfile = "testfile"; +#define TESTSIZE ( 1024 * 1024 ) +#define MAXCOUNT 100 + +static smcd_t *this; +static void myexit( void ) +{ + if( this ) { + smcClose( this ); + } +} + +int main( int argc, char** argv ) +{ + struct stat st; + int fd, i, k; + int *x; + int len, xmin, xmax; + struct flock fl; + + atexit( myexit ); + openlog( "shmemtest", LOG_PID, LOG_DAEMON ); + + /* open the testfile */ + fd = creat(testfile, 0660); + if( fd < 0 ) { + printf( "cant open %s", testfile ); + exit(1); + } + + /* lock the file */ + if( flock( fd, LOCK_EX ) < 0 ) { + printf( "cant get flock" ); + exit(1); + } + + /* try to get shared memory buffer */ + this = smcGetShmemBuffer(testfile, TESTSIZE); + if( !this ) { + /* because there's no shared memory, create one. */ + this = smcCreateShmemBuffer(testfile, TESTSIZE); + if( !this ) { + printf( "cant create shmem buffer" ); + exit(1); + } + } + + /* unlock the file */ + if( flock( fd, LOCK_UN ) < 0 ) { + printf( "cant unflock %s", testfile ); + exit(1); + } + + x = (int *)this->addr; + len = this->size / sizeof(int); + for( k=0; kaddr, this->size) != this->size ) { + printf( "cant write" ); + exit(1); + } + if( smcReleaseExclusiveLock( this ) ) { + printf( "cant release exclusive lock" ); + exit(1); + } + } + /* write the minimum and maximum */ + xmin = xmax = x[0]; + for( i=1; i xmax ) xmax = x[i]; + } + printf( "min %d max %d\n", xmin, xmax ); + + return(0); +} +#endif /* _TEST_ */ diff -uNr buffindexed.orig/shmem.h buffindexed/shmem.h --- buffindexed.orig/shmem.h Thu Jan 1 09:00:00 1970 +++ buffindexed/shmem.h Sat Dec 21 11:03:04 2002 @@ -0,0 +1,26 @@ +/* +** shared memory control utility +*/ + +#ifndef SHMEM_H +#define SHMEM_H + +#include + +typedef struct { + caddr_t addr; /* attached shared memory address */ + size_t size; /* size of the shared memory */ + int shmid; /* shared memory segment id */ + int semap; /* semaphore id */ + int locktype; /* current lock type */ +} smcd_t; + +int smcGetExclusiveLock(smcd_t *this); +int smcGetSharedLock(smcd_t *this); +int smcReleaseSharedLock(smcd_t *this); +int smcReleaseExclusiveLock(smcd_t *this); +smcd_t* smcGetShmemBuffer(const char *name, int mapSize); +smcd_t* smcCreateShmemBuffer(const char *name, int mapSize); +void smcClose( smcd_t *this ); + +#endif /* SHMEM_H */