large history: 2GB shared memory limit

[Russ Allbery]

Patrick Schreurs <patricks at support.nl> writes:

> Today we hit the 2GB limit in shared memory (mmap) with innd. This is
> mainly caused by a large history:

> -rw-rw-r--    1 news     news     9343510653 Jun  9 21:56 history
> -rw-rw-r--    1 news     news     1352977086 Jun  9 21:56 history.hash
> -rw-rw-r--    1 news     news     1803969448 Jun  9 21:56 history.index

> This resulted is this error:
> Jun  9 16:26:54 news3 innd: CNFS-sm: CNFSinitdisks: mmap for /spool1/13 offset 0 len 4620288 failed: Cannot allocate memory

> This 2GB limit seems to be a known issue on 32-bit platforms.

Yeah, there's a limit on the amount of addressable memory (in theory, 4GB,
but conventionally the kernel reserves 2GB for its own purposes).

> As a workaround i've disabled INND_DBZINCORE in config.h

That's the workaround that I'm aware of.

> Is the only solution for this problem to use a 64-bit platform?

Right now, yes.

> I've read something about a history api. Any development on this subject?

Not yet.  I have a lot of ideas, but haven't had a chance to implement
anything.  It should be pretty easy to make the text history file smaller,
but making the hash and/or index smaller is going to be more difficult.

> Does anyone have any experience with history in ramdisk with inn?

Not personally, but there isn't any reason I can think of why it wouldn't
work.

-- 
Russ Allbery (rra at stanford.edu)             <http://www.eyrie.org/~eagle/>

    Please send questions to the list rather than mailing me directly.
     <http://www.eyrie.org/~eagle/faqs/questions.html> explains why.