CNFS and FreeBSD

[The Hermit Hacker]

Craig, I'm running INN+CNFS under FreeBSD 3.2-STABLE with 12GB
buffers, and have never seen the error that you are reporting...what
version of FreeBSD are you running? *raised eyebrow*



On Mon, 13 Sep 1999, Craig Norborg wrote:

>          After struggling with getting CNFS up and running on a FreeBSD 
> box, it all came down to a fairly easy thing that I thought you might want 
> to at least elude to in your "INSTALL" file.
> 
>          The paragraph that states:
> "For each metacycbuff, you now need to determine how many cycbuffs will 
> make up the metacycbuff, the size of those cycbuffs and where they will be 
> stored. Some OSs do not support files larger than 2 GB which will force all 
> of your cycbuffs to be < 2GB (even if they are stored on raw disk 
> partitions). Linux is known to have this limitation, FreeBSD does not. If 
> in  doubt, keep your cycbuffs smaller than 2 GB. Also when laying out 
> your  cycbuffs you will want to try to arrange them across as many physical 
> disks as possible (or use a striped disk array and put them all on that)."
> 
> while it is true that FreeBSD does not have the 2Gb partition, you will 
> continually get the error messages in your /var/log/messages:
> 
> innd: CNFS-sm: CNFSinitdisks: mmap for /var/news/spool/buffers/buf06 offset 
> 0 len 500006912 failed: Cannot allocate memory
> innd: SM storage method 'cnfs' failed initialization
> innd: SM one or more storage methods failed initialization
> innd: SERVER cant initialize the storage subsystem one or more storage 
> methods failed initialization
> 
>          After trying different things, I finally traced the message down 
> in the code, and its a problem in a "mmap" call.  In the FreeBSD man pages 
> under mmap:
> "BUGS
>       len is limited to 2GB.  Mmapping slightly more than 2GB doesn't work, 
> but
>       it is possible to map a window of size (filesize % 2GB) for file 
> sizes of
>       slightly less than 2G, 4GB, 6GB and 8GB.
> 
>       The limit is imposed for a variety of reasons. Most of them have to do
>       with FreeBSD not wanting to use 64 bit offsets in the VM system due to
>       the extreme performance penalty. So FreeBSD uses 32bit page indexes and
>       this gives FreeBSD a maximum of 8TB filesizes. It's actually bugs in the
>       filesystem code that causes the limit to be further restricted to 1TB
>       (loss of precision when doing blockno calculations).
> 
>       Another reason for the 2GB limit is that filesystem metadata can reside
>       at negative offsets."
> 
> 
> 
>          So, even though FreeBSD doesn't have the filesize limitation, it 
> does have a problem with mmap'ing a file over 2Gb in size.  Rather than 
> mess around with "mapping a window" or trying to convert the kernel to 
> 64bit offsets in the VM system, I decided that recreating the buffers as 
> <2Gb files would probably be easier...  Everything worked like a champ 
> right away...
>          Now to see how this puppy works...  Not worrying about expiring 
> articles will be a strange thing....
> Craig Norborg
> craig at norb.org 
> 
> 

Marc G. Fournier                   ICQ#7615664               IRC Nick: Scrappy
Systems Administrator @ hub.org 
primary: scrappy at hub.org           secondary: scrappy@{freebsd|postgresql}.org