CNFS and FreeBSD
craig at norb.org
Tue Sep 14 00:40:34 UTC 1999
After struggling with getting CNFS up and running on a FreeBSD
box, it all came down to a fairly easy thing that I thought you might want
to at least elude to in your "INSTALL" file.
The paragraph that states:
"For each metacycbuff, you now need to determine how many cycbuffs will
make up the metacycbuff, the size of those cycbuffs and where they will be
stored. Some OSs do not support files larger than 2 GB which will force all
of your cycbuffs to be < 2GB (even if they are stored on raw disk
partitions). Linux is known to have this limitation, FreeBSD does not. If
in doubt, keep your cycbuffs smaller than 2 GB. Also when laying out
your cycbuffs you will want to try to arrange them across as many physical
disks as possible (or use a striped disk array and put them all on that)."
while it is true that FreeBSD does not have the 2Gb partition, you will
continually get the error messages in your /var/log/messages:
innd: CNFS-sm: CNFSinitdisks: mmap for /var/news/spool/buffers/buf06 offset
0 len 500006912 failed: Cannot allocate memory
innd: SM storage method 'cnfs' failed initialization
innd: SM one or more storage methods failed initialization
innd: SERVER cant initialize the storage subsystem one or more storage
methods failed initialization
After trying different things, I finally traced the message down
in the code, and its a problem in a "mmap" call. In the FreeBSD man pages
len is limited to 2GB. Mmapping slightly more than 2GB doesn't work,
it is possible to map a window of size (filesize % 2GB) for file
slightly less than 2G, 4GB, 6GB and 8GB.
The limit is imposed for a variety of reasons. Most of them have to do
with FreeBSD not wanting to use 64 bit offsets in the VM system due to
the extreme performance penalty. So FreeBSD uses 32bit page indexes and
this gives FreeBSD a maximum of 8TB filesizes. It's actually bugs in the
filesystem code that causes the limit to be further restricted to 1TB
(loss of precision when doing blockno calculations).
Another reason for the 2GB limit is that filesystem metadata can reside
at negative offsets."
So, even though FreeBSD doesn't have the filesize limitation, it
does have a problem with mmap'ing a file over 2Gb in size. Rather than
mess around with "mapping a window" or trying to convert the kernel to
64bit offsets in the VM system, I decided that recreating the buffers as
<2Gb files would probably be easier... Everything worked like a champ
Now to see how this puppy works... Not worrying about expiring
articles will be a strange thing....
craig at norb.org
More information about the inn-bugs